Target audience: Beginner
Estimated reading time: 10'
This post describes a methodology to manage the Spark context while testing you application using ScalaTest.
This post introduces two basic ScalaTest methods beforeAll and afterAll of the trait BeforeAndAfterAll to manage the context life-cyle of your test application.
Introduction
Debugging Apache Spark application using ScalaTest seems quite simple when dealing with a single test:
- Specify your Spark context configuration SparkConf
- Create the Spark context
- Add test code related to your application
- Clean up resources (Spark context, Akka context, File handles...
This post introduces two basic ScalaTest methods beforeAll and afterAll of the trait BeforeAndAfterAll to manage the context life-cyle of your test application.
Wrapping the Spark context
The objective is to create a small framework that create or retrieve an existing Spark context before executing a test and closing it after the test is completed, independently of the status of the test.
The initialization of the Spark context consist of specifying the configuration for the sequence of tests. Some of these parameters can be dynamically defined through a simple parameterization. The context is created only if it is not already defined within the scope of the test using the getOrCreate method.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 | trait SparkContextWrapper { protected[this] var sc: SparkContext = _ def getContext: SparkContext = { val conf = new SparkConf().setAppName(s"Test-App") .set("spark.executor.instances", "2") .set("spark.driver.memory", "2g") .set("spark.executor.memory", "2g") .set("spark.executor.cores", "2") .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer") .set("spark.rdd.compress", "true") .set("spark.shuffle.spill", "true") .set("spark.shuffle.spill.compress", "true") .set("spark.shuffle.memoryFraction", "0.4") .set("spark.io.compression.codec", "snappy") .set("spark.network.timeout", "600") sc = SparkContext.getOrCreate(conf.setMaster("local[4]")) sc.setLogLevel("ERROR") sc } def close: Unit = { if (sc != null) sc.stop } } |
The solution is to let ScalaTest method manage the lifecycle of the Spark context for testing.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | trait SparkContextForTesting extends SparkContextWrapper with BeforeAndAfterAll { self: Suite => override def beforeAll: Unit = { getContext super.beforeAll } override def afterAll: Unit = { close super.afterAll } } |
Note: The BeforeAndAfterAll trait can only be sub-classed by a test suite inheriting the Suite trait. The method beforeAll (line: 5) is automatically called before the first test of your test suite and the method afterAll (line 10)is called after your last test is completed, whether those tests succeeded or not
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | class MyTest extends FlatSpec with Matchers with SparkContextForTesting { val x: Int = -4 it should "My first test" in { val sqlContext = new SQLContext(sc) // .... } // ... other tests it should "My last test" in { // clean up after it completes } } |
As they say at the end of the movie, "That's all folks!"
This comment has been removed by the author.
ReplyDeleteGood blog for the people who are seeking information about the technology.
ReplyDeleteAwesome work keep it up. Simply superb.
machine learning course
machine learning certification
machine learning training
machine learning training course
Great job. Keep updating this article by posting new informations.
ReplyDeleteSpoken English Classes in Chennai
English Coaching Classes in Chennai
Japanese Language Classes in Chennai
French Language Classes in Chennai
pearson vue exam centers in chennai
German Classes in Chennai
Spoken English Classes in Tnagar
Spoken English Classes in OMR
Great job done. Excellent post for the freshers. Keep updating it.
ReplyDeleteTOEFL Coaching in Chennai
TOEFL Coaching Centres in Chennai
German Language Course in Chennai
IELTS Coaching centre in Chennai
Japanese Language Classes in Chennai
spanish classes in chennai
Japanese Classes in Tambaram
Japanese Classes in Anna Nagar
Really very nice blog information for this one and more technical skills are improve,i like that kind of post.
ReplyDeleteGuest posting sites
Technology
Thanks for posting this. Its really informative.
ReplyDeleteGerman Classes in Chennai
German Training in Chennai
IELTS Coaching in OMR
TOEFL Coaching Centres in Chennai
french classes
pearson vue
German Courses in Chennai
German classes in Anna Nagar
A very awesome blog post. We are really grateful for your blog post. You will find a lot of approaches after visiting your post. KISS English Youtube
ReplyDeleteI have a hard time describing my thoughts on content, but I really felt I should here. Your article is really great. I like the way you wrote this information. watch video
ReplyDeleteNice information thank you,if you want more information please visit our link machine learning online course
ReplyDelete
ReplyDeleteمكافحة حشرات بالخبر افضل شركة مكافحة حشرات بالخبر
مكافحة حشرات بمكة افضل شركة رش حشرات بمكة
مكافحة حشرات بالمدينة المنورة افضل شركة رش حشرات بالمدينة المنورة
مكافحة حشرات بالدمام افضل شركة مكافحة حشرات بالدمام
Nice informative blog, it shares more intresting information. This blog is useful to me.
ReplyDeleteDevOps Training in Bangalore
Best DevOps Training in Bangalore
DevOps Course in Bangalore
DevOps Training Bangalore
DevOps Training Institutes in Bangalore
DevOps Training in btm
Best DevOps Training in Chennai
DevOps Training institute in Chennai
Spoken English Classes in Bangalore
ielts coaching in bangalore
The blog which you have shared is more creative... Waiting for your upcoming data...
ReplyDeletePython Training in Chennai
Python course in Chennai
Python Training Institute in Chennai
Python Training course
Python training in Guindy
Python Training in Tambaram
Hadoop Training in Chennai
Big data training in chennai
SEO training in chennai
JAVA Training in Chennai
this is really good post here. Thanks for taking the time to post such valuable information. Quality content is what always gets the visitors coming.
ReplyDeleteIELTS Coaching in chennai
German Classes in Chennai
GRE Coaching Classes in Chennai
TOEFL Coaching in Chennai
spoken english classes in chennai | Communication training
Movie-watching websites that are more than movie-watching websites Because we are the number 1 free movie site in Thailand for a long time, including new movies, Thai movies, Western movies, Asian movies, we have all kinds of ways for you Including new series Full of all stories without interstitial ads to keep annoying anymore. One place sa-movie.com.
ReplyDeleteAndroid and IOS operating systems. Watch online movies, Thai movies, Western movies, Asian movies, Cartoon movies, Netflix Movie, Action Movies, Comedy Movies, Crime Movies, Drama Movies, Horror Movies, Adventure Movies, Crash Movies and still have many new movies to watch. You can watch for free anytime, anywhere 24 hours a day at see4k.com.
GangManga read manga, read manga, read manga online for free, fast loading, clear images in HD quality, all titles, anywhere, anytime, on mobile, tablet, computer. Android and IOS operating systems. Read top comics, action dramas, comedy, adventure, horror and manga. New coming every day to watch many more. Can be read for free anytime anywhere 24 hours a day at gangmanga.com..
It is no secret that football is among the most popular and widely watched sports. Everybody who likes football tries to find the best platform for free soccer streaming. So, what are the best free sports streaming sites? We are going to answer this question. On this page, you can find a detailed overview of the most widespread soccer streaming websites. Keep on reading and make the best choice for you live24th.me.
Took me technology to entre all of the observations, however I absolutely loved the article. It proved to be Very cordial to me and i am unconditional to all the commenters here! Its constantly clean whilst you can't on your own be informed, however with entertained! Avast Driver Updater Crack
ReplyDeleteWhat a fantabulous observe this has been. in no way seen this understandable of beneficial claim. i'm grateful to you and assume more quantity of posts at the side of those. thank you intensely a lot. Brothers Day Wishes Quotes
ReplyDeleteThanks for making this super blog its too good.
ReplyDeleteTeamviewer Crack
Nordvpn Crack
Recuva Crack
Macbooster Crack