Hi, Can you give some tutorials/examples how to write test case based on the mentioned framework?
Thanks, Sourav On Tue, Apr 28, 2015 at 9:22 PM, Silvio Fiorito < silvio.fior...@granturing.com> wrote: > Sorry that’s correct, I was thinking you were maybe trying to mock > certain aspects of Spark core to write your tests. This is a library to > help write unit tests by managing the SparkContext and StreamingContext. So > you can test your transformations as necessary. More importantly on the > streaming side it really helps simplify running tests on batch outputs. > > If you’re having serialization issues you may need to look at using > transient lazy initializers, to see if that helps? > > From: Michal Michalski > Date: Tuesday, April 28, 2015 at 11:42 AM > To: Silvio Fiorito > Cc: user > Subject: Re: Best practices on testing Spark jobs > > Thanks Silvio. I might be missing something, but it looks like this > project is a kind of a "framework" for setting up Spark for a testing, but > after taking a quick look at the code it doesn't seem like it's solving the > problem with mocking which is my main concern now - am I wrong? > > Kind regards, > Michał Michalski, > michal.michal...@boxever.com > > On 28 April 2015 at 16:35, Silvio Fiorito <silvio.fior...@granturing.com> > wrote: > >> Hi Michal, >> >> Please try spark-testing-base by Holden. I’ve used it and it works well >> for unit testing batch and streaming jobs >> >> https://github.com/holdenk/spark-testing-base >> >> Thanks, >> Silvio >> >> From: Michal Michalski >> Date: Tuesday, April 28, 2015 at 11:32 AM >> To: user >> Subject: Best practices on testing Spark jobs >> >> Hi, >> >> I have two questions regarding testing Spark jobs: >> >> 1. Is it possible to use Mockito for that purpose? I tried to use it, >> but it looks like there are no interactions with mocks. I didn't dive into >> the details of how Mockito works, but I guess it might be because of the >> serialization and how Spark distributes tasks. I'm not sure about it though >> and I'm looking for confirmation. >> >> 2. If not mockito, what's the alternative? What's the recommended way >> to test Spark jobs? Should I manually create mocks by e.g. extending all >> the classes I'd normally mock and changing the implementation of some >> methods? I don't like this idea but I can't really see any other options >> now. >> >> >> Kind regards, >> Michał Michalski, >> michal.michal...@boxever.com >> > > -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o: +91 80 4121 8723 m: +91 988 699 3746 skype: sourav.chandra Livestream "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area, Bangalore 560034 www.livestream.com
