Thanks Silvio. I might be missing something, but it looks like this project is a kind of a "framework" for setting up Spark for a testing, but after taking a quick look at the code it doesn't seem like it's solving the problem with mocking which is my main concern now - am I wrong?
Kind regards, Michał Michalski, michal.michal...@boxever.com On 28 April 2015 at 16:35, Silvio Fiorito <silvio.fior...@granturing.com> wrote: > Hi Michal, > > Please try spark-testing-base by Holden. I’ve used it and it works well > for unit testing batch and streaming jobs > > https://github.com/holdenk/spark-testing-base > > Thanks, > Silvio > > From: Michal Michalski > Date: Tuesday, April 28, 2015 at 11:32 AM > To: user > Subject: Best practices on testing Spark jobs > > Hi, > > I have two questions regarding testing Spark jobs: > > 1. Is it possible to use Mockito for that purpose? I tried to use it, > but it looks like there are no interactions with mocks. I didn't dive into > the details of how Mockito works, but I guess it might be because of the > serialization and how Spark distributes tasks. I'm not sure about it though > and I'm looking for confirmation. > > 2. If not mockito, what's the alternative? What's the recommended way to > test Spark jobs? Should I manually create mocks by e.g. extending all the > classes I'd normally mock and changing the implementation of some methods? > I don't like this idea but I can't really see any other options now. > > > Kind regards, > Michał Michalski, > michal.michal...@boxever.com >
