I am pretty new to spark/scala myself, but I just recently implemented unit tests to test my transformations/aggregations and such myself.
I’m using the mrpowers spark-fast-tests <https://github.com/MrPowers/spark-fast-tests> and spark-daria <https://github.com/MrPowers/spark-daria> libraries. I am also using a JDBC sink in the foreach writer. I’ve mocked the sink to place the generated MYSQL statements in a global object and then I compare the output there to an expected set of mysql statements. I’m running this with FlatSpec ScalaTests, where my spark inputs are manually generated fixtures for each test case. Everything seems to be running well and it's nice and quick. On Tue, May 1, 2018 at 8:25 AM, marcos rebelo <ole...@gmail.com> wrote: > Hey all, > > We are using Scala and SQL heavily, but I have a problem with VERY SLOW > Unit Tests. > > Is there a way to do fast Unit Tests on Spark? > > How are you guys going around it? > > Best Regards > Marcos Rebelo >
