Re: Unit testing Spark/Scala code with Mockito

AFAICT, depends on testing goals, Unit Test, Integration Test or E2E Test. For Unit Test, mostly, it tests individual class or class methods. Mockito can help mock and verify dependent instances or methods. For Integration Test, some Spark testing helper methods can setup the environment, such as

Re: Unit testing Spark/Scala code with Mockito

On a second note with regard Spark and read writes as I understand unit tests are not meant to test database connections. This should be done in integration tests to check that all the parts work together. Unit tests are just meant to test the functional logic, and not spark's ability to read from

Unit testing Spark/Scala code with Mockito

Hi, I have a spark job that reads an XML file from HDFS, process it and port data to Hive tables, one good and one exception table The Code itself works fine. I need to create Unit Test with Mockito for it.. A unit test should test functiona