Why do you need to mock the read/write at all? Why not have your test CSV file, and invoke it (which will perform the real Spark DF read of CSV), write it, and assert on the output?
On Tue, Jul 14, 2020 at 12:19 PM Dark Crusader <[email protected]> wrote: > Sorry I wasn't very clear in my last email. > > I have a function like this: > > def main( read_file): > df = spark.read.csv(read_file) > ****** Some other code ****** > df.write.csv(path) > > Which I need to write a unit test for. > Would pythons unittest mock help me here? > > When I googled this, I mostly see that we shouldn't mock these reads and > writes, but this doesn't solve the problem of how I unittest helper > functions/main method that will have to read and write files. > > An example of the proper way to do this in python would be really helpful. > > Thanks a lot. >
