[pyspark 2.4.3] small input csv ~3.4GB gets 40K tasks created

[Rishi Shah]

Hi All,

I am scratching my head against this weird behavior, where df (read from
.csv) of size ~3.4GB gets cross joined with itself and creates 50K tasks!
How to correlate input size with number of tasks in this case?

-- 
Regards,

Rishi Shah