Sorry for the noise, this was definitely me doing something wrong. In particular, I added maxFilesPerTrigger only after running the code first. So it was picking up the batch generated with the previous checkpoint, when I did not have maxFilesPerTrigger set. Removing the stale checkpoint fixed my problem.
-Dima -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
