Re: [structured-streaming][parquet] readStream files order in Parquet

2018-06-15 Thread Tathagata Das
The files are processed in the order the file last modified timestamp. The path and partitioning scheme are not used for ordering. On Thu, Jun 14, 2018 at 6:59 AM, karthikjay wrote: > My parquet files are first partitioned by environment and then by date > like: > > env=testing/ >date=2018-

[structured-streaming][parquet] readStream files order in Parquet

2018-06-14 Thread karthikjay
My parquet files are first partitioned by environment and then by date like: env=testing/ date=2018-03-04/ part1.parquet part2.parquet part3.parquet date=2018-03-05/ part1.parquet part2.parquet part3.parquet date=2018-03-06/