Hi Luke, It is the order the record appears in the source file.
Basically each record corresponding to a key depends on the previous occurrence of the same key and hence parallel processing does not seem to be a good idea. Is there a possibility where bounded source + fixed window based on the timestamp in the record can be used to somehow batch the whole data into smaller chunks for processing and at the same time can maintain the ordering provided a sorting based on the timestamp? Something like this: Read from Bounded Source -> Fixed window to make smaller batches -> Sorting based on timestamp -> Processing Regards, Neha On Thu, Apr 16, 2020, 6:57 PM Luke Cwik <[email protected]> wrote: > What do you mean by in sequential order, order across files, keys, ...? > Is this an ordering that is based on data such as a timestamp of the > record or the order in which the records appear in the source files? > Do you have a lot of keys or very few? > > If you want to process all the data across all the files in sequential > order with no parallelism then Apache Beam may not provide much value since > its basis is all about parallel data processing. > > On Wed, Apr 15, 2020 at 10:30 PM Neha Sharma <[email protected]> > wrote: > >> Hello, >> >> I have a use case where I have a bounded source and I am reading Avro >> files from Google Cloud Storage. I am also using group by transform.The >> amount of data is huge and I need to process the data in sequential order. >> >> But as Bounded source reads everything it seemed to be a good idea fixed >> window on top of the global window. But it does not seem to be working as >> expected. >> >> Can you please tell me how to handle such scenarios where a bounded >> source with large dataset can be broken down into smaller chunks for >> processing, using windows such that the window for a key should always be >> processed in order. >> >> >> Regards, >> Neha >> >
