What do you mean by in sequential order, order across files, keys, ...? Is this an ordering that is based on data such as a timestamp of the record or the order in which the records appear in the source files? Do you have a lot of keys or very few?
If you want to process all the data across all the files in sequential order with no parallelism then Apache Beam may not provide much value since its basis is all about parallel data processing. On Wed, Apr 15, 2020 at 10:30 PM Neha Sharma <[email protected]> wrote: > Hello, > > I have a use case where I have a bounded source and I am reading Avro > files from Google Cloud Storage. I am also using group by transform.The > amount of data is huge and I need to process the data in sequential order. > > But as Bounded source reads everything it seemed to be a good idea fixed > window on top of the global window. But it does not seem to be working as > expected. > > Can you please tell me how to handle such scenarios where a bounded source > with large dataset can be broken down into smaller chunks for processing, > using windows such that the window for a key should always be processed in > order. > > > Regards, > Neha >
