I'm not sure if I fully understand the use case. When you require ordering, do you need a set of transforms completed on all data before moving to the next set of transforms? Or do you need transforms to complete on a subset of the data before moving to the next subset of the data for the same transforms?
If it is the former, you could consider using the Wait transform <https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/Wait.html>. If it's the latter, you probably need something like OrderedListState as XQ described above. Alternatively, if your data is partitioned in a way which is known ahead of time, you could transform the latter problem into the former problem by defining a transform per subset of data. Thanks, Danny On Sat, Sep 28, 2024 at 1:18 PM XQ Hu via dev <dev@beam.apache.org> wrote: > Not exactly sure what your use case is. This year, at our Beam Summit, > Shunping talked about Beam State and OrderedListStates: > https://beamsummit.org/sessions/2024/introducing-ordered-list-states/. > This might be helpful for you. > > On Sat, Sep 28, 2024 at 10:30 AM Settara Pramod <pramodsettar...@gmail.com> > wrote: > >> Hi Apache Beam Dev Team, >> >> First of all, thank you for developing such an amazing project and making >> it open-source. >> >> I have a use case where I encountered some limitations in using Apache >> Beam to solve my problem. I am working with workloads that are tied to >> specific sequences. My goal is to process workloads in order, where >> workloads with the same sequence should run concurrently, but workloads in >> the next sequence must wait until the previous sequence has fully completed. >> >> With Apache Beam, I found it difficult to achieve this balance. I was >> only able to process workloads either sequentially by enforcing strict >> ordering or run all workloads in parallel. Neither approach fully met my >> needs. >> >> I considered a workaround where one pipeline calls another, but I believe >> this would be inefficient and resource-intensive, leading to poor design >> and scalability. >> >> Is there a way to extend or customize Apache Beam to handle this use case >> more effectively? I would greatly appreciate any guidance or suggestions on >> this. >> >> I would also be happy to set up a call if further clarification is needed >> regarding my use case. >> >> Thanks & Regards >> Pramod >> >>
