Thank you! On Wed, Aug 14, 2024 at 4:55 PM Jaehyeon Kim <[email protected]> wrote:
> Hello, > > I created a PR that adds examples in the common pipeline patterns section > for using a shared object as a cache - > https://github.com/apache/beam/pull/32187. > > On Wed, 14 Aug 2024 at 07:59, Jaehyeon Kim <[email protected]> wrote: > >> Thank you for the suggestion. Let me think about how to contribute and >> take an action. >> >> On Tue, 13 Aug 2024, 8:50 am Ahmet Altay via user, <[email protected]> >> wrote: >> >>> Thank you for the follow up. >>> >>> If you think that presentation is useful, and this is not properly >>> captured in docs, would you be kind enough to help us improve our docs? :) >>> >>> It could be a link to that deck, and github issue, or new content in >>> docs based on that presentation. >>> >>> On Sat, Aug 3, 2024 at 6:37 PM Jaehyeon Kim <[email protected]> wrote: >>> >>>> Hello, >>>> >>>> What I look into can actually be achieved by implementing one of the >>>> caching strategies in a talk at Beam Summit 2022. >>>> >>>> - Strategies for caching data in Dataflow using Beam SDK >>>> >>>> <https://2022.beamsummit.org/sessions/strategies-for-caching-data-in-dataflow-using-beam-sdk/> >>>> >>>> Among the 4 options, I'd try a side input and the shared module >>>> (with/without side input) first. >>>> >>>> Cheers, >>>> Jaehyeon >>>> >>>> >>>> On Thu, 1 Aug 2024 at 13:30, Jaehyeon Kim <[email protected]> wrote: >>>> >>>>> Thank you for letting me know. It is also available in the Python SDK >>>>> - >>>>> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/transforms/util.py#L1656 >>>>> >>>>> However, it doesn't seem to meet the requirement that the side input >>>>> values can change over time because the mentioned transform just seems to >>>>> wait until the previous one gets completed. What I look into is, let's >>>>> say, >>>>> a customer attribute changes then order records should be enriched with >>>>> the >>>>> updated attribute. >>>>> >>>>> On Thu, 1 Aug 2024 at 13:14, LDesire <[email protected]> wrote: >>>>> >>>>>> Hello. I found a similar example code. >>>>>> You can use `Wait` PTransform. >>>>>> Wait (Apache Beam 2.13.0) >>>>>> <https://beam.apache.org/releases/javadoc/2.13.0/org/apache/beam/sdk/transforms/Wait.html> >>>>>> beam.apache.org >>>>>> <https://beam.apache.org/releases/javadoc/2.13.0/org/apache/beam/sdk/transforms/Wait.html> >>>>>> [image: favicon.ico] >>>>>> <https://beam.apache.org/releases/javadoc/2.13.0/org/apache/beam/sdk/transforms/Wait.html> >>>>>> <https://beam.apache.org/releases/javadoc/2.13.0/org/apache/beam/sdk/transforms/Wait.html> >>>>>> >>>>>> Hope this helps. >>>>>> >>>>>> [image: stateful-beam-realtime.png] >>>>>> >>>>>> stateful-beam-realtime/pipeline/src/main/java/org/stjimmy/beam/LtvPipelineSqlLookup.java >>>>>> at 2cc16a9cf8460c5b0e4d749e81654273c14ffb00 · >>>>>> Jimmyst/stateful-beam-realtime >>>>>> <https://github.com/Jimmyst/stateful-beam-realtime/blob/2cc16a9cf8460c5b0e4d749e81654273c14ffb00/pipeline/src/main/java/org/stjimmy/beam/LtvPipelineSqlLookup.java#L92> >>>>>> github.com >>>>>> <https://github.com/Jimmyst/stateful-beam-realtime/blob/2cc16a9cf8460c5b0e4d749e81654273c14ffb00/pipeline/src/main/java/org/stjimmy/beam/LtvPipelineSqlLookup.java#L92> >>>>>> >>>>>> <https://github.com/Jimmyst/stateful-beam-realtime/blob/2cc16a9cf8460c5b0e4d749e81654273c14ffb00/pipeline/src/main/java/org/stjimmy/beam/LtvPipelineSqlLookup.java#L92> >>>>>> >>>>>> >>>>>> 2024. 8. 1. 오전 11:52, Jaehyeon Kim <[email protected]> 작성: >>>>>> >>>>>> Hello, >>>>>> >>>>>> I'm looking into side input patterns especially slowly updating >>>>>> global window side inputs - >>>>>> https://beam.apache.org/documentation/patterns/side-inputs/ >>>>>> >>>>>> It'd be useful if we need to enrich eg) order records with customer >>>>>> details where customer details would be taken as a side input. >>>>>> >>>>>> Let's say we have two Kafka topics, one for client records and the >>>>>> other for order records. For the enrichment to work properly, consumption >>>>>> of order records should wait until all customer records are read. >>>>>> >>>>>> Can you please inform me if it is achievable? >>>>>> >>>>>> Cheers, >>>>>> Jaehyeon >>>>>> >>>>>> >>>>>>
favicon.ico
Description: Binary data
