Re: Best patterns for a polling transform

2023-06-22 Thread Sam Bourne
The streaming support in Python direct runner is currently rather limited In this experiment I was running a batch pipeline instead of a streaming one. Are there any known issues using timers with a batch pipeline? It sounds like we should identify whether this is a problem in the SDK or in the D

Re: Best patterns for a polling transform

2023-06-22 Thread Valentyn Tymofieiev via user
> The below code runs fine with a single worker but with multiple workers there are duplicate values. > I’m using TimeDomain.WATERMARK here due to it simply not working when using REAL_TIME. The docs seem to suggest REAL_TIME would be the way to do this, however there seems to be no guarantee that

Re: Best patterns for a polling transform

2023-06-22 Thread Chad Dombrova
I’m also interested in the answer to this. This is essential for reading from many types of data sources. On Tue, Jun 20, 2023 at 2:57 PM Sam Bourne wrote: > +dev to see if anyone has any suggestions. > > On Fri, Jun 16, 2023 at 5:46 PM Sam Bourne wrote: > >> Hello beam community! >> >> I’m h

Re: Best patterns for a polling transform

2023-06-20 Thread Sam Bourne
+dev to see if anyone has any suggestions. On Fri, Jun 16, 2023 at 5:46 PM Sam Bourne wrote: > Hello beam community! > > I’m having trouble coming up with the best pattern to *eagerly* poll. By > eagerly, I mean that elements should be consumed and yielded as soon as > possible. There are a hand