Hi. So far 2.28 and 2.29
Serge Sent from my iPhone 6 > On 24 May 2021, at 19:37, Boyuan Zhang <boyu...@google.com> wrote: > > > Hi Serge, > > Which Beam version are you using when you notice the problem? > >> On Mon, May 24, 2021 at 6:36 AM Sozonoff Serge <se...@sozonoff.com> wrote: >> Hi, >> >> Tested with 2.28 that flag does not seem to make any difference. >> >> BR, >> Serge >> >>> On 24 May 2021 at 15:06:11, Steve Niemitz (sniem...@apache.org) wrote: >>> >>> Out of curiosity, does adding the "--experiments=use_deprecated_read" >>> argument fix things? (note, this flag was broken in beam 2.29 on the direct >>> runner and didn't do anything, so you'd need to test on 2.28 or 2.30) >>> >>>> On Mon, May 24, 2021 at 4:44 AM Sozonoff Serge <se...@sozonoff.com> wrote: >>>> Hi. >>>> >>>> OK thanks. Just to clarify, in my case the message throughput is zero >>>> when I start the Beam pipeline up and it will still crash once all file >>>> handles are consumed even if I dont send a single message to the kafka >>>> topic. >>>> >>>> Thanks, >>>> Serge >>>> >>>>> On 24 May 2021 at 10:14:33, Jan Lukavský (je...@seznam.cz) wrote: >>>>> >>>>> It is not 100 consumers, the checkpoint is created every 100 records. So, >>>>> if your message throughput is high enough, the consumers might be created >>>>> really often. But most importantly - DirectRunner is really not intended >>>>> for performance sensitive applications. You should use a different runner >>>>> for that. >>>>> >>>>> Best, >>>>> >>>>> Jan >>>>> >>>>>>>> >>>>>>>>
