Hi Serge, Which Beam version are you using when you notice the problem?
On Mon, May 24, 2021 at 6:36 AM Sozonoff Serge <se...@sozonoff.com> wrote: > Hi, > > Tested with 2.28 that flag does not seem to make any difference. > > BR, > Serge > > On 24 May 2021 at 15:06:11, Steve Niemitz (sniem...@apache.org) wrote: > > Out of curiosity, does adding the "--experiments=use_deprecated_read" > argument fix things? (note, this flag was broken in beam 2.29 on the direct > runner and didn't do anything, so you'd need to test on 2.28 or 2.30) > > On Mon, May 24, 2021 at 4:44 AM Sozonoff Serge <se...@sozonoff.com> wrote: > >> Hi. >> >> OK thanks. Just to clarify, in my case the message throughput is zero >> when I start the Beam pipeline up and it will still crash once all file >> handles are consumed even if I dont send a single message to the kafka >> topic. >> >> Thanks, >> Serge >> >> On 24 May 2021 at 10:14:33, Jan Lukavský (je...@seznam.cz) wrote: >> >> It is not 100 consumers, the checkpoint is created every 100 records. So, >> if your message throughput is high enough, the consumers might be created >> really often. But most importantly - DirectRunner is really not intended >> for performance sensitive applications. You should use a different runner >> for that. >> >> Best, >> >> Jan >> >> >> >>
