Hi, Tested with 2.28 that flag does not seem to make any difference.
BR, Serge On 24 May 2021 at 15:06:11, Steve Niemitz (sniem...@apache.org) wrote: Out of curiosity, does adding the "--experiments=use_deprecated_read" argument fix things? (note, this flag was broken in beam 2.29 on the direct runner and didn't do anything, so you'd need to test on 2.28 or 2.30) On Mon, May 24, 2021 at 4:44 AM Sozonoff Serge <se...@sozonoff.com> wrote: Hi. OK thanks. Just to clarify, in my case the message throughput is zero when I start the Beam pipeline up and it will still crash once all file handles are consumed even if I dont send a single message to the kafka topic. Thanks, Serge On 24 May 2021 at 10:14:33, Jan Lukavský (je...@seznam.cz) wrote: It is not 100 consumers, the checkpoint is created every 100 records. So, if your message throughput is high enough, the consumers might be created really often. But most importantly - DirectRunner is really not intended for performance sensitive applications. You should use a different runner for that. Best, Jan
