Hi, For all these connections, are they existing at the same time(connection is created without closing)?
Yes. Serge On 24 May 2021 at 20:41:06, Boyuan Zhang (boyu...@google.com) wrote: One more question : ) For all these connections, are they existing at the same time(connection is created without closing)? On Mon, May 24, 2021 at 10:48 AM Serge Sozonoff <se...@sozonoff.com> wrote: Hi. So far 2.28 and 2.29 Serge Sent from my iPhone 6 On 24 May 2021, at 19:37, Boyuan Zhang <boyu...@google.com> wrote: Hi Serge, Which Beam version are you using when you notice the problem? On Mon, May 24, 2021 at 6:36 AM Sozonoff Serge <se...@sozonoff.com> wrote: Hi, Tested with 2.28 that flag does not seem to make any difference. BR, Serge On 24 May 2021 at 15:06:11, Steve Niemitz (sniem...@apache.org) wrote: Out of curiosity, does adding the "--experiments=use_deprecated_read" argument fix things? (note, this flag was broken in beam 2.29 on the direct runner and didn't do anything, so you'd need to test on 2.28 or 2.30) On Mon, May 24, 2021 at 4:44 AM Sozonoff Serge <se...@sozonoff.com> wrote: Hi. OK thanks. Just to clarify, in my case the message throughput is zero when I start the Beam pipeline up and it will still crash once all file handles are consumed even if I dont send a single message to the kafka topic. Thanks, Serge On 24 May 2021 at 10:14:33, Jan Lukavský (je...@seznam.cz) wrote: It is not 100 consumers, the checkpoint is created every 100 records. So, if your message throughput is high enough, the consumers might be created really often. But most importantly - DirectRunner is really not intended for performance sensitive applications. You should use a different runner for that. Best, Jan
