KafkaIO with DirectRunner is creating tons of connections to Kafka Brokers

Hi, I would like to refer to the following Stackoverflow issue I found. https://stackoverflow.com/questions/56496611/apache-beam-kafka-consumer-restarted-over-and-over-again#new-answer I have the very same issue when developing my Pipeline. Originally the pipeline was bound and would read and p

Re: KafkaIO with DirectRunner is creating tons of connections to Kafka Brokers

8:40 AM, Sozonoff Serge wrote: Hi, I would like to refer to the following Stackoverflow issue I found. https://stackoverflow.com/questions/56496611/apache-beam-kafka-consumer-restarted-over-and-over-again#new-answer I have the very same issue when developing my Pipeline. Originally the pipeline

File processing triggered from external source

Hi, Referring to the explanation found at the following link under (Stream processing triggered from an external source) https://beam.apache.org/documentation/patterns/file-processing/ While implementing this solution I am trying to figure out how to deal with the fact that my pipeline, which

Re: KafkaIO with DirectRunner is creating tons of connections to Kafka Brokers

different runner for that. Best,  Jan On 5/24/21 10:03 AM, Sozonoff Serge wrote: Hi Jan, So if I read your SO answer correctly and looking at the Github link you provided we are talking about ~100 consumers ? Since I am developing locally with a dockerized minimal Kafka broker it is possible that

Re: KafkaIO with DirectRunner is creating tons of connections to Kafka Brokers

.29 on the direct runner and didn't do anything, so you'd need to test on 2.28 or 2.30) On Mon, May 24, 2021 at 4:44 AM Sozonoff Serge wrote: Hi. OK thanks.  Just to clarify, in my case the message throughput is zero when I start the Beam pipeline up and it will still crash once all file h

Re: KafkaIO with DirectRunner is creating tons of connections to Kafka Brokers

Sozonoff Serge wrote: Hi, Tested with 2.28 that flag does not seem to make any difference. BR, Serge On 24 May 2021 at 15:06:11, Steve Niemitz (sniem...@apache.org) wrote: Out of curiosity, does adding the "--experiments=use_deprecated_read" argument fix things? (note, this flag was brok

Re: File processing triggered from external source

On 24 May 2021, at 10:25, Sozonoff Serge wrote: Hi, Referring to the explanation found at the following link under (Stream processing triggered from an external source) https://beam.apache.org/documentation/patterns/file-processing/ While implementing this solution I am trying to figure out h

Re: KafkaIO with DirectRunner is creating tons of connections to Kafka Brokers

ating another one. On Tue, May 25, 2021 at 9:05 AM Alexey Romanenko wrote: On 24 May 2021, at 10:43, Sozonoff Serge wrote: OK thanks.  Just to clarify, in my case the message throughput is zero when I start the Beam pipeline up and it will still crash once all file handles are consumed even if I

Windowing

Hi All, I find myself having to pepper Window transforms all over my pipeline, I count about 9 in order to get my pipeline to run. Aside from the class type they window, all the statements are identical. A window into new GlobalWindows() using an identical trigger. Is there a way to change the

Re: Windowing

: You can use Window.configure() to only set the values you want to change. Is that what you mean? Kenn On Wed, May 26, 2021 at 8:42 AM Sozonoff Serge wrote: Hi All, I find myself having to pepper Window transforms all over my pipeline, I count about 9 in order to get my pipeline to run. Aside

Re: Windowing

you can share more about your pipeline - especially code - then we could probably help more. Kenn On Wed, May 26, 2021 at 9:11 AM Sozonoff Serge wrote: Hi, Well not exactly but maybe I can do something with that. Is there no way to simply assign some sort of Global trigger to the entire pipeline