Re: Timeout Exception When Producing/Consuming Messages to Hundreds of Topics

2021-03-04 Thread Arvid Heise
Hi Claude, Are you sure that the level of abstraction is fitting to your application? Maybe you could describe your use case a bit. Since you use one consumer, I'm assuming that all topics have the same schema. Then, for me, hundreds of similarly structured topics with X partitions sound counter-

Re: Timeout Exception When Producing/Consuming Messages to Hundreds of Topics

2021-03-04 Thread Piotr Nowojski
Hi, Sorry, I don't know. I've heard that this kind of pattern is discouraged by Confluent. At least it used to be. Maybe someone else from the community will be able to help from his experience, however keep in mind that under the hood Flink is just simply using KafkaConsumer and KafkaProducer pr

Timeout Exception When Producing/Consuming Messages to Hundreds of Topics

2021-03-01 Thread Claude M
Hello, I'm trying to run an experiment w/ two flink jobs: - A producer producing messages to hundreds of topics - A consumer consuming the messages from all the topics After the job runs after a few minutes, it will fail w/ following error: Caused by: org.apache.kafka.common.errors.Timeou