I've been trying to upgrade the pyflink-walkthrough to Flink 1.13.1, but without any success.
Unless I give it a lot of resources the data generator times out trying to connect to Kafka. If I give it 6 cores and 11GB (which is about all I can offer it) it does manage to connect, but then fails trying to write to kafka. Not sure what's wrong? Any suggestions? See [1] to review what I tried. Best, David [1] https://github.com/alpinegizmo/flink-playgrounds/commit/777274355ba04de6d8c8f1308b24be99ec86a0d6 21:40 $ docker-compose logs -f generator Attaching to pyflink-walkthrough_generator_1 generator_1 | Connecting to Kafka brokers generator_1 | Waiting for brokers to become available generator_1 | Waiting for brokers to become available generator_1 | Connected to Kafka generator_1 | Traceback (most recent call last): generator_1 | File "./generate_source_data.py", line 61, in <module> generator_1 | write_data(producer) generator_1 | File "./generate_source_data.py", line 42, in write_data generator_1 | producer.send(topic, value=cur_data) generator_1 | File "/usr/local/lib/python3.7/site-packages/kafka/producer/kafka.py", line 576, in send generator_1 | self._wait_on_metadata(topic, self.config['max_block_ms'] / 1000.0) generator_1 | File "/usr/local/lib/python3.7/site-packages/kafka/producer/kafka.py", line 703, in _wait_on_metadata generator_1 | "Failed to update metadata after %.1f secs." % (max_wait,)) generator_1 | kafka.errors.KafkaTimeoutError: KafkaTimeoutError: Failed to update metadata after 60.0 secs.
