Joey Pereira created FLINK-37864:
------------------------------------
Summary: Extend DynamicKafkaSource to allow for cluster-specific
OffsetsInitializer
Key: FLINK-37864
URL: https://issues.apache.org/jira/browse/FLINK-37864
Project: Flink
Issue Type: Improvement
Components: Connectors / Kafka
Affects Versions: 2.0.0
Reporter: Joey Pereira
At the moment, the DynamicKafkaSource cannot have OffsetsInitializers that vary
per-cluster but only per-topic and partition provided the parameterization of
{{org.apache.kafka.common.TopicPartition}}.
For more complex use-cases with consumption from multiple Kafka sources, it
would be ideal to allow for offset customization per-cluster.
One way this can be added while keeping things simple would be to:
- Add {{setStartingOffsets}} and {{setBounded}} with the arguments
{{Map<String, OffsetsInitializer>}}
- Plumb that map as a seperate property through to the
{{DynamicKafkaSourceEnumerator}} and select the appropriate one during
{{createEnumeratorWithAssignedTopicPartitions}}
Additionally, we'll want to:
- Include adjustments to a Python interface FLINK-37863
- Adjust the documentation
(cc [~shuyichen])
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
