Thanks, Yanquan. We can also introduce some restart/recovery strategies on
how to handle MDS unavailability.
On Sat, Apr 5, 2025 at 10:57 PM Yanquan Lv wrote:
> Your explanation is helpful.
>
> For question3, I think the key lies in whether the situation where MDS is
> unavailable is considered
Your explanation is helpful.
For question3, I think the key lies in whether the situation where MDS is
unavailable is considered permissible, as subsequent updates to
DynamicKafkaWriteState depend on this component.
I approve of the usage of state if it is acceptable (for fault tolerance
reasons).
1. Yes, we can add Table API support in the future
2. The failover is coordinated by MDS itself
3. State is there to support situations where MDS is not accessible, it
stores the cached metadata in the state
I hope it clarifies,
Matyas
On Sat, Apr 5, 2025 at 9:55 AM Yanquan Lv wrote:
> Hi, Maty
Hi, Matyas. Thanks for driving this.
I have some questions and I hope you can help explain them:
1. The dynamic Kafka Sink looks like only be used in DataStream jobs
currently, will it be considered to be supported in TableAPI in the future?
2. Even though we have Kafka MetaData Service, due to th
Hi Matyas,
I also have another question.
This FLIP-515 seems based on FLIP-246 and then I read FLIP-246. It seems
that KafkaMetadataService is a key concept in the FLIP.
In this FLIP-515 and FLIP-246, KafkaMetadataServiceImpl is mentioned ,but
I didn't found it in flink-connector-kafka repo wh
Hi Matyas,
Sorry for the late reply. Dynamic Kafka Source has been introduced for a
long time, and it is a little confusing for developers like me
that KafkaDynamicSource and DynamicKafkaSource have the same kafka
connector repo for different things.
Now DynamicKafkaSink will be introduced again.
Hi Matyas,
thanks for clarifying. The design LGTM.
Based on your description, I'd phrase the state management a bit differently:
* The source of truth is and always will be the MDS.
* The response is cached/replicated in Flink state to bridge MDS downtime.
* Since MDS is queried periodically, sta
Hi Arvin,
Thank you for taking time to look at this proposal.
The concept of KafkaMetadataService is not new, it has been introduced with
Dynamic Kafka Source already:
https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/dynamic-kafka/#kafka-metadata-service
This is an
Hi Matyas,
could you please provide more details on the KafkaMetadataService?
Where does it live? How does it work? How does "Metadata state will be
stored in Flink state as the source of truth for the Flink job" work?
Also nit: the test plan contains copy&paste errors.
Best,
Arvid
On Thu, Mar
Hi devs,
I'd like to start a discussion on FLIP-515: Dynamic Kafka Sink [1]. This is
an addition to the existing Dynamic Kafka Source [2] to make the
functionality complete.
Feel free to share your thoughts and suggestions to make this feature
better.
+ Mason Chen
Thanks,
Matyas
[1]
https://cw
10 matches
Mail list logo
