Your explanation is helpful. For question3, I think the key lies in whether the situation where MDS is unavailable is considered permissible, as subsequent updates to DynamicKafkaWriteState depend on this component. I approve of the usage of state if it is acceptable (for fault tolerance reasons). And we need to remind users to check the status of MDS (add logs, depending on the implementation) to avoid subsequent unrecoverable failures.
Őrhidi Mátyás <matyas.orh...@gmail.com> 于2025年4月6日周日 04:11写道： > 1. Yes, we can add Table API support in the future > 2. The failover is coordinated by MDS itself > 3. State is there to support situations where MDS is not accessible, it > stores the cached metadata in the state > > I hope it clarifies, > Matyas > > On Sat, Apr 5, 2025 at 9:55 AM Yanquan Lv <decq12y...@gmail.com> wrote: > > > Hi, Matyas. Thanks for driving this. > > I have some questions and I hope you can help explain them: > > > > 1. The dynamic Kafka Sink looks like only be used in DataStream jobs > > currently, will it be considered to be supported in TableAPI in the > future? > > 2. Even though we have Kafka MetaData Service, due to the lag of Kafka > > MetaData Service, we still encounter situations where the target cluster > is > > unavailable when calling the SinkWriter.write() method. Therefore, a > > failover seems inevitable, but we can run normally after failover. Is my > > understanding correct? > > 3. Because we only support at least once, we do not need to save > > transaction information. I still have doubts about the necessity of > > DynamicKafkaWriteState. When a job fails, can we directly build this > > streamDataMap information from Kafka MetaData Service, or is it because > our > > Kafka MetaData Service has not completed initialization or we cannot > > directly use the latest information obtained from Kafka MetaData Service? > > > > > > > > > > > > > > > > > > > > Őrhidi Mátyás <matyas.orh...@gmail.com> 于2025年3月14日周五 05:04写道： > > > > > Hi devs, > > > > > > I'd like to start a discussion on FLIP-515: Dynamic Kafka Sink [1]. > This > > is > > > an addition to the existing Dynamic Kafka Source [2] to make the > > > functionality complete. > > > > > > Feel free to share your thoughts and suggestions to make this feature > > > better. > > > > > > + Mason Chen > > > > > > Thanks, > > > Matyas > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-515%3A+Dynamic+Kafka+Sink > > > > > > [2] > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217389320 > > > > > >
