Hi Hongshun, this thread is about addressing your first pain point. It would be awesome if you could read the FLIP and provide feedback whether this approach would work in your setup.
The other issues won't need a FLIP but need a smaller bugfix. Best, Arvid On Tue, Mar 25, 2025 at 7:57 AM Hongshun Wang <loserwang1...@gmail.com> wrote: > > Hi heise, > Thanks for pushing this job. Currently, kafka sinks in transactions are not > productive, which will influence the whole kafka cluster. > As we discussed before, I have three problems when using Kafka > exactly-once sink : > 1. Too much transaction id https://issues.apache.org/jira/browse/FLINK-34554 > 2. Too much unclosed producer ( if commit is lag too much from pre-commit: > https://issues.apache.org/jira/browse/FLINK-36569 > 3. maybe Invalid state: https://issues.apache.org/jira/browse/FLINK-37356 > > I do hope this flip can show how to solve this problem. > > Best > Hongshun Wang > > > > On Fri, Feb 28, 2025 at 10:01 PM Arvid Heise <ar...@apache.org> wrote: > > > Dear Flink devs, > > > > I'd like to start discussion around an improvement of the exactly-once > > Kafka sink. Because of its current design, the sink currently puts a > > large strain on the main memory of the Kafka broker [1], which hinders > > adoption of the KafkaSink. Since the KafkaSink will be the only way to > > produce into Kafka for Flink 2.0, this limitation may in turn also > > limit Flink 2 adoption. > > > > I appreciate feedback not only on technical aspects but also on the > > requirements of the new approach [2]: > > - Are most users/customers already on Kafka 3.0 broker? > > - The new approach requires READ permission on the target topic (for > > DescribeProducer API) and DESCRIBE permission on the transaction (for > > ListTransaction API). Are there any concerns around this? > > > > The new behavior would be opt-in in the first phase while the current > > behavior remains the DEFAULT. If the community accepts the new > > behavior, we can turn it on by default and provide an option to > > opt-out. > > > > Note that this improvement is borderline FLIP-worthy; it touches the > > public interfaces of the connector only. However, by doing an official > > FLIP, I'm positive that we can gather feedback more quickly. > > > > I'm also looking for volunteers that test the implementation on their > > dev clusters [3]. > > > > Best, > > > > Arvid > > > > [1] https://issues.apache.org/jira/browse/FLINK-34554 > > [2] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-511%3A+Support+transaction+id+pooling+in+Kafka+connector > > [3] https://github.com/apache/flink-connector-kafka/pull/154 > >
