[
https://issues.apache.org/jira/browse/FLINK-34693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18113945#comment-18113945
]
Sylwester Lachiewicz commented on FLINK-34693:
----------------------------------------------
Triage finding (2026-09-10, connector main at 48ddbf1f):
The retained objects in the attached heap dump ({{java.lang.Object[9019]}},
1.54 GB) sit in {{KafkaWriter.producerCloseables}}, an append-only deque that
is only drained by {{close()}}. A producer that leaves the pool through a
committable and never comes back stays referenced there for the life of the
writer, which is what happens on 3.x whenever writer and committer are not
chained, because {{KafkaCommittableSerializer}} carries only the transactional
id, producer id and epoch. This is the mechanism Robert Metzger described on
[FLINK-29492|https://issues.apache.org/jira/browse/FLINK-29492].
Per line: absent on main, 5.0 and 4.0, where
[FLINK-37282|https://issues.apache.org/jira/browse/FLINK-37282]
([df353e9d|https://github.com/apache/flink-connector-kafka/commit/df353e9d2543eb5babb325a4f9895fe89623a53f],
[f4015d17|https://github.com/apache/flink-connector-kafka/commit/f4015d17bd204b8decaf8d5337de011107e808e4])
replaced the deque with a producer pool and
[FLINK-37613|https://issues.apache.org/jira/browse/FLINK-37613] closed the
remaining failure path. Present unchanged on the live v3.4 branch
([KafkaWriter.java|https://github.com/apache/flink-connector-kafka/blob/v3.4/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/sink/KafkaWriter.java],
field plus three add and closeAll sites).
Recommendation: keep open for a v3.4-only fix and link FLINK-29492, FLINK-37282
and FLINK-37613. FLINK-37282 does not cherry-pick (31 files, +2637/-689 in the
sink package between 3.4.0 and 4.0.0). A scoped change that stops tracking a
producer in {{producerCloseables}} once it is handed to a committable, and
closes the intermediate producers created by the transactional id gap-fill loop
in {{getTransactionalProducer()}}, is in the range of 50 to 100 lines. I can
prepare that if a 3.4.x release is still planned.
_This comment was created with AI assistance._
> Memory leak in KafkaWriter
> --------------------------
>
> Key: FLINK-34693
> URL: https://issues.apache.org/jira/browse/FLINK-34693
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
> Affects Versions: kafka-3.1.0, kafka-3.0.2
> Reporter: Stefan Richter
> Priority: Blocker
> Attachments: image-2024-03-15-10-30-08-280.png
>
>
> KafkaWriter is keeping objects in Dequeue of closeables
> ({{{}producerCloseables{}}}) that are never removed so that the can be GC’ed.
> From heap
> dump:!04599375-f923-4d1a-8d68-9e17e54b363c#media-blob-url=true&id=9d1e022e-8762-45b3-877b-d298ec956078&collection=&contextId=870337&height=306&width=2284&alt=!
> !image-2024-03-15-10-30-08-280.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
