Omnia Ibrahim created KAFKA-15063:
-------------------------------------
Summary: Throttle number of active PIDs
Key: KAFKA-15063
URL: https://issues.apache.org/jira/browse/KAFKA-15063
Project: Kafka
Issue Type: New Feature
Components: core, producer
Affects Versions: 3.4.0, 3.2.0, 3.0.0, 3.1.0, 2.8.0, 3.3
Reporter: Omnia Ibrahim
{color:#172b4d}Ticket to track to track KIP-936. Since KIP-679
i{color:#172b4d}dempotent{color} {color:#172b4d}producers became the default in
Kafka {color:#172b4d}as a result of this all producer instances will be
assigned PID. The increase of number of PIDs stored in Kafka brokers by
{color}{{ProducerStateManager}}{color:#172b4d} exposes the broker to OOM errors
if it has a high number of producers, rogue or misconfigured client(s).{color}
{color}{color}
{color:#172b4d}{color:#172b4d}{color:#172b4d}{color:#172b4d}{color:#172b4d}{color:#172b4d}The
broker is still exposed to OOM{color}{color}{color} even after KIP-854
introduced a separated config to expire PID from transaction IDs if there is
high number of PID before {color}{{producer.id.expiration.ms}}{color:#172b4d}
is exceeded.
{color}{color}{color}
As a result of this the broker will keep experincing OOM and become offline.
The only way to recover from this is to increase the heap.
{color:#172b4d}KIP-936 is proposing throttling number of PIDs per
KafkaPrincipal {color}
{color:#172b4d}See
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-936%3A+Throttle+number+of+active+PIDs]
{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
