Matthias J. Sax created KAFKA-13183:
---------------------------------------
Summary: Dropping nul key/value records upstream to repartiton
topic not tracked via metrics
Key: KAFKA-13183
URL: https://issues.apache.org/jira/browse/KAFKA-13183
Project: Kafka
Issue Type: Bug
Components: streams
Reporter: Matthias J. Sax
For joins and aggregation, we consider records with null key or value as
invalid, and drop them. Inside the aggregate and join processors, we record
dropped record with a corresponding metric (cf `droppedRecrodSensor`).
However, we also apply an upstream optimization if we need to repartition data.
As we know that the downstream aggregation / join will drop those records
anyway, we drop them _before_ we write them into the repartition topic (we
still need the drop logic in the processor for the case we don't have a
repartition topic).
We add a `KStreamFilter` (cf `KStreamImpl#createRepartiitonSource()`) upstream
but this filter does not update the corresponding metric to record dropped
records.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
