[
https://issues.apache.org/jira/browse/KAFKA-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayoub Omari updated KAFKA-19096:
--------------------------------
Description:
Currently, we are logging all skipped records in FK join as dropped.
This seems incorrect, as a dropped record essentially means an invalid record.
In the case of FK join, we are logging as dropped, records which are totally
valid, that simply don't affect the join result (i.e. no output record, no FK
subscription, no FK subscription removal).
Quote from @mjsax:
??That's is not really the definition of "dropped records" – "dropped records"
is a metric for malformed records that could not be processed. – Eg, a
{{{}null{}}}-key record for a table is not valid, because we need a valid PK,
and thus we "drop" such a record??
Check this
[link|https://github.com/apache/kafka/pull/19303#discussion_r2029558102] for
complete discussion.
The goal of this ticket is:
- Fix the code in FK join classes by not marking skipped records as dropped
when they aren't really so.
was:
Currently, we are logging all skipped records in FK join as dropped.
This seems incorrect, as a dropped record essentially means an invalid record.
In the case of FK join, we are logging as dropped, records which are totally
valid, that simply don't affect the join result (i.e. no output record, no FK
subscription, no FK subscription removal).
Quote from @mjsax:
??That's is not really the definition of "dropped records" -- "dropped records"
is a metric for malformed records that could not be processed. -- Eg, a
{{{}null{}}}-key record for a table is not valid, because we need a valid PK,
and thus we "drop" such a record ??
Check this
[link|https://github.com/apache/kafka/pull/19303#discussion_r2029558102] for
complete discussion.
The goal of this ticket is:
- Fix the code in FK join classes by not marking skipped records as dropped
when they aren't really so.
> Skipped Records in FK join should not be marked as dropped
> ----------------------------------------------------------
>
> Key: KAFKA-19096
> URL: https://issues.apache.org/jira/browse/KAFKA-19096
> Project: Kafka
> Issue Type: Improvement
> Affects Versions: 4.0.0
> Reporter: Ayoub Omari
> Priority: Major
>
> Currently, we are logging all skipped records in FK join as dropped.
> This seems incorrect, as a dropped record essentially means an invalid record.
> In the case of FK join, we are logging as dropped, records which are totally
> valid, that simply don't affect the join result (i.e. no output record, no FK
> subscription, no FK subscription removal).
> Quote from @mjsax:
> ??That's is not really the definition of "dropped records" – "dropped
> records" is a metric for malformed records that could not be processed. – Eg,
> a {{{}null{}}}-key record for a table is not valid, because we need a valid
> PK, and thus we "drop" such a record??
> Check this
> [link|https://github.com/apache/kafka/pull/19303#discussion_r2029558102] for
> complete discussion.
> The goal of this ticket is:
> - Fix the code in FK join classes by not marking skipped records as dropped
> when they aren't really so.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
