Given that duplicate tombstones are not a correctness issue, the bug is
not considered critical.
Of course, we want to merge the fix, but there was no capacity so far to
review the PR. But given your inquire about it, its priority was just
bumped up :)
Sorry of the inconvenience, but Kafka is a very busy project and
sometimes it takes much longer to get fixes done due to limited capacity
on the committer side to review and merge PR :(
-Matthias
On 8/30/24 1:36 AM, Vogel, Kevin, DP EPS, BN, extern, external wrote:
Yes, that's it. Thank you. Unfortunately, the issue is not fixed yet. There is
a PR from March '24. This bug seems urgent to me. Are there any plans about
merging the PR?
-Kevin
-----Ursprüngliche Nachricht-----
Von: Matthias J. Sax <mj...@apache.org>
Gesendet: Freitag, 30. August 2024 04:19
An: users@kafka.apache.org
Betreff: Re: Unexpected Tombstone records after kafka streams update 3.7.1
Sounds like https://issues.apache.org/jira/browse/KAFKA-16394
-Matthias
On 8/29/24 02:35, Vogel, Kevin, DP EPS, BN, extern, external wrote:
Hello there,
I searched the Apache Jira for a bug report on this topic but couldn't find
one. Maybe anyone else has noticed something similar or knows more about this.
After we updated our Spring Boot Kafka Streams application kafka-streams
dependency from 3.6.2 to 3.7.1, we noticed some failing tests. The expected
behavior of the streams-processor is, to join two input topics together and
produce a single output record. Then the test injects a third record with a
null value and expects another output record with a null value. But since the
update we are getting two records with null value instead of one.
I stripped down the processor to the absolute minimum, so you have some example
code. Don't worry about the strange names. We are working with a legacy system
and are unfortunately bound to them:
@Bean
public BiFunction<
KStream<Gidpf01iKey, Gidpf01iValue>,
KTable<String, Gidpf01gAggregate>,
KStream<UuidString, Teil>>
processTeil() {
return (gidpf01i, gidpf01gAggregate) -> {
var gidpf01iTable =
gidpf01i
.toTable();
var joined =
gidpf01iTable.leftJoin(
gidpf01gAggregate,
new Gidpf01iGidpf01gAggregateForeignKeyExtractor(),
new Gidpf01iGidpf01gAggregateValueJoiner(),
TableJoined.as("gidpf01i-gidpf01g-aggregate-to-teil"));
return joined.toStream().selectKey(new GenericSbamUuidGenerator<>());
};
}
You see nothing fancy. Just a KTable - Ktable left-join. Even more strange is,
that not all streams-processors behave differently after the update. But they
are all very similar and I can not see any significant difference.
The test looks like this:
@Test
void testProcessGidpf01iTombstone() {
// Key und Value für Teilestamm und Teiletext
final var gidpf01i = createGidpf01iKeyValue("A", REFRESH);
final var gidpf01gAggregate =
createGidpf01gAggregate("T123456789", "Test Beschreibung",
"Test Zusatzinformation");
gipf01gAggregateInputTopic.pipeInput(gidpf01gAggregate);
gidpf01iInputTopic.pipeInput(gidpf01i);
// Datensatz gelesen
var results = teilOutputTopic.readKeyValuesToList();
assertThat(results).hasSize(1);
assertThat(results.get(0).key).isInstanceOf(UuidString.class);
assertThat(results.get(0).value).isNotNull();
// Tombstone für GIDPF01I senden
gidpf01iInputTopic.pipeInput(new TestRecord<>(gidpf01i.key(),
null));
results = teilOutputTopic.readKeyValuesToList();
assertThat(results).hasSize(1); // <--- failing here
because results contain two identical records with null value instead of one
assertThat(results.get(0).key).isInstanceOf(UuidString.class);
assertThat(results.get(0).value).isNull();
assertThat(gidpf01iDlqOutputTopic.isEmpty()).isTrue();
}
Does anyone know about a bug or behavioral change like this in Kafka streams
3.7.1? I'm very grateful for any response.
Kind Regards
Kevin Vogel
Software Developer extern
Mitarbeiter der Qvest Digital AG
Am Dickobskreuz 10, D-53121 Bonn
Tel.: +49 228 54881-0
HRB AG Bonn 18196 Ust-ID (VAT): DE274355441
Vorstand: Dr. Stefan Barth, Kai Ebenrett, Boris Esser, Alexander Steeg
Vorsitzender Aufsichtsrat: Stefan Nöthen
