Hi,
thanks for getting back. I'll try and illustrate the issue.
I've got an input topic 'folderTopicName' fed by a database CDC system.
Messages then pass a series of FK left joins and are eventually sent to an
output topic like this ('agencies' and 'documents' being KTables):
streamsBuilder //
.table( //
folderTopicName, //
Consumed.with( //
folderKeySerde, //
folderSerde)) //
.leftJoin( //
agencies, //
Folder::agencyIdValue, //
AggregateFolder::new, //
TableJoined.as("folder-to-agency"), //
Materializer //
.<FolderId,
AggregateFolder>named("folder-to-agency-materialized") //
.withKeySerde(folderKeySerde) //
.withValueSerde(aggregateFolderSerde))
//
.leftJoin( //
documents, //
.toStream(...
.to(...
...
As far as I understand, left join sematics should be similar to those of
relational databases, i.e. the left hand value always passes the join with
the right hand value set as <null> if not present. Whereas what I am
observing is this: lots of messages on the input topic are even absent on
the first left join changelog topic
('folder-to-agency-materialized-changelog'). But: this seems to happen only
in case the Streams application is fired up for the first time, i.e.
intermediate topics do not yet exist. When streaming another table snapshot
to the input topic, things seem (!) to work as expected...
Best wishes,
Karsten
Bruno Cadonna <cado...@apache.org> schrieb am Mo., 25. März 2024, 17:01:
> Hi,
>
> That sounds worrisome!
>
> Could you please provide us with a minimal example that shows the issue
> you describe?
>
> That would help a lot!
>
> Best,
> Bruno
>
> On 3/25/24 4:07 PM, Karsten Stöckmann wrote:
> > Hi,
> >
> > are there circumstances that might lead to messages silently (i.e.
> without
> > any logged warnings or errors) disappearing from a topology?
> >
> > Specifically, I've got a rather simple topology doing a series of FK left
> > joins and notice severe message loss in case the application is fired up
> > for the first time, i.e. intermediate topics not existing yet. I'd
> > generally expect the message count on the output topic resemble that from
> > the input topic, yet it doesn't (about half only).
> >
> > Any hints on this?
> >
> > Best wishes
> > Karsten
> >
>
