Hi Mangat,
back to work now. I've configured out Streams applications to use
exacly-once semantics, but to no avail. Actually, after some. more
investigation I've come to suspect that the issue is somehow related
to rebalancing.
The initially shown topology lives inside a Quarkus Kafka Streams
ap
Hi Mangat,
thank you for clarification. I'll try the suggested configuration as soon
as I get back to work. Will keep you posted then.
Best wishes
Karsten
mangat rai schrieb am Mi., 27. März 2024, 11:07:
> Hey Karsten,
>
> You don't need to do any other configuration to enable EOS. See here -
Hey Karsten,
You don't need to do any other configuration to enable EOS. See here -
https://docs.confluent.io/platform/current/streams/concepts.html#processing-guarantees
It mentions that the producer will be idempotent. That also mans ack=all
will be considered. Not that if you have any other ack
Hi Mangat,
thanks for clarification. So to my knowledge exactly-once is configured
using the 'processing.guarantee=exactly_once_v2' setting? Is the
configuration setting 'acks=all' somehow related and would you advise
setting that as well?
Best wishes
Karsten
mangat rai schrieb am Di., 26. Mär
Hey Karsten,
So if a topic has not been created yet. Streams app will keep the data in
memory, and then write it later when it is available. if your app is
restarted (or thread is killed), you may lose data but it depends if the
app will commit in the source topics. If there is no errors, then it
Hi Mangat,
thanks for your thoughts. I had actually considered exactly-once semantics
already, was unsure whether it would help, and left it aside for once then.
I'll try that immediately when I get back to work.
About snapshots and deserialization - I doubt that the issue is caused by
deserializ
Hey Karsten,
There could be several reasons this could happen.
1. Did you check the error logs? There are several reasons why the Kafka
stream app may drop incoming messages. Use exactly-once semantics to limit
such cases.
2. Are you sure there was no error when deserializing the records from
`fol
Hi,
thanks for getting back. I'll try and illustrate the issue.
I've got an input topic 'folderTopicName' fed by a database CDC system.
Messages then pass a series of FK left joins and are eventually sent to an
output topic like this ('agencies' and 'documents' being KTables):
strea
Hi,
That sounds worrisome!
Could you please provide us with a minimal example that shows the issue
you describe?
That would help a lot!
Best,
Bruno
On 3/25/24 4:07 PM, Karsten Stöckmann wrote:
Hi,
are there circumstances that might lead to messages silently (i.e. without
any logged warnin
Hi,
are there circumstances that might lead to messages silently (i.e. without
any logged warnings or errors) disappearing from a topology?
Specifically, I've got a rather simple topology doing a series of FK left
joins and notice severe message loss in case the application is fired up
for the fi
10 matches
Mail list logo
