Hi Navneeth
Your configuration looks correct to me.
If you observe that compaction is not cleaning up old records, it could
either be due to slow compaction or could be due to incorrect configuration.
Here are a few things that I would check:
First, validate that the log cleaner is running. The
Hi All,
We are using AWS MSK with kafka version 2.6.1. There is a compacted topic
with the below configurations. After reading the documentation my
understanding was that null values in the topic can be removed using delete
retention time but I can see months old keys having null values. Is there
These properties can't be triggered programatically. Kafka uses an
internal thread pool called "Log Cleaner Thread" that does the job
asynchronously of deleting old segments ("delete") and deleting
repeated records ("compact").
Whatever the S3 connector picks up is already compacted and/or deleted.
Hi,
I have a KStreams app that outputs a KTable
to a topic with cleanup policy "compact,delete".
I have the Confluent S3 Connector to store this
table in S3 where I do further analysis with hive.
Now my question is, if there's a way to trigger
log compaction right before the S3 Connector
reads t
ce message processing happens asynchronously to pulling
> messages from Kafka, and we hit an issue a while back related to topic
> compaction. I think we can solve it, but I'd like confirmation that the way
> we're going about it isn't wrong.
>
> The connector keeps track of w
offset should be retried the connector fetches the message
from Kafka again (it is not kept in-memory once emitted). We only clean up
the state for an offset once it is fully processed.
The issue we hit is that if topic compaction is enabled, we need to know
that the offset is no longer avail
