Hi Kafka people, I'm running an application that pushes database changes into a Kafka topic. I'm also running a Kafka streams application that listens to these topics, and groups them using the high level API, and inserts them to another database.
All topics are compacted, with the exception of the 'repartition topics', which are configured to be retained for 36 hours. Note that the changes in the original kafka topics can be old (generally more than 36 hours), as they only change when the data changes. When I start an instance of the Kafka Streams application, I see the repartition topics being deleted immediately, sometimes before they are processed, and it looks like the repartition messages use the same timestamp as the original message. I see that the repartition topics have timestamp.type = CreateTime, does that mean it uses the timestamp of the original message? Shouldn't that be LogAppendTime for repartition topics? Or is there a way to configure that? If I hack into my Kafka streams code to force it to use LogAppendTime seems to solve my problem, but that appears to take a huge toll on the brokers. Throughput plummets, and I don't really know why. Any ideas? Frank
