Re: Using encrypted distributed worker file

2020-06-15 Thread Tom Bentley
Hi Ashish, KIP-297[1] added support for "config providers", which allow a config file to have an indirect reference to secrets stored elsewhere. While it doesn't sound like the provided FileConfigProvider would be suitable for your needs, you could provide your own ConfigProvider to implement a se

kafka log retention problem

2020-06-15 Thread Ncir, Kawther (external - Service)
Hi, I am working on a project using kafka broker and I face a problem with kafka log retention , We need to delete logs every 24h but after the retention time the broker was stopped and make an issue like this : "ERROR Uncaught exception in scheduled task 'kafka-log-retention' (kafka.utils.Kaf

Re: kafka log retention problem

2020-06-15 Thread Ricardo Ferreira
It sounds like you are trying to forcibly delete the files that build up the segments used by the partitions. If that is the case then I would recommend not using external tools and leave to Kafka manage its filesystem. If you set the retention policy (either by size or time) in your topics the

Unexpected behavior of log.message.format.version

2020-06-15 Thread Jordan Pilat
Heya, I am on a Kafka 1.1.1 cluster, with inter.broker.protocol.version=1.1, but log.message.format.version=0.9.0 (magic v0) I am planning out an upgrade of log.message.format.version, and in doing so, running some tests locally, and have a couple questions: The behavior I see is, when an updat