customising security across the whole of Confluent platform

2019-09-17 Thread Joris Peeters
Hello, I am trying to come up with a good security approach for a Kafka project inside our company. I see there's a variety of options available (ACL/RBAC/certificates/...), and I'm hoping someone can suggest a few possibilities. Some points of interest, - Kafka will be used both by end-users/res

Re: Kafka consumer Fetcher several Ignoring fetched records logs

2019-09-17 Thread Aminouvic
Hello, My apologies for the late reply. No the data isn't getting deleted faster than the consumer is reading it. We also notice when this happens the number of Replica Fetchers doubles on the broker side and the total of out bytes drops significantly. Regards, Amine On 2019/09/07 04:18:18,

Re: customising security across the whole of Confluent platform

2019-09-17 Thread Rajini Sivaram
Hi Joris, I have forwarded your mail to *secur...@confluent.io * since it is about security in the Confluent Platform rather than in Apache Kafka. Regards, Rajini On Tue, Sep 17, 2019 at 11:35 AM Joris Peeters wrote: > Hello, > > I am trying to come up with a good security approach for a Kafk

log.retention.bytes not working

2019-09-17 Thread Vinay Kumar
Hi, I have the same problem mentioned here by Neha. Attached the comment. https://grokbase.com/t/kafka/users/138hje002v/questtion-about-log-retention-bytes Can someone please provide how effectively to impose the retention for all topics.

Re: log.retention.bytes not working

2019-09-17 Thread M. Manna
Hi, This is a very old post. Kafka has come far. Are you saying that log.retention.bytes (broker leave - all topics) and/or retention.bytes (topic level - individual topic) isn’t working for you ? Thanks, On Tue, 17 Sep 2019 at 20:02, Vinay Kumar wrote: > Hi, > I have the same problem mention

Client development - OffsetFetch request always returns UNKNOWN_TOPIC_OR_PARTITION

2019-09-17 Thread Dan Swain
Hi! I'm a maintainer of an open source Kafka client and working on adding support for kafka-stored offsets. The client works fine with v0 of the OffsetCommit/OffsetFetch messages, but with v1+ it always fails to fetch the offset. I've been digging quite a bit and found that the OffsetCommit appe

Re: log.retention.bytes not working

2019-09-17 Thread Vinay Kumar
I don't see log.retention.bytes only is not working. Even after the specified size in log.retention.bytes reached, the topic partition with segments grows much beyond. On Wed, Sep 18, 2019, 1:31 AM M. Manna wrote: > Hi, > > This is a very old post. Kafka has come far. > > Are you saying that log