New consumer - mirror maker - blacklist

Hi All, The docs currently say the following: However, --blacklist is not supported when the new consumer has been enabled (i.e. when bootstrap.servers has been defined in the consumer configuration) Is there an alternative to using a blacklist when using the new consumer type? So that it is pos

Kafka broker crash - broker id then changed

Hi All, We are running Kafka version 0.9.0.1, at the time the brokers crashed yesterday we were running in a 2 mode cluster. This has now been increased to 3. We are not specifying a broker id and relying on kafka generating one. After the brokers crashed (at exactly the same time) we left kafka

Re: Kafka broker crash - broker id then changed

ther files present in temp. Thanks! On Thu, May 26, 2016 at 9:19 AM, Ben Davison wrote: > Possibly tmp got cleaned up? > > Seems like one of the log files where deleted while a producer was writing > messages to it: > > On Thursday, 26 May 2016, cs user wrote: > > >

Kafka behind a load balancer

Hi All, Does anyone have any experience of using kafka behind a load balancer? Would this work? Are there any reasons why you would not want to do it? Thanks!

Re: Kafka behind a load balancer

s not need, and indeed will not work behind a > load balancer. I'd recommend reading the docs for more, but > http://kafka.apache.org/documentation.html#design_loadbalancing is a good > start. > > Thanks > > Tom Crayford > Heroku Kafka > > On Fri, Jun 3, 2016 at 1:15 P

Mirror maker setup - multi node

Hi All, So I understand I can run the following to aggregate topics from two different clusters into a mirror: bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config sourceCluster1Consumer.config --consumer.config sourceCluster2Consumer.config --num.streams 2 --producer.config targetClu

Re: Mirror maker setup - multi node

etty > easy to change the behaviour of the mirrormaker, for example to copy it to > $topic-aggregation instead of $topic, and to not copy it when the topic > ends with aggregation > > On Tue, Jun 28, 2016 at 10:15 AM cs user wrote: > > > Hi All, > > > > So I understa

Enabling PLAINTEXT inter broker security

Hi All, I'm dipping my toes into kafka security, I'm following the guide here: http://kafka.apache.org/documentation.html#security_sasl_plain_brokerconfig and http://kafka.apache.org/documentation.html#security_sasl_brokerconfig My jaas config file looks like: KafkaServer { org.apac

Re: Enabling PLAINTEXT inter broker security

at 11:50 AM, cs user wrote: > Hi All, > > I'm dipping my toes into kafka security, I'm following the guide here: > http://kafka.apache.org/documentation.html#security_sasl_plain_brokerconfig > and http://kafka.apache.org/documentation.html#security_sasl_brokerconfig >

Re: Enabling PLAINTEXT inter broker security

gt; Manikumar > > On Fri, Jul 15, 2016 at 4:22 PM, cs user wrote: > > > Apologies, just to me clear, my broker settings are actually as below, > > using PLAINTEXT throughout > > > > listeners=SASL_PLAINTEXT://host.name:p

Re: Enabling PLAINTEXT inter broker security

PM, cs user wrote: > Yep, tried 0.10.0.0, all working fine :-) > > I was using 0.9. > > Apologies for the spam! > > On Fri, Jul 15, 2016 at 12:05 PM, Manikumar Reddy < > manikumar.re...@gmail.com> wrote: > >> Hi, >> >> Which Kafka version you are u

SSL / SASL_SSL questions

Hi All, I have a question about the config I have working, and whether or not all traffic is being encrypted when sent via the client. Lets say I have the following settings, I'm only including the relevant parameters: Broker config: listeners=SASL_SSL://:9092,SSL://:9093 log.message.format.ve

kafka mirrormaker - new consumer version - message delay

Hi All, I've recently enabled ssl for our cluster, and as we are using a mirror maker I'm now starting our mirror maker processes with the --new.consumer switch. I can see that now the mirror maker consumer group type has switched from ZK to KF. However I've started to notice a delay when sending

Re: kafka mirrormaker - new consumer version - message delay

Hi All, Adding this setting does the trick it seems: --offset.commit.interval.ms 5000 This defaults to 60,000. Not sure if this has any adverse affects by lowering it to 5 seconds. Cheers! On Thu, Jul 21, 2016 at 10:50 AM, cs user wrote: > Hi All, > > I've recently enabl

Re: Question regarding functionality of MirrorMaker

Hi Umesh, I am new to kafka as well, and configuring the MirrorMaker. I got mine working in the following way. I run the mirror maker instance on the mirror cluster, as in where you want to mirror the topics to, although I'm not sure it matters. I use the following options when starting my servi

Re: Question regarding functionality of MirrorMaker

eeper . > > However, did you notice that MM picks the topics which are present at the > time of its startup and mirrors the data. When you add some new topics > after its startup it doesn't pick it automatically? > > Regards, > Umesh Chaudhary > > On Thu, 25 Aug 2016 at 1

Re: Kafka broker crash - broker id then changed

# Exclude namespace mountpoints created with PrivateTmp=yes x /tmp/systemd-private-%b-* X /tmp/systemd-private-%b-*/tmp x /var/tmp/systemd-private-%b-* X /var/tmp/systemd-private-%b-*/tmp Cheers! On Thu, May 26, 2016 at 9:27 AM, cs user wrote: > Hi Ben, > > Thanks for responding