poll(0) after subscribe

Hi, I had a question around expected behavior / API expectations. Is subscribe a lazy call? What I mean by that is if I subscribe to a topic and then call poll(0), then is this first poll guaranteed to never return data? Will the data pre-fetch not start as soon as subscribe is called? Is this

java.nio.channels.ClosedChannelException

Hello community, I am running a simple consumer for kafka such as this int timeout = 8; int bufferSize = 64*1024; consumer = new SimpleConsumer(host, port,timeout, bufferSize, clientName); This runs fine for a couple for hours but I get an exception later on kafka.consumer.SimpleCo

Re: Kafka broker decommission steps

The command line tool that ships with Kafka is error prone. Our standard procedure is: 1. spin up the new broker 2. use `kafkat drain [--brokers ] 3. shut down old broker The `drain` command will generate and submit a partition assignment plan where the new broker id replaces the old one. It's p

Re: Kafka | Unable to publish data to broker - ClosedChannelException

It sounds like an issue with the topic. You can try describe the page_vists topic like: ./kafka-topics.sh --zookeeper localhost:2181 --describe and make sure it has a leader. If not, I would suggest creating a new topic and try if you can send messages to it. On Fri, Mar 4, 2016 at 3:34 AM, Sha

Re: Mirrormaker only publishing to a single partition on destination cluster

Took me a bit but worked out my issue. Appears like all of the messages in my topics were being hashed to a single partition using the DefaultPartitioner. Ended up rolling a simple round robin partitioner and contributed that back via this ticket: https://issues.apache.org/jira/browse/KAFKA-

Re: [sdc-user] Re: Having trouble to connect StreamSets to Kafka with Kerberos authentication

Hi Michal, The configuration in consumer.properties is not correct. The 'sasl.kerberos.service.name' option expects the kerberos principal that Kafka runs as. In your case it should be '*sasl.kerberos.service.name =*kafka*' * Can you please test using the Kafka

Uneven GC behavior between nodes

We're seeing something funny in one of our production clusters that I cannot explain away. Everything works fine, but as we're ramping up on Kafka, I really want to get at the root cause before we push a ton of traffic through it :) We have 6 nodes over three DCs in the cluster. Currently it's run

Re: Kafka Applicability - Large Messages

1GB sounds like a tad steep, you may want to do some testing, as Kafka needs to be told that such large messages can arrive and broker will then pre-allocate buffers for that. Personally, I'd stop short of low megabytes, anything bigger can be dropped off in e.g. S3 and then you just queue a link f

Re: Kafka Applicability - Large Messages

I have used messages upto 20MB and while not ideal, works fairly well. But if you are stepping into GB of data, you may need to chunk them up and reassemble. On Friday, March 4, 2016, Mahesh Dharmasena wrote: > We have a client with several thousand stores which send and receive > messages to m

Kafka Applicability - Large Messages

We have a client with several thousand stores which send and receive messages to main system that resides on the headquarters. A single Store sends and receive around 50 to 100 messages per day. Average Message size could be from 2KB to 1GB. Please let me know whether I can adapt Apache Kafka fo

Re: Kafka broker decommission steps

To answer your questions… 1 - Not in the way you want it to. There is a setting for automatic leader election (which I do not recommend anyone use at this time), but all that does is pick which of the currently assigned replicas should be the leader. It does not reassign partitions from one broker

Re: [sdc-user] Re: Having trouble to connect StreamSets to Kafka with Kerberos authentication

Dear Hari, Thank you for your reply. Replying to your questions: Yes, I have all needed entries in etc/hosts and hosts can 'see' each other. I followed your suggestion and added mentioned entries in server.properties_krb5. Now when starting Kafka Broker I see: listeners = PLAINTEXT://:9092,SASL_

Re: [sdc-user] Re: Having trouble to connect StreamSets to Kafka with Kerberos authentication

Dear Hari, Thank you for your reply. Replying to your questions: Yes, I have all needed entries in etc/hosts and hosts can 'see' each other. I followed your suggestion and added mentioned entries in server.properties_krb5. Now when starting Kafka Broker I see: listeners = PLAINTEXT://:9092,SASL_

GetOffsetShell doesn't work with SASL enabled Kafka

Hi team, I found that GetOffsetShell doesn't work with SASL enabled Kafka. I believe this is due to old producer being used in GetOffsetShell. I want to if there is any alternative to provide the same information with secure Kafka Kafka version 0.9.0.1 Exception % bin/kafka-run-class.sh kafka.t

Re: Kafka Security

Hi Martin, I suggest reading http://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption for an end to end example of how to secure Kafka. Ismael On Fri, Mar 4, 2016 at 12:38 PM, Martin Gainty wrote: > Although authors suggest using existing Cloud security produc

Kafka broker decommission steps

dear Kafka users, I have some questions regarding decommissioning kafka broker node and replacing it with the new one. Lets say that we have three broker nodes and each topic in Kafka has replication factor = 3, we upgrade one node with the following steps : 1. add one broker node to cluster 2. sh

RE: Kafka Security

Although authors suggest using existing Cloud security products such as Sentry (Cloudera) or Argus (Hortonworks) once Zookeeper adopted SASL integration ..kafka folk agreed SASL would be the best way to implement securing the following Kafka features : Authentication via SSL & Kerberos through

Kafka topic deletion still not working well

Hell Apache Kafka community, I'm still investigating an incident; from initial findings topic deletion doesn't seem to work well still with Kafka 0.9.0.1, likely some edge case not covered. Before with 0.8.2.x it used to happen that non-lead replica would be stuck in topic deletion process, and w

Lost broker node

Dear all, On our development cluster we have a topic which only has a single replica. Since creating it we lost a node. By using the kafka-topics -describe command I can see that where the replica was on the lost node (broker #404), the leader has now become "-1". Topic: alphaP

Re: Kafka | Unable to publish data to broker - ClosedChannelException

Hi I changed command to ./kafka-console-producer.sh --broker-list sandbox.hortonworks.com:6667 --topic page_visits Exception changed but no success. [2016-03-04 09:26:12,742] WARN Error while fetching metadata [{TopicMetadata for topic page_visits -> No partition metadata for topic page_visit

RE: Writing a Producer from Scratch

Thanks James. I will repost my question on dev mailing list and checkout the forum you mention. HTTP is a possibility but not the preferred option. Cheers. Steve Stephen Hopson | Infrastructure Architect | Enterprise Solutions Unisys | +44 (0)1908 805010 | +44 (0)7557 303321 | stephen.hop...@gb.