Kafka Connect usage

Hi, I tried executing the following, bin/connect-standalone.sh config/connect-standalone.properties config/connect-file-source.properties config/connect-console-sink.properties I created a file text.txt in kafka directory but get the error, ERROR Failed to flush WorkerSourceTask{id=local-file-s

Stalling behaviour with 0.9 console consumer

Hi, I started with a clean install of 0.9 Kafka broker and populated a test topic with 1 million messages. I then used the console consumer to read from beginning offset. Using --new-consumer reads the messages, but it stalls after every x number of messages or so, and then continues again. It is v

Re: fallout from upgrading to the new Kafka producers

Hi Rajiv, This warning could be ignored and is indeed done in 0.9.0, where we downgrade the logging level for it from WARN to DEBUG. So if you upgrade to 0.9.0 Java producer you should not see this warning. A bit more context on the EOFException, a socket closure could result this; and a server c

Re: error in KafkaConsumer javadocs?

Hey Richard, Yeah, I think you're right. I think this is the same issue from KAFKA-2478, which appears to have been forgotten about. I'll see if we can get the patch merged. -Jason On Mon, Jan 11, 2016 at 4:27 PM, Richard Lee wrote: > Apologies if this has been discussed already... > > The ‘Ma

error in KafkaConsumer javadocs?

Apologies if this has been discussed already... The ‘Manual Offset Control’ section of the KafkaConsumer javadocs has some code that looks like this: ConsumerRecords records = consumer.poll(100); for (ConsumerRecord record : records) { buffer.add(record);

Re: Java client 0.9 poll doesn't return

Looks like you might have bootstrap.servers pointed at Zookeeper. It should point to the Kafka brokers instead. The behavior of poll() currently is to block until the group's coordinator is found, but sending the wrong kind of request to Zookeeper probably results in a server-side disconnect. In th

Java client 0.9 poll doesn't return

Calling the stand alone client (org.apache.kafka" % "kafka-clients" % “0.9.0.0”) from Scala, consumer.poll never returns. I’ve tried both assign TopicPartition and subscribe and various timeouts and I’ve quintuple checked config properties. Here’s a Scala-Ide worksheet val props = loadProper

Re: Kafka 0.9 Consumer Group

Ah, that actually makes sense. The consumer-groups.sh script only returns offset data when the group is active. The offsets should still be there, but it does seem unfortunate that there's no way to view them. We have KAFKA-3059 which adds some additional capabilities for managing offsets with this

fallout from upgrading to the new Kafka producers

We have recently upgraded some of our applications to use the Kafka 0.8.2 Java producers from the old Java wrappers over Scala producers. We've noticed these log messages on our application since the upgrade: 2016-01-11T20:56:43.023Z WARN [roducer-network-thread | producer-2] [s.o.a.kafka.common

Re: kafka-producer-perf-test.sh - 0.8.2.1

Thanks! I also found old thread and Ewen did reply to that. So I will just re-post so somebody else might find it helpful: Ewen: ... EndToEndLatency works with a single message at a time. It produces the message then waits for the consumer to receive it. This approach guarantees there is no delay

Re: Kafka 0.9 Consumer Group

Hi Jason, I used the kafka-consumer-groups.sh to check my consumer group : ~/GitHub/kafka/bin/kafka-consumer-groups.sh --bootstrap-server‹group test.group --describe --new-consumer . I ran this command several times after my app was shut down. I always get "Consumer group `test.group` doe

Re: kafka-producer-perf-test.sh - 0.8.2.1

If you use the perf test without any bound on throughput it will always try to send data faster than it can go out and build up a queue of unsent data. So e.g. if your buffer is 1MB each send will be blocked on waiting for the full 1MB of queued data to clear out and get sent. This makes sense if y

Re: mirror maker against 0.8.2 source cluster and 0.9.0 destination cluster

The MirrorMaker 0.9.0 has some nice features like messageHandler which would be great to use. Any ideas how this new functionality could be used in a 0.8.2 source cluster and 0.9.0 destination cluster in an efficient way? Thanks On Wed, Jan 6, 2016 at 11:30 AM, Ismael Juma wrote: > Hi Stephen,

Re: trouble sending produce requests to 0.9.0.0 broker cluster

I created the following bug report: https://issues.apache.org/jira/browse/KAFKA-3088 -Original Message- From: "Dave Peterson" Sent: Monday, January 11, 2016 9:56am To: users@kafka.apache.org Subject: Re: trouble sending produce requests to 0.9.0.0 broker cluster Ok, thanks for the infor

Re: kafka-producer-perf-test.sh - 0.8.2.1

Ewen, One more question. I mentioned that *kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance* latency is a lot higher than *kafka-run-class.sh kafka.tools.TestEndToEndLatency.* Example: *ProducerPerformance:* 5000 records sent, 337463.891364 records/sec (32.18 MB/sec), *

Re: trouble sending produce requests to 0.9.0.0 broker cluster

Ok, thanks for the information. I tried setting the client ID to "bruce" and the problem disappeared. I looked at https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol and it doesn't say anything about an empty client ID not being allowed (even if it isn't allowed, Kaf

Re: Kafka 0.9 Consumer Group

Sorry, wrong property, I meant enable.auto.commit. -Jason On Mon, Jan 11, 2016 at 9:52 AM, Jason Gustafson wrote: > Hi Howard, > > The offsets are persisted in the __consumer_offsets topic indefinitely. > Since you're using manual commit, have you ensured that auto.offset.reset > is disabled? I

Re: Kafka 0.9 Consumer Group

Hi Howard, The offsets are persisted in the __consumer_offsets topic indefinitely. Since you're using manual commit, have you ensured that auto.offset.reset is disabled? It might also help if you provide a little more detail on how you're verifying that offsets were lost. -Jason On Mon, Jan 11,

Kafka Is Featured on HPE Matter

Hello! HPE Matter, the digital magazine from HPE where the brightest minds in business share their perspectives on a technology-driven world, recently launched its seventh issue. HPE Matter’s The Next Enterprise Issue explores how tech-driven enterprises of the future

Re: best python library to use?

Agree - kafka-python was in hibernation waiting for 0.9.0.0 Kafka release, so a few issues lingered longer than I would have liked. Most of my comments relate to latest master, which we are hoping to release after a bit more testing and polish. Re librdkafka -- to be honest, I'm skeptical that C p

RE: trouble sending produce requests to 0.9.0.0 broker cluster

Looks like you aren't setting the request client-id, and server is crashing on it. I'm not sure whether server api is expected to work w/o client-id, but you can probably fix by sending one. Fwiw, kafka-python sends 'kafka-python' unless user specifies something else. -Dana On Jan 11, 2016 8:41 AM

Re: best python library to use?

kafka-python (https://github.com/dpkp/kafka-python) has also just merged performance improvements to the consumer in https://github.com/dpkp/kafka-python/issues/290 which should see a pretty decent boost in throughput. We were somewhat put off by the poor performance in earlier versions, I imagine

Broker Health check

Hi, I and my team are beginning with kafka so I/we may not have read everything or understood everything yet. The context is we’d like to bootstrap kafka in our own codebase, and first step is to have a deployable walking skeleton that the team can grow along our experience and needs. We are sta

RE: trouble sending produce requests to 0.9.0.0 broker cluster

I forgot to include some information about this problem. When I sent the produce request, the following appeared in server.log: [2016-01-10 23:03:44,957] ERROR [KafkaApi-3] error when handling request Name: ProducerRequest; Version: 0; CorrelationId: 1; ClientId: null; RequiredAcks: 1; AckT

Re: best python library to use?

pykafka’s balanced consumer is very useful. pykafka also has Python bindings to the librdkafka C library that you can optionally enable, which might get you some speed boosts. python-kafka (oh, I just saw this 0.9x version, hm!) was better at producing than pykafka for us, so we am currently using

Kafka 0.9 Consumer Group

Hi, I have a question regarding the Kafka 0.9 Consumer Group . I manually commit offsets using the Kafka 0.9 Consumer created with a consumer group. However, after my app restarted totally from scratch, the consumer group seems to lose all the offsets. Is that true that the consumer offsets ar

Re: Upgraded from 0.8.2 to 0.9.0 and FetchFollower request time has increased

Hmm over the weekend looks like this slowly crept back up as shown in this graph: http://i.imgur.com/DH9Qa09.png I performed the upgrade on 1/8 around 12am as shown by the spike in the graph. Here's a graph with a little more history going back ~1 month. This was pretty stable prior to the upgra

Apache Kafka Usage

Dear Apache Kafka users, I'm currently looking into using Apache Kafka on our infrastructure and I came up with lots of questions that I would like to clarify. I have created a Stackoverflow post here: http://stackoverflow.com/questions/34715556/apache-kafka-for-time-series-data-persistence Coul

trouble sending produce requests to 0.9.0.0 broker cluster

Hi, I'm having trouble sending produce requests to a Kafka 0.9.0.0 broker cluster consisting of 4 brokers with IDs 0, 1, 2, and 3. All 4 brokers are running locally on my CentOS 7 development box, listening on ports 9092, 9093, 9094, and 9095 respectively. I am running my own producer code (Bruc