Consumer pause/resume & partition assignment race condition

While performing some prototyping on 0.10.0.0 using the new client API I noticed that some some clients fail to drain their topic partitions. The Kafka cluster is comprised of 3 nodes. The topic in question has been preloaded with messages. The topic has 50 partitions. The messages were loaded

Re: Kafka streams for out of order density aggregation

I just want to add something: If I understand the question correctly, you are asking for a strong ordering guarantee. I personally doubt that out-of-order on count-based windows can be supported with strong consistency guarantee in an efficient manner. If a late record arrives for window X, the n

Re: log4j setting for embedded kafka server

Siyuan, log4j.properties only gets read by the kafka-run-class.sh scripts: KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/config/tools-log4j.properties If you start the server within your Java application, you need to try to pass "log4j.configuration" to Kafka. Guozhang On Fri, Jun

Re: issue in SimpleConsumerDemo

Did you run it multiple times, and did not clean the committed offsets? This may be a common root cause of seeing fewer messages. Guozhang On Fri, Jun 24, 2016 at 2:51 AM, hengheng0h...@163.com < hengheng0h...@163.com> wrote: > hi, > I got an issue when i run SimpleConsumerDemo(source:kafka > <

Producer Properties

I started porting our code from Kafka 0.8.2.1 to 0.10.0.0 and found my producer code blowing up because of some changes to the config. For example, metadata.broker.list is now bootstrap.servers. I discovered the ProducerConfig class which has, at least, some of the config keys. Before I scr

Re: Kafka producer metadata issue

Intersting. So if we introduce a sleep after the first send then it produces properly? Here is my log. Clearly there is a conn reset. [2016-06-24 13:42:48,620] ERROR Closing socket for /127.0.0.1 because of error (kafka.network.Processor) java.io.IOException: Connection reset by peer at sun.nio

Re: Kafka producer metadata issue

I'm seeing similar with the v9 producer. Here is some test code: @Test public void test1() throws InterruptedException { Producer producer = createProducer(BROKER_DEV); producer.send(new ProducerRecord<>(TOPIC, "value")); producer.send(new ProducerRecord<>(TOPIC, "key2", "value2"));

Re: Kafka producer metadata issue

I just see this on kafka.log file [2016-06-24 13:27:14,346] INFO Closing socket connection to /127.0.0.1. (kafka.network.Processor) On Fri, Jun 24, 2016 at 1:05 PM, Shekar Tippur wrote: > Hello, > > I have a simple Kafka producer directly taken off of > > > https://kafka.apache.org/090/javadoc/

Kafka producer metadata issue

Hello, I have a simple Kafka producer directly taken off of https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html I have changed the bootstrap.servers property. props.put("bootstrap.servers", "localhost:9092"); I dont see any events added to the t

[kafka-connect] multiple or single clusters?

I'm having some trouble figuring out the right way to run Kafka Connect in production. We will have multiple sink connectors that we need to remain running indefinitely and have at least once semantics (with as little duplication as possible) so it seems clear that we need to run in distributed mod

Re: log.retention.bytes

Hi Dave, log.retention.bytes is per partition. If you change it after the topic was created, you'll see the behavior you expect -- namely that the new value is used when the log is cleaned. The frequency that the log is cleaned is controlled by log.retention.check.interval.ms, with a default value

log4j setting for embedded kafka server

Hi guys, I start server grammatically in my application using KafkaStatableServer.startup() method. And in the log4j.properties setting. I add this log4j.logger.org.apacke.kafka=WARN log4j.logger.kafka=WARN But I always get INFO log, Do you guys know how to enforce the log level here? Thanks!

log.retention.bytes

Is the log.retention.bytes setting per partition or for the whole topic?If I change it after a topic has been created do the changes apply to the existing topics? Thanks, Dave This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are inte

issue in SimpleConsumerDemo

hi, I got an issue when i run SimpleConsumerDemo(source:kafka/examples/src/main/java/kafka/examples/SimpleConsumerDemo.java) I only got 2 messages when i set fetchSize to 100. thanks! hengheng0h...@163.com

RE: Setting max fetch size for the console consumer

Thanks! I also had to pass --consumer.config=/etc/kafka/consumer.properties to the command line consumer. -Dave Dave Tauzell | Senior Software Engineer | Surescripts O: 651.855.3042 | www.surescripts.com | dave.tauz...@surescripts.com Connect with us: Twitter I LinkedIn I Facebook I YouTube

Re: Setting max fetch size for the console consumer

It’s actually more than one setting: http://stackoverflow.com/questions/21020347/kafka-sending-a-15mb-message B > On 24 Jun 2016, at 14:31, Tauzell, Dave wrote: > > How do I set the maximum fetch size for the console co

Setting max fetch size for the console consumer

How do I set the maximum fetch size for the console consumer? I'm getting this error when doing some testing with large messages: kafka.common.MessageSizeTooLargeException: Found a message larger than the maximum fetch size of this consumer on topic replicated_twice partition 28 at fetch offset

Re: is kafka the right choice

correction: elevates => alleviates > On 24 Jun 2016, at 11:13, Ben Stopford wrote: > > Kafka uses a long poll > . So requests > effectively block on the server, if there is insufficient data available. > This elevates many of the issues

kafka lost data when use scala API to send data.

Hello With Respect, Here I met a problem when use scala API to send/receive data to/from kafka brokers. I write a very simple producer and consumer code(just like the official examples), I found the code with Java API can work correctly, but the code with Scala API may lost data. Here is detai

Re: is kafka the right choice

Kafka uses a long poll . So requests effectively block on the server, if there is insufficient data available. This elevates many of the issues associated with traditional polling approaches. Service-based applications often require direc