hello Kafka

Hello Kafka 2014-08-27 yanghaogn

How to upgrade Kafka broker to a newer version without impacting live traffic?

Hi All, If I want to upgrade from Kafka 0.8.1.1 to a newer version, what are the steps that I need to do without impacting production traffic? More specifically, how do I have the new brokers to continue writing to the existing logs assuming the file format has not changed? Is there a migration

Re: kafka unit test in Java, TestUtils choosePort sends NoSuchMethodError

As far as I understand it shouldnt have such a dependency. JVM should be able to compile this. I am using JDK1.6. Also this is a unit test. Correct me if I am wrong. I have not downloaded any scala before even till this far. -Parin On Tue, Aug 26, 2014 at 10:12 PM, Jun Rao wrote: > Could you

Re: How to upgrade Kafka broker to a newer version without impacting live traffic?

Assuming the partitions are replicated, a migration plan could be: - install latest kafka on broker $i, /etc/kakfa-NEW - update server config in /etc/kakfa-NEW/config/server.properties to match the old one (in particular zookeeper.connect and log.dirs) - stop broker $i - start broker $i from /etc/

message size limit

Hello, my brokers are reporting that some received messages exceed the `message.max.bytes` value. I'd like to know what producers are at fault but It is pretty much impossible: - the brokers don't log the content of the rejected messages - the log messages do not contain the IP of the producers -

Re: Consumer sensitive expiration of topic

Kafka is designed to maintain persistent backlog of data on disk efficiently and at scale. Unlike other messaging systems, doing so does not affect the performance of the system. If you are worried about the messages occupying disk space, you can always set a lower retention on the topic that is hi

Re: kafka unit test in Java, TestUtils choosePort sends NoSuchMethodError

Kafka is written in scala. So to run Kafka, you need a scala jar. Thanks, Jun On Wed, Aug 27, 2014 at 4:05 PM, Parin Jogani wrote: > As far as I understand it shouldnt have such a dependency. JVM should be > able to compile this. I am using JDK1.6. Also this is a unit test. > Correct me if I

Re: message size limit

The message size check is currently only done on the broker. If you enable trace level logging in RequestChannel, you will see the produce request, which includes the size of each partition. Thanks, Jun On Wed, Aug 27, 2014 at 4:40 PM, Alexis Midon < alexis.mi...@airbedandbreakfast.com> wrote:

Re: message size limit

Hi Jun, thanks for you answer. Unfortunately the size won't help much, I'd like to see the actual message data. By the way what are the things to consider when deciding on `message.max.bytes` value? On Wed, Aug 27, 2014 at 9:06 PM, Jun Rao wrote: > The message size check is currently only

Re: message size limit

am I miss reading this loop: https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/log/Log.scala#L265-L269 it seems like all messages from `validMessages` (which is ByteBufferMessageSet) are NOT appended if one of the message size exceeds the limit. I hope I'm missing something.