Re: Possible DEAD LOCK for one day at broker controller?

It is log.deleteOldSegments(startMs - _.lastModified > log.config.retentionMs) You might have missed the startMs. I have tested it myself. I created a test topic with retention.ms equal to 20 minutes and added some messages. Later I changed the retention.ms to 2 min. I can see whenever the delete

Re: Possible DEAD LOCK for one day at broker controller?

Hi Kishore Senji, The size of segement file is default 1GB. According to the LogManager.scala#cleanupExpiredSegments, Kafka will only delete segments whose lastModTime is older than retention.ms, so I dont think this is the reason for my data loss. Actually I lost some data in topic other than th

Re: Error handling in New AsyncProducer

But this will reduce the throughput in a good scenario. May be we need to enhance the Callback interface appropriately. On Mon, Aug 17, 2015 at 7:15 PM, sunil kalva wrote: > tx jeff, > Actually we need to set "buffer.memory" to minimum (default is ~35 MB) and > "block.on.buffer.full" to "true"

Re: 0.8.2 producer and single message requests

If linger.ms is 0, batching does not add to the latency. It will actually improve throughput without affecting latency. Enabling batching does not mean it will wait for the batch to be full. Whatever gets filled during the previous batch send will be sent in the current batch even if it count is le

Re: Reduce latency

Yuheng, >From the command you gave it looks like you are configuring the perf test to send data as fast as possible (the -1 for target throughput). This means it will always queue up a bunch of unsent data until the buffer is exhausted and then block. The larger the buffer, the bigger the queue. T

Re: Reduce latency

Just to clarify it is 1 thread / Broker / Producer. Javadoc recommends to use one Producer across multiple threads and batching is done behind the scenes so throughput should not be a problem. So if you have more brokers, the same Producer instance will use multiple threads to send to each Broker,

Re: Reduce latency

If you run producerPerformance test, there is only one thread per KafkaProducer doing the actual sending. But one network request could contain multiple batches from what I understand. On Mon, Aug 17, 2015 at 5:42 PM, Yuheng Du wrote: > Thank you Kishore, I made the buffer twice the size of the

Re: Error handling in New AsyncProducer

tx jeff, Actually we need to set "buffer.memory" to minimum (default is ~35 MB) and "block.on.buffer.full" to "true" so that the sender will block as soon as these conditions met. And then release once the cluster is healthy. -- SunilKalva On Mon, Aug 17, 2015 at 11:20 PM, Jeff Holoman wrote:

Re: remote Kafka producer configuration and kafka.common.FailedToSendMessageException

Hi, Hawin: You are right. The main problem is my advertised.host.name at config/server.properites was wrong. I followed your instruction. And also http://edbaker.weebly.com/blog/installing-kafka-on-amazons-ec2 this instruction helped me a lot. Thanks a lot. Sincerely, Selina O

Re: remote Kafka producer configuration and kafka.common.FailedToSendMessageException

Hi, All: Thanks so much. Finally I fixed this bug. 1. set advertised.host.name at config/server.properites as AWS *private IP* address (not public *DNS*) 2. comment host.name at config/server.properites 3. In remote java producer: props.put("metadata.broker.list", borkerPrivateIp+ ":9092");

Re: Reduce latency

Thank you Kishore, I made the buffer twice the size of the batch size and the latency has reduced significantly. But is there only one thread io thread sending the batches? Can I increase the number of threads sending the batches so more than one batch could be sent at the same time? Thanks. O

Re: [DISCUSSION] Kafka 0.8.2.2 release?

Hi, Grant, I took a look at that list. None of those is really critical as you said. So, I'd suggest that we not include those to minimize the scope of the release. Thanks, Jun On Mon, Aug 17, 2015 at 5:16 PM, Grant Henke wrote: > Thanks Gwen. > > I updated a few small things on the wiki page

Re: [DISCUSSION] Kafka 0.8.2.2 release?

Gwen, Thanks for putting the list together. I'd recommend that we exclude the following: KAFKA-1702: This is for the old producer and is only a problem if there are some unexpected exceptions (e.g. UnknownClass). KAFKA-2336: Most people don't change offsets.topic.num.partitions. KAFKA-1724: The p

Re: [DISCUSSION] Kafka 0.8.2.2 release?

Thanks Gwen. I updated a few small things on the wiki page. Below is a list of jiras I think could also be marked as included. All of these, though not super critical, seem like fairly small and low risk changes that help avoid potentially confusing issues or errors for users. KAFKA-2012 KAFKA-9

Re: remote Kafka producer configuration and kafka.common.FailedToSendMessageException

if you want to connect remote kafka producer. Maybe you can try to use IP to start your producer. Making sure your producer and consumer are working fine. Then trying advertised host name to connect it. props.put("*metadata.broker.list", "I P to start your producer")

Re: Possible DEAD LOCK for one day at broker controller?

What is the size of the segment file? You are reducing the retention from 10 days to 1 day. The moment you do this, it will delete all segments which are older than 1 day. So for example, if your latest segment is older than 1 day and if there are consumers which are still catching up (let us say 1

Re: [DISCUSSION] Kafka 0.8.2.2 release?

Thanks for creating a list, Grant! I placed it on the wiki with a quick evaluation of the content and whether it should be in 0.8.2.2: https://cwiki.apache.org/confluence/display/KAFKA/Proposed+patches+for+0.8.2.2 I'm attempting to only cherrypick fixes that are both important for large number of

Re: remote Kafka producer configuration and kafka.common.FailedToSendMessageException

Hi, All: When I tried find the bug in previous email, I tried to produce message on Kafka broker server on AWS instance by command line below: *$** bin/kafka-console-producer.sh --broker-list localhost:9092 --topic temp1* [2015-08-17 21:42:59,468] WARN Property topic is not valid (kafka.utils.

Does Simple Consumer Shell work with Kafka committed offsets?

I'm trying to use the simple consumer shell to read a particular message, but I get no results for any parition+offset in my topic... I run something like this: ``` [kafka_2.10-0.8.2.1] # bin/kafka-simple-consumer-shell.sh --broker-list broker-01:9092,broker-02:9092,broker-03:9092 --offset 1 --par

remote Kafka producer configuration and kafka.common.FailedToSendMessageException

Dear All: I am looking for experts on Kafka to help me on remote Kafka java producer configuration . My Kafka java broker and producer are at different *AWS* instances. How should I set "metadata.broker.list" value. According to https://kafka.apache.org/08/configuration.html, The form

Re: Error handling in New AsyncProducer

Thanks for you explanation Jeff ! On Mon, Aug 17, 2015 at 11:23 PM, Jeff Holoman wrote: > I should've been more specific...if the producer loses total access to all > brokers...eg. some kind of network issue. > > On Mon, Aug 17, 2015 at 1:50 PM, Jeff Holoman > wrote: > > > Actually this won't w

Re: Compaction per topic

Hi Elias, You can set compaction on a per topic basis while leaving "log.cleanup.policy=delete" as the default on the broker. See "Topic-level configuration" here: http://kafka.apache.org/documentation.html#brokerconfigs An example usage of the command line tools to do this is: bin/kafka-topics.

Compaction per topic

Hi all, This is my first post here so please bear with me. I would like to have compaction enabled in some topics, but in others have purge after x amount of days. I did some searches and I couldn't find anything related to this and it appears that the compaction is enabled globally for all topi

Re: Error handling in New AsyncProducer

I should've been more specific...if the producer loses total access to all brokers...eg. some kind of network issue. On Mon, Aug 17, 2015 at 1:50 PM, Jeff Holoman wrote: > Actually this won't work. The problem is if the producer loses > connectivity to the broker, then messages will continue to

Re: Error handling in New AsyncProducer

Actually this won't work. The problem is if the producer loses connectivity to the broker, then messages will continue to queue up until batch.size is exhausted. Then the send will block. At this point, if you gain connectivity again, then the messages will be resent. If all brokers die, you shou

Re: 0.8.2 producer and single message requests

Thanks for the answers. Indeed, the callback model is the same regardless of batching. But for a synchronous web service, batching creates a latency issue. linger.ms is by default set to zero. Also, java futures are hard to work with compared to Scala futures. The current API also returns one fut

Question about unavailable partitions

Hi all, I'm using Kafka 8.2.1 with two brokers hosting a handful of topics. Unfortunately, both broker service died. When I brought them back up, several partitions are unavailable: > /opt/kafka/kafka-current/bin/kafka-topics.sh --zookeeper zoo1,zoo2,zoo3 --unavailable-partitions --describe

Re: [DISCUSSION] Kafka 0.8.2.2 release?

+1 for 0.8.2.2 release with fixes for KAFKA-2189, 2114 and 2308. On Mon, Aug 17, 2015 at 9:16 AM, Flavio Junqueira wrote: > It is pretty typical that Apache projects have a release manager for each > release: > > http://www.apache.org/dev/release-publishing.html < > http://www.apache.org/dev/rel

Re: Error handling in New AsyncProducer

Hi Sunil, Producer will throw an Exception in callback if there is problem while sending data. You can check like: public void onCompletion(RecordMetadata arg0, Exception arg1) { if (arg1 != null) { System.out.println("exception occured"); } System.out.println("sent") On Mon, Aug 17, 2015 at

Re: [DISCUSSION] Kafka 0.8.2.2 release?

It is pretty typical that Apache projects have a release manager for each release: http://www.apache.org/dev/release-publishing.html It doesn't have to be the same person every time, though, not even for the same branch. -Flavio > On 17 Au

Re: [DISCUSSION] Kafka 0.8.2.2 release?

but +1 for 0.8.2 patch that marks the new consumer API as unstable (or unimplemented ;) On Mon, Aug 17, 2015 at 9:12 AM, Gwen Shapira wrote: > The network refactoring portion was not tested well enough yet for me to > feel comfortable pushing it into a bugfix release. The new purgatory and > Mi

Re: [DISCUSSION] Kafka 0.8.2.2 release?

The network refactoring portion was not tested well enough yet for me to feel comfortable pushing it into a bugfix release. The new purgatory and MirrorMaker changes are also pretty big. The whole goal of bugfix releases is to make sure it is more stable than the previous releases. On Mon, Aug 17

Re: [DISCUSSION] Kafka 0.8.2.2 release?

Instead of cherrypicking, why not just make 0.8.2.2 of off current trunk, with new consumer API appropriately annotated/documented as unstable? On Mon, Aug 17, 2015, 17:17 Grant Henke wrote: > +dev > > Adding dev list back in. Somehow it got dropped. > > > On Mon, Aug 17, 2015 at 10:16 AM, Grant

Error handling in New AsyncProducer

Hi all I am using new java producer in async mode, when my entire cluster is down i am loosing all my messages. How do we get notification when the cluster is down so that i can send messages to another cluster. The callback is only triggered when the cluster is reachable . --SK

Re: [DISCUSSION] Kafka 0.8.2.2 release?

Below is a list of candidate bug fix jiras marked fixed for 0.8.3. I don't suspect all of these will (or should) make it into the release but this should be a relatively complete list to work from: - KAFKA-2114 : Unable to change min.insync.r

Re: [DISCUSSION] Kafka 0.8.2.2 release?

+dev Adding dev list back in. Somehow it got dropped. On Mon, Aug 17, 2015 at 10:16 AM, Grant Henke wrote: > Below is a list of candidate bug fix jiras marked fixed for 0.8.3. I don't > suspect all of these will (or should) make it into the release but this > should be a relatively complete li

Re: New producer hangs inifitely when it looses connection to Kafka cluster

Is there a known workaround for this? I believe I'm also encountering this issue. We are being affected by a separate issue that causes the network to drop periodically. When this happens, my producer hangs for up to 10 minutes minutes. The retries work immediately. Also, this seems to only affect

Re: thread that handle client request in Kafka brokers

Hi Tao This is unlikely to be a problem. The producer is threadsafe (see here ) so you can happily share it between your pool of message producers. Kafka also provides a range of facilities for

Re: Possible DEAD LOCK for one day at broker controller?

Hi Kishore Senji, Thanks for the reply. Do you have some suggestions before the fix came up? Try not to modify the retention.ms? Or disable the auto rebalance? Cause this problem is 100% reproduceable in my scenario (two times got dead lock in two retention.ms modification), and I even found some

Re: Kafka Cluster behind Proxy

Thanks Sriharsha. I used the above properties and it works for me. On Mon, Aug 17, 2015 at 7:11 PM, Sriharsha Chintalapani wrote: > Ankit, >Did you try using advertised.host.name and advertised.port by > specifying proxy host & port details. > > -- > Harsha > Sent with Airmail > > On A

Re: Kafka Cluster behind Proxy

Ankit,        Did you try using advertised.host.name and advertised.port by specifying proxy host & port details. --  Harsha Sent with Airmail On August 17, 2015 at 12:28:19 AM, Ankit Jain (ankitjainc...@gmail.com) wrote: Hi All, We want to deploy the Kafka cluster behind the proxy. We are e

Re: Describe command does not update details about killed brokers

I think you are running into KAFKA-972 (MetadataRequest returns stale list of brokers) which is marked to be fixed in the 0.8.3 release. Thank you, Grant On Mon, Aug 17, 2015 at 3:52 AM, Priya Darsini wrote: > Hi, > I have 3 brokers and created a

Topic partitions randomly failed on live system

Hi, We've had an issue on a live system (3 brokers, ~10 topics, some replicated, some partitioned) where a partition wasn't properly reassigned, causing several other partitions to go down. First, this exception happened on broker 1 (we weren't doing anything particular on the system at the ti

Describe command does not update details about killed brokers

Hi, I have 3 brokers and created a topic with replication factor 3. I described topic after killing the brokers one by one. When i deleted my last live broker and described topic it still shows value for leader and isr. I've enabled auto.leader.rebalance Any clarifications for this confusing beha

why does producer fail and does not try other brokers when one of brokers in the cluster fails?

Hi, I have a kafka cluster consisting of two servers. I created a topic XYZ with 3 partitions and replication factor of 2. In the producer side, the producer is configured with broker list of both brokers broker0 and broker1. Topic:XYZ PartitionCount:3 ReplicationFactor:2 Configs: Topic: Replica

Kafka Cluster behind Proxy

Hi All, We want to deploy the Kafka cluster behind the proxy. We are exposing the proxy IP to client. The client is using the proxy IP to connect to zookeeper server to consume the data from Kafka cluster, but as we know, the zookeeper return the broker IP's to client and then client directly comm

Publish large number of messages for load testing

Hello, I am load testing my streaming application. Currently, I am calling the producer.send a large number of times to queue up the messages, then running the consumer to check the throughput. The producer code looks like this: for (long nEvents = 0; nEvents < loopCntrMax; nEvents++) { long