Re: kafka consumer rebalance

ers > to new consumers is not available. More details here: > https://issues.apache.org/jira/browse/KAFKA-4513 > > On Wed, Aug 30, 2017 at 11:05 AM, sunil kalva > wrote: > > > Hi > > Can someone help me here ..! > > > > On Tue, Aug 29, 2017 at 11:57 PM, s

Re: kafka consumer rebalance

Hi Can someone help me here ..! On Tue, Aug 29, 2017 at 11:57 PM, sunil kalva wrote: > Hi > Is there any change in consumer rebalancing algorithm from 0.8.2 and > 0.10.2 versions. If yes how do i do rolling upgrades of consumer client > applications, while doing rolling upgrade fe

kafka consumer rebalance

Hi Is there any change in consumer rebalancing algorithm from 0.8.2 and 0.10.2 versions. If yes how do i do rolling upgrades of consumer client applications, while doing rolling upgrade few of clients might be with older algo and few might be with newer algo then rebalancing might not be working we

consumer rebalancing (0.8.2.2) not happening

*I read this from kafka docs (https://github.com/SOHU-Co/kafka-node/issues/124 ), "When rebalancing is triggered at one consumer, rebalancing should be triggered in other consumers", I am emitting rebalance metrics for jmx bean (kafka.consumer:type

Re: Removing kafka topic contents from Java

Are you looking something like this String command = String.format("--delete --zookeeper %s --topic %s", "localhost:2181", "test_topic"); String[] split = command.split(" "); TopicCommand.main(split); t SunilKalva On Mon, Oct 24, 2016 at 9:09 PM, Ali Akhtar wrote: > There isn't a java API for

Re: delete topic causing spikes in fetch/metadata requests

Hi Can you guys help me with this issue On Oct 12, 2016 10:35 PM, "sunil kalva" wrote: > > We are using kafka 0.8.2.2 (client and server), when ever we delete a > topic we see lot of errors in broker logs like below, and there is also a > spike in fetch/metadata requests.

delete topic causing spikes in fetch/metadata requests

We are using kafka 0.8.2.2 (client and server), when ever we delete a topic we see lot of errors in broker logs like below, and there is also a spike in fetch/metadata requests. Can i correlate these errors with topic delete or its a known issue. Since there is spike in metadata requests and fetch

NotLeaderForPartitionException causing event loss

We are seeing data loss, whenever we see "NotLeaderForPartitionException" exception, We are using 0.8.2 java client publisher API with callback, when i get callback with the error i am logging them in a file and retrying them later. So number of errors = number of logged events are matching but ove

Re: Messages corrupted in kafka

re you > able to see some corrupted messages if you produce, say, 10M messages? > > On Wed, Mar 23, 2016 at 9:40 PM, sunil kalva > wrote: > > > I am using java client and kafka 0.8.2, since events are corrupted in > > kafka broker i cant read and replay them again. > &

Re: Messages corrupted in kafka

ause the broker actually validate the CRC before it stores the messages > on to the disk. > > Is this problem reproduceable? If so, can you find out the messages that > are corrupted? Also, are you using the Java clients or some other clients? > > Jiangjie (Becket) Qin > > On We

Re: Messages corrupted in kafka

can some one help me out here. On Wed, Mar 23, 2016 at 7:36 PM, sunil kalva wrote: > Hi > I am seeing few messages getting corrupted in kafka, It is not happening > frequently and percentage is also very very less (less than 0.1%). > > Basically i am publishing thrift event

Messages corrupted in kafka

Hi I am seeing few messages getting corrupted in kafka, It is not happening frequently and percentage is also very very less (less than 0.1%). Basically i am publishing thrift events in byte array format to kafka topics(with out encoding like base64), and i also see more events than i publish (i c

retrieve commit time for messages

Hi Is there any way to get the commit timestamp of the messages which are retrieved using kafka consumer API. t SunilKalva

Re: replication factor increased automatically !

We found the issue, it was with our UI tool which had a bug in partition reassignment tool. Nothing todo with kafka. Sorry for spamming. t Sunil On Mon, Nov 23, 2015 at 4:13 PM, sunil kalva wrote: > hi > sorry missed the header in my prev mail > > kafka_2.11-0.8.2.1/bin/kaf

Re: replication factor increased automatically !

_3 Partition: 1 Leader: 6 Replicas: 6,4,7 Isr: 7,4,6 > > > The line in bold (i.e. the first line in the output of the kafka-topics.sh) > gives on a topic level - #partitions, #replicationfactor etc > Please paste the full output > > Regards, > Prabhjot > >

Re: replication factor increased automatically !

u able to see 8 replicas(i.e. in ISR) for all the partitions? > > Can you paste the output of *--describe *topic command? > kafka-topics --zookeeper *-describe* > > On Mon, Nov 23, 2015 at 4:29 PM, sunil kalva > wrote: > > > Hi > > We have a 10 node kafka cl

replication factor increased automatically !

Hi We have a 10 node kafka cluster, and we have created a topic with replication factor "3" . It was running fine for 10 days, after that the replication factor is increased to "8" with out running any command to increase the replication. We have verified in server logs and also confirmed with the

Re: kafka metrics emitting to graphite

https://github.com/jmxtrans/jmxtrans > > > > B > > On 11 Oct 2015, at 11:19, sunil kalva wrote: > > > > How to configure, to emit kafka broker metrics to graphite. > > > > t > > SunilKalva > >

kafka metrics emitting to graphite

How to configure, to emit kafka broker metrics to graphite. t SunilKalva

Re: producer hung after some time

We also checked the jconsole for memory graph, it was taking only 80% of the heap (i can see still 20% remaining). -SK On Tue, Sep 29, 2015 at 8:13 PM, Gerrit Jansen van Vuuren < gerrit...@gmail.com> wrote: > also check your'e GC, caused by using async without backpressuce, with the > latest jdk

producer hung after some time

Hi Iam using kafka async producer, after publisher 1M+ messages, all my producer threads get hung and not doing any thing. Following is my thread dump. "KafkaMessageWorker-georrlite-2" #70 daemon prio=5 os_prio=0 tid=0x7f85d00cc800 nid=0x56b4 waiting on condition [0x7f859faf9000] java.l

New Async producer hangs

Hi I am using new asyc java producer , after couple of minutes i am getting following error prio=5 os_prio=31 tid=0x7f9383685800 nid=0xea03 waiting for monitor entry [0x000123a6a000] java.lang.Thread.State: BLOCKED (on object monitor) at org.xerial.snappy.buffer.CachedBufferAl

Re: Error handling in New AsyncProducer

> > On Mon, Aug 17, 2015 at 7:15 PM, sunil kalva > wrote: > > > tx jeff, > > Actually we need to set "buffer.memory" to minimum (default is ~35 MB) > and > > "block.on.buffer.full" to "true" so that the sender will block as soon > a

Re: Error handling in New AsyncProducer

gt; > public void onCompletion(RecordMetadata arg0, Exception arg1) { > > if (arg1 != null) { > > System.out.println("exception occured"); > > } > > System.out.println("sent") > > > > > > > > > > On Mon, Aug 17, 2015 at 9:20 PM, sunil kalv

Error handling in New AsyncProducer

Hi all I am using new java producer in async mode, when my entire cluster is down i am loosing all my messages. How do we get notification when the cluster is down so that i can send messages to another cluster. The callback is only triggered when the cluster is reachable . --SK

Fwd: Spooling support for kafka publishers !

-- Forwarded message -- From: sunil kalva Date: Fri, Aug 7, 2015 at 2:12 PM Subject: Spooling support for kafka publishers ! To: d...@kafka.apache.org Hi What are the best practises to achieve spooling support on producer end if the kafka cluster is not reachable or degraded

simple consumer giving error always

Hi For few topics i always get FetchResponse.error code as "1", i am using simple consumer 0.8.1. What are the reasons to get this error. and any fix for this ? -- SunilKalva

Re: running kafka/zookeeper running locally

get away with mocking out the Kafka > broker, depending on what/how you like to test. > > On Sun, Apr 19, 2015 at 10:47 PM, sunil kalva > wrote: > > > Hi > > Any one tried running zookeeper and kafka locally , which can be useful > for > > automating the test

Re: running kafka/zookeeper running locally

; Also consider with whether you could get away with mocking out the Kafka > > broker, depending on what/how you like to test. > > > > On Sun, Apr 19, 2015 at 10:47 PM, sunil kalva > > wrote: > > > > > Hi > > > Any one tried running zookeeper and kaf

running kafka/zookeeper running locally

Hi Any one tried running zookeeper and kafka locally , which can be useful for automating the test cases for API built on kafka ? SunilKalva

Re: blocking KafkaProducer call

> > On Wed, Apr 1, 2015 at 12:34 PM, sunil kalva wrote: > >> Does these config params has effect when i try to simulate "sync" mode by >> not passing callback ? >> >> On Wed, Apr 1, 2015 at 10:32 PM, Mayuresh Gharat < >> gharatmayures

Re: blocking KafkaProducer call

Does these config params has effect when i try to simulate "sync" mode by not passing callback ? On Wed, Apr 1, 2015 at 10:32 PM, Mayuresh Gharat wrote: > Whats your "linger.ms" and "batch.size" ? > > Thanks, > > Mayuresh > > On Wed, Apr 1, 201

Re: blocking KafkaProducer call

throughput. Please advice me hot to fix this. On Tue, Mar 31, 2015 at 11:21 PM, sunil kalva wrote: > thanks ghenke, that was a quick response. I will test and will let you > know if i have some questions. > > On Tue, Mar 31, 2015 at 11:17 PM, Grant Henke wrote: > >> I think

Re: blocking KafkaProducer call

xception is > thrown. > > The documentation for Future is here: > > http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/Future.html#get() > > On Tue, Mar 31, 2015 at 12:30 PM, sunil kalva > wrote: > > > Hi > > According to this > > > > &g

blocking KafkaProducer call

Hi According to this http://kafka.apache.org/082/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html documentation, if i don't pass callback it will work as blocking call, Does it mean that message will be immediately sent to kafka cluster and all possible exceptions will be thr

kafka audit

Hi What is best practice for adding audit feature in kafka, Is there any framework available for enabling audit feature at producer and consumer level and any UI frameworks for monitoring. tx SunilKalva

Re: New Java Producer Client handling case where Kafka is unreachable

5 at 3:36 PM, sunil kalva wrote: > > I think KafkaProducer.send method blocks until it fetches partition > > metadata for configured time using "metadata.fetch.timeout.ms", once > time > > out it throws TimeoutException. You might be experiencing > TimeoutExcepti

Re: New Java Producer Client handling case where Kafka is unreachable

I think KafkaProducer.send method blocks until it fetches partition metadata for configured time using "metadata.fetch.timeout.ms", once time out it throws TimeoutException. You might be experiencing TimeoutException ? ref: KafkaProducer.java(waitOnMetadata) On Fri, Mar 20, 2015 at 2:42 PM, Samue

Re: Async Producer Callback

gt; Mayuresh > > On Thu, Mar 19, 2015 at 11:05 AM, sunil kalva > wrote: > > > future returns RecordMetadata class which contains only metadata not the > > actual message. > > But i think *steven* had a point like saving the reference in impl class > > and retry

Re: Alternative to camus

Koert I am very new to spark, is it ok to you to share the code base for dumping data into hdfs from kafka using spark ? On Fri, Mar 20, 2015 at 12:20 AM, Koert Kuipers wrote: > we load from kafka into hdfs using spark in batch mode, once a day. it's > very simple (74 lines of code) and works f

Re: Async Producer Callback

2015 at 9:19 AM, Steven Wu wrote: > > > in your callback impl object, you can save a reference to the actual > > message. > > > > On Wed, Mar 18, 2015 at 10:45 PM, sunil kalva > > wrote: > > > > > Hi > > > How do i access the actual message whi

Re: publisher spooling ....!

ype patch to add limited persistence to the > producer. You can see it here: > https://issues.apache.org/jira/browse/KAFKA-1955 > > -Jay > > On Tue, Mar 3, 2015 at 8:15 AM, sunil kalva wrote: > > > Hi > > Is there any way to spool messages to disk at publisher side when kaf

Async Producer Callback

Hi How do i access the actual message which is failed to send to cluster using Callback interface and onCompletion method. Basically if the sender is failed to send, i want to add it to a temp queue and retry them later. --Sunil

Re: producer queue size

essentially i want to use this property "queue.buffering.max.messages" with new KafkaProducer class, and also want to access the current value of the queue SunilKalva On Wed, Mar 18, 2015 at 9:51 PM, sunil kalva wrote: > > Hi > How do i get the size of the inmemory queu

producer queue size

Hi How do i get the size of the inmemory queue which are holding messages and ready to send in async producer, i am using new KafkaProducer class in 0.8.2. Basically instead of dropping the messages silently, i want to avoid sending messages if the queue is already full. I am using async KafkaProd

Re: High level consumer group

he topic, you > need to set a different group.id for each. > > On Mon, Mar 16, 2015 at 10:10 AM, sunil kalva > wrote: > > How do i deploy high level consumer group for one topic in multiple > > instances of consumer applications, i mean i have two consumer > application

High level consumer group

How do i deploy high level consumer group for one topic in multiple instances of consumer applications, i mean i have two consumer applications deployed in two different boxes reading same topic which contains multiple partitions. Can i achieve this ? -- SunilKalva

Re: Kafka elastic no downtime scalability

Joe "Well, I know it is semantic but right now it "can" be elastically scaled without down time but you have to integrate into your environment for what that means it has been that way since 0.8.0 imho" here what do you mean "you have to integrate into your environment", how do i achieve elas

Re: kafka mirroring ...!

pic name in the > current mirror maker implementation. Mirror maker sends the message to > destination topic based on the topic name it reads from source > > On Mon, Mar 9, 2015 at 5:00 PM, sunil kalva wrote: > > > Can i configure different topic name in destination cluster, i

Re: kafka mirroring ...!

Another thing about duplicate messages is that if mirror maker is > > shutdown > > > cleanly, next time when you start it again with same consumer group id, > > > there should be no duplicates. But if mirror maker shutdown > > uncleanly(e.g. > > > By a kill -9), th

Re: kafka mirroring ...!

cer, the default behavior is > sticking to one partition for 10 min and then move to the next partition. > So if you wait for more than 10 min, you should see messages in two > different partitions. > > Jiangjie (Becket) Qin > > On 3/7/15, 8:28 AM, "sunil kalva" wrote: > &

Re: kafka mirroring ...!

And i also observed ,all the data is moving to one partition in destination cluster though i have multiple partitions for that topic in source and destination clusters. SunilKalva On Sat, Mar 7, 2015 at 9:54 PM, sunil kalva wrote: > I ran kafka mirroring tool after producing data in sou

kafka mirroring ...!

I ran kafka mirroring tool after producing data in source cluster, and this is not copied to destination cluster. If i produce data after running tool those data are copied to destination cluster. Am i missing something ? -- SunilKalva

Re: mapping between disk and partition

please be advice on this. On Fri, Mar 6, 2015 at 2:02 AM, sunil kalva wrote: > Hi > > Can i map a specific partition to a different disk in a broker. And what > is the general recommendations for disk to partition mapping for which that > broker is leader. and also for replicati

mapping between disk and partition

Hi Can i map a specific partition to a different disk in a broker. And what is the general recommendations for disk to partition mapping for which that broker is leader. and also for replications that broker handles. -- SunilKalva

Re: moving replications

Is there any way to automate On Mar 3, 2015 11:57 AM, "sunil kalva" wrote: > Why can't kafka automatically rebalances partitions with new broker and > adjust with existing brokers ? > Why should we run manually ? > > On Tue, Mar 3, 2015 at 6:41 AM, Gwen Shap

publisher spooling ....!

https://issues.apache.org/jira/browse/KAFKA-156, Is there any plan to support this in future or any alternatives. What is broker replication some one mentioned in the same jira, is there any documentation for that ? please advise me on this, my entire design depends on this feature. t Sunil Kalva

Re: moving replications

s to the list of brokers specified by > the --broker-list option. The format to use is... > > So it sounds like what you looked for? > > Since the tool runs in "dry run" mode, you can try and see without > actually moving anything. > > Gwen > > > > On Mar

Re: moving replications

isplay/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool > > Its a bit tricky to use, so feel free to follow up with more questions :) > > Gwen > > On Mon, Mar 2, 2015 at 7:59 AM, sunil kalva wrote: > > Hi > > How to move repl

moving replications

Hi How to move replications from one broker to another broker ? -- SunilKalva

load balancing

Is kafka load balancing based on number of partitions of a topic or number partitions of all topics in a cluster ? -- SunilKalva

question related to offset reset

Can i do a offset reset based on time not by message id, since most of times we know when the processing engine failed (based on time), so it is easy to reset the offset to that time for reprocessing. -- SunilKalva

Re: Custom partitioner in kafka-0.8.2.0

; "* > > > http://kafka.apache.org/082/javadoc/org/apache/kafka/clients/producer/ProducerRecord.html > > > ManiKumar > > On Thu, Feb 19, 2015 at 6:05 PM, sunil kalva > wrote: > > > Hi > > I could not find a way to customize "Partitioner" class in new > KafaProducer > > class, is it intentional ? > > > > tx > > SunilKalva > > >

Re: Custom partitioner in kafka-0.8.2.0

> Hi > I could not find a way to customize "Partitioner" class in new > KafaProducer class, is it intentional ? > > tx > SunilKalva >

Custom partitioner in kafka-0.8.2.0

Hi I could not find a way to customize "Partitioner" class in new KafaProducer class, is it intentional ? tx SunilKalva