Re: producer-consumer issues during deployments

you will be able to consumer from it. If the leader is on a > machine that goes offline then consumption will pause whilst leadership > switches over to a replica. > > All the best > B > > > On 26 Nov 2015, at 17:58, Prabhjot Bharaj wrote: > > > > Hi, > > &g

Re: producer-consumer issues during deployments

Hi, Request your expertise on these doubts of mine Thanks, Prabhjot On Thu, Nov 26, 2015 at 4:43 PM, Prabhjot Bharaj wrote: > Hi, > > We arrange our kafka machines in groups and deploy these phases. > > For kafka, we’ll have to map groups with phases. During each phase of the

producer-consumer issues during deployments

Hi, We arrange our kafka machines in groups and deploy these phases. For kafka, we’ll have to map groups with phases. During each phase of the release, all the machines in that group can go down. When this happens, there are a couple of cases:- 1. All replicas are residing in a group of mach

Re: kafka 0.8 producer issue

Hi, >From the information that you've provided, I think your callback is the culprit here. It is seen from the stacktrace:- at com.hpe.ssmamp.kafka.KafkaPDFAProducer$1.onCompletion( KafkaPDFAProducer.java:62) Please provide more information like a code snippet etc, so that we can tell more

Re: kafka producer and client use same groupid

Hi Surender, Please elaborate on your design Consumers don't talk to producers directly, Kafka is a brokered system, and Kafka sits between producers and consumers Also, consumers consume from partitions of a topic and producers write to partitions in a topic These partitions and the logical abstr

Re: All brokers are running but some partitions' leader is -1

Hi, With the information provided, these are the steps I can think of (based on the experience I had with kafka):- 1. do a describe on the topic. See if the partitions and replicas are evenly distributed amongst all. If not, you might want to try the 'Reassign Partitions Tool' - https://cwiki.apa

Re: Kafka Scalability & Partition

Hi, Having such a model will not scale. I think it it's mentioned in earlier posts as well as some wikis available out there Kafka works very well if you have more partitions, so you can massively parallel writes to Kafka. Also, your application need not send partition ids/numbers, unless you've

Re: Debug kafka code in intellij

Hello Folks, Requesting your expertise on this Thanks, Prabhjot On Thu, Nov 5, 2015 at 4:46 PM, Prabhjot Bharaj wrote: > Hi, > > I'm using kafka 0.8.2.1 version with IntelliJ. > Sometimes, I change the code and build it using this command: > > ./gradlew -PscalaVers

Re: New and updated producers and consumers

t fully supports security features like SSL > > > that are slated to be released into the platform. > > > > > > Your choice of development language is entirely up to you. Note that > the > > > only version of clients that will be maintained in the

Re: New and updated producers and consumers

as simple) Thanks, Prabhjot Thanks, Prabhjot On Thu, Nov 5, 2015 at 3:55 PM, Prabhjot Bharaj wrote: > Adding users as well > > On Thu, Nov 5, 2015 at 3:37 PM, Prabhjot Bharaj > wrote: > >> Hi, >> >> I'm using the latest update: 0.8.2.2 >> I would

Re: New and updated producers and consumers

Adding users as well On Thu, Nov 5, 2015 at 3:37 PM, Prabhjot Bharaj wrote: > Hi, > > I'm using the latest update: 0.8.2.2 > I would like to use the latest producer and consumer apis > over the past few weeks, I have tried to do some performance benchmarking > using th

New and updated producers and consumers

Hi, I'm using the latest update: 0.8.2.2 I would like to use the latest producer and consumer apis over the past few weeks, I have tried to do some performance benchmarking using the producer and consumer scripts provided in the bin directory. It was a fun activity and I have learnt a lot about ka

Re: Producer becomes slow over time

machine that is not part of the kafka cluster? > > Can you include your topic description/configuration, producer > configuration, and broker configuration? > > On 9/24/15, 1:44 AM, "Prabhjot Bharaj" wrote: > > >Hi, > > > >I would like to dig deep into t

Re: Producer becomes slow over time

46 PM, Prabhjot Bharaj wrote: > Hello Folks, > > I've noticed that 2 producer machines, that I had configured, have become > very slow over time > They are giving 17-19 MB/s > > But, a producer that I setup today is giving 70MB/s as the write throughput > > If I

Re: [jira] [Commented] (KAFKA-2569) Kafka should write its metrics to a Kafka topic

Hi, I have setup a Ganglia host to get all the stats from my 5 node kafka cluster. I run jmxtrans on Kafka nodes which export to Ganglia. I have exposed all of the available stats to Ganglia and have grouped them as under:- Kafka Cluster Stats metrics (18)Kafka Consumer Stats metrics (208)Kafka C

Fwd: Useful metric to check slow ISR catchup

Hi Dev Folks, Request your expertise on this doubt of mine Thanks, Prabhjot -- Forwarded message -- From: Prabhjot Bharaj Date: Mon, Sep 21, 2015 at 2:59 PM Subject: Re: Useful metric to check slow ISR catchup To: us...@kafka.apache.org Hi, Attaching a screenshot of bytes/sec

Re: [ANNOUNCE] New Committer Sriharsha Chintalapani

Congratulations. It's inspiring for newbies like me Regards, Prabhjot On Sep 22, 2015 10:30 AM, "Ashish Singh" wrote: > Congrats Harsha! > > On Monday, September 21, 2015, Manikumar Reddy > wrote: > > > congrats harsha! > > > > On Tue, Sep 22, 2015 at 9:48 AM, Dong Lin > > wrote: > > > > > Con

Re: Contributor List

My username: pbharaj Thanks, Prabhjot On Sep 8, 2015 10:14 AM, "Prabhjot Bharaj" wrote: > Hi, > > Request you to add me as well for code contributions > > Regards, > Prabhjot > On Sep 8, 2015 2:10 AM, "Gwen Shapira" wrote: > >> Done :) >>

Re: Contributor List

Hi, Request you to add me as well for code contributions Regards, Prabhjot On Sep 8, 2015 2:10 AM, "Gwen Shapira" wrote: > Done :) > > Happy hacking. > > On Mon, Sep 7, 2015 at 11:44 AM, Bill Bejeck wrote: > > > Hi Just a reminder to add me to the contributors list! I've started > looking > >

Re: Contributor List

Hi Gwen, I'm new to Kafka codebase, however I'm using it for evaluation right now I'm trying to understand the code right now On this note, I would like to ask of I can also be a part of the contributor list Thanks, Prabhjot On Sep 5, 2015 4:28 PM, "Bill Bejeck" wrote: > My Jira user name is

Re: Hello!

Hi Sudhanshu, You can go through the details mentioned here: http://kafka.apache.org/contributing.html Also, in case you need to setup your IDE, you can refer to these links:- http://www.lewuathe.com/blog/2014/10/16/build-apache-kafka-with-intellij-idea/ https://cwiki.apache.org/confluence/displ

Re: kafka producer-perf-test.sh compression-codec not working

ucer > produces, and the consumer looks at the difference between between the > timestamp in the body and the current timestamp. > > Or, if you were looking for ack latency, you can use the producer’s async > callback to measure latency. > -Erik > > From: Prabhjot Bharaj m

Re: kafka producer-perf-test.sh compression-codec not working

compression is concerned. If you do implement an > improvement, a patch might be helpful to the community. But something to > consider is that threwput alone isn’t the only important performance > measure. Round trip latency is also important. > Thanks, > -Erik > > > Fr

Re: kafka producer-perf-test.sh compression-codec not working

> goal is for each message to be unique, but still have similar chunks that > a compression algorithm can detect and compress. > > -Erik > > > On 8/25/15, 6:47 AM, "Prabhjot Bharaj" wrote: > > >Hi, > > > >I have bene trying to use kafka-producer-

kafka producer-perf-test.sh compression-codec not working

Hi, I have bene trying to use kafka-producer-perf-test.sh to arrive at certain benchmarks. When I try to run it with --compression-codec values of 1, 2 and 3, I notice increased throughput compared to NoCompressionCodec But, When I checked the Producerperformance.scala, I saw that the the `produc

Re: Best practices - Using kafka (with http server) as source-of-truth

riety of other solutions) in > front of REST proxies for load balancing, HA, SSL termination, etc. This is > yet another hop, so it might affect throughput and latency. > > -Ewen > > On Mon, Jul 27, 2015 at 6:55 AM, Prabhjot Bharaj > wrote: > > > Hi Folks, > > > &g

Re: Connection to zk shell on Kafka

You need the jline JAR file that ships with ZooKeeper. > > Chris > > IBM Tivoli Systems > Research Triangle Park, NC > (919) 224-2240 > Internet: barl...@us.ibm.com > > > > From: Prabhjot Bharaj > To: us...@kafka.apache.org, u...@zookeeper.apache

KAfka Mirror Maker

Hi, I'm using Mirror Maker with a cluster of 3 nodes and cluster of 5 nodes. I would like to ask - is the number of nodes a restriction for Mirror Maker? Also, are there any other restrictions or properties that should be common across both the clusters so that they continue mirroring. I'm aski

Re: Number of kafka topics/partitions supported per cluster of n nodes

I'd > like to see these! > > On Mon, Jul 27, 2015 at 10:51 PM, Prabhjot Bharaj > wrote: > > > Hi, > > > > I'm looking forward to a benchmark which can explain how many total > number > > of topics and partitions can be created in a cluster of n nod

Re: Number of kafka topics/partitions supported per cluster of n nodes

@Jiefu Gong, Are the results of your tests available publicly? Regards, Prabhjot On Tue, Jul 28, 2015 at 10:35 PM, Prabhjot Bharaj wrote: > I would be using the servers available at my place of work. I dont have > access to AWS servers. I would starting off with a small number of no

Re: Number of kafka topics/partitions supported per cluster of n nodes

; as to test Zookeeper's expansibility and disk IO performance . > > 2015-07-28 13:51 GMT+08:00 Prabhjot Bharaj : > > > Hi, > > > > I'm looking forward to a benchmark which can explain how many total > number > > of topics and partitions can be created in a c

Number of kafka topics/partitions supported per cluster of n nodes

Hi, I'm looking forward to a benchmark which can explain how many total number of topics and partitions can be created in a cluster of n nodes, given the message size varies between x and y bytes and how does it vary with varying heap sizes and how it affects the system performance. e.g. the resu

Best practices - Using kafka (with http server) as source-of-truth

Hi Folks, I would like to understand the best practices when using kafka as the source-of-truth, given the fact that I want to pump in data to Kafka using http methods. What are the current production configurations for such a use case:- 1. Kafka-http-client - is it scalable the way Nginx is ??

Zookeeper use cases with Kafka

Hello Folks, I wish to contribute to Kafka internals. And, one of the things which can help me do that is understanding how kafka uses zookeeper. I have some of these basic doubts:- 1. Is zookeeper primarily used for locking ? If yes, in what cases and what kind of nodes does it use - sequential/