Number of partition on a single broker

Can I have 20k partition on a single kafka broker ?

Re: Kafka Monitoring

There are a couple of tools for this. 1. Linkedin Burrow - https://github.com/linkedin/Burrow 2. Yahoo Kafka Monitoring Tool - https://github.com/yahoo/kafka-manager 3. Uber Kafka Monitoring Tool - https://github.com/uber/chaperone Regards, Abhimanyu On Wed, Dec 6, 2017 at 3:14 PM, Ir

org.apache.kafka.common.errors.TimeoutException: Failed to send request after 5000 ms

I am using Kafka V 0.10.2 (8 Cores, 16 GB RAM). While executing the command > ./kafka-consumer-groups.sh --bootstrap-server localhost:9092 > --describe --group Sometimes I am getting this exception : Exception in consumer group command (kafka.admin.ConsumerGroupCommand$) java.lang.Runti

Re: 答复: kafka.admin.TopicCommand Failing

, 2017 at 11:38 AM, Hu Xi wrote: > Increasing `zookeeper.connection.timeout.ms` to a relatively larger value > might be a help. Besides, you could check GC log to see if the STW pauses > expired the Zk sessions. > > > ____ > 发件人: Abhimanyu Nagrath

Re: kafka.admin.TopicCommand Failing

. Regards, Abhimanyu On Thu, Nov 16, 2017 at 5:08 PM, Abhimanyu Nagrath < abhimanyunagr...@gmail.com> wrote: > Hi, I am using a single node Kafka V 0.10.2 (16 GB RAM, 8 cores) and a > single node zookeeper V 3.4.9 (4 GB RAM, 1 core ). I am having 64 consumer > groups and 500 topics

kafka.admin.TopicCommand Failing

Hi, I am using a single node Kafka V 0.10.2 (16 GB RAM, 8 cores) and a single node zookeeper V 3.4.9 (4 GB RAM, 1 core ). I am having 64 consumer groups and 500 topics each having 250 partitions. I am able to execute the commands which require only Kafka broker and its running fine ex. > ./kafka-

Re: consumer group offset lag

I am also facing the same issue. On Tue, Jul 25, 2017 at 9:58 PM, Tarun Garg wrote: > I am sending this mail to DEV because i think this is a bug in AdminClient > while getting the group offset. > > Hi, > > I run through a issue. > i am trying to get the status of a consumer group. > > for this

Re: KIP-162: Enable topic deletion by default

+1 On Wed, Jul 19, 2017 at 8:35 PM, Bill Bejeck wrote: > +1 > > On Wed, Jul 19, 2017 at 10:51 AM, Damian Guy wrote: > > > +1 > > > > On Wed, 31 May 2017 at 13:36 Jim Jagielski wrote: > > > > > +1 > > > > On May 27, 2017, at 9:27 PM, Vahid S Hashemian < > > > vahidhashem...@us.ibm.com> wrote: >

Re: ticketing system Design

Hi , Can anyone suggest me where I can get the answer for these type of questions? Regards, Abhimanyu On Thu, Jun 8, 2017 at 6:49 PM, Abhimanyu Nagrath < abhimanyunagr...@gmail.com> wrote: > Hi , > > Is Apache Kafka along with storm can be used to design a ticketing system.

Re: Kafka Monitoring

Hi Muhammad, You can use the monitoring tool developed by LinkedIn https://github.com/linkedin/Burrow to get the data information within a time window. Regards, Abhimanyu On Tue, Jun 20, 2017 at 4:22 AM, Muhammad Arshad < muhammad.ars...@alticeusa.com> wrote: > Hi, > > wanted to see if there is

Re: Re-Balancing Kafka topics - Best practices

Hi Tod, I am not able to access the link https://github.com/linkedin/kafka-toolskafka-assigner Regards, Abhimanyu On Fri, Jun 16, 2017 at 12:26 AM, karan alang wrote: > Thanks Todd.. for the detailed reply. > > regds, > Karan Alang > > On Tue, Jun 13, 2017 at 3:19 PM, Todd Palino wrote: > >

ticketing system Design

Hi , Is Apache Kafka along with storm can be used to design a ticketing system. By ticketing system, I mean that there are millions of tasks stored in Kafka queues and there are processes/humans to take some actions on the task. there are come constraints that same task should not be assigned to t

Re: Producer Async Issue

Hey Jay, What will be the disadvantages of setting max.block.ms=0? Regards, Abhimanyu On Mon, May 29, 2017 at 3:15 AM, Jay Kreps wrote: > I think setting max.block.ms=0 does what you want. > > -Jay > > On Sat, May 27, 2017 at 12:40 PM, Abhimanyu Nagrath < > abhimanyunag

Queries regarding kafka Monitoring tool burrow.

Hi , I am using burrow to monitor kafka Lags and I am having following queries : 1.On hitting the API /v2/kafka/local/consumer/group1/lag I am not able to view all the topics details present in that group and getting complete: false in the above JSON. What does this mean? Below mentioned is the j

Re: Producer Async Issue

; > > On May 27, 2017, at 5:15 AM, Abhimanyu Nagrath < > abhimanyunagr...@gmail.com> wrote: > > > > Hi, > > I am using Kafka 0.10.2 single node cluster and I want to know > > Kafka producer completely asynchronous. So what configuration I need to

Re: Trouble with querying offsets when using new consumer groups API

Hi Jerry, I am also facing the same issue. Did you found the solution? Regards, Abhimanyu On Fri, May 26, 2017 at 7:24 PM, Jerry George wrote: > Hi > > I had question about the new consumer APIs. > > I am having trouble retrieving the offsets once the consumers are > *disconnected* when using

Producer Async Issue

Hi, I am using Kafka 0.10.2 single node cluster and I want to know Kafka producer completely asynchronous. So what configuration I need to change in order to make producer completely asynchronous. Regards, Abhimanyu

Re: Messages are repeating in kafka

hat is the problem with setting offsets.retention.minutes == > log.retention.hours > ? Did it fix the problem you were facing before? > > > On Tue, May 23, 2017 at 11:40 PM, Abhimanyu Nagrath < > abhimanyunagr...@gmail.com> wrote: > > > Hi Kant, > > > > After setting thi

Re: Messages are repeating in kafka

offsets.retention.minutes= 1440 * 30 ? Regards, Abhimanyu On Mon, May 22, 2017 at 4:09 PM, Abhimanyu Nagrath < abhimanyunagr...@gmail.com> wrote: > @Kant I was going through the offset related configurations before setting > offsets.retention.minutes so came accross this configuration and thou

Re: Messages are repeating in kafka

do you think you need to set that? Did you try setting > offsets.retention.minutes > = 1440 * 30 and still seeing duplicates? > > On Mon, May 22, 2017 at 12:37 AM, Abhimanyu Nagrath < > abhimanyunagr...@gmail.com> wrote: > > > Hi Girish , > > &

Re: Messages are repeating in kafka

ion of the offsets topic is an upper bound > across all topics. So in this case, don't create any other topics in the > future with retention of more than 30 days or otherwise they may have the > same problem too. > > On May 21, 2017 03:25, "Abhimanyu Nagrath" > wr

Re: Messages are repeating in kafka

9 AM, kant kodali wrote: > > > >> What is your average message size and network speed? > >> > >> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath < > >> abhimanyunagr...@gmail.com> wrote: > >> > >>> Hi Girish, > >>> >

Re: Messages are repeating in kafka

Hi, My average message size is less 30 Kb and I get 1 near about 5 Million message per day On Sun, May 21, 2017 at 2:49 PM, kant kodali wrote: > What is your average message size and network speed? > > On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath < > abhimanyunagr...@g

Re: Messages are repeating in kafka

Hi Girish, I did not set any value for offsets.retention.minutes so therefore what I think is picking its default value i.e 1440 minutes so what do you think what should I set if I am keeping my data for 30 days? Regards, Abhimanyu

Messages are repeating in kafka

I am using single node Kafka broker (v 0.10.2) Hardware (8 core, 16 GB RAM, 1 TD HardDisk) and zookeeper (v 3.4.8). I have a topic with 200 partition in which messages contains the total of 3 Million messages. It took 5 days to completely process all the messages and as soon as message got processe

Re: Heartbeat failed w/ multi-group/consumers

I am also facing the same issue . Regards, Abhimanyu On Thu, May 18, 2017 at 5:29 AM, Cédric Chantepie < c.chante...@yahoo.fr.invalid> wrote: > Hi, > > I have a test app using Java lib for consumers with Kafka 0.10, using > Kafka storage for offset. > > This app is managing 190 consumers, accro

Kafka Consumer Attached to partition but not consuming messages

https://issues.apache.org/jira/browse/KAFKA-5221 . Regards, Abhimanyu

Re: Kafka - Move topics to different Partition in same hardware

Hi Srinivasa, If you are using ebs disk then you can increase the size of your disk at runtime. else stop your Kafka node, attach a bigger disk then manually copy data to the newly attached disk specify the new path in Kafka server.properties and restart kafka. Regards, Abhimanyu On Wed, May 10,

Re: Issue with kafka-server-stop on RedHat7

Hi Vedant, Just try to run kill -s TERM $KafkaProcessPID . On Wed, May 10, 2017 at 12:31 AM, Vedant Nighojkar wrote: > Hi Team, > > We are using Apache Kafka in one of our products. We support Windows, AIX > and Linux RedHat6 and above. > > I am seeing an issue with the kafka-server-stop.sh scr

WARN Attempting to send response via channel for which there is no open connection, connection id 0 (kafka.network.Processor)

Hi, I am using single node Kafka 0.10.2 . and my server.log file is full with this warning *WARN Attempting to send response via channel for which there is no open connection, connection id 0 (Kafka.network.Processor)* . Can anyone suggest how to fix this warning. Regards, Abhimanyu

Error: Executing consumer group command failed due to Request GROUP_COORDINATOR failed on brokers List(localhost:9092 (id: -1 rack: null))

I am using single node Kafka 0.10.2 and while running this command *./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group * on my Kafka broker server 80 % time its throwing this error. Error: Executing consumer group command failed due to Request GROUP_COORDINATOR failed o

Fwd: Error: Executing consumer group command failed due to Request GROUP_COORDINATOR failed on brokers List(localhost:9092 (id: -1 rack: null))

-- Forwarded message -- From: Abhimanyu Nagrath Date: Thu, May 4, 2017 at 4:21 PM Subject: Error: Executing consumer group command failed due to Request GROUP_COORDINATOR failed on brokers List(localhost:9092 (id: -1 rack: null)) To: d...@kafka.apache.org Hi, I am using single