Kafka 0.8's VerifyConsumerRebalance reports an error

[Yury Ruchin]

Hi,

I've run into the following problem. I try to read from a 50-partition
Kafka topic using high level consumer with 8 streams. I'm using 8-thread
pool, each thread handling one stream. After a short time, the threads
reading from the stream stop reading. Lag between topic latest offset and
the consumer constantly increases as new messages come in.

I looked into ZK:  /consumers//owners/ and see a
list of znodes corresponding to the full list of partitions: [1, 2, 3,
...]. When I do zk get on e. g.
/consumers//owners//1 - I see a valid consumer
name corresponding to Kafka client logs, e. g.
_-140304960-abc12345-0. However, when I run
the VerifyConsumerRebalance tool, I see the following:

No owner for partition [,1]
(kafka.tools.VerifyConsumerRebalance$)

No owner for partition [,2]
(kafka.tools.VerifyConsumerRebalance$)

...

No owner for partition [,50]
(kafka.tools.VerifyConsumerRebalance$)

According to this output, no partition has owner, which seemingly
contradicts to what I see in ZK.

What would cause such a problem and how can I troubleshoot it further?

Thanks!

How does number of partitions affect sequential disk IO

[Daniel Compton]

I’ve been reading the Kafka docs and one thing that I’m having trouble 
understanding is how partitions affect sequential disk IO. One of the reasons 
Kafka is so fast is that you can do lots of sequential IO with read-ahead cache 
and all of that goodness. However, if your broker is responsible for say 20 
partitions, then won’t the disk be seeking to 20 different spots for its writes 
and reads? I thought that maybe letting the OS handle fsync would make this 
less of an issue but it still seems like it could be a problem.

In our particular situation, we are going to have 6 brokers, 3 in each DC, with 
mirror maker replication from the secondary DC to the primary DC. We aren’t 
likely to need to add more nodes for a while so would it be faster to have 1 
partition/node than say 3-4/node to minimise the seek times on disk?

Are my assumptions correct or is this not an issue in practice? There are some 
nice things about having more partitions like rebalancing more evenly if we 
lose a broker but we don’t want to make things significantly slower to get 
this.  

Thanks, Daniel.

Re: How does number of partitions affect sequential disk IO

[Paul Mackles]

You'll want to account for the number of disks per node. Normally,
partitions are spread across multiple disks. Even more important, the OS
file cache reduces the amount of seeking provided that you are reading
mostly sequentially and your consumers are keeping up.

On 6/24/14 3:58 AM, "Daniel Compton"  wrote:

>I¹ve been reading the Kafka docs and one thing that I¹m having trouble
>understanding is how partitions affect sequential disk IO. One of the
>reasons Kafka is so fast is that you can do lots of sequential IO with
>read-ahead cache and all of that goodness. However, if your broker is
>responsible for say 20 partitions, then won¹t the disk be seeking to 20
>different spots for its writes and reads? I thought that maybe letting
>the OS handle fsync would make this less of an issue but it still seems
>like it could be a problem.
>
>In our particular situation, we are going to have 6 brokers, 3 in each
>DC, with mirror maker replication from the secondary DC to the primary
>DC. We aren¹t likely to need to add more nodes for a while so would it be
>faster to have 1 partition/node than say 3-4/node to minimise the seek
>times on disk?
>
>Are my assumptions correct or is this not an issue in practice? There are
>some nice things about having more partitions like rebalancing more
>evenly if we lose a broker but we don¹t want to make things significantly
>slower to get this.
>
>Thanks, Daniel.

Re: Consumer offset is getting reset back to some old value automatically

[Hemath Kumar]

Yes kane i have the replication factor configured as 3


On Tue, Jun 24, 2014 at 2:42 AM, Kane Kane  wrote:

> Hello Neha, can you explain your statements:
> >>Bringing one node down in a cluster will go smoothly only if your
> replication factor is 1 and you enabled controlled shutdown on the brokers.
>
> Can you elaborate your notion of "smooth"? I thought if you have
> replication factor=3 in this case, you should be able to tolerate loss
> of a node?
>
> >>Also, bringing down 1 node our of a 3 node zookeeper cluster is risky,
> since any subsequent leader election might not reach a quorum.
>
> So, you mean ZK cluster of 3 nodes can't tolerate 1 node loss? I've
> seen many recommendations to run 3-nodes cluster, does it mean in
> cluster of 3 you won't be able to operate after loosing 1 node?
>
> Thanks.
>
> On Mon, Jun 23, 2014 at 9:04 AM, Neha Narkhede 
> wrote:
> > Bringing one node down in a cluster will go smoothly only if your
> > replication factor is 1 and you enabled controlled shutdown on the
> brokers.
> > Also, bringing down 1 node our of a 3 node zookeeper cluster is risky,
> > since any subsequent leader election might not reach a quorum. Having
> said
> > that, a partition going offline shouldn't cause a consumer's offset to
> > reset to an old value. How did you find out what the consumer's offset
> was?
> > Do you have your consumer's logs around?
> >
> > Thanks,
> > Neha
> >
> >
> > On Mon, Jun 23, 2014 at 12:28 AM, Hemath Kumar 
> > wrote:
> >
> >> We have a 3 node cluster ( 3 kafka + 3 ZK nodes ) . Recently we came
> across
> >> a strange issue where we wanted to bring one of the node down from
> cluster
> >> ( 1 kafka + 1 zookeeper) for doing a maintenance . But the movement we
> >> brought it to down on some of the topics ( only some partitions)
> consumers
> >> offset is reset some old value.
> >>
> >> Any reason why this is happened?. As of my knowledge when brought one
> node
> >> down its should work smoothly with out any impact.
> >>
> >> Thanks,
> >> Murthy Chelankuri
> >>
>

Re: How does number of partitions affect sequential disk IO

[Daniel Compton]

Good point. We've only got two disks per node and two topics so I was planning 
to have one disk/partition. 

Our workload is very write heavy so I'm mostly concerned about write 
throughput. Will we get write speed improvements by sticking to 1 
partition/disk or will the difference between 1 and 3 partitions/node be 
negligible?

> On 24/06/2014, at 9:42 pm, Paul Mackles  wrote:
> 
> You'll want to account for the number of disks per node. Normally,
> partitions are spread across multiple disks. Even more important, the OS
> file cache reduces the amount of seeking provided that you are reading
> mostly sequentially and your consumers are keeping up.
> 
>> On 6/24/14 3:58 AM, "Daniel Compton"  wrote:
>> 
>> I¹ve been reading the Kafka docs and one thing that I¹m having trouble
>> understanding is how partitions affect sequential disk IO. One of the
>> reasons Kafka is so fast is that you can do lots of sequential IO with
>> read-ahead cache and all of that goodness. However, if your broker is
>> responsible for say 20 partitions, then won¹t the disk be seeking to 20
>> different spots for its writes and reads? I thought that maybe letting
>> the OS handle fsync would make this less of an issue but it still seems
>> like it could be a problem.
>> 
>> In our particular situation, we are going to have 6 brokers, 3 in each
>> DC, with mirror maker replication from the secondary DC to the primary
>> DC. We aren¹t likely to need to add more nodes for a while so would it be
>> faster to have 1 partition/node than say 3-4/node to minimise the seek
>> times on disk?
>> 
>> Are my assumptions correct or is this not an issue in practice? There are
>> some nice things about having more partitions like rebalancing more
>> evenly if we lose a broker but we don¹t want to make things significantly
>> slower to get this.
>> 
>> Thanks, Daniel.
> 

Re: How does number of partitions affect sequential disk IO

[Paul Mackles]

Its probably best to run some tests that simulate your usage patterns. I
think a lot of it will be determined by how effectively you are able to
utilize the OS file cache in which case you could have many more
partitions. Its a delicate balance but you definitely want to err on the
side of having more partitions. Keep in mind that you are only able to
parallelize down to the partition level so if you have only have 2
partitions, you can only have 2 consumers. Depending on your volume, that
might not be enough.

On 6/24/14 6:44 AM, "Daniel Compton"  wrote:

>Good point. We've only got two disks per node and two topics so I was
>planning to have one disk/partition.
>
>Our workload is very write heavy so I'm mostly concerned about write
>throughput. Will we get write speed improvements by sticking to 1
>partition/disk or will the difference between 1 and 3 partitions/node be
>negligible?
>
>> On 24/06/2014, at 9:42 pm, Paul Mackles  wrote:
>> 
>> You'll want to account for the number of disks per node. Normally,
>> partitions are spread across multiple disks. Even more important, the OS
>> file cache reduces the amount of seeking provided that you are reading
>> mostly sequentially and your consumers are keeping up.
>> 
>>> On 6/24/14 3:58 AM, "Daniel Compton"  wrote:
>>> 
>>> I¹ve been reading the Kafka docs and one thing that I¹m having trouble
>>> understanding is how partitions affect sequential disk IO. One of the
>>> reasons Kafka is so fast is that you can do lots of sequential IO with
>>> read-ahead cache and all of that goodness. However, if your broker is
>>> responsible for say 20 partitions, then won¹t the disk be seeking to 20
>>> different spots for its writes and reads? I thought that maybe letting
>>> the OS handle fsync would make this less of an issue but it still seems
>>> like it could be a problem.
>>> 
>>> In our particular situation, we are going to have 6 brokers, 3 in each
>>> DC, with mirror maker replication from the secondary DC to the primary
>>> DC. We aren¹t likely to need to add more nodes for a while so would it
>>>be
>>> faster to have 1 partition/node than say 3-4/node to minimise the seek
>>> times on disk?
>>> 
>>> Are my assumptions correct or is this not an issue in practice? There
>>>are
>>> some nice things about having more partitions like rebalancing more
>>> evenly if we lose a broker but we don¹t want to make things
>>>significantly
>>> slower to get this.
>>> 
>>> Thanks, Daniel.
>> 

What's the behavior when Kafka is deleting messages and consumers are still reading

[Lian, Li]

When Kafka broker is trying to delete message according to the log retention 
setting, either triggered by log age or topic partition size, if the same time 
there are still consumers reading the topic, what will happen?

Best regards,
Lex Lian

Email: ll...@ebay.com

Announcing Kafka Web Console v2.0.0

[Claude Mamo]

Announcing the second major release of Kafka Web Console:
https://github.com/claudemamo/kafka-web-console/releases/tag/v2.0.0.
Highlights:

- I've borrowed some ideas from Kafka Offset Monitor and added graphs to
show the history of consumers offsets and lag as well as message throughput

- Added partition leaders, log size, offset and lag

- Bug fixes!

Claude

Re: Announcing Kafka Web Console v2.0.0

[Joe Stein]

Awesome Claude, thanks!

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/


On Tue, Jun 24, 2014 at 10:57 AM, Claude Mamo  wrote:

> Announcing the second major release of Kafka Web Console:
> https://github.com/claudemamo/kafka-web-console/releases/tag/v2.0.0.
> Highlights:
>
> - I've borrowed some ideas from Kafka Offset Monitor and added graphs to
> show the history of consumers offsets and lag as well as message throughput
>
> - Added partition leaders, log size, offset and lag
>
> - Bug fixes!
>
> Claude
>

Re: high level consumer not working

[Guozhang Wang]

Hi Li Li,

If you use the same consumer group id then offsets may have already been
committed to Kafka, hence messages before that will not be consumed.

Guozhang


On Mon, Jun 23, 2014 at 6:09 PM, Li Li  wrote:

> no luck by adding props.put("auto.offset.reset", "smallest");
> but running consumer after producer works.
> But in my use case, it's not alwasy true for this.
> Another question, The consumer should remember the offset. it's not
> very easy to use.
>
> On Mon, Jun 23, 2014 at 11:05 PM, Guozhang Wang 
> wrote:
> > Did you start the consumer after the producer? The default behavior of
> the
> > consumer is to "consume from the tail of the log", and hence if there is
> no
> > new messages coming in after the consumer started, it will get nothing.
> You
> > may set
> >
> > auto.offset.reset="smallest"
> >
> > and try again.
> >
> > Guozhang
> >
> >
> > On Sun, Jun 22, 2014 at 9:10 PM, Li Li  wrote:
> >
> >> hi all,
> >>I am reading the book "apache kafka" and write a simple producer
> >> and consumer class. the producer works but the consumer hangs.
> >>The producer class:
> >> public static void main(String[] args) {
> >> String topic="test-topic";
> >> Properties props = new Properties();
> >> props.put("metadata.broker.list","linux157:9092");
> >> props.put("serializer.class","kafka.serializer.StringEncoder");
> >> props.put("request.required.acks", "1");
> >> ProducerConfig config = new ProducerConfig(props);
> >> Producer producer = new
> >> Producer(config);
> >> for(int i=0;i<100;i++){
> >> KeyedMessage data = new
> >> KeyedMessage(topic, "msg"+i);
> >> producer.send(data);
> >> }
> >> producer.close();
> >>
> >> }
> >>
> >> public class TestKafkaConsumer {
> >> private final ConsumerConnector consumer;
> >> private final String topic;
> >>
> >> public TestKafkaConsumer(String zookeeper, String groupId, String
> topic) {
> >> Properties props = new Properties();
> >> props.put("zookeeper.connect", zookeeper);
> >> props.put("group.id", groupId);
> >> props.put("zookeeper.session.timeout.ms", "500");
> >> props.put("zookeeper.sync.time.ms", "250");
> >> props.put("auto.commit.interval.ms", "1000");
> >> consumer = Consumer.createJavaConsumerConnector(new ConsumerConfig(
> >> props));
> >> this.topic = topic;
> >> }
> >>
> >> public void testConsumer() {
> >> Map topicCount = new HashMap();
> >> // Define single thread for topic
> >> topicCount.put(topic, new Integer(1));
> >> Map>> consumerStreams =
> >> consumer
> >>  .createMessageStreams(topicCount);
> >> List> streams =
> consumerStreams.get(topic);
> >> for (final KafkaStream stream : streams) {
> >> ConsumerIterator consumerIte =
> stream.iterator();
> >> while (consumerIte.hasNext())
> >> System.out.println("Message from Single Topic :: "
> >>+ new String(consumerIte.next().message()));
> >> }
> >> if (consumer != null)
> >> consumer.shutdown();
> >> }
> >>
> >> public static void main(String[] args) {
> >> String topic = "test-topic";
> >> TestKafkaConsumer simpleHLConsumer = new
> >> TestKafkaConsumer("linux157:2181",testgroup22", topic);
> >> simpleHLConsumer.testConsumer();
> >>
> >> }
> >>
> >> }
> >>
> >
> >
> >
> > --
> > -- Guozhang
>



-- 
-- Guozhang

Re: What's the behavior when Kafka is deleting messages and consumers are still reading

[Guozhang Wang]

Hi Li,

The log operations are protected by a lock, so if there is a concurrent
read on this partition it will not be deleted. But then when it is deleted
the next fetch/read will result in a OffsetOutOfRange exception and the
consumer needs to restart from a offset reset value.

Guozhang


On Tue, Jun 24, 2014 at 5:20 AM, Lian, Li  wrote:

> When Kafka broker is trying to delete message according to the log
> retention setting, either triggered by log age or topic partition size, if
> the same time there are still consumers reading the topic, what will happen?
>
> Best regards,
> Lex Lian
>
> Email: ll...@ebay.com
>
>


-- 
-- Guozhang

Experiences with larger message sizes

[Denny Lee]

By any chance has anyone worked with using Kafka with message sizes that are 
approximately 50MB in size?  Based on from some of the previous threads there 
are probably some concerns on memory pressure due to the compression on the 
broker and decompression on the consumer and a best practices on ensuring batch 
size (to ultimately not have the compressed message exceed message size limit). 
 

Any other best practices or thoughts concerning this scenario?

Thanks!
Denny

Apacha Kafka Commercial Support

[Diego Alvarez Zuluaga]

Hello

Are there any vendors who provide commercial support for Kafka?

We're very interested in using kafka, but our infraestructure team ask us
(DevTeam) for a commercial support.

Tks

diegoalva...@sura.com.co
http://www.gruposuramericana.com/en/Pages/OurPortfolio/Suramericana.aspx

Re: restarting a broker during partition reassignment

[Luke Forehand]

My hypothesis for how Partition [luke3,3] with leader 11, had offset reset
to zero, caused by reboot of leader broker during partition reassignment:

The replicas for [luke3,3] were in progress being reassigned from broker
10,11,12 -> 11,12,13
I rebooted broker 11 which was the leader for [luke3.3]
Broker 12 and 13 logs indicate replica fetch failures from leader 11 due
to connection time out

Broker 10 attempts to become the leader for [luke3,3] but has an issue (I
see a zk exception but I'm unsure what is happening)

Broker 11 eventually comes online and attempts to fetch from new leader 10
Broker 11 completes fetch from leader 10 at offset 0
Broker 10 is leader but is serving a new data log and offset has been reset
Remaining brokers truncate logs and follow broker 10

Gist of logs for brokers 13,11,12 that I think backs up this summary:
https://gist.github.com/anonymous/cb79dc251d87e334cfff



Thanks,
Luke Forehand |  Networked Insights  |  Software Engineer



On 6/23/14, 5:57 PM, "Guozhang Wang"  wrote:

>Hi Luke,
>
>What are the exceptions/warnings you saw in the broker and controller
>logs?
>
>Guozhang
>
>
>On Mon, Jun 23, 2014 at 2:03 PM, Luke Forehand <
>luke.foreh...@networkedinsights.com> wrote:
>
>> Hello,
>>
>> I am testing kafka 0.8.1.1 in preparation for an upgrade from
>> kafka-0.8.1-beta.  I have a 4 node cluster with one broker per node,
>>and a
>> topic with 8 partitions and 3 replicas.  Each partition has about 6
>> million records.
>>
>> I generated a partition reassignment json that basically causes all
>> partitions to be shifted by one broker.  As the reassignment was in
>> progress I bounced one of the servers.  After the server came back up
>>and
>> the broker started, I waited for the server logs to stop complaining and
>> then ran the reassignment verify script and all partitions were verified
>> as completed reassignment.
>>
>> However, one of the partition offsets was reset to 0, and 4 out of 8
>> partitions only had 2 in-sync-replicas instead of 3 (in-sync came back
>>to
>> 3 but only after I again bounced the server I had previously bounced
>> during reassignment).
>>
>> Is this considered a bug?  I ask because we use the SimpleConsumer API
>>so
>> we keep track of our own offset "pointers".  If it is not a bug then I
>> could reset the pointer to "earliest" and continue reading, but I was
>> wondering if there is a potential for data loss in my scenario.  I have
>> plenty of logs and can reproduce but before I spam I was wondering if
>> there is already a jira task for this issue or if anybody else is aware.
>>
>> Thanks,
>> Luke Forehand |  Networked Insights  |  Software Engineer
>>
>>
>
>
>-- 
>-- Guozhang

Re: Using GetOffsetShell against non-existent topic creates the topic which cannot be deleted

[Luke Forehand]

Definitely, this is version 0.8.1.1

https://issues.apache.org/jira/browse/KAFKA-1507


Luke Forehand |  Networked Insights  |  Software Engineer





On 6/23/14, 6:58 PM, "Guozhang Wang"  wrote:

>Luke,
>
>Thanks for the findings, could you file a JIRA to keep track of this bug?
>
>Guozhang
>
>
>On Mon, Jun 23, 2014 at 2:42 PM, Luke Forehand <
>luke.foreh...@networkedinsights.com> wrote:
>
>> Slight nitpick, but a typo in using GetOffsetShell command can cause a
>> topic to be created which cannot be deleted (because deletion is still
>>in
>> progress)
>>
>> ./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list
>> kafka10:9092,kafka11:9092,kafka12:9092,kafka13:9092 --topic typo --time
>>1
>>
>> ./kafka-topics.sh --zookeeper stormqa1/kafka-prod --describe --topic
>>typo
>> Topic:typo  PartitionCount:8ReplicationFactor:1 Configs:
>> Topic: typo Partition: 0Leader: 10  Replicas: 10
>>  Isr: 10
>> ...
>>
>>
>>
>>
>> Luke Forehand |  Networked Insights  |  Software Engineer
>>
>>
>>
>>
>>
>>
>
>
>-- 
>-- Guozhang

Re: Apacha Kafka Commercial Support

[Joe Stein]

Hi Diego,

Big Data Open Source Security LLC
https://www.linkedin.com/company/big-data-open-source-security-llc provides
commercial support around Apache Kafka.

We currently do this as a retained professional service rate (so the cost
is not nodes or volume like product vendors).

We have also been discussing releasing and maintaing our own
distribution(s) as there are patches folks need/want now (e.g.
https://issues.apache.org/jira/browse/KAFKA-1477 and
https://issues.apache.org/jira/browse/KAFKA-1215) and new features (e.g.
moving Zookeeper functionality off onto Cassandra and also better support
Kafka running on Mesos or YARN) that the Apache project community may not
need/want but would be desirable for many implementations.  We have been
doing these as client engagements for our clients (rolling patched versions
for them) but it might make more sense to make it more widely available
unless we can start to get these changes upstream for releases.

Feel free to ping me directly and we can setup a time to discuss your
specific requirements.

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/


On Tue, Jun 24, 2014 at 11:39 AM, Diego Alvarez Zuluaga <
diego.alvarez.zulu...@gmail.com> wrote:

> Hello
>
> Are there any vendors who provide commercial support for Kafka?
>
> We're very interested in using kafka, but our infraestructure team ask us
> (DevTeam) for a commercial support.
>
> Tks
>
> diegoalva...@sura.com.co
> http://www.gruposuramericana.com/en/Pages/OurPortfolio/Suramericana.aspx
>

Re: How does number of partitions affect sequential disk IO

[Jay Kreps]

The primary relevant factor here is the fsync interval. Kafka's replication
guarantees do not require fsyncing every message, so the reason for doing
so is to handle correlated power loss (a pretty uncommon failure in a real
data center). Replication will handle most other failure modes with much
much lower overhead.

With a lax fsync interval you can have LOTS of partitions with very little
impact. The OS will do a good job of merging physical writes and scheduling
them in an order that is convenient.

On the other hand if you fsync on every message you will see a pretty big
drop off with more than one partition per disk as you immediately incur a
5-10ms seek on each write.

For a policy in between "on every message" and "when the os feels like it"
you will see performance somewhere in between.

-Jay


On Tue, Jun 24, 2014 at 4:00 AM, Paul Mackles  wrote:

> Its probably best to run some tests that simulate your usage patterns. I
> think a lot of it will be determined by how effectively you are able to
> utilize the OS file cache in which case you could have many more
> partitions. Its a delicate balance but you definitely want to err on the
> side of having more partitions. Keep in mind that you are only able to
> parallelize down to the partition level so if you have only have 2
> partitions, you can only have 2 consumers. Depending on your volume, that
> might not be enough.
>
> On 6/24/14 6:44 AM, "Daniel Compton"  wrote:
>
> >Good point. We've only got two disks per node and two topics so I was
> >planning to have one disk/partition.
> >
> >Our workload is very write heavy so I'm mostly concerned about write
> >throughput. Will we get write speed improvements by sticking to 1
> >partition/disk or will the difference between 1 and 3 partitions/node be
> >negligible?
> >
> >> On 24/06/2014, at 9:42 pm, Paul Mackles  wrote:
> >>
> >> You'll want to account for the number of disks per node. Normally,
> >> partitions are spread across multiple disks. Even more important, the OS
> >> file cache reduces the amount of seeking provided that you are reading
> >> mostly sequentially and your consumers are keeping up.
> >>
> >>> On 6/24/14 3:58 AM, "Daniel Compton"  wrote:
> >>>
> >>> I¹ve been reading the Kafka docs and one thing that I¹m having trouble
> >>> understanding is how partitions affect sequential disk IO. One of the
> >>> reasons Kafka is so fast is that you can do lots of sequential IO with
> >>> read-ahead cache and all of that goodness. However, if your broker is
> >>> responsible for say 20 partitions, then won¹t the disk be seeking to 20
> >>> different spots for its writes and reads? I thought that maybe letting
> >>> the OS handle fsync would make this less of an issue but it still seems
> >>> like it could be a problem.
> >>>
> >>> In our particular situation, we are going to have 6 brokers, 3 in each
> >>> DC, with mirror maker replication from the secondary DC to the primary
> >>> DC. We aren¹t likely to need to add more nodes for a while so would it
> >>>be
> >>> faster to have 1 partition/node than say 3-4/node to minimise the seek
> >>> times on disk?
> >>>
> >>> Are my assumptions correct or is this not an issue in practice? There
> >>>are
> >>> some nice things about having more partitions like rebalancing more
> >>> evenly if we lose a broker but we don¹t want to make things
> >>>significantly
> >>> slower to get this.
> >>>
> >>> Thanks, Daniel.
> >>
>
>

Re: Experiences with larger message sizes

[Joe Stein]

Hi Denny, have you considered saving those files to HDFS and sending the
"event" information to Kafka?

You could then pass that off to Apache Spark in a consumer and get data
locality for the file saved (or something of the sort [no pun intended]).

You could also stream every line (or however you want to "chunk" it) in the
file as a separate message to the broker with a wrapping message object (so
you know which file you are dealing with when consuming).

What you plan to-do with the data has a lot to-do with how you are going to
process and manage it.

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/


On Tue, Jun 24, 2014 at 11:35 AM, Denny Lee  wrote:

> By any chance has anyone worked with using Kafka with message sizes that
> are approximately 50MB in size?  Based on from some of the previous threads
> there are probably some concerns on memory pressure due to the compression
> on the broker and decompression on the consumer and a best practices on
> ensuring batch size (to ultimately not have the compressed message exceed
> message size limit).
>
> Any other best practices or thoughts concerning this scenario?
>
> Thanks!
> Denny
>
>

Re: Experiences with larger message sizes

[Denny Lee]

Hey Joe,

Yes, I have - my original plan is to do something similar to what you suggested 
which was to simply push the data into HDFS / S3 and then having only the event 
information within Kafka so that way multiple consumers can just read the event 
information and ping HDFS/S3 for the actual message itself.  

Part of the reason for considering just pushing the entire message up is due to 
the potential where we will have a firehose of messages of this size and we 
will need to push this data to multiple locations.

Thanks,
Denny

On June 24, 2014 at 9:26:49 AM, Joe Stein (joe.st...@stealth.ly) wrote:

Hi Denny, have you considered saving those files to HDFS and sending the  
"event" information to Kafka?  

You could then pass that off to Apache Spark in a consumer and get data  
locality for the file saved (or something of the sort [no pun intended]).  

You could also stream every line (or however you want to "chunk" it) in the  
file as a separate message to the broker with a wrapping message object (so  
you know which file you are dealing with when consuming).  

What you plan to-do with the data has a lot to-do with how you are going to  
process and manage it.  

/***  
Joe Stein  
Founder, Principal Consultant  
Big Data Open Source Security LLC  
http://www.stealth.ly  
Twitter: @allthingshadoop   
/  


On Tue, Jun 24, 2014 at 11:35 AM, Denny Lee  wrote:  

> By any chance has anyone worked with using Kafka with message sizes that  
> are approximately 50MB in size? Based on from some of the previous threads  
> there are probably some concerns on memory pressure due to the compression  
> on the broker and decompression on the consumer and a best practices on  
> ensuring batch size (to ultimately not have the compressed message exceed  
> message size limit).  
>  
> Any other best practices or thoughts concerning this scenario?  
>  
> Thanks!  
> Denny  
>  
>  

Re: Experiences with larger message sizes

[Joe Stein]

You could then chunk the data (wrapped in an outer message so you have meta
data like file name, total size, current chunk size) and produce that with
the partition key being filename.

We are in progress working on a system for doing file loading to Kafka
(which will eventually support both chunked and pointers [initially
chunking line by line since use case 1 is to read from a closed file handle
location]) https://github.com/stealthly/f2k (there is not much there yet
maybe in the next few days / later this week) maybe useful for your use
case or we could eventually add your use case to it.

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/


On Tue, Jun 24, 2014 at 12:37 PM, Denny Lee  wrote:

> Hey Joe,
>
> Yes, I have - my original plan is to do something similar to what you
> suggested which was to simply push the data into HDFS / S3 and then having
> only the event information within Kafka so that way multiple consumers can
> just read the event information and ping HDFS/S3 for the actual message
> itself.
>
> Part of the reason for considering just pushing the entire message up is
> due to the potential where we will have a firehose of messages of this size
> and we will need to push this data to multiple locations.
>
> Thanks,
> Denny
>
> On June 24, 2014 at 9:26:49 AM, Joe Stein (joe.st...@stealth.ly) wrote:
>
> Hi Denny, have you considered saving those files to HDFS and sending the
> "event" information to Kafka?
>
> You could then pass that off to Apache Spark in a consumer and get data
> locality for the file saved (or something of the sort [no pun intended]).
>
> You could also stream every line (or however you want to "chunk" it) in
> the
> file as a separate message to the broker with a wrapping message object
> (so
> you know which file you are dealing with when consuming).
>
> What you plan to-do with the data has a lot to-do with how you are going
> to
> process and manage it.
>
> /***
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop 
> /
>
>
> On Tue, Jun 24, 2014 at 11:35 AM, Denny Lee 
> wrote:
>
> > By any chance has anyone worked with using Kafka with message sizes that
> > are approximately 50MB in size? Based on from some of the previous
> threads
> > there are probably some concerns on memory pressure due to the
> compression
> > on the broker and decompression on the consumer and a best practices on
> > ensuring batch size (to ultimately not have the compressed message
> exceed
> > message size limit).
> >
> > Any other best practices or thoughts concerning this scenario?
> >
> > Thanks!
> > Denny
> >
> >
>
>

Re: What's the behavior when Kafka is deleting messages and consumers are still reading

[Neha Narkhede]

The behavior on the consumer in this case is governed by the value of the
"auto.offset.reset" config. Depending on this config, it will reset it's
offset to either the earliest or the latest in the log.

Thanks,
Neha


On Tue, Jun 24, 2014 at 8:17 AM, Guozhang Wang  wrote:

> Hi Li,
>
> The log operations are protected by a lock, so if there is a concurrent
> read on this partition it will not be deleted. But then when it is deleted
> the next fetch/read will result in a OffsetOutOfRange exception and the
> consumer needs to restart from a offset reset value.
>
> Guozhang
>
>
> On Tue, Jun 24, 2014 at 5:20 AM, Lian, Li  wrote:
>
> > When Kafka broker is trying to delete message according to the log
> > retention setting, either triggered by log age or topic partition size,
> if
> > the same time there are still consumers reading the topic, what will
> happen?
> >
> > Best regards,
> > Lex Lian
> >
> > Email: ll...@ebay.com
> >
> >
>
>
> --
> -- Guozhang
>

Re: Consumer offset is getting reset back to some old value automatically

[Neha Narkhede]

Can you elaborate your notion of "smooth"? I thought if you have
replication factor=3 in this case, you should be able to tolerate loss
of a node?

Yes, you should be able to tolerate the loss of a node but if controlled
shutdown is not enabled, the delay between loss of the old leader and
election of the new leader will be longer.

So, you mean ZK cluster of 3 nodes can't tolerate 1 node loss? I've
seen many recommendations to run 3-nodes cluster, does it mean in
cluster of 3 you won't be able to operate after loosing 1 node?

ZK cluster of 3 nodes will tolerate the loss of 1 node, but if there is a
subsequent leader election for any reason, there is a chance that the
cluster does not reach a quorum. It is less likely but still risky to some
extent.


On Tue, Jun 24, 2014 at 2:44 AM, Hemath Kumar 
wrote:

> Yes kane i have the replication factor configured as 3
>
>
> On Tue, Jun 24, 2014 at 2:42 AM, Kane Kane  wrote:
>
> > Hello Neha, can you explain your statements:
> > >>Bringing one node down in a cluster will go smoothly only if your
> > replication factor is 1 and you enabled controlled shutdown on the
> brokers.
> >
> > Can you elaborate your notion of "smooth"? I thought if you have
> > replication factor=3 in this case, you should be able to tolerate loss
> > of a node?
> >
> > >>Also, bringing down 1 node our of a 3 node zookeeper cluster is risky,
> > since any subsequent leader election might not reach a quorum.
> >
> > So, you mean ZK cluster of 3 nodes can't tolerate 1 node loss? I've
> > seen many recommendations to run 3-nodes cluster, does it mean in
> > cluster of 3 you won't be able to operate after loosing 1 node?
> >
> > Thanks.
> >
> > On Mon, Jun 23, 2014 at 9:04 AM, Neha Narkhede 
> > wrote:
> > > Bringing one node down in a cluster will go smoothly only if your
> > > replication factor is 1 and you enabled controlled shutdown on the
> > brokers.
> > > Also, bringing down 1 node our of a 3 node zookeeper cluster is risky,
> > > since any subsequent leader election might not reach a quorum. Having
> > said
> > > that, a partition going offline shouldn't cause a consumer's offset to
> > > reset to an old value. How did you find out what the consumer's offset
> > was?
> > > Do you have your consumer's logs around?
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Mon, Jun 23, 2014 at 12:28 AM, Hemath Kumar  >
> > > wrote:
> > >
> > >> We have a 3 node cluster ( 3 kafka + 3 ZK nodes ) . Recently we came
> > across
> > >> a strange issue where we wanted to bring one of the node down from
> > cluster
> > >> ( 1 kafka + 1 zookeeper) for doing a maintenance . But the movement we
> > >> brought it to down on some of the topics ( only some partitions)
> > consumers
> > >> offset is reset some old value.
> > >>
> > >> Any reason why this is happened?. As of my knowledge when brought one
> > node
> > >> down its should work smoothly with out any impact.
> > >>
> > >> Thanks,
> > >> Murthy Chelankuri
> > >>
> >
>

Re: Kafka 0.8's VerifyConsumerRebalance reports an error

[Neha Narkhede]

Is it possible that maybe the zookeeper url used for the
VerifyConsumerRebalance tool is incorrect?


On Tue, Jun 24, 2014 at 12:02 AM, Yury Ruchin  wrote:

> Hi,
>
> I've run into the following problem. I try to read from a 50-partition
> Kafka topic using high level consumer with 8 streams. I'm using 8-thread
> pool, each thread handling one stream. After a short time, the threads
> reading from the stream stop reading. Lag between topic latest offset and
> the consumer constantly increases as new messages come in.
>
> I looked into ZK:  /consumers//owners/ and see a
> list of znodes corresponding to the full list of partitions: [1, 2, 3,
> ...]. When I do zk get on e. g.
> /consumers//owners//1 - I see a valid consumer
> name corresponding to Kafka client logs, e. g.
> _-140304960-abc12345-0. However, when I run
> the VerifyConsumerRebalance tool, I see the following:
>
> No owner for partition [,1]
> (kafka.tools.VerifyConsumerRebalance$)
>
> No owner for partition [,2]
> (kafka.tools.VerifyConsumerRebalance$)
>
> ...
>
> No owner for partition [,50]
> (kafka.tools.VerifyConsumerRebalance$)
>
> According to this output, no partition has owner, which seemingly
> contradicts to what I see in ZK.
>
> What would cause such a problem and how can I troubleshoot it further?
>
> Thanks!
>

Re: Kafka 0.8's VerifyConsumerRebalance reports an error

[Yury Ruchin]

I've just double-checked. The URL is correct, the same one is used by Kafka
clients.


2014-06-24 22:21 GMT+04:00 Neha Narkhede :

> Is it possible that maybe the zookeeper url used for the
> VerifyConsumerRebalance tool is incorrect?
>
>
> On Tue, Jun 24, 2014 at 12:02 AM, Yury Ruchin 
> wrote:
>
> > Hi,
> >
> > I've run into the following problem. I try to read from a 50-partition
> > Kafka topic using high level consumer with 8 streams. I'm using 8-thread
> > pool, each thread handling one stream. After a short time, the threads
> > reading from the stream stop reading. Lag between topic latest offset and
> > the consumer constantly increases as new messages come in.
> >
> > I looked into ZK:  /consumers//owners/ and
> see a
> > list of znodes corresponding to the full list of partitions: [1, 2, 3,
> > ...]. When I do zk get on e. g.
> > /consumers//owners//1 - I see a valid consumer
> > name corresponding to Kafka client logs, e. g.
> > _-140304960-abc12345-0. However, when I run
> > the VerifyConsumerRebalance tool, I see the following:
> >
> > No owner for partition [,1]
> > (kafka.tools.VerifyConsumerRebalance$)
> >
> > No owner for partition [,2]
> > (kafka.tools.VerifyConsumerRebalance$)
> >
> > ...
> >
> > No owner for partition [,50]
> > (kafka.tools.VerifyConsumerRebalance$)
> >
> > According to this output, no partition has owner, which seemingly
> > contradicts to what I see in ZK.
> >
> > What would cause such a problem and how can I troubleshoot it further?
> >
> > Thanks!
> >
>

Re: Consumer offset is getting reset back to some old value automatically

[Kane Kane]

Hello Neha,

>>ZK cluster of 3 nodes will tolerate the loss of 1 node, but if there is a
subsequent leader election for any reason, there is a chance that the
cluster does not reach a quorum. It is less likely but still risky to some
extent.

Does it mean if you have to tolerate 1 node loss without any issues,
you need *at least* 4 nodes?

On Tue, Jun 24, 2014 at 11:16 AM, Neha Narkhede  wrote:
> Can you elaborate your notion of "smooth"? I thought if you have
> replication factor=3 in this case, you should be able to tolerate loss
> of a node?
>
> Yes, you should be able to tolerate the loss of a node but if controlled
> shutdown is not enabled, the delay between loss of the old leader and
> election of the new leader will be longer.
>
> So, you mean ZK cluster of 3 nodes can't tolerate 1 node loss? I've
> seen many recommendations to run 3-nodes cluster, does it mean in
> cluster of 3 you won't be able to operate after loosing 1 node?
>
> ZK cluster of 3 nodes will tolerate the loss of 1 node, but if there is a
> subsequent leader election for any reason, there is a chance that the
> cluster does not reach a quorum. It is less likely but still risky to some
> extent.
>
>
> On Tue, Jun 24, 2014 at 2:44 AM, Hemath Kumar 
> wrote:
>
>> Yes kane i have the replication factor configured as 3
>>
>>
>> On Tue, Jun 24, 2014 at 2:42 AM, Kane Kane  wrote:
>>
>> > Hello Neha, can you explain your statements:
>> > >>Bringing one node down in a cluster will go smoothly only if your
>> > replication factor is 1 and you enabled controlled shutdown on the
>> brokers.
>> >
>> > Can you elaborate your notion of "smooth"? I thought if you have
>> > replication factor=3 in this case, you should be able to tolerate loss
>> > of a node?
>> >
>> > >>Also, bringing down 1 node our of a 3 node zookeeper cluster is risky,
>> > since any subsequent leader election might not reach a quorum.
>> >
>> > So, you mean ZK cluster of 3 nodes can't tolerate 1 node loss? I've
>> > seen many recommendations to run 3-nodes cluster, does it mean in
>> > cluster of 3 you won't be able to operate after loosing 1 node?
>> >
>> > Thanks.
>> >
>> > On Mon, Jun 23, 2014 at 9:04 AM, Neha Narkhede 
>> > wrote:
>> > > Bringing one node down in a cluster will go smoothly only if your
>> > > replication factor is 1 and you enabled controlled shutdown on the
>> > brokers.
>> > > Also, bringing down 1 node our of a 3 node zookeeper cluster is risky,
>> > > since any subsequent leader election might not reach a quorum. Having
>> > said
>> > > that, a partition going offline shouldn't cause a consumer's offset to
>> > > reset to an old value. How did you find out what the consumer's offset
>> > was?
>> > > Do you have your consumer's logs around?
>> > >
>> > > Thanks,
>> > > Neha
>> > >
>> > >
>> > > On Mon, Jun 23, 2014 at 12:28 AM, Hemath Kumar > >
>> > > wrote:
>> > >
>> > >> We have a 3 node cluster ( 3 kafka + 3 ZK nodes ) . Recently we came
>> > across
>> > >> a strange issue where we wanted to bring one of the node down from
>> > cluster
>> > >> ( 1 kafka + 1 zookeeper) for doing a maintenance . But the movement we
>> > >> brought it to down on some of the topics ( only some partitions)
>> > consumers
>> > >> offset is reset some old value.
>> > >>
>> > >> Any reason why this is happened?. As of my knowledge when brought one
>> > node
>> > >> down its should work smoothly with out any impact.
>> > >>
>> > >> Thanks,
>> > >> Murthy Chelankuri
>> > >>
>> >
>>

Re: Consumer offset is getting reset back to some old value automatically

[Kane Kane]

Sorry, i meant 5 nodes in previous question.

On Tue, Jun 24, 2014 at 12:36 PM, Kane Kane  wrote:
> Hello Neha,
>
>>>ZK cluster of 3 nodes will tolerate the loss of 1 node, but if there is a
> subsequent leader election for any reason, there is a chance that the
> cluster does not reach a quorum. It is less likely but still risky to some
> extent.
>
> Does it mean if you have to tolerate 1 node loss without any issues,
> you need *at least* 4 nodes?
>
> On Tue, Jun 24, 2014 at 11:16 AM, Neha Narkhede  
> wrote:
>> Can you elaborate your notion of "smooth"? I thought if you have
>> replication factor=3 in this case, you should be able to tolerate loss
>> of a node?
>>
>> Yes, you should be able to tolerate the loss of a node but if controlled
>> shutdown is not enabled, the delay between loss of the old leader and
>> election of the new leader will be longer.
>>
>> So, you mean ZK cluster of 3 nodes can't tolerate 1 node loss? I've
>> seen many recommendations to run 3-nodes cluster, does it mean in
>> cluster of 3 you won't be able to operate after loosing 1 node?
>>
>> ZK cluster of 3 nodes will tolerate the loss of 1 node, but if there is a
>> subsequent leader election for any reason, there is a chance that the
>> cluster does not reach a quorum. It is less likely but still risky to some
>> extent.
>>
>>
>> On Tue, Jun 24, 2014 at 2:44 AM, Hemath Kumar 
>> wrote:
>>
>>> Yes kane i have the replication factor configured as 3
>>>
>>>
>>> On Tue, Jun 24, 2014 at 2:42 AM, Kane Kane  wrote:
>>>
>>> > Hello Neha, can you explain your statements:
>>> > >>Bringing one node down in a cluster will go smoothly only if your
>>> > replication factor is 1 and you enabled controlled shutdown on the
>>> brokers.
>>> >
>>> > Can you elaborate your notion of "smooth"? I thought if you have
>>> > replication factor=3 in this case, you should be able to tolerate loss
>>> > of a node?
>>> >
>>> > >>Also, bringing down 1 node our of a 3 node zookeeper cluster is risky,
>>> > since any subsequent leader election might not reach a quorum.
>>> >
>>> > So, you mean ZK cluster of 3 nodes can't tolerate 1 node loss? I've
>>> > seen many recommendations to run 3-nodes cluster, does it mean in
>>> > cluster of 3 you won't be able to operate after loosing 1 node?
>>> >
>>> > Thanks.
>>> >
>>> > On Mon, Jun 23, 2014 at 9:04 AM, Neha Narkhede 
>>> > wrote:
>>> > > Bringing one node down in a cluster will go smoothly only if your
>>> > > replication factor is 1 and you enabled controlled shutdown on the
>>> > brokers.
>>> > > Also, bringing down 1 node our of a 3 node zookeeper cluster is risky,
>>> > > since any subsequent leader election might not reach a quorum. Having
>>> > said
>>> > > that, a partition going offline shouldn't cause a consumer's offset to
>>> > > reset to an old value. How did you find out what the consumer's offset
>>> > was?
>>> > > Do you have your consumer's logs around?
>>> > >
>>> > > Thanks,
>>> > > Neha
>>> > >
>>> > >
>>> > > On Mon, Jun 23, 2014 at 12:28 AM, Hemath Kumar >> >
>>> > > wrote:
>>> > >
>>> > >> We have a 3 node cluster ( 3 kafka + 3 ZK nodes ) . Recently we came
>>> > across
>>> > >> a strange issue where we wanted to bring one of the node down from
>>> > cluster
>>> > >> ( 1 kafka + 1 zookeeper) for doing a maintenance . But the movement we
>>> > >> brought it to down on some of the topics ( only some partitions)
>>> > consumers
>>> > >> offset is reset some old value.
>>> > >>
>>> > >> Any reason why this is happened?. As of my knowledge when brought one
>>> > node
>>> > >> down its should work smoothly with out any impact.
>>> > >>
>>> > >> Thanks,
>>> > >> Murthy Chelankuri
>>> > >>
>>> >
>>>

Re: Kafka 'reassign-partitions' behavior if the replica does not catches up

[Virendra Pratap Singh]

In process of giving 0.8.1.1 a try.

However I believe the question still holds true.
If the amount of data getting pumped in a partition is such that any new
replica partition can never catch up, then what would the partition
reassignment tool behavior?
If it will be in an infinitely wait mode waiting for the replica to be in
sync, then is there an alternative to cancel the reassignment operation?

Regards,
Virendra
On 6/23/14, 4:49 PM, "Neha Narkhede"  wrote:

>However, in my case I realized that the new re-assigned partitions are not
>getting in sync at all. The in-rate in the existing partition is far more
>then the sync rate between earlier and newly reassigned partitions.
>
>This is probably due to a bug. Partition reassignment is stable in
>0.8.1.1.
>I recommend you upgrade and retry.
>
>Thanks
>Neha
>
>
>On Mon, Jun 23, 2014 at 3:24 PM, Virendra Pratap Singh <
>vpsi...@yahoo-inc.com.invalid> wrote:
>
>> I attempted bin/kafka-reassign-partitions.sh for the topics in my
>>cluster.
>> It was automatic reassignment of topics to new brokers.
>>
>> Kafka duly initiated the partition reassignment. I believe kafka
>> controller will do the new leader election once the re-assigned
>>partitions
>> are in sync.
>>
>> However, in my case I realized that the new re-assigned partitions are
>>not
>> getting in sync at all. The in-rate in the existing partition is far
>>more
>> then the sync rate between earlier and newly reassigned partitions.
>>
>> In this scenario, will kafka wait indefinitely for the newly assigned
>> partition to become sync (which will never be).
>>
>> Secondly what are the options to speed up the partition replica sync
>>rate?
>>
>> Running kafka 0.8.0
>>
>> Regards,
>> Virendra
>>
>>
>>
>>

Re: How does number of partitions affect sequential disk IO

[Daniel Compton]

Thanks Jay, that's exactly what I was looking for.


On 25 June 2014 04:18, Jay Kreps  wrote:

> The primary relevant factor here is the fsync interval. Kafka's replication
> guarantees do not require fsyncing every message, so the reason for doing
> so is to handle correlated power loss (a pretty uncommon failure in a real
> data center). Replication will handle most other failure modes with much
> much lower overhead.
>
> With a lax fsync interval you can have LOTS of partitions with very little
> impact. The OS will do a good job of merging physical writes and scheduling
> them in an order that is convenient.
>
> On the other hand if you fsync on every message you will see a pretty big
> drop off with more than one partition per disk as you immediately incur a
> 5-10ms seek on each write.
>
> For a policy in between "on every message" and "when the os feels like it"
> you will see performance somewhere in between.
>
> -Jay
>
>
> On Tue, Jun 24, 2014 at 4:00 AM, Paul Mackles  wrote:
>
> > Its probably best to run some tests that simulate your usage patterns. I
> > think a lot of it will be determined by how effectively you are able to
> > utilize the OS file cache in which case you could have many more
> > partitions. Its a delicate balance but you definitely want to err on the
> > side of having more partitions. Keep in mind that you are only able to
> > parallelize down to the partition level so if you have only have 2
> > partitions, you can only have 2 consumers. Depending on your volume, that
> > might not be enough.
> >
> > On 6/24/14 6:44 AM, "Daniel Compton"  wrote:
> >
> > >Good point. We've only got two disks per node and two topics so I was
> > >planning to have one disk/partition.
> > >
> > >Our workload is very write heavy so I'm mostly concerned about write
> > >throughput. Will we get write speed improvements by sticking to 1
> > >partition/disk or will the difference between 1 and 3 partitions/node be
> > >negligible?
> > >
> > >> On 24/06/2014, at 9:42 pm, Paul Mackles  wrote:
> > >>
> > >> You'll want to account for the number of disks per node. Normally,
> > >> partitions are spread across multiple disks. Even more important, the
> OS
> > >> file cache reduces the amount of seeking provided that you are reading
> > >> mostly sequentially and your consumers are keeping up.
> > >>
> > >>> On 6/24/14 3:58 AM, "Daniel Compton"  wrote:
> > >>>
> > >>> I¹ve been reading the Kafka docs and one thing that I¹m having
> trouble
> > >>> understanding is how partitions affect sequential disk IO. One of the
> > >>> reasons Kafka is so fast is that you can do lots of sequential IO
> with
> > >>> read-ahead cache and all of that goodness. However, if your broker is
> > >>> responsible for say 20 partitions, then won¹t the disk be seeking to
> 20
> > >>> different spots for its writes and reads? I thought that maybe
> letting
> > >>> the OS handle fsync would make this less of an issue but it still
> seems
> > >>> like it could be a problem.
> > >>>
> > >>> In our particular situation, we are going to have 6 brokers, 3 in
> each
> > >>> DC, with mirror maker replication from the secondary DC to the
> primary
> > >>> DC. We aren¹t likely to need to add more nodes for a while so would
> it
> > >>>be
> > >>> faster to have 1 partition/node than say 3-4/node to minimise the
> seek
> > >>> times on disk?
> > >>>
> > >>> Are my assumptions correct or is this not an issue in practice? There
> > >>>are
> > >>> some nice things about having more partitions like rebalancing more
> > >>> evenly if we lose a broker but we don¹t want to make things
> > >>>significantly
> > >>> slower to get this.
> > >>>
> > >>> Thanks, Daniel.
> > >>
> >
> >
>

Re: Kafka 'reassign-partitions' behavior if the replica does not catches up

[Neha Narkhede]

We have a JIRA to track the cancel feature -
https://issues.apache.org/jira/browse/KAFKA-1506.

Thanks,
Neha


On Tue, Jun 24, 2014 at 1:32 PM, Virendra Pratap Singh <
vpsi...@yahoo-inc.com.invalid> wrote:

> In process of giving 0.8.1.1 a try.
>
> However I believe the question still holds true.
> If the amount of data getting pumped in a partition is such that any new
> replica partition can never catch up, then what would the partition
> reassignment tool behavior?
> If it will be in an infinitely wait mode waiting for the replica to be in
> sync, then is there an alternative to cancel the reassignment operation?
>
> Regards,
> Virendra
> On 6/23/14, 4:49 PM, "Neha Narkhede"  wrote:
>
> >However, in my case I realized that the new re-assigned partitions are not
> >getting in sync at all. The in-rate in the existing partition is far more
> >then the sync rate between earlier and newly reassigned partitions.
> >
> >This is probably due to a bug. Partition reassignment is stable in
> >0.8.1.1.
> >I recommend you upgrade and retry.
> >
> >Thanks
> >Neha
> >
> >
> >On Mon, Jun 23, 2014 at 3:24 PM, Virendra Pratap Singh <
> >vpsi...@yahoo-inc.com.invalid> wrote:
> >
> >> I attempted bin/kafka-reassign-partitions.sh for the topics in my
> >>cluster.
> >> It was automatic reassignment of topics to new brokers.
> >>
> >> Kafka duly initiated the partition reassignment. I believe kafka
> >> controller will do the new leader election once the re-assigned
> >>partitions
> >> are in sync.
> >>
> >> However, in my case I realized that the new re-assigned partitions are
> >>not
> >> getting in sync at all. The in-rate in the existing partition is far
> >>more
> >> then the sync rate between earlier and newly reassigned partitions.
> >>
> >> In this scenario, will kafka wait indefinitely for the newly assigned
> >> partition to become sync (which will never be).
> >>
> >> Secondly what are the options to speed up the partition replica sync
> >>rate?
> >>
> >> Running kafka 0.8.0
> >>
> >> Regards,
> >> Virendra
> >>
> >>
> >>
> >>
>
>

Re: Kafka 0.8's VerifyConsumerRebalance reports an error

[Neha Narkhede]

I would turn on DEBUG on the tool to see which url it reads and doesn't
find the owners.




On Tue, Jun 24, 2014 at 11:28 AM, Yury Ruchin  wrote:

> I've just double-checked. The URL is correct, the same one is used by Kafka
> clients.
>
>
> 2014-06-24 22:21 GMT+04:00 Neha Narkhede :
>
> > Is it possible that maybe the zookeeper url used for the
> > VerifyConsumerRebalance tool is incorrect?
> >
> >
> > On Tue, Jun 24, 2014 at 12:02 AM, Yury Ruchin 
> > wrote:
> >
> > > Hi,
> > >
> > > I've run into the following problem. I try to read from a 50-partition
> > > Kafka topic using high level consumer with 8 streams. I'm using
> 8-thread
> > > pool, each thread handling one stream. After a short time, the threads
> > > reading from the stream stop reading. Lag between topic latest offset
> and
> > > the consumer constantly increases as new messages come in.
> > >
> > > I looked into ZK:  /consumers//owners/ and
> > see a
> > > list of znodes corresponding to the full list of partitions: [1, 2, 3,
> > > ...]. When I do zk get on e. g.
> > > /consumers//owners//1 - I see a valid
> consumer
> > > name corresponding to Kafka client logs, e. g.
> > > _-140304960-abc12345-0. However, when I
> run
> > > the VerifyConsumerRebalance tool, I see the following:
> > >
> > > No owner for partition [,1]
> > > (kafka.tools.VerifyConsumerRebalance$)
> > >
> > > No owner for partition [,2]
> > > (kafka.tools.VerifyConsumerRebalance$)
> > >
> > > ...
> > >
> > > No owner for partition [,50]
> > > (kafka.tools.VerifyConsumerRebalance$)
> > >
> > > According to this output, no partition has owner, which seemingly
> > > contradicts to what I see in ZK.
> > >
> > > What would cause such a problem and how can I troubleshoot it further?
> > >
> > > Thanks!
> > >
> >
>

Re: Consumer offset is getting reset back to some old value automatically

[Neha Narkhede]

See the explanation from the zookeeper folks here
 -

" Because Zookeeper requires a majority, it is best to use an odd number of
machines. For example, with four machines ZooKeeper can only handle the
failure of a single machine; if two machines fail, the remaining two
machines do not constitute a majority. However, with five machines
ZooKeeper can handle the failure of two machines."

Hope that helps.



On Tue, Jun 24, 2014 at 12:36 PM, Kane Kane  wrote:

> Sorry, i meant 5 nodes in previous question.
>
> On Tue, Jun 24, 2014 at 12:36 PM, Kane Kane  wrote:
> > Hello Neha,
> >
> >>>ZK cluster of 3 nodes will tolerate the loss of 1 node, but if there is
> a
> > subsequent leader election for any reason, there is a chance that the
> > cluster does not reach a quorum. It is less likely but still risky to
> some
> > extent.
> >
> > Does it mean if you have to tolerate 1 node loss without any issues,
> > you need *at least* 4 nodes?
> >
> > On Tue, Jun 24, 2014 at 11:16 AM, Neha Narkhede 
> wrote:
> >> Can you elaborate your notion of "smooth"? I thought if you have
> >> replication factor=3 in this case, you should be able to tolerate loss
> >> of a node?
> >>
> >> Yes, you should be able to tolerate the loss of a node but if controlled
> >> shutdown is not enabled, the delay between loss of the old leader and
> >> election of the new leader will be longer.
> >>
> >> So, you mean ZK cluster of 3 nodes can't tolerate 1 node loss? I've
> >> seen many recommendations to run 3-nodes cluster, does it mean in
> >> cluster of 3 you won't be able to operate after loosing 1 node?
> >>
> >> ZK cluster of 3 nodes will tolerate the loss of 1 node, but if there is
> a
> >> subsequent leader election for any reason, there is a chance that the
> >> cluster does not reach a quorum. It is less likely but still risky to
> some
> >> extent.
> >>
> >>
> >> On Tue, Jun 24, 2014 at 2:44 AM, Hemath Kumar 
> >> wrote:
> >>
> >>> Yes kane i have the replication factor configured as 3
> >>>
> >>>
> >>> On Tue, Jun 24, 2014 at 2:42 AM, Kane Kane 
> wrote:
> >>>
> >>> > Hello Neha, can you explain your statements:
> >>> > >>Bringing one node down in a cluster will go smoothly only if your
> >>> > replication factor is 1 and you enabled controlled shutdown on the
> >>> brokers.
> >>> >
> >>> > Can you elaborate your notion of "smooth"? I thought if you have
> >>> > replication factor=3 in this case, you should be able to tolerate
> loss
> >>> > of a node?
> >>> >
> >>> > >>Also, bringing down 1 node our of a 3 node zookeeper cluster is
> risky,
> >>> > since any subsequent leader election might not reach a quorum.
> >>> >
> >>> > So, you mean ZK cluster of 3 nodes can't tolerate 1 node loss? I've
> >>> > seen many recommendations to run 3-nodes cluster, does it mean in
> >>> > cluster of 3 you won't be able to operate after loosing 1 node?
> >>> >
> >>> > Thanks.
> >>> >
> >>> > On Mon, Jun 23, 2014 at 9:04 AM, Neha Narkhede <
> neha.narkh...@gmail.com>
> >>> > wrote:
> >>> > > Bringing one node down in a cluster will go smoothly only if your
> >>> > > replication factor is 1 and you enabled controlled shutdown on the
> >>> > brokers.
> >>> > > Also, bringing down 1 node our of a 3 node zookeeper cluster is
> risky,
> >>> > > since any subsequent leader election might not reach a quorum.
> Having
> >>> > said
> >>> > > that, a partition going offline shouldn't cause a consumer's
> offset to
> >>> > > reset to an old value. How did you find out what the consumer's
> offset
> >>> > was?
> >>> > > Do you have your consumer's logs around?
> >>> > >
> >>> > > Thanks,
> >>> > > Neha
> >>> > >
> >>> > >
> >>> > > On Mon, Jun 23, 2014 at 12:28 AM, Hemath Kumar <
> hksrckmur...@gmail.com
> >>> >
> >>> > > wrote:
> >>> > >
> >>> > >> We have a 3 node cluster ( 3 kafka + 3 ZK nodes ) . Recently we
> came
> >>> > across
> >>> > >> a strange issue where we wanted to bring one of the node down from
> >>> > cluster
> >>> > >> ( 1 kafka + 1 zookeeper) for doing a maintenance . But the
> movement we
> >>> > >> brought it to down on some of the topics ( only some partitions)
> >>> > consumers
> >>> > >> offset is reset some old value.
> >>> > >>
> >>> > >> Any reason why this is happened?. As of my knowledge when brought
> one
> >>> > node
> >>> > >> down its should work smoothly with out any impact.
> >>> > >>
> >>> > >> Thanks,
> >>> > >> Murthy Chelankuri
> >>> > >>
> >>> >
> >>>
>

Kafka 0.8/VIP/SSL

[Reiner Stach]

I'm looking for advice on running Kafka 0.8 behind VIPs. The goal is to
support SSL traffic, with encryption and decryption being performed by
back-to-back VIPs at the client and in front of the broker. That is:

Kafka client
--> vip1a.myco.com:8080 (SSL encrypt) --- WAN ---> VIP 1b (SSL decrypt)
--> Kafka broker 1
--> vip2a.myco.com:8080 (SSL encrypt) --- WAN ---> VIP 2b (SSL decrypt)
--> Kafka broker 2

Note there is a separate VIP pipeline per broker, no multiplexing involved.
We don't care about ordering of messages within partitions or elsewhere.

I've configured each broker to register in ZK and advertise itself as being
the VIP endpoint:
broker 1:
advertised.host.name=vip1a.myco.com
advertised.port=8080
broker 2:
advertised.host.name=vip2a.myco.com
advertised.port=8080

Should this work? Any issues?

And since it *isn't* working for me yet, any idea why I might consistently
see a LeaderNotAvailableException when trying to publish or consume?

One possible issue: the brokers need to talk directly to each other,
broker-to-broker, right? And they will try to talk to each other via the
VIP endpoints (vip1a, vip2a)? Seems like I will need to either make the VIP
address routeable from the Kafka broker hosts to allow the broker-to-broker
comms, or else hack the Kafka code so the brokers present the advertised
host/port in response to metadata requests from clients - but talk to each
other on their 'real' addresses.

Many thanks
Reiner

Re: What's the behavior when Kafka is deleting messages and consumers are still reading

[Lian, Li]

GuoZhang,

Thanks for explaining. I thought there might be such kind of lock
mechanism but cannot confirm it in any documentation on Kafka website.

It will be better if this could be written down in some Wiki or FAQ.


Best regards,
Lex Lian

Email: ll...@ebay.com





On 6/24/14, 11:17 PM, "Guozhang Wang"  wrote:

>Hi Li,
>
>The log operations are protected by a lock, so if there is a concurrent
>read on this partition it will not be deleted. But then when it is deleted
>the next fetch/read will result in a OffsetOutOfRange exception and the
>consumer needs to restart from a offset reset value.
>
>Guozhang
>
>
>On Tue, Jun 24, 2014 at 5:20 AM, Lian, Li  wrote:
>
>> When Kafka broker is trying to delete message according to the log
>> retention setting, either triggered by log age or topic partition size,
>>if
>> the same time there are still consumers reading the topic, what will
>>happen?
>>
>> Best regards,
>> Lex Lian
>>
>> Email: ll...@ebay.com
>>
>>
>
>
>-- 
>-- Guozhang

Re: What's the behavior when Kafka is deleting messages and consumers are still reading

[Guozhang Wang]

I think we can probably update the documentation page for this update:

https://kafka.apache.org/documentation.html#compaction


On Tue, Jun 24, 2014 at 3:54 PM, Lian, Li  wrote:

> GuoZhang,
>
> Thanks for explaining. I thought there might be such kind of lock
> mechanism but cannot confirm it in any documentation on Kafka website.
>
> It will be better if this could be written down in some Wiki or FAQ.
>
>
> Best regards,
> Lex Lian
>
> Email: ll...@ebay.com
>
>
>
>
>
> On 6/24/14, 11:17 PM, "Guozhang Wang"  wrote:
>
> >Hi Li,
> >
> >The log operations are protected by a lock, so if there is a concurrent
> >read on this partition it will not be deleted. But then when it is deleted
> >the next fetch/read will result in a OffsetOutOfRange exception and the
> >consumer needs to restart from a offset reset value.
> >
> >Guozhang
> >
> >
> >On Tue, Jun 24, 2014 at 5:20 AM, Lian, Li  wrote:
> >
> >> When Kafka broker is trying to delete message according to the log
> >> retention setting, either triggered by log age or topic partition size,
> >>if
> >> the same time there are still consumers reading the topic, what will
> >>happen?
> >>
> >> Best regards,
> >> Lex Lian
> >>
> >> Email: ll...@ebay.com
> >>
> >>
> >
> >
> >--
> >-- Guozhang
>
>


-- 
-- Guozhang

Uneven distribution of kafka topic partitions across multiple brokers

[Virendra Pratap Singh]

Have a kafka cluster with 10 brokers (kafka 0.8.0).  All of the brokers were 
setup upfront. None was added later. Default number of partition is set to 4 
and default replication to 2.
Have 3 topics in the system. None of these topics are manually created upfront, 
when the cluster is setup. So relying on kafka to automatically create these 
topics when the producer(s) send data first time for each of these topics.
We have multiple producer which will emit data for all of these topics at any 
point of time. What it means is that kafka will be hit with producer request 
simultaneously from multiple producer for producer request for these 3 topics.

What is observed is the topics partitions do not get spread out evenly in this 
scenario. There are 10 brokers (ids 1-10) so expectation is that 3 * 4 = 12 
topic partitions should be spread out on all 10 servers. However in this case 
the first 2 brokers share most of the load and few partitions are spread out. 
The same is true for the replicated instances also.

Here is the dump of list topic

topic: topic1  partition: 0leader: 1   replicas: 1,2   isr: 1,2
topic: topic1  partition: 1leader: 2   replicas: 2,1   isr: 2,1
topic: topic1  partition: 2leader: 1   replicas: 1,2   isr: 1,2
topic: topic1  partition: 3leader: 2   replicas: 2,1   isr: 2,1
topic: topic2partition: 0leader: 9   replicas: 9,4   isr: 9,4
topic: topic2partition: 1leader: 10  replicas: 10,5  isr: 10,5
topic: topic2partition: 2leader: 1   replicas: 1,6   isr: 1,6
topic: topic2partition: 3leader: 2   replicas: 2,7   isr: 2,7
topic: topic3 partition: 0leader: 2   replicas: 2,1   isr: 2,1
topic: topic3 partition: 1leader: 1   replicas: 1,2   isr: 1,2
topic: topic3 partition: 2leader: 2   replicas: 2,1   isr: 2,1
topic: topic3 partition: 3leader: 1   replicas: 1,2   isr: 1,2

So what is my options to have kafka evenly distribute the topic partitions? 
Would pre creating the topics via create topic command help?

Regards,
Virendra

Monitoring Producers at Large Scale

[Bhavesh Mistry]

We use Kafka as Transport Layer to transport application logs.  How do we
monitor Producers at large scales about 6000 boxes x 4 topic per box so
roughly 24000 producers (spread across multiple data center.. we have
brokers per DC).  We do the monitoring based on logs.  I have tried
intercepting logs via Log4J custom implementation which only intercept WARN
and ERROR and FATAL events  org.apache.log4j.AppenderSkeleton append method
which send its logs to brokers (This is working but after load testing it
is causing deadlock some times between ProducerSendThread and Producer).

I know there are JMX monitoring MBeans available which we can pull the
data, but I would like to monitor Exceptions eg Leader Not Found, Queue is
full, resend fail etc in Kafka Library.

How does LinkedIn monitor the Producers ?

Thanks,

Bhavesh

Blacklisting Brokers

[Lung, Paul]

Hi All,

Is there anyway to blacklist brokers? Sometimes we run into situations where 
there are certain hardware failures on a broker machine, and the machines goes 
into a “half dead” state. The broker process is up and participating in the 
cluster, but it can’t actually transmit messages properly. Sometimes, even SSH 
fails on the machines, and we can’t log in to shutdown the broker process. The 
only way to resolve this issue at this point is to shutdown the machine, which 
is not always easy in a large data center scenario. The consequences are 
disastrous if this half dead broker is elected as the leader. So in this 
situation, it would be nice to be able to quickly black list a broker.

Thank you.
Paul Lung

kafka.common.LeaderNotAvailableException

[Zack Payton]

Hi all,

I have 3 zookeeper servers and 2 Kafka servers.
Running Kafka version 0.8.1.1.
Running zookeeper 3.3.5-cdh3u6.
>From the Kafka servers I can access the zookeeper servers on 2181.
>From one of the Kafka servers I can create a topic no problem:
[root@kafka1 kafka-0.8.1.1-src]# bin/kafka-topics.sh --zookeeper
zookeeper1:2181,zookeeper2:2181,zookeeper3 --create --topic test1
--replication-factor 2 --partitions 2
Created topic "test1".

But, when I go to write to it I get the following:
[root@kafka1 kafka-0.8.1.1-src]# bin/kafka-console-producer.sh
--broker-list kafka1.queue.customer.sea1.ops.riotgames.com:9092,
kafka2.queue.customer.sea1.ops.riotgames.com:9092 --topic test1
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
TEST1
[2014-06-24 18:49:15,753] WARN Error while fetching metadata
[{TopicMetadata for topic test1 ->
No partition metadata for topic test1 due to
kafka.common.LeaderNotAvailableException}] for topic [test1]: class
kafka.common.LeaderNotAvailableException
 (kafka.producer.BrokerPartitionInfo)
[2014-06-24 18:49:15,813] WARN Error while fetching metadata partition
0 leader:
none replicas: isr: isUnderReplicated: false for topic partition [test1,0]:
[class kafka.common.LeaderNotAvailableException]
(kafka.producer.BrokerPartitionInfo)
[2014-06-24 18:49:15,814] WARN Error while fetching metadata partition
1 leader:
none replicas: isr: isUnderReplicated: false for topic partition [test1,1]:
[class kafka.common.LeaderNotAvailableException]
(kafka.producer.BrokerPartitionInfo)
[2014-06-24 18:49:15,821] WARN Failed to collate messages by
topic,partition due to: No leader for any partition in topic test1
(kafka.producer.async.DefaultEventHandler)
[2014-06-24 18:49:15,933] WARN Error while fetching metadata
[{TopicMetadata for topic test1 ->
No partition metadata for topic test1 due to
kafka.common.LeaderNotAvailableException}] for topic [test1]: class
kafka.common.LeaderNotAvailableException
 (kafka.producer.BrokerPartitionInfo)
[2014-06-24 18:49:15,933] WARN Failed to collate messages by
topic,partition due to: No leader for any partition in topic test1
(kafka.producer.async.DefaultEventHandler)
[2014-06-24 18:49:16,041] WARN Error while fetching metadata
[{TopicMetadata for topic test1 ->
No partition metadata for topic test1 due to
kafka.common.LeaderNotAvailableException}] for topic [test1]: class
kafka.common.LeaderNotAvailableException
 (kafka.producer.BrokerPartitionInfo)
[2014-06-24 18:49:16,042] WARN Failed to collate messages by
topic,partition due to: No leader for any partition in topic test1
(kafka.producer.async.DefaultEventHandler)
[2014-06-24 18:49:16,148] WARN Error while fetching metadata partition
0 leader:
none replicas: isr: isUnderReplicated: false for topic partition [test1,0]:
[class kafka.common.LeaderNotAvailableException]
(kafka.producer.BrokerPartitionInfo)
[2014-06-24 18:49:16,149] WARN Error while fetching metadata partition
1 leader:
none replicas: isr: isUnderReplicated: false for topic partition [test1,1]:
[class kafka.common.LeaderNotAvailableException]
(kafka.producer.BrokerPartitionInfo)
[2014-06-24 18:49:16,149] WARN Failed to collate messages by
topic,partition due to: No leader for any partition in topic test1
(kafka.producer.async.DefaultEventHandler)
[2014-06-24 18:49:16,257] WARN Error while fetching metadata
[{TopicMetadata for topic test1 ->
No partition metadata for topic test1 due to
kafka.common.LeaderNotAvailableException}] for topic [test1]: class
kafka.common.LeaderNotAvailableException
 (kafka.producer.BrokerPartitionInfo)
[2014-06-24 18:49:16,260] ERROR Failed to send requests for topics test1
with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
[2014-06-24 18:49:16,262] ERROR Error in handling batch of 1 events
(kafka.producer.async.ProducerSendThread)
kafka.common.FailedToSendMessageException: Failed to send messages after 3
tries.
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
at scala.collection.immutable.Stream.foreach(Stream.scala:254)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66)
at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)



Documentation on this issue is sparse.  Any ideas?

Thanks,
Z

Re: kafka.common.LeaderNotAvailableException

[Joe Stein]

Are there any errors in the broker's logs?

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/


On Tue, Jun 24, 2014 at 9:56 PM, Zack Payton  wrote:

> Hi all,
>
> I have 3 zookeeper servers and 2 Kafka servers.
> Running Kafka version 0.8.1.1.
> Running zookeeper 3.3.5-cdh3u6.
> From the Kafka servers I can access the zookeeper servers on 2181.
> From one of the Kafka servers I can create a topic no problem:
> [root@kafka1 kafka-0.8.1.1-src]# bin/kafka-topics.sh --zookeeper
> zookeeper1:2181,zookeeper2:2181,zookeeper3 --create --topic test1
> --replication-factor 2 --partitions 2
> Created topic "test1".
>
> But, when I go to write to it I get the following:
> [root@kafka1 kafka-0.8.1.1-src]# bin/kafka-console-producer.sh
> --broker-list kafka1.queue.customer.sea1.ops.riotgames.com:9092,
> kafka2.queue.customer.sea1.ops.riotgames.com:9092 --topic test1
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> TEST1
> [2014-06-24 18:49:15,753] WARN Error while fetching metadata
> [{TopicMetadata for topic test1 ->
> No partition metadata for topic test1 due to
> kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> kafka.common.LeaderNotAvailableException
>  (kafka.producer.BrokerPartitionInfo)
> [2014-06-24 18:49:15,813] WARN Error while fetching metadata partition
> 0 leader:
> none replicas: isr: isUnderReplicated: false for topic partition [test1,0]:
> [class kafka.common.LeaderNotAvailableException]
> (kafka.producer.BrokerPartitionInfo)
> [2014-06-24 18:49:15,814] WARN Error while fetching metadata partition
> 1 leader:
> none replicas: isr: isUnderReplicated: false for topic partition [test1,1]:
> [class kafka.common.LeaderNotAvailableException]
> (kafka.producer.BrokerPartitionInfo)
> [2014-06-24 18:49:15,821] WARN Failed to collate messages by
> topic,partition due to: No leader for any partition in topic test1
> (kafka.producer.async.DefaultEventHandler)
> [2014-06-24 18:49:15,933] WARN Error while fetching metadata
> [{TopicMetadata for topic test1 ->
> No partition metadata for topic test1 due to
> kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> kafka.common.LeaderNotAvailableException
>  (kafka.producer.BrokerPartitionInfo)
> [2014-06-24 18:49:15,933] WARN Failed to collate messages by
> topic,partition due to: No leader for any partition in topic test1
> (kafka.producer.async.DefaultEventHandler)
> [2014-06-24 18:49:16,041] WARN Error while fetching metadata
> [{TopicMetadata for topic test1 ->
> No partition metadata for topic test1 due to
> kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> kafka.common.LeaderNotAvailableException
>  (kafka.producer.BrokerPartitionInfo)
> [2014-06-24 18:49:16,042] WARN Failed to collate messages by
> topic,partition due to: No leader for any partition in topic test1
> (kafka.producer.async.DefaultEventHandler)
> [2014-06-24 18:49:16,148] WARN Error while fetching metadata partition
> 0 leader:
> none replicas: isr: isUnderReplicated: false for topic partition [test1,0]:
> [class kafka.common.LeaderNotAvailableException]
> (kafka.producer.BrokerPartitionInfo)
> [2014-06-24 18:49:16,149] WARN Error while fetching metadata partition
> 1 leader:
> none replicas: isr: isUnderReplicated: false for topic partition [test1,1]:
> [class kafka.common.LeaderNotAvailableException]
> (kafka.producer.BrokerPartitionInfo)
> [2014-06-24 18:49:16,149] WARN Failed to collate messages by
> topic,partition due to: No leader for any partition in topic test1
> (kafka.producer.async.DefaultEventHandler)
> [2014-06-24 18:49:16,257] WARN Error while fetching metadata
> [{TopicMetadata for topic test1 ->
> No partition metadata for topic test1 due to
> kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> kafka.common.LeaderNotAvailableException
>  (kafka.producer.BrokerPartitionInfo)
> [2014-06-24 18:49:16,260] ERROR Failed to send requests for topics test1
> with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler)
> [2014-06-24 18:49:16,262] ERROR Error in handling batch of 1 events
> (kafka.producer.async.ProducerSendThread)
> kafka.common.FailedToSendMessageException: Failed to send messages after 3
> tries.
> at
>
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> at
>
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
> at
>
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87)
> at
>
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67)
> at scala.collection.immutable.

Re: kafka.common.LeaderNotAvailableException

[Zack Payton]

server.log has a lot of these errors:
[2014-06-24 20:07:16,124] ERROR [KafkaApi-6] error when handling request
Name: FetchRequest; Version: 0; CorrelationId: 81138; ClientId:
ReplicaFetche
rThread-0-5; ReplicaId: 6; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo:
[test1,0] -> PartitionFetchInfo(0,1048576) (kafka.server.KafkaApis)
*kafka.common.KafkaException: Shouldn't set logEndOffset for replica 6
partition [test1,0] since it's local*
at kafka.cluster.Replica.logEndOffset_$eq(Replica.scala:46)
at
kafka.cluster.Partition.updateLeaderHWAndMaybeExpandIsr(Partition.scala:236)
at
kafka.server.ReplicaManager.recordFollowerPosition(ReplicaManager.scala:432)
at
kafka.server.KafkaApis$$anonfun$maybeUpdatePartitionHw$2.apply(KafkaApis.scala:460)
at
kafka.server.KafkaApis$$anonfun$maybeUpdatePartitionHw$2.apply(KafkaApis.scala:458)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:105)
at
kafka.server.KafkaApis.maybeUpdatePartitionHw(KafkaApis.scala:458)
at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:424)
at kafka.server.KafkaApis.handle(KafkaApis.scala:186)
at
kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
at java.lang.Thread.run(Thread.java:701)



On Tue, Jun 24, 2014 at 8:10 PM, Joe Stein  wrote:

> Are there any errors in the broker's logs?
>
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop 
> /
>
>
> On Tue, Jun 24, 2014 at 9:56 PM, Zack Payton  wrote:
>
> > Hi all,
> >
> > I have 3 zookeeper servers and 2 Kafka servers.
> > Running Kafka version 0.8.1.1.
> > Running zookeeper 3.3.5-cdh3u6.
> > From the Kafka servers I can access the zookeeper servers on 2181.
> > From one of the Kafka servers I can create a topic no problem:
> > [root@kafka1 kafka-0.8.1.1-src]# bin/kafka-topics.sh --zookeeper
> > zookeeper1:2181,zookeeper2:2181,zookeeper3 --create --topic test1
> > --replication-factor 2 --partitions 2
> > Created topic "test1".
> >
> > But, when I go to write to it I get the following:
> > [root@kafka1 kafka-0.8.1.1-src]# bin/kafka-console-producer.sh
> > --broker-list kafka1.queue.customer.sea1.ops.riotgames.com:9092,
> > kafka2.queue.customer.sea1.ops.riotgames.com:9092 --topic test1
> > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> > SLF4J: Defaulting to no-operation (NOP) logger implementation
> > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for
> further
> > details.
> > TEST1
> > [2014-06-24 18:49:15,753] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException
> >  (kafka.producer.BrokerPartitionInfo)
> > [2014-06-24 18:49:15,813] WARN Error while fetching metadata partition
> > 0 leader:
> > none replicas: isr: isUnderReplicated: false for topic partition
> [test1,0]:
> > [class kafka.common.LeaderNotAvailableException]
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-06-24 18:49:15,814] WARN Error while fetching metadata partition
> > 1 leader:
> > none replicas: isr: isUnderReplicated: false for topic partition
> [test1,1]:
> > [class kafka.common.LeaderNotAvailableException]
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-06-24 18:49:15,821] WARN Failed to collate messages by
> > topic,partition due to: No leader for any partition in topic test1
> > (kafka.producer.async.DefaultEventHandler)
> > [2014-06-24 18:49:15,933] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException
> >  (kafka.producer.BrokerPartitionInfo)
> > [2014-06-24 18:49:15,933] WARN Failed to collate messages by
> > topic,partition due to: No leader for any partition in topic test1
> > (kafka.producer.async.DefaultEventHandler)
> > [2014-06-24 18:49:16,041] WARN Error while fetching metadata
> > [{TopicMetadata for topic test1 ->
> > No partition metadata for topic test1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [test1]: class
> > kafka.common.LeaderNotAvailableException
> >  (kafka.producer.BrokerPartitionInfo)
> > [2014-06-24 18:49:16,042] WARN Failed to collate messages by
> > topic,partition due to: No leader for any partition in topic test1
> > (kafka.producer.async.DefaultEventHandler)
> > [2014-06-24 18:49:16,148] WARN Error while fetching metadata partition
> > 0 leader:
> > none replicas: isr: isUnderReplicated: false for topic partition
> [test1,0]:
> > [class kafka.common.LeaderNotAvailableException]
> > (kafka.producer.BrokerPartitionInfo)
> > [2014-

Re: Uneven distribution of kafka topic partitions across multiple brokers

[Joe Stein]

Take a look at

bin/kafka-reassign-partitions.sh

Option  Description

--  ---

--broker-list   The list of brokers to which the

  partitions need to be reassigned
in
  the form "0,1,2". This is
required
  if --topics-to-move-json-file is

  used to generate reassignment

  configuration

--execute   Kick off the reassignment as
specified
  by the --reassignment-json-file

  option.

--generate  Generate a candidate partition

  reassignment configuration. Note

  that this only generates a
candidate
  assignment, it does not execute
it.
--reassignment-json-file   reassignment configurationThe
format
  to use is -

{"partitions":

[{"topic": "foo",

  "partition": 1,

  "replicas": [1,2,3] }],

"version":1

}

--topics-to-move-json-file to move the partitions of the

  specified topics to the list of

  brokers specified by the
--broker-
  list option. The format to use is
-
{"topics":

[{"topic": "foo"},{"topic":
"foo1"}],
"version":1

}

--verifyVerify if the reassignment
completed
  as specified by the
--reassignment-
  json-file option.

--zookeeper   REQUIRED: The connection string for

  the zookeeper connection in the
form
  host:port. Multiple URLS can be

  given to allow fail-over.

Command must include exactly one action: --generate, --execute or --verify

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/


On Tue, Jun 24, 2014 at 7:44 PM, Virendra Pratap Singh <
vpsi...@yahoo-inc.com.invalid> wrote:

> Have a kafka cluster with 10 brokers (kafka 0.8.0).  All of the brokers
> were setup upfront. None was added later. Default number of partition is
> set to 4 and default replication to 2.
> Have 3 topics in the system. None of these topics are manually created
> upfront, when the cluster is setup. So relying on kafka to automatically
> create these topics when the producer(s) send data first time for each of
> these topics.
> We have multiple producer which will emit data for all of these topics at
> any point of time. What it means is that kafka will be hit with producer
> request simultaneously from multiple producer for producer request for
> these 3 topics.
>
> What is observed is the topics partitions do not get spread out evenly in
> this scenario. There are 10 brokers (ids 1-10) so expectation is that 3 * 4
> = 12 topic partitions should be spread out on all 10 servers. However in
> this case the first 2 brokers share most of the load and few partitions are
> spread out. The same is true for the replicated instances also.
>
> Here is the dump of list topic
>
> topic: topic1  partition: 0leader: 1   replicas: 1,2   isr: 1,2
> topic: topic1  partition: 1leader: 2   replicas: 2,1   isr: 2,1
> topic: topic1  partition: 2leader: 1   replicas: 1,2   isr: 1,2
> topic: topic1  partition: 3leader: 2   replicas: 2,1   isr: 2,1
> topic: topic2partition: 0leader: 9   replicas: 9,4   isr:
> 9,4
> topic: topic2partition: 1leader: 10  replicas: 10,5  isr:
> 10,5
> topic: topic2partition: 2leader: 1   replicas: 1,6   isr:
> 1,6
> topic: topic2partition: 3leader: 2   replicas: 2,7   isr:
> 2,7
> topic: topic3 partition: 0leader: 2   replicas: 2,1   isr: 2,1
> topic: topic3 partition: 1leader: 1   replicas: 1,2   isr: 1,2
> topic: topic3 partition: 2leader: 2   replicas: 2,1   isr: 2,1
> topic: topic3 partition

Getting "java.io.IOException: Too many open files"

[Lung, Paul]

Hi All,


I just upgraded my cluster from 0.8.1 to 0.8.1.1. I’m seeing the following 
error messages on the same 3 brokers once in a while:


[2014-06-24 21:43:44,711] ERROR Error in acceptor (kafka.network.Acceptor)

java.io.IOException: Too many open files

at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

at 
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)

at kafka.network.Acceptor.accept(SocketServer.scala:200)

at kafka.network.Acceptor.run(SocketServer.scala:154)

at java.lang.Thread.run(Thread.java:679)

[2014-06-24 21:43:44,711] ERROR Error in acceptor (kafka.network.Acceptor)

java.io.IOException: Too many open files

at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

at 
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)

at kafka.network.Acceptor.accept(SocketServer.scala:200)

at kafka.network.Acceptor.run(SocketServer.scala:154)

at java.lang.Thread.run(Thread.java:679)

When this happens, these 3 brokers essentially go out of sync when you do a 
“kafka-topics.sh —describe”.

I tracked the number of open files by doing “watch –n 1 ‘sudo lsof | wc –l’”, 
which basically counts all open files on the system. The numbers for the 
systems are basically in the 6000 range, with one system going to 9000. I 
presume the 9000 machine is the controller. Looking at the ulimit of the user, 
both the hard limit and the soft limit for open files is 100,000. Using sysctl, 
the max file is fs.file-max = 9774928. So we seem to be way under the limit.

What am I missing here? Is there some JVM limit around 10K open files or 
something?

Paul Lung

Re: Getting "java.io.IOException: Too many open files"

[Prakash Gowri Shankor]

How many files does each broker itself have open ? You can find this from
'ls -l /proc//fd'




On Tue, Jun 24, 2014 at 10:18 PM, Lung, Paul  wrote:

> Hi All,
>
>
> I just upgraded my cluster from 0.8.1 to 0.8.1.1. I’m seeing the following
> error messages on the same 3 brokers once in a while:
>
>
> [2014-06-24 21:43:44,711] ERROR Error in acceptor (kafka.network.Acceptor)
>
> java.io.IOException: Too many open files
>
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
>
> at kafka.network.Acceptor.accept(SocketServer.scala:200)
>
> at kafka.network.Acceptor.run(SocketServer.scala:154)
>
> at java.lang.Thread.run(Thread.java:679)
>
> [2014-06-24 21:43:44,711] ERROR Error in acceptor (kafka.network.Acceptor)
>
> java.io.IOException: Too many open files
>
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)
>
> at kafka.network.Acceptor.accept(SocketServer.scala:200)
>
> at kafka.network.Acceptor.run(SocketServer.scala:154)
>
> at java.lang.Thread.run(Thread.java:679)
>
> When this happens, these 3 brokers essentially go out of sync when you do
> a “kafka-topics.sh —describe”.
>
> I tracked the number of open files by doing “watch –n 1 ‘sudo lsof | wc
> –l’”, which basically counts all open files on the system. The numbers for
> the systems are basically in the 6000 range, with one system going to 9000.
> I presume the 9000 machine is the controller. Looking at the ulimit of the
> user, both the hard limit and the soft limit for open files is 100,000.
> Using sysctl, the max file is fs.file-max = 9774928. So we seem to be way
> under the limit.
>
> What am I missing here? Is there some JVM limit around 10K open files or
> something?
>
> Paul Lung
>

Re: Getting "java.io.IOException: Too many open files"

[Lung, Paul]

The controller machine has 3500 or so, while the other machines have
around 1600.

Paul Lung

On 6/24/14, 10:31 PM, "Prakash Gowri Shankor" 
wrote:

>How many files does each broker itself have open ? You can find this from
>'ls -l /proc//fd'
>
>
>
>
>On Tue, Jun 24, 2014 at 10:18 PM, Lung, Paul  wrote:
>
>> Hi All,
>>
>>
>> I just upgraded my cluster from 0.8.1 to 0.8.1.1. I¹m seeing the
>>following
>> error messages on the same 3 brokers once in a while:
>>
>>
>> [2014-06-24 21:43:44,711] ERROR Error in acceptor
>>(kafka.network.Acceptor)
>>
>> java.io.IOException: Too many open files
>>
>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>
>> at
>> 
>>sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:16
>>3)
>>
>> at kafka.network.Acceptor.accept(SocketServer.scala:200)
>>
>> at kafka.network.Acceptor.run(SocketServer.scala:154)
>>
>> at java.lang.Thread.run(Thread.java:679)
>>
>> [2014-06-24 21:43:44,711] ERROR Error in acceptor
>>(kafka.network.Acceptor)
>>
>> java.io.IOException: Too many open files
>>
>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>
>> at
>> 
>>sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:16
>>3)
>>
>> at kafka.network.Acceptor.accept(SocketServer.scala:200)
>>
>> at kafka.network.Acceptor.run(SocketServer.scala:154)
>>
>> at java.lang.Thread.run(Thread.java:679)
>>
>> When this happens, these 3 brokers essentially go out of sync when you
>>do
>> a ³kafka-topics.sh ‹describe².
>>
>> I tracked the number of open files by doing ³watch ­n 1 Œsudo lsof | wc
>> ­l¹², which basically counts all open files on the system. The numbers
>>for
>> the systems are basically in the 6000 range, with one system going to
>>9000.
>> I presume the 9000 machine is the controller. Looking at the ulimit of
>>the
>> user, both the hard limit and the soft limit for open files is 100,000.
>> Using sysctl, the max file is fs.file-max = 9774928. So we seem to be
>>way
>> under the limit.
>>
>> What am I missing here? Is there some JVM limit around 10K open files or
>> something?
>>
>> Paul Lung
>>

Re: Uneven distribution of kafka topic partitions across multiple brokers

[Neha Narkhede]

Looking at the output of list topics, here is what I think happened. When
topic1 and topic3 were created, only brokers 1&2 were online and alive.
When topic2 was created, almost all brokers were online. Only brokers that
are alive at the time of topic creation can be assigned replicas for the
topic. I would suggest ensuring that all brokers are alive and repeating
the experiment with 0.8.1.1, which is the latest stable release.

Thanks,
Neha


On Tue, Jun 24, 2014 at 4:44 PM, Virendra Pratap Singh <
vpsi...@yahoo-inc.com.invalid> wrote:

> Have a kafka cluster with 10 brokers (kafka 0.8.0).  All of the brokers
> were setup upfront. None was added later. Default number of partition is
> set to 4 and default replication to 2.
> Have 3 topics in the system. None of these topics are manually created
> upfront, when the cluster is setup. So relying on kafka to automatically
> create these topics when the producer(s) send data first time for each of
> these topics.
> We have multiple producer which will emit data for all of these topics at
> any point of time. What it means is that kafka will be hit with producer
> request simultaneously from multiple producer for producer request for
> these 3 topics.
>
> What is observed is the topics partitions do not get spread out evenly in
> this scenario. There are 10 brokers (ids 1-10) so expectation is that 3 * 4
> = 12 topic partitions should be spread out on all 10 servers. However in
> this case the first 2 brokers share most of the load and few partitions are
> spread out. The same is true for the replicated instances also.
>
> Here is the dump of list topic
>
> topic: topic1  partition: 0leader: 1   replicas: 1,2   isr: 1,2
> topic: topic1  partition: 1leader: 2   replicas: 2,1   isr: 2,1
> topic: topic1  partition: 2leader: 1   replicas: 1,2   isr: 1,2
> topic: topic1  partition: 3leader: 2   replicas: 2,1   isr: 2,1
> topic: topic2partition: 0leader: 9   replicas: 9,4   isr:
> 9,4
> topic: topic2partition: 1leader: 10  replicas: 10,5  isr:
> 10,5
> topic: topic2partition: 2leader: 1   replicas: 1,6   isr:
> 1,6
> topic: topic2partition: 3leader: 2   replicas: 2,7   isr:
> 2,7
> topic: topic3 partition: 0leader: 2   replicas: 2,1   isr: 2,1
> topic: topic3 partition: 1leader: 1   replicas: 1,2   isr: 1,2
> topic: topic3 partition: 2leader: 2   replicas: 2,1   isr: 2,1
> topic: topic3 partition: 3leader: 1   replicas: 1,2   isr: 1,2
>
> So what is my options to have kafka evenly distribute the topic
> partitions? Would pre creating the topics via create topic command help?
>
> Regards,
> Virendra
>

Re: Getting "java.io.IOException: Too many open files"

[Lung, Paul]

Ok. What I just saw was that when the controller machine reaches around
4100+ files, it crashes. Then I think the controller bounced between 2
other machines, taking them down too, and the circled back to the original
machine.

Paul Lung

On 6/24/14, 10:51 PM, "Lung, Paul"  wrote:

>The controller machine has 3500 or so, while the other machines have
>around 1600.
>
>Paul Lung
>
>On 6/24/14, 10:31 PM, "Prakash Gowri Shankor" 
>wrote:
>
>>How many files does each broker itself have open ? You can find this from
>>'ls -l /proc//fd'
>>
>>
>>
>>
>>On Tue, Jun 24, 2014 at 10:18 PM, Lung, Paul  wrote:
>>
>>> Hi All,
>>>
>>>
>>> I just upgraded my cluster from 0.8.1 to 0.8.1.1. I¹m seeing the
>>>following
>>> error messages on the same 3 brokers once in a while:
>>>
>>>
>>> [2014-06-24 21:43:44,711] ERROR Error in acceptor
>>>(kafka.network.Acceptor)
>>>
>>> java.io.IOException: Too many open files
>>>
>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>>
>>> at
>>> 
>>>sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:1
>>>6
>>>3)
>>>
>>> at kafka.network.Acceptor.accept(SocketServer.scala:200)
>>>
>>> at kafka.network.Acceptor.run(SocketServer.scala:154)
>>>
>>> at java.lang.Thread.run(Thread.java:679)
>>>
>>> [2014-06-24 21:43:44,711] ERROR Error in acceptor
>>>(kafka.network.Acceptor)
>>>
>>> java.io.IOException: Too many open files
>>>
>>> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>>>
>>> at
>>> 
>>>sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:1
>>>6
>>>3)
>>>
>>> at kafka.network.Acceptor.accept(SocketServer.scala:200)
>>>
>>> at kafka.network.Acceptor.run(SocketServer.scala:154)
>>>
>>> at java.lang.Thread.run(Thread.java:679)
>>>
>>> When this happens, these 3 brokers essentially go out of sync when you
>>>do
>>> a ³kafka-topics.sh ‹describe².
>>>
>>> I tracked the number of open files by doing ³watch ­n 1 Œsudo lsof | wc
>>> ­l¹², which basically counts all open files on the system. The numbers
>>>for
>>> the systems are basically in the 6000 range, with one system going to
>>>9000.
>>> I presume the 9000 machine is the controller. Looking at the ulimit of
>>>the
>>> user, both the hard limit and the soft limit for open files is 100,000.
>>> Using sysctl, the max file is fs.file-max = 9774928. So we seem to be
>>>way
>>> under the limit.
>>>
>>> What am I missing here? Is there some JVM limit around 10K open files
>>>or
>>> something?
>>>
>>> Paul Lung
>>>
>