Best partition configuration

[Tom Brown]

I have not been able to find reliable advice regarding how many partitions
should exist on a single broker. How many partitions have you used, and
what kind of throughput have you seen?

Thanks in advance!

--Tom

RE: Best partition configuration

[Yu, Libo]

Say if you have 3 brokers and 2 consumers in a consumer group, you may choose 6.
So when one broker is offline, the load is still well balanced.
If you use 12, when you add a fouth broker, the load is still well balanced.


Regards,

Libo


-Original Message-
From: Tom Brown [mailto:tombrow...@gmail.com] 
Sent: Wednesday, August 21, 2013 10:07 AM
To: users@kafka.apache.org
Subject: Best partition configuration

I have not been able to find reliable advice regarding how many partitions 
should exist on a single broker. How many partitions have you used, and what 
kind of throughput have you seen?

Thanks in advance!

--Tom

RE: issue with kafka-preferred-replica-election.sh

[Yu, Libo]

For the path /brokers/topics/my_topic/partitions, if you remove partitions and 
run
kafka-preferred-replica-election.sh, it terminates right away with keeper 
exception.

Regards,

Libo


-Original Message-
From: Guozhang Wang [mailto:wangg...@gmail.com] 
Sent: Wednesday, August 21, 2013 12:23 AM
To: users@kafka.apache.org
Subject: Re: issue with kafka-preferred-replica-election.sh

Libo,

Sorry for the late reply. I will file a JIRA for this one. Could you please 
provide the full process and environment to reproduce this issue?

Guozhang


On Tue, Aug 20, 2013 at 1:10 PM, Yu, Libo  wrote:

> Hi Guozhang,
>
> Thanks for the detailed reply. All the brokers are running.
> The path not created is /brokers/topics/uattoqaaa.default/partitions
> not /brokers/topics/uattoqaaa.default/partitions/[partition-id]/state
> Actually, I found quite a few topics don't have the issue. The 
> controller must have failed to create them. I will keep monitoring this issue.
>
> Currently, when /brokers/topics/my_path/partitions is not there, 
> kafka-preferred-replica-election.sh stops right away with nonode 
> keeper exception. Could you file a JIRA to make it keep working in 
> that scenario?
>
>
>
> Regards,
>
> Libo
>
>
> -Original Message-
> From: Guozhang Wang [mailto:wangg...@gmail.com]
> Sent: Tuesday, August 20, 2013 3:56 PM
> To: users@kafka.apache.org
> Subject: Re: issue with kafka-preferred-replica-election.sh
>
> The create command tool will only create the path 
> /brokers/topics/uattoqaaa.default, and then controller, once noticing 
> the change in ZK about the added topic, will elect leaders from the 
> assigned replicas written in /brokers/topics/uattoqaaa.default and 
> then create /brokers/topics/uattoqaaa.default/partitions/[partition-id]/state.
>
> If you have created the topic for a long time but 
> /brokers/topics/uattoqaaa.default/partitions/[partition-id]/state is 
> not created by controller, it could because the replicas specified in 
> /brokers/topics/uattoqaaa.default does not exist yet. Could you verify 
> the brokers specified in this path are up and running?
>
> Guozhang
>
>
> On Tue, Aug 20, 2013 at 12:14 PM, Yu, Libo  wrote:
>
> > One more question:
> > "The create topic command will not immediately create the path; it 
> > will be created by the controller later during the creation of 
> > partitions and assign replicas to partitions."
> >
> > So when will the controller create the path and partitions? When the 
> > first message is published to the topic and partition?
> > All the partitions are created at once?
> >
> > Regards,
> >
> > Libo
> >
> >
> > -Original Message-
> > From: Guozhang Wang [mailto:wangg...@gmail.com]
> > Sent: Tuesday, August 20, 2013 1:48 PM
> > To: users@kafka.apache.org
> > Subject: Re: issue with kafka-preferred-replica-election.sh
> >
> > Could you check that before you run
> > kafka-preferred-replica-election.sh
> > the /brokers/topics/uattoqaaa.default/partitions Node has been 
> > created or not?
> > The create topic command will not immediately create the path; it 
> > will be created by the controller later during the creation of 
> > partitions and assign replicas to partitions. If you start the 
> > election command right after the create topic tool, it might not be created 
> > yet.
> >
> > Guozhang
> >
> >
> > On Tue, Aug 20, 2013 at 10:24 AM, Yu, Libo  wrote:
> >
> > > It is created with 3 partitions.
> > >
> > > Regards,
> > >
> > > Libo
> > >
> > >
> > > -Original Message-
> > > From: Guozhang Wang [mailto:wangg...@gmail.com]
> > > Sent: Tuesday, August 20, 2013 12:21 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: issue with kafka-preferred-replica-election.sh
> > >
> > > Hello Libo,
> > >
> > > Before you call the kafka-preferred-replica-election.sh have you 
> > > already created the topic uattoqaaa.default with the number of
> > partitions?
> > >
> > > Guozhang
> > >
> > >
> > > On Tue, Aug 20, 2013 at 8:35 AM, Yu, Libo  wrote:
> > >
> > > > Hi,
> > > >
> > > > I tried to run kafka-preferred-replica-election.sh on our kafka
> > cluster.
> > > > But I got this expection:
> > > > Failed to start preferred replica election
> > > > org.I0Itec.zkclient.exception.ZkNoNodeException:
> > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > > > KeeperErrorCode = NoNode for
> > > > /brokers/topics/uattoqaaa.default/partitions
> > > >
> > > > I checked zookeeper and there is no 
> > > > /brokers/topics/uattoqaaa.default/partitions. All I found is 
> > > > /brokers/topics/uattoqaaa.default.
> > > >
> > > > I believe this is a bug and I am not sure if there is a JIRA for it.
> > > > Could you confirm? Thanks.
> > > >
> > > > Regards,
> > > >
> > > > Libo
> > > >
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
> -- Guozhang
>



--
-- Guozhang

mx4j-tools.jar missing

[Yu, Libo]

Hi team,

I got the message from a broker log:

[2013-08-21 08:22:55,502] INFO Will not load MX4J, mx4j-tools.jar is not in th
e classpath (kafka.utils.Mx4jLoader$)

Should it be in kafka_2.8.0-0.8.0-SNAPSHOT.jar or  
kafka-assembly-0.8.0-SNAPSHOT-deps.jar?
Is that a packaging error with sbt? Thanks.

Regards,

Libo

broker never comes back to ISR

[Yu, Libo]

Hi team,

We have three kafka brokers in a production cluster. We use replication factor 
3 for all topics.
We notice quite frequently one broker is not in isr. Sometimes after it is 
restarted, it
will go back to isr. Sometimes even after it is restarted, it will not go back 
to isr.

In today's case, after a broker is restarted, this is what we found from the 
log:

[2013-08-21 08:22:55,524] INFO [Kafka Server 2], started 
(kafka.server.KafkaServer)
[2013-08-21 08:25:06,621] INFO Closing socket connection to /xxx.xx.xx.xx. 
(kafka.network.Processor)
[2013-08-21 08:25:06,716] INFO Closing socket connection to / xxx.xx.xx.xx. 
(kafka.network.Processor)
[2013-08-21 08:27:19,824] INFO Closing socket connection to / xxx.xx.xx.xx. 
(kafka.network.Processor)
[2013-08-21 08:28:16,711] INFO Closing socket connection to / xxx.xx.xx.xx. 
(kafka.network.Processor)
[2013-08-21 08:28:17,978] INFO Closing socket connection to / xxx.xx.xx.xx. 
(kafka.network.Processor)
...
Numerous "Closing socket connection" and nothing else.

Any guidance will be appreciated.

Regards,

Libo

RE: broker never comes back to ISR

[Yu, Libo]

I checked the log of normal restart. The replication manager should start to 
handle
leader and isr request after the server is up. What may stop it from doing that?
Is it because of missing mx4j-tools.jar?

Regards,

Libo

From: Yu, Libo [ICG-IT]
Sent: Wednesday, August 21, 2013 10:51 AM
To: 'users@kafka.apache.org'
Subject: broker never comes back to ISR

Hi team,

We have three kafka brokers in a production cluster. We use replication factor 
3 for all topics.
We notice quite frequently one broker is not in isr. Sometimes after it is 
restarted, it
will go back to isr. Sometimes even after it is restarted, it will not go back 
to isr.

In today's case, after a broker is restarted, this is what we found from the 
log:

[2013-08-21 08:22:55,524] INFO [Kafka Server 2], started 
(kafka.server.KafkaServer)
[2013-08-21 08:25:06,621] INFO Closing socket connection to /xxx.xx.xx.xx. 
(kafka.network.Processor)
[2013-08-21 08:25:06,716] INFO Closing socket connection to / xxx.xx.xx.xx. 
(kafka.network.Processor)
[2013-08-21 08:27:19,824] INFO Closing socket connection to / xxx.xx.xx.xx. 
(kafka.network.Processor)
[2013-08-21 08:28:16,711] INFO Closing socket connection to / xxx.xx.xx.xx. 
(kafka.network.Processor)
[2013-08-21 08:28:17,978] INFO Closing socket connection to / xxx.xx.xx.xx. 
(kafka.network.Processor)
...
Numerous "Closing socket connection" and nothing else.

Any guidance will be appreciated.

Regards,

Libo

Re: issue with kafka-preferred-replica-election.sh

[Guozhang Wang]

Libo,

Just want to clarify, in your case after you created your topic, the
/brokers/topics/my_topic/partitions was never created or it was deleted
somehow?

Guozhang



On Wed, Aug 21, 2013 at 7:25 AM, Yu, Libo  wrote:

> For the path /brokers/topics/my_topic/partitions, if you remove partitions
> and run
> kafka-preferred-replica-election.sh, it terminates right away with keeper
> exception.
>
> Regards,
>
> Libo
>
>
> -Original Message-
> From: Guozhang Wang [mailto:wangg...@gmail.com]
> Sent: Wednesday, August 21, 2013 12:23 AM
> To: users@kafka.apache.org
> Subject: Re: issue with kafka-preferred-replica-election.sh
>
> Libo,
>
> Sorry for the late reply. I will file a JIRA for this one. Could you
> please provide the full process and environment to reproduce this issue?
>
> Guozhang
>
>
> On Tue, Aug 20, 2013 at 1:10 PM, Yu, Libo  wrote:
>
> > Hi Guozhang,
> >
> > Thanks for the detailed reply. All the brokers are running.
> > The path not created is /brokers/topics/uattoqaaa.default/partitions
> > not /brokers/topics/uattoqaaa.default/partitions/[partition-id]/state
> > Actually, I found quite a few topics don't have the issue. The
> > controller must have failed to create them. I will keep monitoring this
> issue.
> >
> > Currently, when /brokers/topics/my_path/partitions is not there,
> > kafka-preferred-replica-election.sh stops right away with nonode
> > keeper exception. Could you file a JIRA to make it keep working in
> > that scenario?
> >
> >
> >
> > Regards,
> >
> > Libo
> >
> >
> > -Original Message-
> > From: Guozhang Wang [mailto:wangg...@gmail.com]
> > Sent: Tuesday, August 20, 2013 3:56 PM
> > To: users@kafka.apache.org
> > Subject: Re: issue with kafka-preferred-replica-election.sh
> >
> > The create command tool will only create the path
> > /brokers/topics/uattoqaaa.default, and then controller, once noticing
> > the change in ZK about the added topic, will elect leaders from the
> > assigned replicas written in /brokers/topics/uattoqaaa.default and
> > then create
> /brokers/topics/uattoqaaa.default/partitions/[partition-id]/state.
> >
> > If you have created the topic for a long time but
> > /brokers/topics/uattoqaaa.default/partitions/[partition-id]/state is
> > not created by controller, it could because the replicas specified in
> > /brokers/topics/uattoqaaa.default does not exist yet. Could you verify
> > the brokers specified in this path are up and running?
> >
> > Guozhang
> >
> >
> > On Tue, Aug 20, 2013 at 12:14 PM, Yu, Libo  wrote:
> >
> > > One more question:
> > > "The create topic command will not immediately create the path; it
> > > will be created by the controller later during the creation of
> > > partitions and assign replicas to partitions."
> > >
> > > So when will the controller create the path and partitions? When the
> > > first message is published to the topic and partition?
> > > All the partitions are created at once?
> > >
> > > Regards,
> > >
> > > Libo
> > >
> > >
> > > -Original Message-
> > > From: Guozhang Wang [mailto:wangg...@gmail.com]
> > > Sent: Tuesday, August 20, 2013 1:48 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: issue with kafka-preferred-replica-election.sh
> > >
> > > Could you check that before you run
> > > kafka-preferred-replica-election.sh
> > > the /brokers/topics/uattoqaaa.default/partitions Node has been
> > > created or not?
> > > The create topic command will not immediately create the path; it
> > > will be created by the controller later during the creation of
> > > partitions and assign replicas to partitions. If you start the
> > > election command right after the create topic tool, it might not be
> created yet.
> > >
> > > Guozhang
> > >
> > >
> > > On Tue, Aug 20, 2013 at 10:24 AM, Yu, Libo  wrote:
> > >
> > > > It is created with 3 partitions.
> > > >
> > > > Regards,
> > > >
> > > > Libo
> > > >
> > > >
> > > > -Original Message-
> > > > From: Guozhang Wang [mailto:wangg...@gmail.com]
> > > > Sent: Tuesday, August 20, 2013 12:21 PM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: issue with kafka-preferred-replica-election.sh
> > > >
> > > > Hello Libo,
> > > >
> > > > Before you call the kafka-preferred-replica-election.sh have you
> > > > already created the topic uattoqaaa.default with the number of
> > > partitions?
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Tue, Aug 20, 2013 at 8:35 AM, Yu, Libo  wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I tried to run kafka-preferred-replica-election.sh on our kafka
> > > cluster.
> > > > > But I got this expection:
> > > > > Failed to start preferred replica election
> > > > > org.I0Itec.zkclient.exception.ZkNoNodeException:
> > > > > org.apache.zookeeper.KeeperException$NoNodeException:
> > > > > KeeperErrorCode = NoNode for
> > > > > /brokers/topics/uattoqaaa.default/partitions
> > > > >
> > > > > I checked zookeeper and there is no
> > > > > /brokers/topics/uattoqaaa.default/partitions. All I found is
> > > 

RE: broker never comes back to ISR

[Neha Narkhede]

Could you take a thread dump on that broker and send it across? One of the
possibilities is the replica fetcher thread is somehow dead.

Thanks,
Neha
On Aug 21, 2013 8:00 AM, "Yu, Libo"  wrote:

> I checked the log of normal restart. The replication manager should start
> to handle
> leader and isr request after the server is up. What may stop it from doing
> that?
> Is it because of missing mx4j-tools.jar?
>
> Regards,
>
> Libo
>
> From: Yu, Libo [ICG-IT]
> Sent: Wednesday, August 21, 2013 10:51 AM
> To: 'users@kafka.apache.org'
> Subject: broker never comes back to ISR
>
> Hi team,
>
> We have three kafka brokers in a production cluster. We use replication
> factor 3 for all topics.
> We notice quite frequently one broker is not in isr. Sometimes after it is
> restarted, it
> will go back to isr. Sometimes even after it is restarted, it will not go
> back to isr.
>
> In today's case, after a broker is restarted, this is what we found from
> the log:
>
> [2013-08-21 08:22:55,524] INFO [Kafka Server 2], started
> (kafka.server.KafkaServer)
> [2013-08-21 08:25:06,621] INFO Closing socket connection to /xxx.xx.xx.xx.
> (kafka.network.Processor)
> [2013-08-21 08:25:06,716] INFO Closing socket connection to /
> xxx.xx.xx.xx. (kafka.network.Processor)
> [2013-08-21 08:27:19,824] INFO Closing socket connection to /
> xxx.xx.xx.xx. (kafka.network.Processor)
> [2013-08-21 08:28:16,711] INFO Closing socket connection to /
> xxx.xx.xx.xx. (kafka.network.Processor)
> [2013-08-21 08:28:17,978] INFO Closing socket connection to /
> xxx.xx.xx.xx. (kafka.network.Processor)
> ...
> Numerous "Closing socket connection" and nothing else.
>
> Any guidance will be appreciated.
>
> Regards,
>
> Libo
>
>

High level java consumer hang in kafka 0.7 because of LinkedBlockingQueue

[Frank Yao]

Hi,

My java consumer client will suddenly stop consuming the stream. While I
enabled the 'TRACE' log, I found that this stop always happened in the end
of a queuechunk consumption - after several times' chunk actually. What
weird is that in log there are not any useful information. It occurred
suddenly.

I use jstack to figure out what happened and I found that perhaps the
thread is locked:
  java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0xfa399970> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:60)
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:32)
at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59)
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51

After google, I found some body has met this problem before, but no
solution there.
http://mail-archives.apache.org/mod_mbox/kafka-users/201211.mbox/%3CCANZjK9i87enoPY15rzh2Bg4D8+H1jvkSCkro=f3EROjn_4T=r...@mail.gmail.com%3E

If any one can offer any help, I'll really appreciate it.

-

Frank Yao
@VIPSHOP, Shanghai, China

Re: Best partition configuration

[Philip O'Toole]

On Wed, Aug 21, 2013 at 10:06:50AM -0400, Tom Brown wrote:
> I have not been able to find reliable advice regarding how many partitions
> should exist on a single broker. How many partitions have you used, and
> what kind of throughput have you seen?

We settled on 8 per broker, and usually run 3 brokers in production
environments, giving us a total of 24 partitions. Throughput has been superb.

For integration testing however, we usually use just 1 or 2 partitions.

Philip

> 
> Thanks in advance!
> 
> --Tom

-- 
Philip O'Toole

Senior Developer
Loggly, Inc.
San Francisco, CA.
www.loggly.com

Come join us!
http://loggly.com/company/careers/

Re: High level java consumer hang in kafka 0.7 because of LinkedBlockingQueue

[Philip O'Toole]

The high-level consumer will block if a) there are no more messages
available, b) the next message available is larger than the maximum fetch size 
you
have specified, or c) your client code simply stops pulling messages from the
iterator (the blocking queue will fill up).

Not sure if this applies to what you are seeing, but you should be aware of
these conditions.

Philip

On Thu, Aug 22, 2013 at 12:10:54AM +0800, Frank Yao wrote:
> Hi,
> 
> My java consumer client will suddenly stop consuming the stream. While I
> enabled the 'TRACE' log, I found that this stop always happened in the end
> of a queuechunk consumption - after several times' chunk actually. What
> weird is that in log there are not any useful information. It occurred
> suddenly.
> 
> I use jstack to figure out what happened and I found that perhaps the
> thread is locked:
>   java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0xfa399970> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:60)
> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:32)
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59)
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51
> 
> After google, I found some body has met this problem before, but no
> solution there.
> http://mail-archives.apache.org/mod_mbox/kafka-users/201211.mbox/%3CCANZjK9i87enoPY15rzh2Bg4D8+H1jvkSCkro=f3EROjn_4T=r...@mail.gmail.com%3E
> 
> If any one can offer any help, I'll really appreciate it.
> 
> -
> 
> Frank Yao
> @VIPSHOP, Shanghai, China

-- 
Philip O'Toole

Senior Developer
Loggly, Inc.
San Francisco, CA.
www.loggly.com

Come join us!
http://loggly.com/company/careers/

Re: High level java consumer hang in kafka 0.7 because of LinkedBlockingQueue

[Neha Narkhede]

Thanks for listing the possible conditions for a stalled consumer, Philip.
I added this to the FAQ -
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped%2Cwhy%3F

Frank,

Let us know if the FAQ above helps to understand your issue or can be
improved.

Thanks,
Neha


On Wed, Aug 21, 2013 at 9:26 AM, Philip O'Toole  wrote:

> The high-level consumer will block if a) there are no more messages
> available, b) the next message available is larger than the maximum fetch
> size you
> have specified, or c) your client code simply stops pulling messages from
> the
> iterator (the blocking queue will fill up).
>
> Not sure if this applies to what you are seeing, but you should be aware of
> these conditions.
>
> Philip
>
> On Thu, Aug 22, 2013 at 12:10:54AM +0800, Frank Yao wrote:
> > Hi,
> >
> > My java consumer client will suddenly stop consuming the stream. While I
> > enabled the 'TRACE' log, I found that this stop always happened in the
> end
> > of a queuechunk consumption - after several times' chunk actually. What
> > weird is that in log there are not any useful information. It occurred
> > suddenly.
> >
> > I use jstack to figure out what happened and I found that perhaps the
> > thread is locked:
> >   java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for  <0xfa399970> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> > at
> >
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> > at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:60)
> > at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:32)
> > at
> kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:59)
> > at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:51
> >
> > After google, I found some body has met this problem before, but no
> > solution there.
> >
> http://mail-archives.apache.org/mod_mbox/kafka-users/201211.mbox/%3CCANZjK9i87enoPY15rzh2Bg4D8+H1jvkSCkro=f3EROjn_4T=r...@mail.gmail.com%3E
> >
> > If any one can offer any help, I'll really appreciate it.
> >
> > -
> >
> > Frank Yao
> > @VIPSHOP, Shanghai, China
>
> --
> Philip O'Toole
>
> Senior Developer
> Loggly, Inc.
> San Francisco, CA.
> www.loggly.com
>
> Come join us!
> http://loggly.com/company/careers/
>

Re: How to get broker back to ISR

[James Wu]

Thank you Neha.

Actually my situation was more like this topic:
http://mail-archives.apache.org/mod_mbox/kafka-users/201308.mbox/browser
My broker didn't back to ISR.

I will try to reproduce later.


On Tue, Aug 20, 2013 at 10:55 PM, Neha Narkhede wrote:

> Once the broker is restarted, the controller broker will send it a list of
> partitions that it should follow. The broker starts fetching from the
> respective leaders and enters the ISR. Depending on the duration of
> shutdown, the broker can take some time to enter ISR.
>
> Thanks,
> Neha
> On Aug 20, 2013 4:26 AM, "James Wu"  wrote:
>
> > Hi,
> >
> > I am wondering if my leader broker crash, how to get it back to ISR after
> > restart kafak ?
> >
> > In the initial status the kafka-list-topic.sh shows:
> > topic: failover-test partition: 0 leader: 0 replicas: 0,1 isr: 0,1
> >
> > If I terminate the leader and kafka-list-topic.sh shows:
> > topic: failover-test partition: 0 leader: 1 replicas: 0,1 isr: 1
> >
> >
> > Is there any document can explain what is the procedure to get my broker0
> > back to isr ?
> >
> >
> > Thanks!
> >
> > --
> > --
> > Friendly regards,
> >
> > *James Wu 
> > *
> >
>



-- 
-- 
Friendly regards,

*James Wu 
*+886.922.199.680

Re: How to get broker back to ISR

[James Wu]

Sorry, the link is :
http://mail-archives.apache.org/mod_mbox/kafka-users/201308.mbox/raw/%3CFF142F6B499AE34CAED4D263F6CA32901D34E289%40EXTXMB19.nam.nsroot.net%3E/1


On Thu, Aug 22, 2013 at 12:46 AM, James Wu  wrote:

> Thank you Neha.
>
> Actually my situation was more like this topic:
> http://mail-archives.apache.org/mod_mbox/kafka-users/201308.mbox/browser
> My broker didn't back to ISR.
>
> I will try to reproduce later.
>
>
> On Tue, Aug 20, 2013 at 10:55 PM, Neha Narkhede 
> wrote:
>
>> Once the broker is restarted, the controller broker will send it a list of
>> partitions that it should follow. The broker starts fetching from the
>> respective leaders and enters the ISR. Depending on the duration of
>> shutdown, the broker can take some time to enter ISR.
>>
>> Thanks,
>> Neha
>> On Aug 20, 2013 4:26 AM, "James Wu"  wrote:
>>
>> > Hi,
>> >
>> > I am wondering if my leader broker crash, how to get it back to ISR
>> after
>> > restart kafak ?
>> >
>> > In the initial status the kafka-list-topic.sh shows:
>> > topic: failover-test partition: 0 leader: 0 replicas: 0,1 isr: 0,1
>> >
>> > If I terminate the leader and kafka-list-topic.sh shows:
>> > topic: failover-test partition: 0 leader: 1 replicas: 0,1 isr: 1
>> >
>> >
>> > Is there any document can explain what is the procedure to get my
>> broker0
>> > back to isr ?
>> >
>> >
>> > Thanks!
>> >
>> > --
>> > --
>> > Friendly regards,
>> >
>> > *James Wu 
>> > *
>> >
>>
>
>
>
> --
> --
>  Friendly regards,
>
> *James Wu 
> *+886.922.199.680
>



-- 
-- 
Friendly regards,

*James Wu 
*+886.922.199.680

Re: Best partition configuration

[Tom Brown]

Philip,

How many topics per broker (just one?) And what is the read/write profile
of your setup?

--Tom


On Wed, Aug 21, 2013 at 12:24 PM, Philip O'Toole  wrote:

> On Wed, Aug 21, 2013 at 10:06:50AM -0400, Tom Brown wrote:
> > I have not been able to find reliable advice regarding how many
> partitions
> > should exist on a single broker. How many partitions have you used, and
> > what kind of throughput have you seen?
>
> We settled on 8 per broker, and usually run 3 brokers in production
> environments, giving us a total of 24 partitions. Throughput has been
> superb.
>
> For integration testing however, we usually use just 1 or 2 partitions.
>
> Philip
>
> >
> > Thanks in advance!
> >
> > --Tom
>
> --
> Philip O'Toole
>
> Senior Developer
> Loggly, Inc.
> San Francisco, CA.
> www.loggly.com
>
> Come join us!
> http://loggly.com/company/careers/
>

Re: questtion about log.retention.bytes

[Jun Rao]

All per topic configuration properties below have the format of csv (e.g.,
"topic1:value1,topic2:value2"). Updated our website to make it clear.

Thanks,

Jun


On Tue, Aug 20, 2013 at 6:16 AM, Paul Christian
wrote:

> Jun,
>
> For my first example is that syntax correct? I.e.
>
> log.retention.bytes.per.topic.A = 15MB
> log.retention.bytes.per.topic.B = 20MB
>
> I totally guessed there and was wondering if I guessed right? Otherwise is
> there a document with the proper formatting to full out this map?
>
> Thank you,
>
> Paul
>

Re: Failed to start preferred replica election

[Jun Rao]

Added to the 0.8 documentation.

Thanks,

Jun


On Tue, Aug 20, 2013 at 9:22 AM, Jay Kreps  wrote:

> Is there any way to channel these many excellent email threads into
> documentation improvements :-)
>
> -Jay
>
>
> On Mon, Aug 19, 2013 at 8:55 PM, Jun Rao  wrote:
>
> > We also have a jmx bean that tracks the lag in messages per partition in
> > the follower broker.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, Aug 19, 2013 at 1:07 PM, Vadim Keylis 
> > wrote:
> >
> > > It does print partitions. I just did not include them in the bug.
> > >
> > > How can I monitor replica resync progress as well as know when resync
> > > process completed using script? That should allow me to better predict
> > when
> > > the tool would run successfully.
> > >
> > > Thanks so much.
> > >
> > >
> > > On Mon, Aug 19, 2013 at 12:59 PM, Neha Narkhede <
> neha.narkh...@gmail.com
> > > >wrote:
> > >
> > > > I think the error message can be improved to at least print which
> > > > partitions it couldn't move the leader for. What could be happening
> is
> > > that
> > > > the 2 brokers that were down might not have entered the ISR yet. So
> the
> > > > tool will not be able to move any leaders to them. You can run
> > > > kafka-list-topics with the --under-replicated-count option to print
> the
> > > > list of under replicated partitions.
> > > >
> > > > Please can you file a bug to improve the error reporting of this
> tool?
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Mon, Aug 19, 2013 at 12:26 PM, Vadim Keylis <
> vkeylis2...@gmail.com
> > > > >wrote:
> > > >
> > > > > I have a cluster of 3 kafka servers. Replication factor is 3. Two
> out
> > > of
> > > > 3
> > > > > servers were shutdown and traffic was sent to only one server that
> > was
> > > > up.
> > > > > I brought second host up and it says according to logs that server
> > has
> > > > > started.
> > > > >
> > > > > I ran ./kafka-list-topic.sh --zookeeper  Still was showing
> > > leaders
> > > > > are not distributed. Then ran
> > > > > kafka-preferred-replica-election.sh which exited with error:
> > > > >
> > > > > kafka.common.AdminCommandFailedException: Admin command failed
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.admin.PreferredReplicaLeaderElectionCommand.moveLeaderToPreferredReplica(PreferredReplicaLeaderElectionCommand.scala:119)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.admin.PreferredReplicaLeaderElectionCommand$.main(PreferredReplicaLeaderElectionCommand.scala:60)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.admin.PreferredReplicaLeaderElectionCommand.main(PreferredReplicaLeaderElectionCommand.scala)
> > > > >
> > > > > Would you please give suggestion what have caused the exception and
> > how
> > > > to
> > > > > recover from it?
> > > > >
> > > > > Thanks so much in advance,
> > > > > Vadim
> > > > >
> > > >
> > >
> >
>

Re: mx4j-tools.jar missing

[Jun Rao]

It's optional. If you want to access jmx through http, you can include that
jar.

Thanks,

Jun


On Wed, Aug 21, 2013 at 7:39 AM, Yu, Libo  wrote:

> Hi team,
>
> I got the message from a broker log:
>
> [2013-08-21 08:22:55,502] INFO Will not load MX4J, mx4j-tools.jar is not
> in th
> e classpath (kafka.utils.Mx4jLoader$)
>
> Should it be in kafka_2.8.0-0.8.0-SNAPSHOT.jar or
>  kafka-assembly-0.8.0-SNAPSHOT-deps.jar?
> Is that a packaging error with sbt? Thanks.
>
> Regards,
>
> Libo
>
>

Re: How to get broker back to ISR

[Jun Rao]

Using list-topics command, you can figure out which replicas are not in
ISR. Using the lag jmx (
http://kafka.apache.org/documentation.html#monitoring), you can figure out
if the follower is catching up.

Thanks,

Jun


On Wed, Aug 21, 2013 at 9:47 AM, James Wu  wrote:

> Sorry, the link is :
>
> http://mail-archives.apache.org/mod_mbox/kafka-users/201308.mbox/raw/%3CFF142F6B499AE34CAED4D263F6CA32901D34E289%40EXTXMB19.nam.nsroot.net%3E/1
>
>
> On Thu, Aug 22, 2013 at 12:46 AM, James Wu  wrote:
>
> > Thank you Neha.
> >
> > Actually my situation was more like this topic:
> > http://mail-archives.apache.org/mod_mbox/kafka-users/201308.mbox/browser
> > My broker didn't back to ISR.
> >
> > I will try to reproduce later.
> >
> >
> > On Tue, Aug 20, 2013 at 10:55 PM, Neha Narkhede  >wrote:
> >
> >> Once the broker is restarted, the controller broker will send it a list
> of
> >> partitions that it should follow. The broker starts fetching from the
> >> respective leaders and enters the ISR. Depending on the duration of
> >> shutdown, the broker can take some time to enter ISR.
> >>
> >> Thanks,
> >> Neha
> >> On Aug 20, 2013 4:26 AM, "James Wu"  wrote:
> >>
> >> > Hi,
> >> >
> >> > I am wondering if my leader broker crash, how to get it back to ISR
> >> after
> >> > restart kafak ?
> >> >
> >> > In the initial status the kafka-list-topic.sh shows:
> >> > topic: failover-test partition: 0 leader: 0 replicas: 0,1 isr: 0,1
> >> >
> >> > If I terminate the leader and kafka-list-topic.sh shows:
> >> > topic: failover-test partition: 0 leader: 1 replicas: 0,1 isr: 1
> >> >
> >> >
> >> > Is there any document can explain what is the procedure to get my
> >> broker0
> >> > back to isr ?
> >> >
> >> >
> >> > Thanks!
> >> >
> >> > --
> >> > --
> >> > Friendly regards,
> >> >
> >> > *James Wu 
> >> > *
> >> >
> >>
> >
> >
> >
> > --
> > --
> >  Friendly regards,
> >
> > *James Wu 
> > *+886.922.199.680
> >
>
>
>
> --
> --
> Friendly regards,
>
> *James Wu 
> *+886.922.199.680
>

Re: Best partition configuration

[Jay Kreps]

Hey Tom,

I tried to expand the FAQ on this, let me know if you feel like it answers
your question:

https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIchoosethenumberofpartitionsforatopic%3F

Folks--let me know if I missed anything there. I tried to give both a
simple answer and a more complete answer.

-Jay



On Wed, Aug 21, 2013 at 9:52 AM, Tom Brown  wrote:

> Philip,
>
> How many topics per broker (just one?) And what is the read/write profile
> of your setup?
>
> --Tom
>
>
> On Wed, Aug 21, 2013 at 12:24 PM, Philip O'Toole 
> wrote:
>
> > On Wed, Aug 21, 2013 at 10:06:50AM -0400, Tom Brown wrote:
> > > I have not been able to find reliable advice regarding how many
> > partitions
> > > should exist on a single broker. How many partitions have you used, and
> > > what kind of throughput have you seen?
> >
> > We settled on 8 per broker, and usually run 3 brokers in production
> > environments, giving us a total of 24 partitions. Throughput has been
> > superb.
> >
> > For integration testing however, we usually use just 1 or 2 partitions.
> >
> > Philip
> >
> > >
> > > Thanks in advance!
> > >
> > > --Tom
> >
> > --
> > Philip O'Toole
> >
> > Senior Developer
> > Loggly, Inc.
> > San Francisco, CA.
> > www.loggly.com
> >
> > Come join us!
> > http://loggly.com/company/careers/
> >
>

Re: Best partition configuration

[Philip O'Toole]

1 topic. 

I don't understand the second question. 

Philip

On Aug 21, 2013, at 9:52 AM, Tom Brown  wrote:

> Philip,
> 
> How many topics per broker (just one?) And what is the read/write profile
> of your setup?
> 
> --Tom
> 
> 
> On Wed, Aug 21, 2013 at 12:24 PM, Philip O'Toole  wrote:
> 
>> On Wed, Aug 21, 2013 at 10:06:50AM -0400, Tom Brown wrote:
>>> I have not been able to find reliable advice regarding how many
>> partitions
>>> should exist on a single broker. How many partitions have you used, and
>>> what kind of throughput have you seen?
>> 
>> We settled on 8 per broker, and usually run 3 brokers in production
>> environments, giving us a total of 24 partitions. Throughput has been
>> superb.
>> 
>> For integration testing however, we usually use just 1 or 2 partitions.
>> 
>> Philip
>> 
>>> 
>>> Thanks in advance!
>>> 
>>> --Tom
>> 
>> --
>> Philip O'Toole
>> 
>> Senior Developer
>> Loggly, Inc.
>> San Francisco, CA.
>> www.loggly.com
>> 
>> Come join us!
>> http://loggly.com/company/careers/
>> 

RE: issue with kafka-preferred-replica-election.sh

[Yu, Libo]

We never deleted it. Either it was never created or deleted somehow.

Regards,

Libo


-Original Message-
From: Guozhang Wang [mailto:wangg...@gmail.com] 
Sent: Wednesday, August 21, 2013 11:41 AM
To: users@kafka.apache.org
Subject: Re: issue with kafka-preferred-replica-election.sh

Libo,

Just want to clarify, in your case after you created your topic, the 
/brokers/topics/my_topic/partitions was never created or it was deleted somehow?

Guozhang



On Wed, Aug 21, 2013 at 7:25 AM, Yu, Libo  wrote:

> For the path /brokers/topics/my_topic/partitions, if you remove 
> partitions and run kafka-preferred-replica-election.sh, it terminates 
> right away with keeper exception.
>
> Regards,
>
> Libo
>
>
> -Original Message-
> From: Guozhang Wang [mailto:wangg...@gmail.com]
> Sent: Wednesday, August 21, 2013 12:23 AM
> To: users@kafka.apache.org
> Subject: Re: issue with kafka-preferred-replica-election.sh
>
> Libo,
>
> Sorry for the late reply. I will file a JIRA for this one. Could you 
> please provide the full process and environment to reproduce this issue?
>
> Guozhang
>
>
> On Tue, Aug 20, 2013 at 1:10 PM, Yu, Libo  wrote:
>
> > Hi Guozhang,
> >
> > Thanks for the detailed reply. All the brokers are running.
> > The path not created is /brokers/topics/uattoqaaa.default/partitions
> > not 
> > /brokers/topics/uattoqaaa.default/partitions/[partition-id]/state
> > Actually, I found quite a few topics don't have the issue. The 
> > controller must have failed to create them. I will keep monitoring 
> > this
> issue.
> >
> > Currently, when /brokers/topics/my_path/partitions is not there, 
> > kafka-preferred-replica-election.sh stops right away with nonode 
> > keeper exception. Could you file a JIRA to make it keep working in 
> > that scenario?
> >
> >
> >
> > Regards,
> >
> > Libo
> >
> >
> > -Original Message-
> > From: Guozhang Wang [mailto:wangg...@gmail.com]
> > Sent: Tuesday, August 20, 2013 3:56 PM
> > To: users@kafka.apache.org
> > Subject: Re: issue with kafka-preferred-replica-election.sh
> >
> > The create command tool will only create the path 
> > /brokers/topics/uattoqaaa.default, and then controller, once 
> > noticing the change in ZK about the added topic, will elect leaders 
> > from the assigned replicas written in 
> > /brokers/topics/uattoqaaa.default and then create
> /brokers/topics/uattoqaaa.default/partitions/[partition-id]/state.
> >
> > If you have created the topic for a long time but 
> > /brokers/topics/uattoqaaa.default/partitions/[partition-id]/state is 
> > not created by controller, it could because the replicas specified 
> > in /brokers/topics/uattoqaaa.default does not exist yet. Could you 
> > verify the brokers specified in this path are up and running?
> >
> > Guozhang
> >
> >
> > On Tue, Aug 20, 2013 at 12:14 PM, Yu, Libo  wrote:
> >
> > > One more question:
> > > "The create topic command will not immediately create the path; it 
> > > will be created by the controller later during the creation of 
> > > partitions and assign replicas to partitions."
> > >
> > > So when will the controller create the path and partitions? When 
> > > the first message is published to the topic and partition?
> > > All the partitions are created at once?
> > >
> > > Regards,
> > >
> > > Libo
> > >
> > >
> > > -Original Message-
> > > From: Guozhang Wang [mailto:wangg...@gmail.com]
> > > Sent: Tuesday, August 20, 2013 1:48 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: issue with kafka-preferred-replica-election.sh
> > >
> > > Could you check that before you run 
> > > kafka-preferred-replica-election.sh
> > > the /brokers/topics/uattoqaaa.default/partitions Node has been 
> > > created or not?
> > > The create topic command will not immediately create the path; it 
> > > will be created by the controller later during the creation of 
> > > partitions and assign replicas to partitions. If you start the 
> > > election command right after the create topic tool, it might not 
> > > be
> created yet.
> > >
> > > Guozhang
> > >
> > >
> > > On Tue, Aug 20, 2013 at 10:24 AM, Yu, Libo  wrote:
> > >
> > > > It is created with 3 partitions.
> > > >
> > > > Regards,
> > > >
> > > > Libo
> > > >
> > > >
> > > > -Original Message-
> > > > From: Guozhang Wang [mailto:wangg...@gmail.com]
> > > > Sent: Tuesday, August 20, 2013 12:21 PM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: issue with kafka-preferred-replica-election.sh
> > > >
> > > > Hello Libo,
> > > >
> > > > Before you call the kafka-preferred-replica-election.sh have you 
> > > > already created the topic uattoqaaa.default with the number of
> > > partitions?
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Tue, Aug 20, 2013 at 8:35 AM, Yu, Libo  wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I tried to run kafka-preferred-replica-election.sh on our 
> > > > > kafka
> > > cluster.
> > > > > But I got this expection:
> > > > > Failed to start preferred replica election
> > > 

RE: broker never comes back to ISR

[Yu, Libo]

Hi Neha,

Here is the thread dump. The process has been up for almost 7 hours and 
Nothing is in the log but numerous " INFO Closing socket connection to "
Thanks 

Libo

Deadlock Detection:

No deadlocks found.

Thread 20075: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may 
be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) 
@bci=20, line=226 (Interpreted frame)
 - 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode,
 boolean, long) @bci=174, line=460 (Interpreted frame)
 - 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, 
boolean, long) @bci=102, line=359 (Interpreted frame)
 - java.util.concurrent.SynchronousQueue.poll(long, 
java.util.concurrent.TimeUnit) @bci=11, line=942 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 
(Compiled frame)
 - 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
 @bci=17, line=1103 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=722 (Interpreted frame)


Thread 9516: (state = BLOCKED)


Thread 22490: (state = IN_NATIVE)
 - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, 
int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, line=150 
(Compiled frame)
 - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121 
(Compiled frame)
 - java.io.BufferedInputStream.fill() @bci=175, line=235 (Compiled frame)
 - java.io.BufferedInputStream.read() @bci=12, line=254 (Compiled frame)
 - java.io.FilterInputStream.read() @bci=4, line=83 (Compiled frame)
 - 
sun.rmi.transport.tcp.TCPTransport.handleMessages(sun.rmi.transport.Connection, 
boolean) @bci=25, line=535 (Compiled frame)
 - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0() @bci=685, 
line=808 (Interpreted frame)
 - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run() @bci=50, line=667 
(Interpreted frame)
 - 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
 @bci=46, line=1110 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=722 (Interpreted frame)


Thread 14658: (state = IN_NATIVE)
 - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, 
int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, line=150 
(Compiled frame)
 - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121 
(Compiled frame)
 - java.io.BufferedInputStream.fill() @bci=175, line=235 (Compiled frame)
 - java.io.BufferedInputStream.read() @bci=12, line=254 (Compiled frame)
 - java.io.FilterInputStream.read() @bci=4, line=83 (Compiled frame)
 - 
sun.rmi.transport.tcp.TCPTransport.handleMessages(sun.rmi.transport.Connection, 
boolean) @bci=25, line=535 (Compiled frame)
 - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0() @bci=685, 
line=808 (Interpreted frame)
 - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run() @bci=50, line=667 
(Interpreted frame)
 - 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
 @bci=46, line=1110 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=722 (Interpreted frame)


Thread 16336: (state = IN_NATIVE)
 - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, 
int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, line=150 
(Compiled frame)
 - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121 
(Compiled frame)
 - java.io.BufferedInputStream.fill() @bci=175, line=235 (Compiled frame)
 - java.io.BufferedInputStream.read() @bci=12, line=254 (Compiled frame)
 - java.io.FilterInputStream.read() @bci=4, line=83 (Compiled frame)
 - 
sun.rmi.transport.tcp.TCPTransport.handleMessages(sun.rmi.transport.Connection, 
boolean) @bci=25, line=535 (Compiled frame)
 - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0() @bci=685, 
line=808 (Interpreted frame)
 - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run() @bci=50, line=667 
(Interpreted frame)
 - 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
 @bci=46, line=1110 (Compiled frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 
(Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=722 (Interpreted frame)


Thread 12630: (state = IN_NATIVE)
 - java.net.So

RE: How to get broker back to ISR

[Yu, Libo]

Hi Neha,

Which broker is controller broker and how is it defined? 

Regards,

Libo


-Original Message-
From: Neha Narkhede [mailto:neha.narkh...@gmail.com] 
Sent: Tuesday, August 20, 2013 10:56 AM
To: users@kafka.apache.org
Subject: Re: How to get broker back to ISR

Once the broker is restarted, the controller broker will send it a list of 
partitions that it should follow. The broker starts fetching from the 
respective leaders and enters the ISR. Depending on the duration of shutdown, 
the broker can take some time to enter ISR.

Thanks,
Neha
On Aug 20, 2013 4:26 AM, "James Wu"  wrote:

> Hi,
>
> I am wondering if my leader broker crash, how to get it back to ISR 
> after restart kafak ?
>
> In the initial status the kafka-list-topic.sh shows:
> topic: failover-test partition: 0 leader: 0 replicas: 0,1 isr: 0,1
>
> If I terminate the leader and kafka-list-topic.sh shows:
> topic: failover-test partition: 0 leader: 1 replicas: 0,1 isr: 1
>
>
> Is there any document can explain what is the procedure to get my 
> broker0 back to isr ?
>
>
> Thanks!
>
> --
> --
> Friendly regards,
>
> *James Wu 
> *
>

ordering

[Yu, Libo]

Hi,

This is from kafka faq:


  *   Each partition is not consumed by more than one consumer thread/process 
in each consumer group. This allows to have each process consume in a single 
threaded fashion to guarantee ordering to the consumer within the partition (if 
we split up a partition of ordered messages and handed them out to multiple 
consumers even though the messages were stored in order they would be processed 
out of order at times).

Is this doable?

Say a topic has 3 partitions. And there are 3 consumer processes in a consumer 
group each of which has a single thread.
When we start the 3 processes, when the first process is up, it will consume 
all 3 partitions. When the second process is up,
1 process consumes 2 partitions and 1 consumes 1 partition. Only when the third 
process is up, will each process consume
One partition. This will cause trouble. There seems to be no way to bound a 
stream to one partition.


Regards,

Libo

controller data

[Yu, Libo]

Hi team,

I found this line from the log:
[2013-08-21 08:22:55,513] INFO conflict in /controller data: 2 stored data: 3
(kafka.utils.ZkUtils$)

I checked the zookeeper and the data in /controller is 3.

We have 3 brokers in our cluster. The broker ids are 0,1,2 or 1,2,3?

Regards,

Libo

Re: ordering

[Philip O'Toole]

No, there isn't, not at the very start when there is no state in
Zookeeper. Once there is state the Kafka team have told me that
rebalancing will not result in any dupes.

However, if there is no state in Zookeeper and your partitions are
empty, simply wait until all consumers have balanced before sending
any data.

Philip

On Wed, Aug 21, 2013 at 12:30 PM, Yu, Libo  wrote:
> Hi,
>
> This is from kafka faq:
>
>
>   *   Each partition is not consumed by more than one consumer thread/process 
> in each consumer group. This allows to have each process consume in a single 
> threaded fashion to guarantee ordering to the consumer within the partition 
> (if we split up a partition of ordered messages and handed them out to 
> multiple consumers even though the messages were stored in order they would 
> be processed out of order at times).
>
> Is this doable?
>
> Say a topic has 3 partitions. And there are 3 consumer processes in a 
> consumer group each of which has a single thread.
> When we start the 3 processes, when the first process is up, it will consume 
> all 3 partitions. When the second process is up,
> 1 process consumes 2 partitions and 1 consumes 1 partition. Only when the 
> third process is up, will each process consume
> One partition. This will cause trouble. There seems to be no way to bound a 
> stream to one partition.
>
>
> Regards,
>
> Libo
>

RE: How to get broker back to ISR

[Yu, Libo]

We have 3 brokers in our kafka cluster (1,2,3). Broker 2 somehow is not in isr.
I restarted it and it did not help at all. And we notice in many case we have to
restart the whole cluster to get it back. This is our top priority concern 
currently.

Here is the log after the restart:

[2013-08-21 16:17:18,992] INFO Registered broker 2 at path /brokers/ids/2 with
 address :1234. (kafka.utils.ZkUtils$)
[2013-08-21 16:17:18,992] INFO [Kafka Server 2], Connecting to ZK: :1234, 
:1234, :1234
 (kafka.server.KafkaServer)
[2013-08-21 16:17:19,061] INFO Will not load MX4J, mx4j-tools.jar is not in th
e classpath (kafka.utils.Mx4jLoader$)
[2013-08-21 16:17:19,072] INFO conflict in /controller data: 2 stored data: 3
(kafka.utils.ZkUtils$)
[2013-08-21 16:17:19,082] INFO [Kafka Server 2], started (kafka.server.KafkaSe
rver)
[2013-08-21 16:17:49,774] INFO Closing socket connection to /123.456.789. 
(kafka.network.Processor)
..

Regards,

Libo


-Original Message-
From: Yu, Libo [ICG-IT] 
Sent: Wednesday, August 21, 2013 3:15 PM
To: 'users@kafka.apache.org'
Subject: RE: How to get broker back to ISR

Hi Neha,

Which broker is controller broker and how is it defined? 

Regards,

Libo


-Original Message-
From: Neha Narkhede [mailto:neha.narkh...@gmail.com]
Sent: Tuesday, August 20, 2013 10:56 AM
To: users@kafka.apache.org
Subject: Re: How to get broker back to ISR

Once the broker is restarted, the controller broker will send it a list of 
partitions that it should follow. The broker starts fetching from the 
respective leaders and enters the ISR. Depending on the duration of shutdown, 
the broker can take some time to enter ISR.

Thanks,
Neha
On Aug 20, 2013 4:26 AM, "James Wu"  wrote:

> Hi,
>
> I am wondering if my leader broker crash, how to get it back to ISR 
> after restart kafak ?
>
> In the initial status the kafka-list-topic.sh shows:
> topic: failover-test partition: 0 leader: 0 replicas: 0,1 isr: 0,1
>
> If I terminate the leader and kafka-list-topic.sh shows:
> topic: failover-test partition: 0 leader: 1 replicas: 0,1 isr: 1
>
>
> Is there any document can explain what is the procedure to get my
> broker0 back to isr ?
>
>
> Thanks!
>
> --
> --
> Friendly regards,
>
> *James Wu 
> *
>

Patch for mmap + windows

[Jay Kreps]

Elizabeth and I have a patch to support our memory mapped offset index
files properly on Windows:
https://issues.apache.org/jira/browse/KAFKA-1008

Question: Do we want this on 0.8 or trunk? I would feel more comfortable
with it in trunk, but that means windows support in 0.8 is known to be
broken (as opposed to "not known to be broken but not known to be working
either" since we are not doing aggressive system testing on windows).

I would feel more comfortable doing the patch on 0.8 if there was someone
who would be willing to take on real load testing and/or production
operation on Windows so we could have some confidence that Kafka on Windows
actually works, otherwise this could just be the tip of the iceberg.

Also it would be great to get review on that patch regardless of the
destination.

-Jay

Re: Patch for mmap + windows

[Timothy Chen]

Hi Jay,

I'm planning to test run Kafka on Windows in our test environments
evaluating if it's suitable for production usage.

I can provide feedback with the patch how well it works and if we encounter
any functional or perf problems.

Tim




On Wed, Aug 21, 2013 at 2:54 PM, Jay Kreps  wrote:

> Elizabeth and I have a patch to support our memory mapped offset index
> files properly on Windows:
> https://issues.apache.org/jira/browse/KAFKA-1008
>
> Question: Do we want this on 0.8 or trunk? I would feel more comfortable
> with it in trunk, but that means windows support in 0.8 is known to be
> broken (as opposed to "not known to be broken but not known to be working
> either" since we are not doing aggressive system testing on windows).
>
> I would feel more comfortable doing the patch on 0.8 if there was someone
> who would be willing to take on real load testing and/or production
> operation on Windows so we could have some confidence that Kafka on Windows
> actually works, otherwise this could just be the tip of the iceberg.
>
> Also it would be great to get review on that patch regardless of the
> destination.
>
> -Jay
>

Re: Patch for mmap + windows

[Jay Kreps]

That would be great!

-Jay


On Wed, Aug 21, 2013 at 3:13 PM, Timothy Chen  wrote:

> Hi Jay,
>
> I'm planning to test run Kafka on Windows in our test environments
> evaluating if it's suitable for production usage.
>
> I can provide feedback with the patch how well it works and if we encounter
> any functional or perf problems.
>
> Tim
>
>
>
>
> On Wed, Aug 21, 2013 at 2:54 PM, Jay Kreps  wrote:
>
> > Elizabeth and I have a patch to support our memory mapped offset index
> > files properly on Windows:
> > https://issues.apache.org/jira/browse/KAFKA-1008
> >
> > Question: Do we want this on 0.8 or trunk? I would feel more comfortable
> > with it in trunk, but that means windows support in 0.8 is known to be
> > broken (as opposed to "not known to be broken but not known to be working
> > either" since we are not doing aggressive system testing on windows).
> >
> > I would feel more comfortable doing the patch on 0.8 if there was someone
> > who would be willing to take on real load testing and/or production
> > operation on Windows so we could have some confidence that Kafka on
> Windows
> > actually works, otherwise this could just be the tip of the iceberg.
> >
> > Also it would be great to get review on that patch regardless of the
> > destination.
> >
> > -Jay
> >
>

Re: How to get broker back to ISR

[Neha Narkhede]

Yu,

The controller is the broker that has the ActiveControllerCount jmx value
of 1. At any point of time, only one broker in a Kafka cluster should have
a value of 1 for this jmx mbean.

I personally find it very complex to find the replica fetcher thread's lag
for a particular partition that is under replicated. I think we should have
a tool that will take in a topic, partition and zookeeper url and give the
lag for all the replicas for that partition. I will file a JIRA for this.

Thanks,
Neha


On Wed, Aug 21, 2013 at 1:41 PM, Yu, Libo  wrote:

>
> We have 3 brokers in our kafka cluster (1,2,3). Broker 2 somehow is not in
> isr.
> I restarted it and it did not help at all. And we notice in many case we
> have to
> restart the whole cluster to get it back. This is our top priority concern
> currently.
>
> Here is the log after the restart:
>
> [2013-08-21 16:17:18,992] INFO Registered broker 2 at path /brokers/ids/2
> with
>  address :1234. (kafka.utils.ZkUtils$)
> [2013-08-21 16:17:18,992] INFO [Kafka Server 2], Connecting to ZK:
> :1234, :1234, :1234
>  (kafka.server.KafkaServer)
> [2013-08-21 16:17:19,061] INFO Will not load MX4J, mx4j-tools.jar is not
> in th
> e classpath (kafka.utils.Mx4jLoader$)
> [2013-08-21 16:17:19,072] INFO conflict in /controller data: 2 stored
> data: 3
> (kafka.utils.ZkUtils$)
> [2013-08-21 16:17:19,082] INFO [Kafka Server 2], started
> (kafka.server.KafkaSe
> rver)
> [2013-08-21 16:17:49,774] INFO Closing socket connection to /123.456.789.
> (kafka.network.Processor)
> ..
>
> Regards,
>
> Libo
>
>
> -Original Message-
> From: Yu, Libo [ICG-IT]
> Sent: Wednesday, August 21, 2013 3:15 PM
> To: 'users@kafka.apache.org'
> Subject: RE: How to get broker back to ISR
>
> Hi Neha,
>
> Which broker is controller broker and how is it defined?
>
> Regards,
>
> Libo
>
>
> -Original Message-
> From: Neha Narkhede [mailto:neha.narkh...@gmail.com]
> Sent: Tuesday, August 20, 2013 10:56 AM
> To: users@kafka.apache.org
> Subject: Re: How to get broker back to ISR
>
> Once the broker is restarted, the controller broker will send it a list of
> partitions that it should follow. The broker starts fetching from the
> respective leaders and enters the ISR. Depending on the duration of
> shutdown, the broker can take some time to enter ISR.
>
> Thanks,
> Neha
> On Aug 20, 2013 4:26 AM, "James Wu"  wrote:
>
> > Hi,
> >
> > I am wondering if my leader broker crash, how to get it back to ISR
> > after restart kafak ?
> >
> > In the initial status the kafka-list-topic.sh shows:
> > topic: failover-test partition: 0 leader: 0 replicas: 0,1 isr: 0,1
> >
> > If I terminate the leader and kafka-list-topic.sh shows:
> > topic: failover-test partition: 0 leader: 1 replicas: 0,1 isr: 1
> >
> >
> > Is there any document can explain what is the procedure to get my
> > broker0 back to isr ?
> >
> >
> > Thanks!
> >
> > --
> > --
> > Friendly regards,
> >
> > *James Wu 
> > *
> >
>

Re: controller data

[Neha Narkhede]

This log message is marked INFO since it is normal, though I think the
message content can be improved. The broker that does not get elected as
the controller will have this log message that indicates it tried to become
controller but some other broker succeeded instead.

Thanks,
Neha


On Wed, Aug 21, 2013 at 12:57 PM, Yu, Libo  wrote:

> Hi team,
>
> I found this line from the log:
> [2013-08-21 08:22:55,513] INFO conflict in /controller data: 2 stored
> data: 3
> (kafka.utils.ZkUtils$)
>
> I checked the zookeeper and the data in /controller is 3.
>
> We have 3 brokers in our cluster. The broker ids are 0,1,2 or 1,2,3?
>
> Regards,
>
> Libo
>
>

Re: broker never comes back to ISR

[Neha Narkhede]

Thanks for sending this across. One thing that is weird about the thread
dump is that the thread names are not included, so it is somewhat difficult
to read. But just from the stack traces, I don't see the
ReplicaFetcherThreads alive. Can you grep your broker logs to see if the
replica fetcher thread died due to some error. Something like -

grep ERROR kafka.log | grep "Error due to"

Thanks,
Neha


On Wed, Aug 21, 2013 at 12:10 PM, Yu, Libo  wrote:

> Hi Neha,
>
> Here is the thread dump. The process has been up for almost 7 hours and
> Nothing is in the log but numerous " INFO Closing socket connection to
> "
> Thanks
>
> Libo
>
> Deadlock Detection:
>
> No deadlocks found.
>
> Thread 20075: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
> may be imprecise)
>  - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object,
> long) @bci=20, line=226 (Interpreted frame)
>  -
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode,
> boolean, long) @bci=174, line=460 (Interpreted frame)
>  -
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object,
> boolean, long) @bci=102, line=359 (Interpreted frame)
>  - java.util.concurrent.SynchronousQueue.poll(long,
> java.util.concurrent.TimeUnit) @bci=11, line=942 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043
> (Compiled frame)
>  -
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
> @bci=17, line=1103 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=722 (Interpreted frame)
>
>
> Thread 9516: (state = BLOCKED)
>
>
> Thread 22490: (state = IN_NATIVE)
>  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[],
> int, int, int) @bci=0 (Compiled frame; information may be imprecise)
>  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
> line=150 (Compiled frame)
>  - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121
> (Compiled frame)
>  - java.io.BufferedInputStream.fill() @bci=175, line=235 (Compiled frame)
>  - java.io.BufferedInputStream.read() @bci=12, line=254 (Compiled frame)
>  - java.io.FilterInputStream.read() @bci=4, line=83 (Compiled frame)
>  -
> sun.rmi.transport.tcp.TCPTransport.handleMessages(sun.rmi.transport.Connection,
> boolean) @bci=25, line=535 (Compiled frame)
>  - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0() @bci=685,
> line=808 (Interpreted frame)
>  - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run() @bci=50,
> line=667 (Interpreted frame)
>  -
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
> @bci=46, line=1110 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=722 (Interpreted frame)
>
>
> Thread 14658: (state = IN_NATIVE)
>  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[],
> int, int, int) @bci=0 (Compiled frame; information may be imprecise)
>  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
> line=150 (Compiled frame)
>  - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121
> (Compiled frame)
>  - java.io.BufferedInputStream.fill() @bci=175, line=235 (Compiled frame)
>  - java.io.BufferedInputStream.read() @bci=12, line=254 (Compiled frame)
>  - java.io.FilterInputStream.read() @bci=4, line=83 (Compiled frame)
>  -
> sun.rmi.transport.tcp.TCPTransport.handleMessages(sun.rmi.transport.Connection,
> boolean) @bci=25, line=535 (Compiled frame)
>  - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0() @bci=685,
> line=808 (Interpreted frame)
>  - sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run() @bci=50,
> line=667 (Interpreted frame)
>  -
> java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
> @bci=46, line=1110 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603
> (Interpreted frame)
>  - java.lang.Thread.run() @bci=11, line=722 (Interpreted frame)
>
>
> Thread 16336: (state = IN_NATIVE)
>  - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[],
> int, int, int) @bci=0 (Compiled frame; information may be imprecise)
>  - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79,
> line=150 (Compiled frame)
>  - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=121
> (Compiled frame)
>  - java.io.BufferedInputStream.fill() @bci=175, line=235 (Compiled frame)
>  - java.io.BufferedInputStream.read() @bci=12, line=254 (Compiled frame)
>  - java.io.FilterInputStream.read() @bci=4, line=83 (Compiled frame)
>  -
> sun.rmi.transport.tcp.TCPTransport.handleMessages(sun.rmi.transport.C

More questions on avro serialization

[Mark]

Does LinkedIn include the SHA of the schema into the header of each Avro 
message they write or do they wrap the avro message and prepend the SHA?

In either case, how does the Hadoop consumer know what schema to read?

Re: use kafka as dependency in sbt from central maven repository

[James Wu]

My way is build kafka and install pom in local.
Then set: "Local Maven Repository" at
"file:///Users/jameswu/.m2/repository" in resolvers.



On Wed, Aug 21, 2013 at 5:21 AM, Lorenz Knies  wrote:

> hi joe,
>
> thanks for the hint with the apache repo.
> the from directive did not work for me. while it stopped sbt from
> complaining during the update, it did not include the jar in the classpath
> either.
> this may be correct behaviour since the sbt documentation states that you
> may specify a direct URL to the *jar file* via the from directive.
>
> but adding the apache repo as a resolver was the solution that fixed my
> setup:
>
> name := "kafka-dependency-test"
>
> scalaVersion := "2.9.2"
>
> resolvers += "Apache repo" at "
> https://repository.apache.org/content/repositories/releases";
>
> libraryDependencies += "org.apache.kafka" % "kafka_2.9.2" % "0.8.0-beta1"
> exclude("com.sun.jmx","jmxri") exclude("com.sun.jdmk","jmxtools")
>
>
> thanks,
>
> lorenz
>
> On Aug 20, 2013, at 4:03 PM, Joe Stein  wrote:
>
> > Lorenz,
> >
> > The Maven central repo got bonked (
> > https://issues.apache.org/jira/browse/KAFKA-974 ) and it looks like it
> is
> > affecting you too
> >
> > Try this
> >
> > "org.apache.kafka" % "kafka_2.8.2" % "0.8.0-beta1" intransitive() from "
> > https://repository.apache.org/content/repositories/releases";
> >
> > I just tried it and it works for me and others have used the main apache
> > repo to work around this same problem
> >
> > this will get fixed in Maven central for beta2 or if we go straight to GA
> >
> > thanks, let me know
> >
> >
> > /***
> > Joe Stein
> > Founder, Principal Consultant
> > Big Data Open Source Security LLC
> > http://www.stealth.ly
> > Twitter: @allthingshadoop 
> > /
> >
> >
> > On Tue, Aug 20, 2013 at 9:46 AM, Lorenz Knies  wrote:
> >
> >> thanks for the reply, but that unfortunately does not help.
> >>
> >> to make my problem easily reproducible:
> >>
> >> creating an empty directory, put just a build.sbt file with the
> following
> >> content into it:
> >>
> >> name := "kafka-dependency-test"
> >>
> >> scalaVersion := "2.8.2"
> >>
> >> libraryDependencies += "org.apache.kafka" % "kafka_2.8.2" %
> "0.8.0-beta1"
> >> intransitive()
> >>
> >> and run
> >>
> >> sbt update
> >>
> >> gives me the following error:
> >>
> >> sbt.ResolveException: unresolved dependency:
> >> org.apache.kafka#kafka_2.8.2;0.8.0-beta1: null name not allowed
> >>
> >>
> >> sbt sbt-version gives me the following:
> >>
> >> [info] Loading global plugins from /Users/lorenz/.sbt/plugins
> >> [info] Set current project to kafka-dependency-test (in build
> >> file:/Users/lorenz/software/kafka_dependency_test/)
> >> [info] 0.12.4
> >>
> >>
> >>
> >> nobody else having these problems?
> >> regards
> >>
> >> lorenz
> >>
> >>
> >>
> >>
> >> On Aug 20, 2013, at 3:15 PM, Joe Stein  wrote:
> >>
> >>> you need to put intransitive() at the end so it ignores dependencies
> that
> >>> are defined to be ignored in maven
> >>>
> >>> so your line would be
> >>>
> >>> libraryDependencies += "org.apache.kafka" % "kafka_2.9.2" %
> "0.8.0-beta1"
> >>> intransitive()
> >>>
> >>> /***
> >>> Joe Stein
> >>> Founder, Principal Consultant
> >>> Big Data Open Source Security LLC
> >>> http://www.stealth.ly
> >>> Twitter: @allthingshadoop 
> >>> /
> >>>
> >>>
> >>> On Tue, Aug 20, 2013 at 2:48 AM, Lorenz Knies  wrote:
> >>>
>  Hi,
> 
>  i am having problems using kafka as a dependency in sbt.
> 
>  with this simple build.sbt:
> 
>  name := "kafka-dependency-test"
> 
>  scalaVersion := "2.9.2"
> 
>  libraryDependencies += "org.apache.kafka" % "kafka_2.9.2" %
> >> "0.8.0-beta1"
> 
> 
>  when i do
> 
>  sbt update
> 
>  i get the following error:
> 
>  sbt.ResolveException: unresolved dependency:
>  org.apache.kafka#kafka_2.9.2;0.8.0-beta1: null name not allowed
> 
>  i am using sbt 0.12.4 but also tried other versions with no luck
> 
>  looks like the ivy implementation ignores the org and name attributes
> >> for
>  the zookeeper dependency in the custom inline xml included in kafkas
>  Build.scala
> 
>  anybody else having these problems and maybe some advice for a
>  solution/workaround?
> 
> 
>  regards
> 
>  lorenz
> 
> 
> >>
> >>
>
>


-- 
-- 
Friendly regards,

*James Wu 
*+886.922.199.680

Re: More questions on avro serialization

[Neha Narkhede]

We define the LinkedIn Kafka message to have a magic byte (indicating Avro
serialization), MD5 header followed by the payload. The Hadoop consumer
reads the MD5, looks up the schema in the repository and deserializes the
message.

Thanks,
Neha


On Wed, Aug 21, 2013 at 8:15 PM, Mark  wrote:

> Does LinkedIn include the SHA of the schema into the header of each Avro
> message they write or do they wrap the avro message and prepend the SHA?
>
> In either case, how does the Hadoop consumer know what schema to read?

Re: ordering

[Jun Rao]

Actually, I am not sure if I understand the trouble that you mentioned.
Could you elaborate that a bit more?

Thanks,

Jun


On Wed, Aug 21, 2013 at 12:30 PM, Yu, Libo  wrote:

> Hi,
>
> This is from kafka faq:
>
>
>   *   Each partition is not consumed by more than one consumer
> thread/process in each consumer group. This allows to have each process
> consume in a single threaded fashion to guarantee ordering to the consumer
> within the partition (if we split up a partition of ordered messages and
> handed them out to multiple consumers even though the messages were stored
> in order they would be processed out of order at times).
>
> Is this doable?
>
> Say a topic has 3 partitions. And there are 3 consumer processes in a
> consumer group each of which has a single thread.
> When we start the 3 processes, when the first process is up, it will
> consume all 3 partitions. When the second process is up,
> 1 process consumes 2 partitions and 1 consumes 1 partition. Only when the
> third process is up, will each process consume
> One partition. This will cause trouble. There seems to be no way to bound
> a stream to one partition.
>
>
> Regards,
>
> Libo
>
>

Re: More questions on avro serialization

[Mark]

Neha, thanks for the response. 

So the only point of the magic byte is to indicate that the rest of the message 
is Avro encoded? I noticed that in Camus a 4 byte int id of the schema is 
written instead of the 16 byte SHA. Is this the new preferred way? Which is 
compatible with https://issues.apache.org/jira/browse/AVRO-1124?

Thanks again

On Aug 21, 2013, at 8:38 PM, Neha Narkhede  wrote:

> We define the LinkedIn Kafka message to have a magic byte (indicating Avro
> serialization), MD5 header followed by the payload. The Hadoop consumer
> reads the MD5, looks up the schema in the repository and deserializes the
> message.
> 
> Thanks,
> Neha
> 
> 
> On Wed, Aug 21, 2013 at 8:15 PM, Mark  wrote:
> 
>> Does LinkedIn include the SHA of the schema into the header of each Avro
>> message they write or do they wrap the avro message and prepend the SHA?
>> 
>> In either case, how does the Hadoop consumer know what schema to read?

Re: Kafka Mirroring setup

[Maxime Brugidou]

We sort of have the same situation where our analytics DC is one of the
main producer DC too. If you use Kafka only for analytics it is fine to
produce directly to the analytics cluster from that DC and mirror the rest.

However we also want to be able to run things locally that will consume
local data from the local clusters for near real-time applications. This
can't be done in the central DC in this situation since all data will be
aggregated. The N+1 solution is more flexible if you need that.
On Aug 20, 2013 7:57 PM, "Andrew Otto"  wrote:

> In our case, our aggregator/analytics cluster is in our main datacenter,
> so there's no risk of the main producers becoming disconnected from it.  It
> seems nicer to have a dedicated aggregator cluster, that only only gets its
> data via MirrorMaker (Option A), but in our case this isn't necessary.
>
> The aggregator cluster could use MirrorMaker to consume from remote
> datacenters, but still have regular local producers send it data directly
> (Option B).
>
>
> On Aug 20, 2013, at 1:47 PM, Jay Kreps  wrote:
>
> > We do something like A (though I'm not sure I understand B):
> > http://kafka.apache.org/documentation.html#datacenters
> >
> > Essentially what we wanted was that each datacenter stood alone so that
> we
> > would not lose data if the datacenters became disconnected. Network
> > partitions within our data centers are extremely rare but between
> > datacenters relatively common.
> >
> > -Jay
> >
> >
> >
> >
> > On Tue, Aug 20, 2013 at 10:35 AM, Andrew Otto 
> wrote:
> >
> >> Hi all!
> >>
> >> Wikimedia is investigating how best to set up Broker clusters in
> multiple
> >> data centers.  Our main analytics Broker cluster is currently in our
> main
> >> datacenter.  It is possible for all of the main DC's frontend producers
> to
> >> produce directly to our analytics cluster, but we're not sure if this
> is a
> >> best practice.  So!  What does LinkedIn recommend?
> >>
> >> Option A: N + 1 clusters.
> >> - N production Broker Clusters (1 for each DC).
> >> - +1 aggregator/analytics Broker cluster that mirrors all of the
> >> production clusters.
> >>
> >> - Option B: N total Broker clusters.
> >> - Frontend producers in the main cluster produce directly to the
> >> aggregator/analytics cluster.
> >> - Other DC's clusters are mirrored to the aggregator/analytics cluster.
> >>
> >> Thanks!
> >> -Andrew
>
>