Re: Failed to produce message from the second producer

[Jun Rao]

Any error in kafka controller and state-change logs?

Thanks,

Jun


On Mon, Nov 11, 2013 at 10:39 PM, ji yan  wrote:

> @Priya, I have two brokers, each running on each of two machines. The
> producer broker-list is pointing to the broker server on the same machine
> as the producer.
>
> @Jun, I created the topic using kafka-create-topic.sh, the error is
> consistently reproduceble
>
>
> On Mon, Nov 11, 2013 at 9:51 AM, Jun Rao  wrote:
>
> > Are those ERROR transient? If so, this is normal when the topic didn't
> > exist and was auto created.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, Nov 11, 2013 at 12:12 AM, ji yan  wrote:
> >
> > > Hi Kafka Users
> > >
> > > I have a test setup at home with one machine hosting a zookeeper server
> > and
> > > a broker server, while the second machine hosting a broker server.
> > >
> > > On the first machine, producing and consuming messages are ok. But on
> the
> > > second one, producing message throws this error in the log.
> > >
> > > The topic test has two partitions and two replicas.
> > >
> > > [2013-11-11 00:07:24,974] WARN Error while fetching metadata
> > > [{TopicMetadata for topic test ->
> > > No partition metadata for topic test due to
> > > kafka.common.LeaderNotAvailableException}] for topic [test]: class
> > > kafka.common.LeaderNotAvailableException
> > >  (kafka.producer.BrokerPartitionInfo)
> > > [2013-11-11 00:07:24,975] ERROR Failed to collate messages by topic,
> > > partition due to: Failed to fetch topic metadata for topic: test
> > > (kafka.producer.async.DefaultEventHandler)
> > >
> > > However, when displaying list of topics, I do see a leader for the
> topic
> > >
> > > topic: test partition: 0 leader: 0 replicas: 0,1 isr: 0
> > > topic: test partition: 1 leader: 0 replicas: 1,0 isr: 0
> > >
> > > Thanks
> > >
> >
>
>
>
> --
> Do not wait to strike till the iron is hot; but make it hot by striking
>
> LinkedIn Profile: http://www.linkedin.com/pub/ji-yan/a/a57/79
>

pom warning

[hsy...@gmail.com]

Hi guys,

When I built my project using maven I got WARNING

[WARNING] The POM for org.apache.kafka:kafka_2.8.0:jar:0.8.0-beta1 is
invalid, transitive dependencies (if any) will not be available: 1 problem
was encountered while building the effective model

And I looked at the kafka_2.8.0-0.8.0-beta1.pom file
I found 2  tags













  


Is it supposed to under one single  ?

Best
Siyuan

Re: pom warning

[Joe Stein]

Hi Siyuan, we have a fix for this in 0.8.0 which is in progress being
released.

If you find the pom breaking for any reason (some build systems have
problems with the bad pom) you can use the direct apache repository
https://repository.apache.org/content/groups/public/ which has a fixed pom
for 0.8.0-beta1 which couldn't get pushed to maven central since it is
immutable.

Thanks!

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/


On Tue, Nov 12, 2013 at 2:56 PM, hsy...@gmail.com  wrote:

> Hi guys,
>
> When I built my project using maven I got WARNING
>
> [WARNING] The POM for org.apache.kafka:kafka_2.8.0:jar:0.8.0-beta1 is
> invalid, transitive dependencies (if any) will not be available: 1 problem
> was encountered while building the effective model
>
> And I looked at the kafka_2.8.0-0.8.0-beta1.pom file
> I found 2  tags
>
> 
> 
> 
> 
> 
> 
>  rev="3.3.4">
> 
> 
> 
> 
> 
>   
> 
>
> Is it supposed to under one single  ?
>
> Best
> Siyuan
>

High level consumer Blocked when there is still message in topic

[李帅]

Hi,

   I use Kafka 0.8 high level consumer reads message from topic
stream, 3 replica and 10 paritions.

   When I use 10 threads read the stream and runing for some time (one
hour or one day),

some threads block at m_stream.iterator().hasNext(), but the parition
still has lots of messages.

   I check consumer's fetch.message.max.bytes and broker's
message.max.bytes, there is no

message size bigger than these values.

   The consumer configure is
   props.put("zookeeper.session.timeout.ms", "4000");
   props.put("zookeeper.sync.time.ms", "200");
   props.put("auto.commit.interval.ms", "1000");


   Please give me some option about how to avoid consumer block.

   Is there some configure parameter can fix this problem.

Thanks!

XiaoTian

Re: High level consumer Blocked when there is still message in topic

[Jun Rao]

What's the max lag (reported in JMX) in the consumer? Can the consumer keep
up with the incoming data rate?

Thanks,

Jun


On Tue, Nov 12, 2013 at 7:19 PM, 李帅  wrote:

> Hi,
>
>I use Kafka 0.8 high level consumer reads message from topic
> stream, 3 replica and 10 paritions.
>
>When I use 10 threads read the stream and runing for some time (one
> hour or one day),
>
> some threads block at m_stream.iterator().hasNext(), but the parition
> still has lots of messages.
>
>I check consumer's fetch.message.max.bytes and broker's
> message.max.bytes, there is no
>
> message size bigger than these values.
>
>The consumer configure is
>props.put("zookeeper.session.timeout.ms", "4000");
>props.put("zookeeper.sync.time.ms", "200");
>props.put("auto.commit.interval.ms", "1000");
>
>
>Please give me some option about how to avoid consumer block.
>
>Is there some configure parameter can fix this problem.
>
> Thanks!
>
> XiaoTian
>