Re: Reg : Unable to produce message

[Mohamed Ashiq]

Sorry I miss to mention the topics which I was trying are already existing
and worked before.
When I try to list topics and try to get topic information I was able to
get all the details including leader selected.
But if I try to produce I am getting this problem. And this happened to
few topics and most of the topics in the same cluster are working properly.

Regards,
Ashiq.

On 20/03/16, 11:58 AM, "Manikumar Reddy"  wrote:

>We may get few warning exceptions, on first produce to unknown topic ,
>with
>default server config property auto.create.topics.enable = true. If this
>is
>the case, then it is harmless exception.
>
>On Sun, Mar 20, 2016 at 11:19 AM, Mohamed Ashiq
>
>wrote:
>
>> All,
>>
>> I am getting this error for few topics
>> WARN Error while fetching metadata with correlation id 4802 :
>> {next=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
>> WARN Error while fetching metadata with correlation id 4803 :
>> {next=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
>>
>> I am using Kafka 0.9.0.0 and getting this error for few topics while
>> trying to produce messages. After googling I got answers to delete those
>> topics and to recreate.
>> But I can¹t take that option since I would like to have the data and the
>> state of available.
>>
>> We got this problem in our integration environment. We are very close to
>> moving to production. It would be great if someone can help to
>>understand
>> this problem better.
>>
>> Regards,
>> Ashiq.
>>
>>
>>

What happens if controlled shutdown can't complete within controlled.shutdown.max.retries attempts?

[James Cheng]

The broker has the following parameters related to controlled shutdown:

controlled.shutdown.enable  Enable controlled shutdown of the server
boolean truemedium
controlled.shutdown.max.retries Controlled shutdown can fail for multiple 
reasons. This determines the number of retries when such failure happens  
int 3   medium
controlled.shutdown.retry.backoff.msBefore each retry, the system needs 
time to recover from the state that caused the previous failure (Controller 
fail over, replica lag etc). This config determines the amount of time to wait 
before retrying. long5000medium


If the broker attempts controlled shutdown and then fails, and this happens 
controlled.shutdown.max.retries number of times, what happens? Does it go back 
into active service and continue participating in the cluster, as if controlled 
shutdown never happened? Or does it do an uncontrolled shutdown?

Thanks,
-James




This email and any attachments may contain confidential and privileged material 
for the sole use of the intended recipient. Any review, copying, or 
distribution of this email (or any attachments) by others is prohibited. If you 
are not the intended recipient, please contact the sender immediately and 
permanently delete this email and any attachments. No employee or agent of TiVo 
Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by 
email. Binding agreements with TiVo Inc. may only be made by a signed written 
agreement.

Help understanding what happened

[Scott Reynolds]

In a test in staging environment, we kill -9 the broker. It was started
back up by runit and started recovering. We are seeing errors like this:

WARN Found an corrupted index file,
/mnt/services/kafka/data/TOPIC-17/16763460.index, deleting and
rebuilding index... (kafka.log.Log)

The file is a multiple of 8 (10485760) and has entries.

So this leads me to believe that lastOffset <= baseOffset (
https://code.hq.twilio.com/data/kafka/blob/35003fd51b80d605d40339dead623012442a92ab/core/src/main/scala/kafka/log/OffsetIndex.scala#L354
)

Was wondering how that could happen ? Isn't baseOffset taken from the
filename and therefore is the FIRST entry in the log ? All other entries
should be greater then that.

RE: Reg : Unable to produce message

[Muthukumaran K]

Hi Ashiq, 

I am going through the same trouble with 0.9.0.0 and 0.9.0.1 for entirely new 
installation on remote VM. 

But from search across mailing lists and other sources, many seem to have had 
success when they have set the advertised.host.name with proper broker host 
name and using the same in the client. I was not able to get this solution 
working in my setup. May be you can try this. 

Regards
Muthu

-Original Message-
From: Mohamed Ashiq [mailto:mohamed_as...@trimble.com] 
Sent: Sunday, March 20, 2016 1:17 PM
To: users@kafka.apache.org
Subject: Re: Reg : Unable to produce message

Sorry I miss to mention the topics which I was trying are already existing and 
worked before.
When I try to list topics and try to get topic information I was able to get 
all the details including leader selected.
But if I try to produce I am getting this problem. And this happened to few 
topics and most of the topics in the same cluster are working properly.

Regards,
Ashiq.

On 20/03/16, 11:58 AM, "Manikumar Reddy"  wrote:

>We may get few warning exceptions, on first produce to unknown topic , 
>with default server config property auto.create.topics.enable = true. 
>If this is the case, then it is harmless exception.
>
>On Sun, Mar 20, 2016 at 11:19 AM, Mohamed Ashiq 
>
>wrote:
>
>> All,
>>
>> I am getting this error for few topics WARN Error while fetching 
>> metadata with correlation id 4802 :
>> {next=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
>> WARN Error while fetching metadata with correlation id 4803 :
>> {next=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
>>
>> I am using Kafka 0.9.0.0 and getting this error for few topics while 
>> trying to produce messages. After googling I got answers to delete 
>> those topics and to recreate.
>> But I can¹t take that option since I would like to have the data and 
>> the state of available.
>>
>> We got this problem in our integration environment. We are very close 
>>to  moving to production. It would be great if someone can help to 
>>understand  this problem better.
>>
>> Regards,
>> Ashiq.
>>
>>
>>

Re: Larger Size Error Message

[Fang Wong]

Hi Guozhang,

The problem is that server "10.225.36.226" is not one of my kafka clients,
nslookup shows it is another internal server, my servers are like
10.224.146.6 #, I can't even login to that
server. All of my messages are at most a few KB.

Is it possible anybody within the internal network can send any message to
kafka? How do I allow a list of fixed servers can send a request to kafka
server?

Thanks,
Fang


On Tue, Mar 15, 2016 at 5:31 PM, Guozhang Wang  wrote:

> Fang,
>
> From the logs you showed above there is a single produce request with very
> large request size:
>
> "[2016-03-14 06:43:03,579] INFO Closing socket connection to
> /10.225.36.226 due to invalid request: Request of length *808124929* is
> not valid, it is larger than the maximum size of 104857600 bytes.
> (kafka.network.Processor)"
>
> Which is about 770MB while the maximum request size is configured as 100MB.
> It is from the client hosted at "10.225.36.226", if you can go to that
> server and checks the producer logs around that time, maybe you can
> discover why there comes a single big produce request.
>
> Guozhang
>
>
> On Mon, Mar 14, 2016 at 1:59 PM, Fang Wong  wrote:
>
> > After changing log level from INFO to TRACE, here is kafka server.log:
> >
> > [2016-03-14 06:43:03,568] TRACE 156 bytes written.
> > (kafka.network.BoundedByteBufferSend)
> >
> > [2016-03-14 06:43:03,575] TRACE 68 bytes read.
> > (kafka.network.BoundedByteBufferReceive)
> >
> > [2016-03-14 06:43:03,575] TRACE [ReplicaFetcherThread-0-7], Follower 8
> > has replica log end offset 0 for partition [__consumer_offsets,20].
> > Received 0 messages and leader hw 0
> > (kafka.server.ReplicaFetcherThread)
> >
> > [2016-03-14 06:43:03,575] TRACE [ReplicaFetcherThread-0-7], Follower 8
> > has replica log end offset 0 after appending 0 bytes of messages for
> > partition [__consumer_offsets,20] (kafka.server.ReplicaFetcherThread)
> >
> > [2016-03-14 06:43:03,575] TRACE Setting high watermark for replica 8
> > partition [__consumer_offsets,20] on broker 8 to [0 [-1 : -1]]
> > (kafka.cluster.Replica)
> >
> > [2016-03-14 06:43:03,575] TRACE [ReplicaFetcherThread-0-7], Follower 8
> > set replica high watermark for partition [__consumer_offsets,20] to 0
> > (kafka.server.ReplicaFetcherThread)
> >
> > [2016-03-14 06:43:03,575] TRACE [ReplicaFetcherThread-0-7], Follower 8
> > has replica log end offset 0 for partition [__consumer_offsets,12].
> > Received 0 messages and leader hw 0
> > (kafka.server.ReplicaFetcherThread)
> >
> > [2016-03-14 06:43:03,575] TRACE [ReplicaFetcherThread-0-7], Follower 8
> > has replica log end offset 0 after appending 0 bytes of messages for
> > partition [__consumer_offsets,12] (kafka.server.ReplicaFetcherThread)
> >
> > [2016-03-14 06:43:03,575] TRACE Setting high watermark for replica 8
> > partition [__consumer_offsets,12] on broker 8 to [0 [-1 : -1]]
> > (kafka.cluster.Replica)
> >
> > [2016-03-14 06:43:03,575] TRACE [ReplicaFetcherThread-0-7], Follower 8
> > set replica high watermark for partition [__consumer_offsets,12] to 0
> > (kafka.server.ReplicaFetcherThread)
> >
> > [2016-03-14 06:43:03,575] TRACE [ReplicaFetcherThread-0-7], Issuing to
> > broker 7 of fetch request Name: FetchRequest; Version: 0;
> > CorrelationId: 397497; ClientId: ReplicaFetcherThread-0-7; ReplicaId:
> > 8; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo:
> > [__consumer_offsets,20] ->
> > PartitionFetchInfo(0,1048576),[__consumer_offsets,12] ->
> > PartitionFetchInfo(0,1048576) (kafka.server.ReplicaFetcherThread)
> >
> > [2016-03-14 06:43:03,575] TRACE 110 bytes written.
> > (kafka.network.BoundedByteBufferSend)
> >
> > [2016-03-14 06:43:03,579] DEBUG Accepted connection from
> > /10.225.36.226 on /10.224.146.63:9092. sendBufferSize
> > [actual|requested]: [102400|102400] recvBufferSize [actual|requested]:
> > [102400|102400] (kafka.network.Acceptor)
> >
> > [2016-03-14 06:43:03,579] TRACE Processor id 2 selection time =
> > 145604848 ns (kafka.network.Processor)
> >
> > [2016-03-14 06:43:03,579] DEBUG Processor 2 listening to new
> > connection from /10.225.36.226:36151 (kafka.network.Processor)
> >
> > [2016-03-14 06:43:03,579] TRACE Processor id 2 selection time = 3588
> > ns (kafka.network.Processor)
> >
> > [2016-03-14 06:43:03,579] INFO Closing socket connection to
> > /10.225.36.226 due to invalid request: Request of length 808124929 is
> > not valid, it is larger than the maximum size of 104857600 bytes.
> > (kafka.network.Processor)
> >
> > [2016-03-14 06:43:03,580] DEBUG Closing connection from
> > /10.225.36.226:36151 (kafka.network.Processor)
> >
> >
> >
> > Here is kafka-request.log:
> >
> >
> > [2016-03-14 06:43:03,463] TRACE Completed request:Name: FetchRequest;
> > Version: 0; CorrelationId: 399486; ClientId: ReplicaFetcherThread-0-8;
> > ReplicaId: 6; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo:
> > [__consumer_offsets,5] -> PartitionFetchInfo(0,1048576) from client
> > /10.224.146.61:10716
> >
> ;totalTime:5

Consumer 0.9 /Broker 0.9.0.1 - for certain partitions, the commit offset in kafka alway gets rewinded to a fixed point

[Saurabh Daftary]

I am seeing a very wired issue with my consumer group when running consumer
0.9 with broker 0.9.0.1. On adding or removing consumers, it seems that for
certain partitions the commit offset in kafka alway gets rewinded to a
fixed point.

When I run the below command over time to look at my consumer group:

bin/kafka-consumer-groups.sh --describe --group consumer-gp-1 --new-consumer
--bootstrap-server localhost:9092

I notice that for certain partitions the consumer offsets gets always reset
back to a fixed point. See an example below(Each line represents the result
of running the above command at different times (~1 min apart) for a
particular partition):

consumer-gp-1, topic-1, 54, 6682915, 8264593, 1581678, consumer-5_/
172.18.5.49
consumer-gp-1, topic-1, 54, 6852270, 8265027, 1412757, consumer-5_/
172.18.5.49
consumer-gp-1, topic-1, 54, 6682915, 8265252, 1582337, consumer-5_/
172.18.5.49
consumer-gp-1, topic-1, 54, 6865345, 8265457, 1400112, consumer-5_/
172.18.5.49
consumer-gp-1, 54, 6682915, 8266152, 1583237, consumer-5_/172.18.5.49

In the above case, for partition 54 - the consumer offset will move ahead
but then always come back again to 6682915

Any inputs on how could I go about debugging this?

Re: Consumer 0.9 /Broker 0.9.0.1 - for certain partitions, the commit offset in kafka alway gets rewinded to a fixed point

[Ismael Juma]

Hi Saurabh,

When you say Consumer 0.9, do you mean 0.9.0.0? If so, you should really
upgrade to 0.9.0.1 as it contains a number of important fixes.

Ismael

On Sun, Mar 20, 2016 at 6:31 PM, Saurabh Daftary 
wrote:

> I am seeing a very wired issue with my consumer group when running consumer
> 0.9 with broker 0.9.0.1. On adding or removing consumers, it seems that for
> certain partitions the commit offset in kafka alway gets rewinded to a
> fixed point.
>
> When I run the below command over time to look at my consumer group:
>
> bin/kafka-consumer-groups.sh --describe --group consumer-gp-1
> --new-consumer
> --bootstrap-server localhost:9092
>
> I notice that for certain partitions the consumer offsets gets always reset
> back to a fixed point. See an example below(Each line represents the result
> of running the above command at different times (~1 min apart) for a
> particular partition):
>
> consumer-gp-1, topic-1, 54, 6682915, 8264593, 1581678, consumer-5_/
> 172.18.5.49
> consumer-gp-1, topic-1, 54, 6852270, 8265027, 1412757, consumer-5_/
> 172.18.5.49
> consumer-gp-1, topic-1, 54, 6682915, 8265252, 1582337, consumer-5_/
> 172.18.5.49
> consumer-gp-1, topic-1, 54, 6865345, 8265457, 1400112, consumer-5_/
> 172.18.5.49
> consumer-gp-1, 54, 6682915, 8266152, 1583237, consumer-5_/172.18.5.49
>
> In the above case, for partition 54 - the consumer offset will move ahead
> but then always come back again to 6682915
>
> Any inputs on how could I go about debugging this?
>

Re: Consumer 0.9 /Broker 0.9.0.1 - for certain partitions, the commit offset in kafka alway gets rewinded to a fixed point

[Saurabh Daftary]

Yes, I do mean 0.9.0.0. I am planning to upgrade to 0.9.0.1 but was
wondering if this was a known issue in 0.9.0.0

On Sun, Mar 20, 2016 at 11:31 AM, Saurabh Daftary  wrote:

> I am seeing a very wired issue with my consumer group when running
> consumer 0.9 with broker 0.9.0.1. On adding or removing consumers, it
> seems that for certain partitions the commit offset in kafka alway gets
> rewinded to a fixed point.
>
> When I run the below command over time to look at my consumer group:
>
> bin/kafka-consumer-groups.sh --describe --group consumer-gp-1 --new-consumer
> --bootstrap-server localhost:9092
>
> I notice that for certain partitions the consumer offsets gets always
> reset back to a fixed point. See an example below(Each line represents the
> result of running the above command at different times (~1 min apart) for a
> particular partition):
>
> consumer-gp-1, topic-1, 54, 6682915, 8264593, 1581678, consumer-5_/
> 172.18.5.49
> consumer-gp-1, topic-1, 54, 6852270, 8265027, 1412757, consumer-5_/
> 172.18.5.49
> consumer-gp-1, topic-1, 54, 6682915, 8265252, 1582337, consumer-5_/
> 172.18.5.49
> consumer-gp-1, topic-1, 54, 6865345, 8265457, 1400112, consumer-5_/
> 172.18.5.49
> consumer-gp-1, 54, 6682915, 8266152, 1583237, consumer-5_/172.18.5.49
>
> In the above case, for partition 54 - the consumer offset will move ahead
> but then always come back again to 6682915
>
> Any inputs on how could I go about debugging this?
>

Re: Consumer 0.9 /Broker 0.9.0.1 - for certain partitions, the commit offset in kafka alway gets rewinded to a fixed point

[Ismael Juma]

See here for important issues fixed in 0.9.0.1:

http://www.confluent.io/blog/announcing-apache-kafka-0.9.0.1-and-confluent-platform-2.0.1

Upgrading is highly recommended.

Ismael

On Sun, Mar 20, 2016 at 11:20 PM, Saurabh Daftary  wrote:

> Yes, I do mean 0.9.0.0. I am planning to upgrade to 0.9.0.1 but was
> wondering if this was a known issue in 0.9.0.0
>
> On Sun, Mar 20, 2016 at 11:31 AM, Saurabh Daftary <
> saurabh.daft...@gmail.com
> > wrote:
>
> > I am seeing a very wired issue with my consumer group when running
> > consumer 0.9 with broker 0.9.0.1. On adding or removing consumers, it
> > seems that for certain partitions the commit offset in kafka alway gets
> > rewinded to a fixed point.
> >
> > When I run the below command over time to look at my consumer group:
> >
> > bin/kafka-consumer-groups.sh --describe --group consumer-gp-1
> --new-consumer
> > --bootstrap-server localhost:9092
> >
> > I notice that for certain partitions the consumer offsets gets always
> > reset back to a fixed point. See an example below(Each line represents
> the
> > result of running the above command at different times (~1 min apart)
> for a
> > particular partition):
> >
> > consumer-gp-1, topic-1, 54, 6682915, 8264593, 1581678, consumer-5_/
> > 172.18.5.49
> > consumer-gp-1, topic-1, 54, 6852270, 8265027, 1412757, consumer-5_/
> > 172.18.5.49
> > consumer-gp-1, topic-1, 54, 6682915, 8265252, 1582337, consumer-5_/
> > 172.18.5.49
> > consumer-gp-1, topic-1, 54, 6865345, 8265457, 1400112, consumer-5_/
> > 172.18.5.49
> > consumer-gp-1, 54, 6682915, 8266152, 1583237, consumer-5_/172.18.5.49
> >
> > In the above case, for partition 54 - the consumer offset will move ahead
> > but then always come back again to 6682915
> >
> > Any inputs on how could I go about debugging this?
> >
>

Re: For SSL config, Any way to avoid the cleartext passwords?

[Adam Kunicki]

One option is that your application could read the password from an access
restricted file (e.g. owner read/write only) or retrieve it from a
credentials server (e.g. hadoop kms, hashicorp vault)

For what its worth, java keystore passwords are pretty useless anyway and
keystores can be read without even knowing it as demonstrated in this code
snippet:

https://gist.github.com/zach-klippenstein/4631307


On Sun, Mar 20, 2016 at 8:18 PM, Linyuxin  wrote:

> Hi All,
> Kafka 0.9.0 support ssl.
> And in the document, password in ssl config is cleartext passwords.
> e.g.
>   ssl.keystore.location=/var/private/ssl/kafka.server.keystore.jks
> ssl.keystore.password=test1234
> ssl.key.password=test1234
>
> ssl.truststore.location=/var/private/ssl/kafka.server.truststore.jks
> ssl.truststore.password=test1234
> any way to avoid this "test1234" cleartext in the file?
> Like some encryption?
>



-- 
Adam Kunicki
StreamSets | Field Engineer
mobile: 415.890.DATA (3282) | linkedin 

two questions

[allen chan]

1) I am using the upgrade instructions to upgrade from 0.8 to 0.9. Can
someone tell me if i need to continue to bump the
inter.broker.protocol.version after each upgrade? Currently the broker code
is 0.9.0.1 but i have the config file listing as inter.broker.protocol.versi
on=0.9.0.0
2) Is it possible to use multiple variations of producers / consumers?
My broker is on 0.9.0.1 and i am currently using 0.8.x producer/consumer. I
want to test the new producer first then the new consumer. So would there
be issues if the setup was:
0.9.x producer -> 0.9.x broker -> 0.8.x consumer

Thanks
-- 
Allen Michael Chan

Would Kafka streams be a good choice for a collaborative web app?

[Mark van Leeuwen]

Hi,

I'm soon to begin design and dev of a collaborative web app where 
changes made by one user should appear to other users in near real time.


I'm new to Kafka, but having read a bit about Kafka streams I'm 
wondering if it would be a good solution. Change events produced by one 
user would be published to multiple consumer clients over a websocket, 
each having their own offset.


Would this be viable?

Are there any considerations I should be aware of?

Thanks,
Mark