Re: Kafka Committed Offset Behavior off by 1

Done: https://issues.apache.org/jira/browse/KAFKA-3191 Feel free to assign it to me (wasn't able to do that myself) On Mon, Feb 1,

Trying to figure out the protocol.

I am trying to write a Kafka client (specifically a consumer) and am using the protocol document at https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol . Had a specific question on the Offset Request API: OffsetRequest => ReplicaId [TopicName [Partition Time MaxNumberO

Re: Kafka Committed Offset Behavior off by 1

This is the second time I see this complaint, so we could probably make the API docs clearer. Adam, feel like submitting a JIRA? On Mon, Feb 1, 2016 at 3:34 PM, Adam Kunicki wrote: > Thanks, actually found this out per: > > http://www.confluent.io/blog/tutorial-getting-started-with-the-new-apac

Re: How to bind all Kafka tcp port to private net address

yes, the host.name is useless in this case. Even if I set the host.name=private ip, the broker also bind on 0.0.0.0 2016-02-01 23:27 GMT+08:00 John Prout : > I have set the host.name option in the server.properties file, but the > Broker is still binding to all interfaces, and logging that's what

Re: Kafka Committed Offset Behavior off by 1

Thanks, actually found this out per: http://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0.9-consumer-client

Re: Message lost after consumer crash in kafka 0.9

One thing to add, is that by doing this you could possibly get duplicates but not data loss, which obeys Kafka's at-least once semantics. Guozhang On Mon, Feb 1, 2016 at 3:17 PM, Guozhang Wang wrote: > Hi Han, > > I looked at your test code and actually the error is in this line: > https://gist

Re: Message lost after consumer crash in kafka 0.9

Hi Han, I looked at your test code and actually the error is in this line: https://gist.github.com/darkjh/437ac72cdd4b1c4ca2e7#file-kafkabug2-scala-L61 where you call "commitSync" in the finally block, which will commit messages that is returned to you from poll() call. More specifically, for e

Re: [produce response] error code

Never mind, Found the documentation, it was crc32 error - I fixed it. On Mon, Feb 1, 2016 at 9:24 PM john pradeep wrote: > Hi, > I am writing a kafka client, I tried to send a produce request and I get > back an error code on the partition as "*2*". but i couldn't find any > documentation on th

Re: Kafka Committed Offset Behavior off by 1

The committed offset is actually the next message to consume, not the last message consumed. So that sounds like expected behavior to me. The consumer code handles this internally, but if you write code to commit offsets manually, it can be a gotcha. -Dana On Mon, Feb 1, 2016 at 1:35 PM, Adam Kun

Kafka Committed Offset Behavior off by 1

Hi, I've been noticing that a restarted consumer in 0.9 will start consuming from the last committed offset (inclusive). This means that any restarted consumer will get the last read (and committed) message causing a duplicate each time the consumer is restarted from the same position if there hav

[produce response] error code

Hi, I am writing a kafka client, I tried to send a produce request and I get back an error code on the partition as "*2*". but i couldn't find any documentation on the description of this error code, any help? Thanks, John

Re: Kafka SSL Configuration Problems

Aha , got it. So thats where I got confused. > On Feb 1, 2016, at 3:04 PM, Ismael Juma wrote: > > Hi Nazario, > > The problem in the original post is that you were setting > advertised.host.name, which means that advertised.listeners won't fall back > to listeners anymore. Yes, it's bit con

Re: Kafka SSL Configuration Problems

Hi Nazario, The problem in the original post is that you were setting advertised.host.name, which means that advertised.listeners won't fall back to listeners anymore. Yes, it's bit confusing given how the configs evolved over time. I have configured several clusters to use SSL by setting listen

Re: Kafka SSL Configuration Problems

On Mon, Feb 1, 2016 at 7:15 PM, Nazario Parsacala wrote: > So it looks like you need both listeners and advertised.listeners ..? > No, you always need to set `listeners` (`advertised.listeners` defaults to `listeners`). If you want `advertised.listeners` to be different than `listeners`, then yo

Re: Kafka SSL Configuration Problems

I dont think that is the behavior I have seen. If I set listeners only ( as per my original post) , SSL will never get registered. [2016-02-01 11:27:49,712] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(servername,9092,PLAINTEXT) (kafka.utils.ZkUtils)

Get messages between offsets? Or determine # of messages between offsets?

What's the best way to consume messages from a starting offset to an ending offset? It seems the low level consumer (API), kafka-simple-consumer-shell.sh, and kafkacat offer an option to consume from a starting offset, but there's no option to tell it to stop consuming after a given offset. One has

Re: Kafka SSL Configuration Problems

So it looks like you need both listeners and advertised.listeners ..? When I set both configs .. It finally worked. Maybe we can update the docs ..? > On Feb 1, 2016, at 1:59 PM, Nazario Parsacala wrote: > > So I made the port 9092 but SSL. But it seems like it is just openning it for > PL

Re: Kafka SSL Configuration Problems

So I made the port 9092 but SSL. But it seems like it is just openning it for PLAINTEXT. Even though it has registered it as SSL [2016-02-01 13:42:20,536] INFO Registered broker 0 at path /brokers/ids/0 with addresses: SSL -> EndPoint(reactor.us.cixsoft.net,9092,SSL) (kafka.utils.ZkUtils) ope

Re: Kafka SSL Configuration Problems

Hmm. So I removed port 9092 and just use port 9093. So no PLAINTEXT just SSL advertised.listeners=SSL://reactor.us.cixsoft.net:9093 Cleared Zookeeper and Kafka store and restart .. You see that it is registering 9093 onbly [2016-02-01 13:35:51,729] INFO Registered broker 0 at path /brokers/ids/0

Re: Kafka SSL Configuration Problems

Ok, This is getting interesting .. On the broker side, it is saying that it is registering 9092 as PLAINTEXT and 9093 as SSL [2016-02-01 13:26:33,796] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(servername,9092,PLAINTEXT),SSL -> EndPoint(servername,909

Re: Kafka SSL Configuration Problems

No juice. /kafka-topics.sh --describe --topic anotherone --zookeeper localhost:2181 Topic:anotheronePartitionCount:4ReplicationFactor:1 Configs: Topic: anotherone Partition: 0Leader: 0 Replicas: 0 Isr: 0 Topic: anotherone Partition: 1

Fwd: Re: Kafka SSL Configuration Problems

Hello Nazario, Could you try it by creating a new topic? Thank you, Anirudh That works. At least it is saying that it is registering now with the SSL side. [2016-02-01 12:29:40,184] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(servername,9092,PLAINTEXT),

Message lost after consumer crash in kafka 0.9

Hi, One of our usage of kafka is to tolerate arbitrary consumer crash without losing or duplicating messages. So in our code we manually commit offset after successfully persisted the consumer state. In prototyping with kafka-0.9's new consumer API, I found that in some cases, kafka failed to sen

Re: Kafka SSL Configuration Problems

That works. At least it is saying that it is registering now with the SSL side. [2016-02-01 12:29:40,184] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(servername,9092,PLAINTEXT),SSL -> EndPoint(servername,9093,SSL) (kafka.utils.ZkUtils) Thank you. Now

Re: Kafka SSL Configuration Problems

Please use advertised.listeners instead of advertised.host.name. See this comment: https://github.com/apache/kafka/pull/793#issuecomment-174287124 Ismael On Mon, Feb 1, 2016 at 4:44 PM, Nazario Parsacala wrote: > Hi, > > We were using kafka for a while now. We have been using the binary releas

Kafka SSL Configuration Problems

Hi, We were using kafka for a while now. We have been using the binary release 2.10-0.8.2.1 . But we have been needing a encrypted communication between our publishers and subscribers. So we got 2.10-0.9.0.0. This works very well with no SSL enabled. But currently have issues with SSL enabled.

RE: How to bind all Kafka tcp port to private net address

I have set the host.name option in the server.properties file, but the Broker is still binding to all interfaces, and logging that's what it is doing. This is with kafka 0.9.0 running on a Solaris 10 server with 3 Virtual interfaces installed, in addition to the Physical interface. John -O