Re: 0.8.2 producer with 0.8.1.1 cluster?

I used the 0.8.2 producer in a 0.8.1 cluster in a nonproduction environment. No problems to report it worked great, but my testing at that time was not particularly extensive for failure scenarios. Christian On Wed, Nov 12, 2014 at 10:37 PM, Shlomi Hazan wrote: > I was asking to know if there's

Re: first message is lost

Got it， thanks Guozhang. On Thu Nov 13 2014 at 12:04:17 AM Guozhang Wang wrote: > Yonghui, > > If consumer is not set with --from-beginning, then this scenario is > expected: KAFKA-1006 > > We are still figuring what is the best way to resolve t

Re: 0.8.2 producer with 0.8.1.1 cluster?

I was asking to know if there's a point in trying... >From your answer I understand the answer is yes. 10x, Shlomi On Wed, Nov 12, 2014 at 7:04 PM, Guozhang Wang wrote: > Shlomi, > > It should be compatible, did you see any issues using it against a 0.8.1.1 > cluster? > > Guozhang > > On Wed, No

Re: Getting Simple consumer details using MBean

Hi Jun Rao, Thanks for your quick reply. I am not able to see this any bean named as "SimpleConsumer". Is there any configuration related to this? How can I see this bean named listing in Jconsole window? Thanks and Regards Madhukar On Thu, Nov 13, 2014 at 6:06 AM, Jun Rao wrote: > Those a

Re: Add partitions with replica assignment in same command

If you add partitions, existing partitions' assignment won't change. Changing the replica assignment subsequently should be fine. Thanks, Jun On Wed, Nov 12, 2014 at 2:24 PM, Allen Wang wrote: > I found this JIRA > > https://issues.apache.org/jira/browse/KAFKA-1656 > > Now, we have to use two

Re: Broker keeps rebalancing

Does this help? https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog ? On Wed, Nov 12, 2014 at 3:53 PM, Chen Wang wrote: > Hi there, > My kafka client is reading a 3 partition topic from kafka with 3 threads > distributed on different machines. I am

Re: Add partitions with replica assignment in same command

When we first add partitions, will it change the assignment of replicas for existing partitions? Nope. It should not touch the existing partitions. Also, will there be any issues executing the second reassignment command which will change the assignment again for the new partitions added? No. 2n

Re: Getting Simple consumer details using MBean

Those are for 0.7. In 0.8, you should see sth like FetchRequestRateAndTimeMs in SimpleConsumer. Thanks, Jun On Wed, Nov 12, 2014 at 5:14 AM, Madhukar Bharti wrote: > Hi, > > I want to get the simple consumer details using MBean as described here > < > https://cwiki.apache.org/confluence/displa

Broker keeps rebalancing

Hi there, My kafka client is reading a 3 partition topic from kafka with 3 threads distributed on different machines. I am seeing frequent owner changes on the topics when running: bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group my_test_group --topic mytopic -zkconnect localhost:21

Re: Add partitions with replica assignment in same command

I found this JIRA https://issues.apache.org/jira/browse/KAFKA-1656 Now, we have to use two commands to accomplish the goal - first add partitions using TopicCommand and then reassign replicas using ReassignPartitionsCommand. When we first add partitions, will it change the assignment of replicas

Kafka in a docker container stops with no errors logged

Hello, We're using Kafka 0.8.1.1 and we're trying to run it in a Docker container. For the most part, this has been fine, however one of the containers has stopped a couple times and when I look at the log output from Docker (E.g. Kafka STDOUT), I don't see any errors. At one point it states

Re: Programmatic Kafka version detection/extraction?

I haven't worked much with JMX before, but some quick googling (10-20 minutes) is very inconclusive as to how I would go about getting the server version I'm connecting to from a Python client. Can someone please reassure me that it's relatively trivial for non Java clients to query JMX for the se

Re: Programmatic Kafka version detection/extraction?

+1 on the JMX + gradle properties. Is there any (seamless) way of including the exact git hash? That would be extremely useful if users need help debugging and happen to be on an unreleased build (say, off trunk) On Wed, Nov 12, 2014 at 09:34:35AM -0800, Gwen Shapira wrote: > Actually, Jun suggest

Re: Programmatic Kafka version detection/extraction?

Right, this should should get exposed in JMX for: * Producer * Broker * Consumer Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Wed, Nov 12, 2014 at 12:34 PM, Gwen Shapira wrote: > Actually, Jun suggested ex

Re: Programmatic Kafka version detection/extraction?

Actually, Jun suggested exposing this via JMX. On Wed, Nov 12, 2014 at 9:31 AM, Gwen Shapira wrote: > Good question. > > The server will need to expose this in the protocol, so Kafka clients will > know what they are talking to. > > We may also want to expose this in the producer and consumer, s

Re: Programmatic Kafka version detection/extraction?

Good question. The server will need to expose this in the protocol, so Kafka clients will know what they are talking to. We may also want to expose this in the producer and consumer, so people who use Kafka's built-in clients will know which version they have in the environment. On Wed, Nov 12

Re: spikes in producer requests/sec

You can also look into the producer time metrics and see if the time goes up before the spike of the request rate. If the time does go up, you can look at the breakdown of the time. Any latency due to local I/O will be included in the local time. Thanks, Jun On Tue, Nov 11, 2014 at 10:50 AM, Wes

Re: Programmatic Kafka version detection/extraction?

Just to be clear: this is going to be exposed via some Api the clients can call at startup? > On Nov 12, 2014, at 08:59, Guozhang Wang wrote: > > Sounds great, +1 on this. > >> On Tue, Nov 11, 2014 at 1:36 PM, Gwen Shapira wrote: >> >> So it looks like we can use Gradle to add properties to

Re: 0.8.2 producer with 0.8.1.1 cluster?

Shlomi, It should be compatible, did you see any issues using it against a 0.8.1.1 cluster? Guozhang On Wed, Nov 12, 2014 at 5:43 AM, Shlomi Hazan wrote: > Hi, > Is the new producer 0.8.2 supposed to work with 0.8.1.1 cluster? > Shlomi > -- -- Guozhang

Re: Order of consumed messages in the same partition.

Hi Filippo, By saying "offset" do you mean the offset field inside the message or you keep track of the ordering from the producer end and found it not consistent as observed from the consumer end? Guozhang On Wed, Nov 12, 2014 at 8:39 AM, Filippo De Luca wrote: > Hi all, > I would like to kno

Re: Programmatic Kafka version detection/extraction?

Sounds great, +1 on this. On Tue, Nov 11, 2014 at 1:36 PM, Gwen Shapira wrote: > So it looks like we can use Gradle to add properties to manifest file and > then use getResourceAsStream to read the file and parse it. > > The Gradle part would be something like: > jar.manifest { > att

Order of consumed messages in the same partition.

Hi all, I would like to know if there is any possibility to consume one message for a given topic and partition, using the high-level consumer, that has the offset lower than the last consumed offset from the same topic and partition couple. It seems that I am experiencing this problem in mine app

Re: first message is lost

Yonghui, If consumer is not set with --from-beginning, then this scenario is expected: KAFKA-1006 We are still figuring what is the best way to resolve this issue. Guozhang On Wed, Nov 12, 2014 at 12:35 AM, Yonghui Zhao wrote: > Hi, > > For a

0.8.2 producer with 0.8.1.1 cluster?

Hi, Is the new producer 0.8.2 supposed to work with 0.8.1.1 cluster? Shlomi

Getting Simple consumer details using MBean

Hi, I want to get the simple consumer details using MBean as described here . But these bean names are not showing in JConsole as well as while trying to read from JMX. Please help me to get simple consumer detail

Re: Security in 0.8.2 beta

Thanks. We use the encryption approach as well. But the 2 topic approach is unique. Thank you. Kashyap. On Nov 12, 2014 1:54 AM, "Joe Stein" wrote: > I know a few implements that do this "encrypt your messages with a PSK > between producers and consumers". One of them actually writes the > "encr

first message is lost

Hi, For a non-existent topic, the consumer and producer are set up. Then if the producer sends the first message, producer gets this exception: [2014-11-12 16:24:28,041] WARN Error while fetching metadata [{TopicMetadata for topic test5 -> No partition metadata for topic test5 due to kafka.commo