Kafka basic doubts

2016-02-22 Thread Pariksheet Barapatre
Hi All, Greetings..!!! This is my first email to Kafka Community. I have just started exploring Kafka on CDH5.5 cluster which ships with Kafka 0.8.2.1. I am able to run sample programs for producer as well as consumer (both high level and low level). Now I am trying to load messages from Kafka

Re: Consumer seek on 0.9.0 API

2016-02-22 Thread Péricé Robin
OK I understand the explanation. Thanks you for sharing your knowledge ! Regards, Robin 2016-02-18 18:56 GMT+01:00 Jason Gustafson : > Woops. Looks like Alex got there first. Glad you were able to figure it > out. > > -Jason > > On Thu, Feb 18, 2016 at 9:55 AM, Jason Gustafson > wrote: > > > H

Re: Kafka 0.9.0.0 on Solaris 5.10 not running

2016-02-22 Thread Marcus Gründler
Hi All, the same happens for Kafka 0.9.0.1. The controller is unable to setup a connection on the local machine. I tried to set the following configurations in config/server.properties host.name=localhost advertised.host.name=localhost controller.socket.timeout.ms=245000 without any luck. The

Does pausing a topic allow other consumers to consume it?

2016-02-22 Thread craig w
Using kafka 0.9.0.0, if there are multiple consumers in a group and one consumer pauses the topic+partition it's consuming, does that allow/cause other consumers in the group to pick it up?

How to recover kafka topic dead partition not served

2016-02-22 Thread Yinping YP12 Wang
kafka.apache.org: The kafka version I use is kafka_2.10-0.8.2.0,I create topic with 3 replication in kafka cluster. When the kafka cluster runs a long time,some topic partitions become dead,so can't work.Restart kafka can't solve recover topic partition.The solution I use is delete all kafka d

Re: Kafka 0.9.0.0 on Solaris 5.10 not running

2016-02-22 Thread Ismael Juma
Hi Marcus, One thing that changed in 0.9.0.0 is that we now use `NetworkClient` for inter-broker communication. `NetworkClient` is also used by the new Java producer and consumer. Does the Java producer in 0.8.2 work fine for you? Ismael On Mon, Feb 22, 2016 at 3:06 PM, Marcus Gründler wrote:

kafka-streams in memory state store state directory usage

2016-02-22 Thread Tom Dearman
Our use case for kafka-streams only needs to use the in memory state store but the current code on trunk still requires a valid ’state.dir’ that the client can write too. A lock file per partition and an offset file is created but the offset file always contains zero values for offsets since al

Kafka Security quality level in 0.9.0.1

2016-02-22 Thread Anatoliy Sokolenko
Hi, Does recently released Kafka 0.9.0.1 have final release of security features, initiated in 0.9.0.0 or it is still should be considered beta quality? I’m asking because Cloudera recently send an announcement of their parcel release that promotes Kafka security features and does not mention th

Re: Does kafka.common.QueueFullException indicate back pressure in Kafka?

2016-02-22 Thread Alex Loddengaard
Hi John, I'm glad the info was helpful. It's hard to diagnose this issue without monitoring. I suggest setting up graphite to graph JMX metrics. There's a good (not designed for production) script here (as part of a Vagrant VM): https://github.com/gwenshap/ops_training_vm/blob/master/bootstrap.sh

Re: Does pausing a topic allow other consumers to consume it?

2016-02-22 Thread Jason Gustafson
Nope. Pausing a partition just stops the consumer from sending any more fetches for it. It will not trigger a partition reassignment. One thing to be wary of, however, is that the partition will automatically be unpaused after the next normal rebalance. -Jason On Mon, Feb 22, 2016 at 7:19 AM, cra

Re: Does pausing a topic allow other consumers to consume it?

2016-02-22 Thread craig w
so if I had a bunch of consumers coming online while some consumers had partitions paused, those consumers that were paused would become unpaused? On Mon, Feb 22, 2016 at 2:02 PM, Jason Gustafson wrote: > Nope. Pausing a partition just stops the consumer from sending any more > fetches for it. I

Re: Does pausing a topic allow other consumers to consume it?

2016-02-22 Thread Jason Gustafson
That is correct. The typical way to handle this (if you need to) is to repause the partitions in the ConsumerRebalanceListener onPartitionsAssigned() hook. -Jason On Mon, Feb 22, 2016 at 11:03 AM, craig w wrote: > so if I had a bunch of consumers coming online while some consumers had > partiti

Re: Does pausing a topic allow other consumers to consume it?

2016-02-22 Thread craig w
I think I'll need to do that. My current setup is where I have 200 partitions for a topic with 200 consumers, all part of the same group. Each consumer does a "poll". Upon receiving messages it will pause the topic/paritions it received, while the messages are processed, as each message is process

Re: Kafka 0.9.0.0 on Solaris 5.10 not running

2016-02-22 Thread Marcus Gründler
Hi Ismael, Yes, producers (and consumers) work just fine on 0.8.2. We see the exceptions even without any producer/consumers. The problem can be reproduced easily - I just start the following scripts an wait for 30 seconds: bin/kafka-zookeeper-start.sh path/to/zkconf bin/kafka-server-start.sh

Re: kafka-streams in memory state store state directory usage

2016-02-22 Thread Guozhang Wang
Hi Tom, That is right that currently for in-memory state store the directory is not actually used, i.e. the directory will be created but no data will be written. But the "state.dir" is not a required config in the sense that if user do not specify a value, it will default to "/tmp/kafka-streams".

Re: Exactly-once publication behaviour

2016-02-22 Thread Mark Harrison
On 2/21/16 10:23 AM, Jay Kreps wrote: Hey Andrew, Yeah I think the current state is that we did several design and prototypes (both the transaction work and the idempotence design and the conditional write KIP), but none of these offshoots is really fully rationalized with the other ones. Slow p

Re: Partitions not assigned on topic auto creation

2016-02-22 Thread Venkatesan Subramanian
Jason: Thanks for your reply. I did turn on logging for kafka as well when I ran the test. I do not see any partition assignment for the topic happening. If you don’t see anything obviously wrong in what I am doing, I can file a ticket with Confluent and attach the logs to it. Regards Venkat

Re: Partitions not assigned on topic auto creation

2016-02-22 Thread Jason Gustafson
Hey Venkat, The logs will help, but I'll probably need to see some code to understand what is going on (consumer usage can be a little subtle sometimes). You're welcome to open a ticket if you like or post here. Thanks, Jason On Mon, Feb 22, 2016 at 5:09 PM, Venkatesan Subramanian < vsubraman...

Question related to ordering of messages

2016-02-22 Thread Rnjai Lamba
Let us say there are two Producers P1 and P2. P1 makes attempt to write log L1 at time t1 to a partition PT however it gets written at t2 due to network latency. P2 makes attempt to write log L2 at time t3 to a partition PT however it gets written at t4 due to network latency. Let us say t1t4 so w

new producer failed with org.apache.kafka.common.errors.TimeoutException

2016-02-22 Thread Kris K
Hi All, I saw an issue today wherein the producers (new producers) started to fail with org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 6 ms. This issue happened when we took down one of the 6 brokers (running version 0.8.2.1) for planned maintenance (graceful

Kafka 2.11-0.9.0.1 - Performance Run

2016-02-22 Thread Harihara Subramaniam
Hi, How do we run Performance Tests on kafka 2.11-0.9.0.1? -- Cheers! Harihara Subramaniam

Re: Kafka 2.11-0.9.0.1 - Performance Run

2016-02-22 Thread allen chan
what kind of performance tests are you looking for? I am currently using kafka-producer-perf-test.sh to benchmark my brokers. There is also a consumer version too On Mon, Feb 22, 2016 at 8:05 PM, Harihara Subramaniam < subramaniam.harih...@gmail.com> wrote: > Hi, > > How do we run Performance Tes

kafka-consumer-perf.sh

2016-02-22 Thread allen chan
Something i do not understand about this perf-test tool. 1. The legend shows 5 columns but the data shows 6 columns. I am assuming the 0 column is the one that is throwing everything off? 2. does nMsg.sec = number of message consumed per second? [bin]$ sudo ./kafka-consumer-perf-test.sh --group

Error from producer sync API "Broker: Request timed out"

2016-02-22 Thread ANKUR JOHRI
Hi, There is a producer which uses librdkafka to send messages over a Kafka infrastructure. There is an error which occurred from sync API call. It gives error message "Broker: Request timed out". To give some context of the scenario, there are two brokers - broker 2 on local host being the pr