Re: more uniform task assignment across kafka stream nodes

Please share the rest of your topology code (without any UDFs / business logic). Otherwise, I cannot give further advice. -Matthias On 3/25/17 6:08 PM, Ara Ebrahimi wrote: > Via: > > builder.stream("topic1"); > builder.stream("topic2"); > builder.stream("topic3”); > > These are different kinds

Re: more uniform task assignment across kafka stream nodes

Via: builder.stream("topic1"); builder.stream("topic2"); builder.stream("topic3”); These are different kinds of topics consuming different avro objects. Ara. On Mar 25, 2017, at 6:04 PM, Matthias J. Sax mailto:matth...@confluent.io>> wrote: This message is

Re: more uniform task assignment across kafka stream nodes

Ara, How do you consume your topics? Via > builder.stream("topic1", "topic2", "topic3); or via > builder.stream("topic1"); > builder.stream("topic2"); > builder.stream("topic3"); Both and handled differently with regard to creating tasks (partition to task assignment also depends on you downst

Re: more uniform task assignment across kafka stream nodes

Mathias, This apparently happens because we have more than 1 source topic. We have 3 source topics in the same application. So it seems like the task assignment algorithm creates topologies not for one specific topic at a time but the total partitions across all source topics consumed in an app

Re: more uniform task assignment across kafka stream nodes

Hi, I am wondering why this happens in the first place. Streams, load-balanced over all running instances, and each instance should be the same number of tasks (and thus partitions) assigned. What is the overall assignment? Do you have StandyBy tasks configured? What version do you use? -Matthi

Re: [DISCUSS] KIP 130: Expose states of active tasks to KafkaStreams public API

Re 1): this is a good point. May be we can move `StreamsMetadata#streamThreads` as `KafkaStreams#localThreadsMetadata`? 3): this is a minor suggestion about function name of `assignedPartitions`, to `topicPartitions` to be consistent with `StreamsMetadata`? Guozhang On Thu, Mar 23, 2017 at 4:30

Fwd: [DISCUSS] KIP 130: Expose states of active tasks to KafkaStreams public API

Forwarding Matthias's reply to the channel (I think he meant to do that but reply only to me). -- Forwarded message -- From: Matthias J. Sax Date: Thu, Mar 23, 2017 at 4:30 PM Subject: Re: [DISCUSS] KIP 130: Expose states of active tasks to KafkaStreams public API To: Guozhang Wan

Re: Kafka streams 0.10.2 Producer throwing exception eventually causing streams shutdown

Hi Sachin, Not in this case. Thanks Eno > On Mar 25, 2017, at 6:19 PM, Sachin Mittal wrote: > > OK. > I will try this out. > > Do I need to change anything for > max.in.flight.requests.per.connection > > Thanks > Sachin > > > On Sat, Mar 25, 2017 at 10:59 PM, Eno Thereska > wrote: > >> H

Re: Kafka streams 0.10.2 Producer throwing exception eventually causing streams shutdown

OK. I will try this out. Do I need to change anything for max.in.flight.requests.per.connection Thanks Sachin On Sat, Mar 25, 2017 at 10:59 PM, Eno Thereska wrote: > Hi Sachin, > > For this particular error, “org.apache.kafka.common.errors. > NotLeaderForPartitionException: This server is not

Re: 3 machines 12 threads all fail within 2 hours of starting the streams application

Hi, The broker is a three machine cluster. The replication factor for input and also internal topics is 3. Brokers don't seem to fail. I always see their instances running. Also note that when an identical streams application with single thread on a single instance is pulling data from some other

Re: 3 machines 12 threads all fail within 2 hours of starting the streams application

Hi Sachin, See my previous email on the NotLeaderForPartitionException error. What is your Kafka configuration, how many brokers are you using? Also could you share the replication level (if different from 1) of your streams topics? Are there brokers failing while Streams is running? Thanks

Re: any way to manually assign listeners to kafka partitions ?

There is a method is consumer is used for getting all partitions for a topic List https://kafka.apache.org/0100/javadoc/org/apache/kafka/common/PartitionInfo.html> > *partitionsFor

any way to manually assign listeners to kafka partitions ?

Hi , Is there anyway to get the list of all paritions and assign them to individual java threads for constantly listening ? Keep learning keep moving .

Re: Kafka streams 0.10.2 Producer throwing exception eventually causing streams shutdown

Hi Sachin, For this particular error, “org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.”, could you try setting the number of retries to something large like this: Properties props = new Properties(); props.put(StreamsConfig

3 machines 12 threads all fail within 2 hours of starting the streams application

Hi All, I am revisiting the ongoing issue of getting a multi instance multi threaded kafka streams cluster to work. Scenario is that we have a 12 partition source topic. (note our server cluster replication factor is 3). We have a 3 machines client cluster with one instance on each. Each instances

Kafka streams 0.10.2 Producer throwing exception eventually causing streams shutdown

Hi, We have encountered another case of series of errors which I would need more help in understanding. In logs we see message like this: ERROR 2017-03-25 03:41:40,001 [kafka-producer-network-thread | 85-StreamThread-3-producer]: org.apache.kafka.streams.processor.internals.RecordCollectorImpl - t

kafka streams 0.10.2 failing while restoring offsets

Hi, So recently we fixed the deadlock issue and also built the streams jar by copying the rocks db configs from trunk. So we don't get any deadlock issue now and also we see that the wait time of CPU cores stays around 5% (down from 50% earlier). However we now get a new exception which is not han