Re: Kafka Streams - Custom processor "init" method called before state store has data restored into it

2017-11-12 Thread Bill Bejeck
Rainer, Thanks for sharing the logs. With respect to option "c" in you orginal email, given that you are using a custom processor and state store would setting a StateRestoreListener in the custom store suit your needs? Would you be comfortable sharing your code so I can see if there is an acce

building kafka minimal kafka cluster that tolerates a loss of one node.

2017-11-12 Thread Vitaliy Semochkin
Hi What should be parameters for a broker, topic and producer if I want to build a cluster of 3+ kafka brokers which will survive occasional one node crashes (including rack crash) and guaranties with at least once delivery. If numbers of brokers>=3 my own thoughts are: broker.rack = each of 3 b

Re: Kafka Streams - Custom processor "init" method called before state store has data restored into it

2017-11-12 Thread Rainer Guessner
Hi Bill,   thanks for the suggestion towards StateRestoreListener, however that does not solve my issue as its a global listener and doesn't help the processor itself.   Please find the simple one-class code for reproducing the issue below. Please create two topics: kafka-topics -zookeeper localho

Kafka Streams question

2017-11-12 Thread Boris Lublinsky
I am writing Kafka Streams implementation (1.0.0), for which I have 2 input streams. Is it possible to have different subscriber IDs for these 2 streams. I see only one place where subscriber’s ID can be specified: streamsConfiguration.put(StreamsConfig.CLIENT_ID_CONFIG, ApplicationKafkaParamete

Fwd: kafka streams with multiple threads and state store

2017-11-12 Thread Ranjit Kumar
Hi All, I am new to kafka. I have some fundamental questions related to choosing architecture for m requirement can please suggest me. I am developing some IOT project, please find the details below. 1. I have moving objects which are continuously sending their position + some other informati