Re: Kafka Streams question

2016-07-14 Thread Michael Noll
Also, in the next version of Kafka / Kafka Streams such "intermediate" topics will automatically be created for you when you do joins or aggregations: https://issues.apache.org/jira/browse/KAFKA-3561 So my previous message explained your options today when using the current release of Kafka Strea

Re: Kafka Streams question

2016-07-14 Thread Michael Noll
Poul, to add to what Matthias said: If you are wondering how to manually create a topic, you have basically two options. A. Use Kafka's CLI tools to create the topic "from the outside". # Example $ kafka-topics.sh --create --topic my-custom-toipc --zookeeper localhost:2181 --partitions

Re: Kafka Streams question

2016-07-14 Thread Matthias J. Sax
Hi, you can manually create a topic with the number of partitions you want to have and use this topic via through() KStream input = ... input.map().through("manually-created-topic").join(...) However, both KStream and KTable need to have the same number of partitions for perform the join. Thus,