Also, in the next version of Kafka / Kafka Streams such "intermediate" topics will automatically be created for you when you do joins or aggregations:
https://issues.apache.org/jira/browse/KAFKA-3561 So my previous message explained your options today when using the current release of Kafka Streams (v0.10.0.0). -Michael On Thu, Jul 14, 2016 at 10:32 AM, Michael Noll <mich...@confluent.io> wrote: > Poul, > > to add to what Matthias said: If you are wondering how to manually create > a topic, you have basically two options. > > A. Use Kafka's CLI tools to create the topic "from the outside". > > # Example > $ kafka-topics.sh --create --topic my-custom-toipc --zookeeper > localhost:2181 --partitions 1 --replication-factor 1 > > B. Use Kafka's API to programmatically create the topic. See [1] for an > example. > > Question for you to learn how we could perhaps improve the status quo: > How would you have expected this to work in the current Kafka Streams > API? For example, would you have expected that, say, the `through()` > method would accept parameters to specify the number of partitions? > > > Hope this helps, > Michael > > [1] > https://github.com/confluentinc/examples/blob/kafka-0.10.0.0-cp-3.0.0/kafka-streams/src/test/java/io/confluent/examples/streams/kafka/KafkaEmbedded.java#L133-L160 > > > > > On Thu, Jul 14, 2016 at 10:08 AM, Matthias J. Sax <matth...@confluent.io> > wrote: > >> Hi, >> >> you can manually create a topic with the number of partitions you want >> to have and use this topic via through() >> >> KStream input = ... >> >> input.map().through("manually-created-topic").join(...) >> >> However, both KStream and KTable need to have the same number of >> partitions for perform the join. Thus, you might need to create a topic >> (with the same number of partitions) for the table, too. >> >> See >> >> http://docs.confluent.io/3.0.0/streams/developer-guide.html#joining-streams >> >> >> -Matthias >> >> On 07/13/2016 11:59 PM, Poul Costinsky wrote: >> > Hi! I am prototyping some code using Kafka Streams, and have a >> question. I need to map a stream into another (with different partition >> key) and join it with a table. How do I control number of partitions of the >> mapped stream? >> > >> > Thanks! >> > >> > Poul Costinsky >> > Chief Architect >> > >> > <http://www.adbrain.com/> >> > (360) 207-1753 <tel:%28360%29%20207-1753> >> > >> > >> > >> > >> > >> >> > > > -- > > *Michael G. Noll | Product Manager | Confluent | +1 650.453.5860 > <%2B1%20650.453.5860>Download Apache Kafka and Confluent Platform: > www.confluent.io/download <http://www.confluent.io/download>* > -- *Michael G. Noll | Product Manager | Confluent | +1 650.453.5860Download Apache Kafka and Confluent Platform: www.confluent.io/download <http://www.confluent.io/download>*
