-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Kafka Stream parallelizes via Kafka partitions -- for each partitions a task is created. If you subscribe to multiple topics, the topics with the most partitions determine the number of task, and each task get partitions from all topics assigned.
Furthermore, you can configure the number to thread (num.stream.threads , see http://docs.confluent.io/current/streams/developer-guide.html#optional-c onfiguration-parameters) - -- the max useful configuration is the number of created tasks. Keep in mind, if you start multiple instanced of you Streams app, partitions are managed in a consumer group fashion, ie, are distributed over the running instances. Please see here for more details http://docs.confluent.io/current/streams/architecture.html#parallelism-m odel - -Matthias On 10/4/16 1:27 PM, Ali Akhtar wrote: > I need to consume a large number of topics, and handle each topic > in a different way. > > I was thinking about creating a different KStream for each topic, > and doing KStream.foreach for each stream, to process incoming > messages. > > However, its unclear if this will be handled in a parallel way by > default, or if I need to create a managed ThreadPool and create the > KStream for each topic within its own thread pool. > > Can anyone shed some light - does KStreamBuilder / KStream handle > concurrency for each KStream, or does this need to be managed? > > Thanks. > -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJX9BRgAAoJECnhiMLycopP9hEQAKgilK2PEaG3QgohJSi8THbb yE/u6vVjSgNpIcZKVULoJ8cWMwh9pHb0LH28B/RNBVujHaBc5WO8YuhpvvlKeBro TPCvbE+IHKJI/R9pD6OdUkeN09SOd6iJ/Bbc6N02+3rsCFTLiFzwusPx8pH9Tx59 RTA6VmWBwkzQt1pCYHIKVYil138jgRh7hjQs/3XYex0vibL3bQBltmZwYnIalcbX n7fAr3rKlrwLMvH1LPr5NPiyzp6al4gdXxeqNAFNI0wwb6y7nqbMeywdOh4KEruC XT8O63O8ykfpL+wNSldT7lnvsxwL5myEp0ONKPRD5S1URzTVEFNj9dzohwGFV7ZE M/1nBu5pxf6BzSBWgi1A30iTUgQo7pP7ManKhRw71kGotD/oLdu2gAL4mHuKiao6 6Z/6prVsDouAk4CbuvXmNlmAFgHHZswtza0qZEG0797Xl3ByhOfAcuREzOJ2c+LE gI1C7E3iV3FgWuOK5B6VIdu0qjC88r8hD7+Q1ep+iDoZUXKqH9LUB06sGE8EaM5Q X67ihVuYo1akG3Hta2JFIHbRuoHLTUnqx3BGMEi8bbZWXfeIY+jl2IwQIqAVaXEo soA6fgDJmGS0vrVKwF7ceT3XmGtkG9h7tHjxnr+TruC6vdWMlNrSg0A6mb158RNd pvdyd9Qf++uud/UZYQb0 =VfUR -----END PGP SIGNATURE-----
