Just added an entry in the FAQ page: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan%27tIspecifythenumberofstreamsparallelismpertopicmapusingwildcardstreamasIusestaticstreamhandler ?
On Mon, Nov 10, 2014 at 7:56 AM, Guozhang Wang <wangg...@gmail.com> wrote: > Hi Alan, > > The reason we do not have per-topic parallelism spec in wildcard is two > folds: 1) we use a per-topic hash-based partition algorithm, and hence > having each topic with the same num. of streams may give us better load > balance, 2) with the topicFilter we will not know exactly which topics to > consume at the construction time, hence no way to specify per-topic specs. > > 1) has been lifted since we have implemented new partitioning algorithm, > and for 2) we need to think about how to support it if we really want to, > perhaps we can also use a regex-ed topic-count map, while ensuring that > each regex in the map is precedent of the topic filter, and no overlap with > each other, etc. What is your usecase that requires per-topic numStream > spec? > > Guozhang > > On Sun, Nov 9, 2014 at 6:03 AM, Alan Lavintman <alan.lavint...@gmail.com> > wrote: > >> Hi guys, i have seen that if create a message stream by using: >> >> createMessageStreams >> >> I can define a map with Topic->#Streams >> >> Is there a reason why createMessageStreamsByFilter us not giving the same >> support? I have only a TopicFilter and numStreams interface such as: >> >> public List<KafkaStream<byte[], byte[]>> >> createMessageStreamsByFilter(TopicFilter topicFilter, int numStreams); >> >> But it does not allow me to specify the parallelism per topic. Am I >> missing >> something or my assumption is correct? >> >> Bests and thanks, >> Alan. >> > > > > -- > -- Guozhang > -- -- Guozhang
