Hi Guozhang,
I do have another Email talking about Partitions per topic. I paste it
within this Email.
I am expecting those consumers will work concurrently. The behavior I
observed here is consumer thread-1 will work a while, then thread-3 will
work, then thread-0 ..., is it normal?
version is 2.2.0.
Best Regards,
Mingtao
On Wed, Jul 23, 2014 at 7:57 PM, Guozhang Wang <wangg...@gmail.com> wrote:
> num.partitions is only used as a default value when the createTopic command
> does not specify the num.partitions or it is automatically created. In your
> case since you always use its value in the createTopic you will always can
> one partition. Try change your code to sth. like:
>
> String[] args = new String[]{
> "--zookeeper", config.getString("zookeeper"),
> "--topic", config.getString("topic"),
> "--replica", config.getString("replicas"),
> "--partition", "8"
> };
>
> CreateTopicCommand.main(args);
>
>
>
> On Wed, Jul 23, 2014 at 4:38 PM, Mingtao Zhang <mail2ming...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > In kafka.properties, I put (forgot to change):
> >
> > num.partitions=1
> >
> > While I create topics programatically:
> >
> > String[] args = new String[]{
> > "--zookeeper", config.getString("zookeeper"),
> > "--topic", config.getString("topic"),
> > "--replica", config.getString("replicas"),
> > "--partition", config.getString("partitions")
> > };
> >
> > CreateTopicCommand.main(args);
> >
> > The performance engineer told me only one consumer thread is actively
> > working even I have 4 consumer threads started (could see when debugging
> or
> > in thread dump); and 4 partitions configured from the args.
> >
> > It seems that num.partitions is still controlling the parallelism. Do I
> > need to change this num.partitions accordingly? Could I remove it? What
> is
> > I have different parallel requirement for different topic?
> >
> > Thank you in advance!
> >
> > Best Regards,
> > Mingtao
> >
>
On Mon, Aug 11, 2014 at 7:37 PM, Guozhang Wang <wangg...@gmail.com> wrote:
> Mingtao,
>
> How many partitions of the consumed topic has? Basically the data is
> distributed per-partition, and hence if the number of consumers is larger
> than the number of partitions, some consumers will not get any data.
>
> Guozhang
>
>
> On Mon, Aug 11, 2014 at 3:29 PM, Mingtao Zhang <mail2ming...@gmail.com>
> wrote:
>
> > Is it anyhow related to the issue?
> >
> > WARN No previously checkpointed highwatermark value found for topic RAW
> > partition 0. Returning 0 as the highwatermark
> > (kafka.server.HighwaterMarkCheckpoint)
> >
> > Mingtao
> >
>
>
>
> --
> -- Guozhang
>
