I have created a stream with topic contains 5 partitions and expected to
create 5 stream tasks ,i got 10 tasks as
0_0  0_1  0_2  0_3  0_4  1_0  1_1  1_2  1_3  1_4


im doing wordcount in this example,

here is my topology in this link: 1.
https://gist.github.com/Pk007790/72b0718f26e6963246e83da992b3e725
2.https://gist.github.com/Pk007790/a05226007ca90cdd36c362d09d19bda6.

On Tue, Oct 24, 2017 at 3:29 PM, Damian Guy <damian....@gmail.com> wrote:

> It would depend on what your topology looks like, which you haven't show
> here. But if there may be internal topics generated due to repartitioning
> which would cause the extra tasks.
> If you provide the topology we would be able to tell you.
> Thanks,
> Damian
>
> On Tue, 24 Oct 2017 at 10:14 pravin kumar <pk007...@gmail.com> wrote:
>
> > I have created a stream with topic contains 5 partitions and expected to
> > create 5 stream tasks ,i got 10 tasks as
> > 0_0  0_1  0_2  0_3  0_4  1_0  1_1  1_2  1_3  1_4
> >
> >
> > my doubt is:im expected to have 5 tasks how it produced 10 tasks
> >
> > here are some logs:
> >             [2017-10-24 10:27:35,284] INFO Kafka commitId :
> > cb8625948210849f (org.apache.kafka.common.utils.AppInfoParser)
> > [2017-10-24 10:27:35,284] DEBUG Kafka consumer created
> > (org.apache.kafka.clients.consumer.KafkaConsumer)
> > [2017-10-24 10:27:35,304] INFO stream-thread
> >
> > [SingleConsumerMultiConsumerUsingStreamx4-4dc5b303-62b4-4898-
> 9d8f-a1a9a8adfb7d-StreamThread-1]
> > State transition from CREATED to RUNNING.
> > (org.apache.kafka.streams.processor.internals.StreamThread)
> > [2017-10-24 10:27:35,306] DEBUG stream-client
> >
> > [SingleConsumerMultiConsumerUsingStreamx4-4dc5b303-62b4-4898-
> 9d8f-a1a9a8adfb7d]
> > Removing local Kafka Streams application data in
> >
> > /home/admin/Documents/kafka_2.11-0.10.2.1/kafka-streams/
> SingleConsumerMultiConsumerUsingStreamx4
> > for application SingleConsumerMultiConsumerUsingStreamx4.
> > (org.apache.kafka.streams.KafkaStreams)
> > [2017-10-24 10:27:35,311] DEBUG stream-thread [cleanup] Acquired state
> dir
> > lock for task 0_0
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,311] INFO stream-thread [cleanup] Deleting obsolete
> > state directory 0_0 for task 0_0 as cleanup delay of 0 ms has passed
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,322] DEBUG stream-thread [cleanup] Released state
> dir
> > lock for task 0_0
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,322] DEBUG stream-thread [cleanup] Acquired state
> dir
> > lock for task 1_0
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,322] INFO stream-thread [cleanup] Deleting obsolete
> > state directory 1_0 for task 1_0 as cleanup delay of 0 ms has passed
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,395] DEBUG stream-thread [cleanup] Released state
> dir
> > lock for task 1_0
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,395] DEBUG stream-thread [cleanup] Acquired state
> dir
> > lock for task 0_1
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,395] INFO stream-thread [cleanup] Deleting obsolete
> > state directory 0_1 for task 0_1 as cleanup delay of 0 ms has passed
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,395] DEBUG stream-thread [cleanup] Released state
> dir
> > lock for task 0_1
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,395] DEBUG stream-thread [cleanup] Acquired state
> dir
> > lock for task 1_1
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,395] INFO stream-thread [cleanup] Deleting obsolete
> > state directory 1_1 for task 1_1 as cleanup delay of 0 ms has passed
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,396] DEBUG stream-thread [cleanup] Released state
> dir
> > lock for task 1_1
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,396] DEBUG stream-thread [cleanup] Acquired state
> dir
> > lock for task 0_2
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,396] INFO stream-thread [cleanup] Deleting obsolete
> > state directory 0_2 for task 0_2 as cleanup delay of 0 ms has passed
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,397] DEBUG stream-thread [cleanup] Released state
> dir
> > lock for task 0_2
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,397] DEBUG stream-thread [cleanup] Acquired state
> dir
> > lock for task 1_2
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,397] INFO stream-thread [cleanup] Deleting obsolete
> > state directory 1_2 for task 1_2 as cleanup delay of 0 ms has passed
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,398] DEBUG stream-thread [cleanup] Released state
> dir
> > lock for task 1_2
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,398] DEBUG stream-thread [cleanup] Acquired state
> dir
> > lock for task 0_3
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,398] INFO stream-thread [cleanup] Deleting obsolete
> > state directory 0_3 for task 0_3 as cleanup delay of 0 ms has passed
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,398] DEBUG stream-thread [cleanup] Released state
> dir
> > lock for task 0_3
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,398] DEBUG stream-thread [cleanup] Acquired state
> dir
> > lock for task 1_3
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,399] INFO stream-thread [cleanup] Deleting obsolete
> > state directory 1_3 for task 1_3 as cleanup delay of 0 ms has passed
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,399] DEBUG stream-thread [cleanup] Released state
> dir
> > lock for task 1_3
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,400] DEBUG stream-thread [cleanup] Acquired state
> dir
> > lock for task 0_4
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,400] INFO stream-thread [cleanup] Deleting obsolete
> > state directory 0_4 for task 0_4 as cleanup delay of 0 ms has passed
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,400] DEBUG stream-thread [cleanup] Released state
> dir
> > lock for task 0_4
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,400] DEBUG stream-thread [cleanup] Acquired state
> dir
> > lock for task 1_4
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,400] INFO stream-thread [cleanup] Deleting obsolete
> > state directory 1_4 for task 1_4 as cleanup delay of 0 ms has passed
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,401] DEBUG stream-thread [cleanup] Released state
> dir
> > lock for task 1_4
> > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > [2017-10-24 10:27:35,401] DEBUG stream-client
> >
> > [SingleConsumerMultiConsumerUsingStreamx4-4dc5b303-62b4-4898-
> 9d8f-a1a9a8adfb7d]
> > Starting Kafka Stream process. (org.apache.kafka.streams.KafkaStreams)
> > [2017-10-24 10:27:35,413] DEBUG Updated cluster metadata version 1 to
> > Cluster(id = null, nodes = [localhost:9092 (id: -1 rack: null)],
> partitions
> > = []) (org.apache.kafka.clients.Metadata)
> >
>

Reply via email to