I have created a stream with topic contains 5 partitions and expected to
create 5 stream tasks ,i got 10 tasks as
0_0 0_1 0_2 0_3 0_4 1_0 1_1 1_2 1_3 1_4
my doubt is:im expected to have 5 tasks how it produced 10 tasks
here are some logs:
[2017-10-24 10:27:35,284] INFO Kafka commitId :
cb8625948210849f (org.apache.kafka.common.utils.AppInfoParser)
[2017-10-24 10:27:35,284] DEBUG Kafka consumer created
(org.apache.kafka.clients.consumer.KafkaConsumer)
[2017-10-24 10:27:35,304] INFO stream-thread
[SingleConsumerMultiConsumerUsingStreamx4-4dc5b303-62b4-4898-9d8f-a1a9a8adfb7d-StreamThread-1]
State transition from CREATED to RUNNING.
(org.apache.kafka.streams.processor.internals.StreamThread)
[2017-10-24 10:27:35,306] DEBUG stream-client
[SingleConsumerMultiConsumerUsingStreamx4-4dc5b303-62b4-4898-9d8f-a1a9a8adfb7d]
Removing local Kafka Streams application data in
/home/admin/Documents/kafka_2.11-0.10.2.1/kafka-streams/SingleConsumerMultiConsumerUsingStreamx4
for application SingleConsumerMultiConsumerUsingStreamx4.
(org.apache.kafka.streams.KafkaStreams)
[2017-10-24 10:27:35,311] DEBUG stream-thread [cleanup] Acquired state dir
lock for task 0_0
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,311] INFO stream-thread [cleanup] Deleting obsolete
state directory 0_0 for task 0_0 as cleanup delay of 0 ms has passed
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,322] DEBUG stream-thread [cleanup] Released state dir
lock for task 0_0
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,322] DEBUG stream-thread [cleanup] Acquired state dir
lock for task 1_0
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,322] INFO stream-thread [cleanup] Deleting obsolete
state directory 1_0 for task 1_0 as cleanup delay of 0 ms has passed
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,395] DEBUG stream-thread [cleanup] Released state dir
lock for task 1_0
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,395] DEBUG stream-thread [cleanup] Acquired state dir
lock for task 0_1
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,395] INFO stream-thread [cleanup] Deleting obsolete
state directory 0_1 for task 0_1 as cleanup delay of 0 ms has passed
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,395] DEBUG stream-thread [cleanup] Released state dir
lock for task 0_1
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,395] DEBUG stream-thread [cleanup] Acquired state dir
lock for task 1_1
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,395] INFO stream-thread [cleanup] Deleting obsolete
state directory 1_1 for task 1_1 as cleanup delay of 0 ms has passed
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,396] DEBUG stream-thread [cleanup] Released state dir
lock for task 1_1
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,396] DEBUG stream-thread [cleanup] Acquired state dir
lock for task 0_2
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,396] INFO stream-thread [cleanup] Deleting obsolete
state directory 0_2 for task 0_2 as cleanup delay of 0 ms has passed
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,397] DEBUG stream-thread [cleanup] Released state dir
lock for task 0_2
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,397] DEBUG stream-thread [cleanup] Acquired state dir
lock for task 1_2
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,397] INFO stream-thread [cleanup] Deleting obsolete
state directory 1_2 for task 1_2 as cleanup delay of 0 ms has passed
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,398] DEBUG stream-thread [cleanup] Released state dir
lock for task 1_2
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,398] DEBUG stream-thread [cleanup] Acquired state dir
lock for task 0_3
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,398] INFO stream-thread [cleanup] Deleting obsolete
state directory 0_3 for task 0_3 as cleanup delay of 0 ms has passed
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,398] DEBUG stream-thread [cleanup] Released state dir
lock for task 0_3
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,398] DEBUG stream-thread [cleanup] Acquired state dir
lock for task 1_3
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,399] INFO stream-thread [cleanup] Deleting obsolete
state directory 1_3 for task 1_3 as cleanup delay of 0 ms has passed
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,399] DEBUG stream-thread [cleanup] Released state dir
lock for task 1_3
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,400] DEBUG stream-thread [cleanup] Acquired state dir
lock for task 0_4
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,400] INFO stream-thread [cleanup] Deleting obsolete
state directory 0_4 for task 0_4 as cleanup delay of 0 ms has passed
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,400] DEBUG stream-thread [cleanup] Released state dir
lock for task 0_4
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,400] DEBUG stream-thread [cleanup] Acquired state dir
lock for task 1_4
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,400] INFO stream-thread [cleanup] Deleting obsolete
state directory 1_4 for task 1_4 as cleanup delay of 0 ms has passed
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,401] DEBUG stream-thread [cleanup] Released state dir
lock for task 1_4
(org.apache.kafka.streams.processor.internals.StateDirectory)
[2017-10-24 10:27:35,401] DEBUG stream-client
[SingleConsumerMultiConsumerUsingStreamx4-4dc5b303-62b4-4898-9d8f-a1a9a8adfb7d]
Starting Kafka Stream process. (org.apache.kafka.streams.KafkaStreams)
[2017-10-24 10:27:35,413] DEBUG Updated cluster metadata version 1 to
Cluster(id = null, nodes = [localhost:9092 (id: -1 rack: null)], partitions
= []) (org.apache.kafka.clients.Metadata)
