ohhh...thank you. Its cleared now

On Tue, Oct 31, 2017 at 4:36 PM, Damian Guy <damian....@gmail.com> wrote:

> Hi, the `map` when it is followed by `groupByKey` will cause a
> repartitioning of the data, so you will have your 5 tasks processing the
> input partitions and 5 tasks processing the partitions from the
> repartitioning.
>
> On Tue, 31 Oct 2017 at 10:56 pravin kumar <pk007...@gmail.com> wrote:
>
> >  I have created a stream with topic contains 5 partitions and expected to
> > create 5 stream tasks ,i got 10 tasks as
> > 0_0  0_1  0_2  0_3  0_4  1_0  1_1  1_2  1_3  1_4
> >
> >
> > im doing wordcount in this example,
> >
> > here is my topology in this link: 1.
> > https://gist.github.com/Pk007790/72b0718f26e6963246e83da992b3e725
> > 2.https://gist.github.com/Pk007790/a05226007ca90cdd36c362d09d19bda6.
> >
> > On Tue, Oct 24, 2017 at 3:29 PM, Damian Guy <damian....@gmail.com>
> wrote:
> >
> > > It would depend on what your topology looks like, which you haven't
> show
> > > here. But if there may be internal topics generated due to
> repartitioning
> > > which would cause the extra tasks.
> > > If you provide the topology we would be able to tell you.
> > > Thanks,
> > > Damian
> > >
> > > On Tue, 24 Oct 2017 at 10:14 pravin kumar <pk007...@gmail.com> wrote:
> > >
> > > > I have created a stream with topic contains 5 partitions and expected
> > to
> > > > create 5 stream tasks ,i got 10 tasks as
> > > > 0_0  0_1  0_2  0_3  0_4  1_0  1_1  1_2  1_3  1_4
> > > >
> > > >
> > > > my doubt is:im expected to have 5 tasks how it produced 10 tasks
> > > >
> > > > here are some logs:
> > > >             [2017-10-24 10:27:35,284] INFO Kafka commitId :
> > > > cb8625948210849f (org.apache.kafka.common.utils.AppInfoParser)
> > > > [2017-10-24 10:27:35,284] DEBUG Kafka consumer created
> > > > (org.apache.kafka.clients.consumer.KafkaConsumer)
> > > > [2017-10-24 10:27:35,304] INFO stream-thread
> > > >
> > > > [SingleConsumerMultiConsumerUsingStreamx4-4dc5b303-62b4-4898-
> > > 9d8f-a1a9a8adfb7d-StreamThread-1]
> > > > State transition from CREATED to RUNNING.
> > > > (org.apache.kafka.streams.processor.internals.StreamThread)
> > > > [2017-10-24 10:27:35,306] DEBUG stream-client
> > > >
> > > > [SingleConsumerMultiConsumerUsingStreamx4-4dc5b303-62b4-4898-
> > > 9d8f-a1a9a8adfb7d]
> > > > Removing local Kafka Streams application data in
> > > >
> > > > /home/admin/Documents/kafka_2.11-0.10.2.1/kafka-streams/
> > > SingleConsumerMultiConsumerUsingStreamx4
> > > > for application SingleConsumerMultiConsumerUsingStreamx4.
> > > > (org.apache.kafka.streams.KafkaStreams)
> > > > [2017-10-24 10:27:35,311] DEBUG stream-thread [cleanup] Acquired
> state
> > > dir
> > > > lock for task 0_0
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,311] INFO stream-thread [cleanup] Deleting
> > obsolete
> > > > state directory 0_0 for task 0_0 as cleanup delay of 0 ms has passed
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,322] DEBUG stream-thread [cleanup] Released
> state
> > > dir
> > > > lock for task 0_0
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,322] DEBUG stream-thread [cleanup] Acquired
> state
> > > dir
> > > > lock for task 1_0
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,322] INFO stream-thread [cleanup] Deleting
> > obsolete
> > > > state directory 1_0 for task 1_0 as cleanup delay of 0 ms has passed
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,395] DEBUG stream-thread [cleanup] Released
> state
> > > dir
> > > > lock for task 1_0
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,395] DEBUG stream-thread [cleanup] Acquired
> state
> > > dir
> > > > lock for task 0_1
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,395] INFO stream-thread [cleanup] Deleting
> > obsolete
> > > > state directory 0_1 for task 0_1 as cleanup delay of 0 ms has passed
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,395] DEBUG stream-thread [cleanup] Released
> state
> > > dir
> > > > lock for task 0_1
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,395] DEBUG stream-thread [cleanup] Acquired
> state
> > > dir
> > > > lock for task 1_1
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,395] INFO stream-thread [cleanup] Deleting
> > obsolete
> > > > state directory 1_1 for task 1_1 as cleanup delay of 0 ms has passed
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,396] DEBUG stream-thread [cleanup] Released
> state
> > > dir
> > > > lock for task 1_1
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,396] DEBUG stream-thread [cleanup] Acquired
> state
> > > dir
> > > > lock for task 0_2
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,396] INFO stream-thread [cleanup] Deleting
> > obsolete
> > > > state directory 0_2 for task 0_2 as cleanup delay of 0 ms has passed
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,397] DEBUG stream-thread [cleanup] Released
> state
> > > dir
> > > > lock for task 0_2
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,397] DEBUG stream-thread [cleanup] Acquired
> state
> > > dir
> > > > lock for task 1_2
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,397] INFO stream-thread [cleanup] Deleting
> > obsolete
> > > > state directory 1_2 for task 1_2 as cleanup delay of 0 ms has passed
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,398] DEBUG stream-thread [cleanup] Released
> state
> > > dir
> > > > lock for task 1_2
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,398] DEBUG stream-thread [cleanup] Acquired
> state
> > > dir
> > > > lock for task 0_3
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,398] INFO stream-thread [cleanup] Deleting
> > obsolete
> > > > state directory 0_3 for task 0_3 as cleanup delay of 0 ms has passed
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,398] DEBUG stream-thread [cleanup] Released
> state
> > > dir
> > > > lock for task 0_3
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,398] DEBUG stream-thread [cleanup] Acquired
> state
> > > dir
> > > > lock for task 1_3
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,399] INFO stream-thread [cleanup] Deleting
> > obsolete
> > > > state directory 1_3 for task 1_3 as cleanup delay of 0 ms has passed
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,399] DEBUG stream-thread [cleanup] Released
> state
> > > dir
> > > > lock for task 1_3
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,400] DEBUG stream-thread [cleanup] Acquired
> state
> > > dir
> > > > lock for task 0_4
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,400] INFO stream-thread [cleanup] Deleting
> > obsolete
> > > > state directory 0_4 for task 0_4 as cleanup delay of 0 ms has passed
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,400] DEBUG stream-thread [cleanup] Released
> state
> > > dir
> > > > lock for task 0_4
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,400] DEBUG stream-thread [cleanup] Acquired
> state
> > > dir
> > > > lock for task 1_4
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,400] INFO stream-thread [cleanup] Deleting
> > obsolete
> > > > state directory 1_4 for task 1_4 as cleanup delay of 0 ms has passed
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,401] DEBUG stream-thread [cleanup] Released
> state
> > > dir
> > > > lock for task 1_4
> > > > (org.apache.kafka.streams.processor.internals.StateDirectory)
> > > > [2017-10-24 10:27:35,401] DEBUG stream-client
> > > >
> > > > [SingleConsumerMultiConsumerUsingStreamx4-4dc5b303-62b4-4898-
> > > 9d8f-a1a9a8adfb7d]
> > > > Starting Kafka Stream process. (org.apache.kafka.streams.
> KafkaStreams)
> > > > [2017-10-24 10:27:35,413] DEBUG Updated cluster metadata version 1 to
> > > > Cluster(id = null, nodes = [localhost:9092 (id: -1 rack: null)],
> > > partitions
> > > > = []) (org.apache.kafka.clients.Metadata)
> > > >
> > >
> >
>

Reply via email to