NIlesh and Shamsul, 2) you don't need to use another database to keep
track processed tuples. Are you sure you are doing tuple ack and fail in
the downstream bolts so that kafkaspout knows it processed the tuple.
Tuple replays occurs if there are timeouts happening or incase of
exceptions where you call fail on a tuple.
>> 3)The consumer group is isn’t working properly for storm Kafka
>> integration.
>> a.When we give same group id to the Kafka consumer of different
>> topology but still both are reading same messages.
>> b.If we have 2 different consumer with different consumer group id in
>> different topology it works fine if both topologies are deployed at
>> the same time, but doesn’t if we deploy one of them after some of
>> the message are already loaded in the topic and read by the first
>> topology.
a. Kafka Spout uses simple consumer api it doesn't need a consumer
group. can you give us more details why you need two topologies to use
coordinate? (i.e use the same consumer group). Thanks, Harsha
On Thu, Nov 6, 2014, at 04:27 AM, Shamsul Haque wrote:
> Hi Nilesh,
>
>
For point 1, try by increasing the 'topology.message.timeout.secs'
to 10 to 15 mins or more then slowly decrease it which suits your
topology. For me that worked for the same case.
>
For point 2, we have used database to made track what we have
processed, so don't process the same tuple again.
>
>
regards
>
Shams
>
> On Thursday 06 November 2014 12:16 PM,
Nilesh Chhapru wrote:
>> Hi All,
>>
>> We are using storm Kafka integration where
a Spout reads from a Kafka topic.
>>
>> Following is the version of storm, Kafka
and zookeeper we are using.
>> *Strom : apache-storm-0.9.2-incubating*
>> *Kafka : kafka_2.8.0-0.8.1.1*
>> *Zookeeper : zookeeper-3.4.6*
>>
>> I am facing following issues at spout.
>> 1)The messages gets failed even if
the average time taken is less than max.topology.timeout
value, also we aren’t getting any exceptions at any of the
bolt.
>> 2)A topology is finally emitting
to the Kafka producer i.e. some other topic, but the messages
are getting duplicated due to replay issues.
>> 3)The consumer group is isn’t
working properly for storm Kafka integration.
>> a.When we give same group id to
the Kafka consumer of different topology but still both are
reading same messages.
>> b.If we have 2 different consumer
with different consumer group id in different topology it
works fine if both topologies are deployed at the same time,
but doesn’t if we deploy one of them after some of the
message are already loaded in the topic and read by the
first topology.
>>
>> Kindly help me with above points as it is
hampering the overall scope of the project and also time
lines.
>>
>> Do call or email in-case you need any other
information.
>>
>>
>> *Nilesh
Chhapru,*
>> (:
+91 9619030491
>>
>> ---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
>>
>>
****Opinions expressed in this e-mail are those of the author
and do not necessarily represent those of Ugam. Ugam does not
accept any responsibility or liability for it. This e-mail
message may contain proprietary, confidential or legally
privileged information for the sole use of the person or entity
to whom this message was originally addressed. Any review,
re-transmission, dissemination or other use of or taking of any
action in reliance upon this information by persons or entities
other than the intended recipient is prohibited. If you have
received this e-mail in error, please delete it and all
attachments from any servers, hard drives or any other media.
>>
>>
Warning: Sufficient measures have been taken to scan any
presence of viruses however the recipient should check this
email and any attachments for the presence of viruses. Ugam
accepts no liability for any damage caused by any virus
transmitted by this email. ****
>
> --
>
> Email had 1 attachment:
> * india-com.jpg 35k (image/jpeg)
