We deploy our applications as war files in tomcat. Most of these applications have a rest API but some also consume from topics. For the apps running in tomcat we use spring and initialize our kafka consumer in the @postConstruct of a spring bean so this only happens once.
The bean itself which is a wrapper for the kafka consumer is annotated with @Component and initialized as a singleton. We also have a shutdown method on this bean annotated with @preDestroy. One thing you'll need to consider if you do publish consumed messages to a thread pool within the application is ordering if your topic is partitioned and uses a partition key for some kind of ordering. You'll consume in order from the topic but lose ordering once you publish to your thread pool. On Fri, Dec 13, 2013 at 6:11 PM, Joe Stein <joe.st...@stealth.ly> wrote: > That error comes if you are trying to create a message/stream on a consumer > instantiated object more than once. > > Why are you using tomcat for the consumers? Is it to see the results of > messages? if so you need to isolate the Consumer in some way so there is a > singleton (assuming one partition or if more than one partition some type > of consumer pool) and the messages being consumed write to a thread safe > memory structure and then the UI can read from that thread safe memory > structure. > > /******************************************* > Joe Stein > Founder, Principal Consultant > Big Data Open Source Security LLC > http://www.stealth.ly > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > ********************************************/ > > > On Fri, Dec 13, 2013 at 5:42 PM, Seshadri, Balaji > <balaji.sesha...@dish.com>wrote: > > > 0.8 > > > > -----Original Message----- > > From: Neha Narkhede [mailto:neha.narkh...@gmail.com] > > Sent: Friday, December 13, 2013 3:33 PM > > To: users@kafka.apache.org > > Subject: Re: Unable to start consumers in Tomcat > > > > Which version of kafka are you using? > > > > > > On Fri, Dec 13, 2013 at 2:29 PM, Seshadri, Balaji > > <balaji.sesha...@dish.com>wrote: > > > > > Any idea on this error guys ?. > > > > > > -----Original Message----- > > > From: Seshadri, Balaji [mailto:balaji.sesha...@dish.com] > > > Sent: Friday, December 13, 2013 9:32 AM > > > To: 'users@kafka.apache.org' > > > Subject: Unable to start consumers in Tomcat > > > > > > Hi , > > > > > > Can you guys lets us know why are we getting this error when try to > > > spawn a consumer ?. > > > > > > ZookeeperConsumerConnector can create message streams at most once > > > > > > Balaji > > > > > >
