We deploy our applications as war files in tomcat. Most of these applications have a rest API but some also consume from topics. For the apps running in tomcat we use spring and initialize our kafka consumer in the @postConstruct of a spring bean so this only happens once.
The bean itself which is a wrapper for the kafka consumer is annotated with @Component and initialized as a singleton. We also have a shutdown method on this bean annotated with @preDestroy. One thing you'll need to consider if you do publish consumed messages to a thread pool within the application is ordering if your topic is partitioned and uses a partition key for some kind of ordering. You'll consume in order from the topic but lose ordering once you publish to your thread pool. On Fri, Dec 13, 2013 at 6:11 PM, Joe Stein <[email protected]> wrote: > That error comes if you are trying to create a message/stream on a consumer > instantiated object more than once. > > Why are you using tomcat for the consumers? Is it to see the results of > messages? if so you need to isolate the Consumer in some way so there is a > singleton (assuming one partition or if more than one partition some type > of consumer pool) and the messages being consumed write to a thread safe > memory structure and then the UI can read from that thread safe memory > structure. > > /******************************************* > Joe Stein > Founder, Principal Consultant > Big Data Open Source Security LLC > http://www.stealth.ly > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > ********************************************/ > > > On Fri, Dec 13, 2013 at 5:42 PM, Seshadri, Balaji > <[email protected]>wrote: > > > 0.8 > > > > -----Original Message----- > > From: Neha Narkhede [mailto:[email protected]] > > Sent: Friday, December 13, 2013 3:33 PM > > To: [email protected] > > Subject: Re: Unable to start consumers in Tomcat > > > > Which version of kafka are you using? > > > > > > On Fri, Dec 13, 2013 at 2:29 PM, Seshadri, Balaji > > <[email protected]>wrote: > > > > > Any idea on this error guys ?. > > > > > > -----Original Message----- > > > From: Seshadri, Balaji [mailto:[email protected]] > > > Sent: Friday, December 13, 2013 9:32 AM > > > To: '[email protected]' > > > Subject: Unable to start consumers in Tomcat > > > > > > Hi , > > > > > > Can you guys lets us know why are we getting this error when try to > > > spawn a consumer ?. > > > > > > ZookeeperConsumerConnector can create message streams at most once > > > > > > Balaji > > > > > >
