Sadly it's not easily reproducible, it seems rather random while we're starting large numbers of consumer processes. Out of the 300 or so we restarted during our last release, I saw this happen to 3 of them. And we stagger our restarts in groups of 100.
-- Ian Friedman On Thursday, September 5, 2013 at 11:27 PM, Jun Rao wrote: > Could you reproduce this easily? > > Thanks, > > Jun > > > On Thu, Sep 5, 2013 at 10:52 AM, Ian Friedman <i...@flurry.com > (mailto:i...@flurry.com)> wrote: > > > It shouldn't be, no… it returns an ImmutableMap.. > > > > setTopicMap(ImmutableMap.of(KafkaLogQueuer.sKafkaDataLogPathsTopic, 1)); > > > > -- > > Ian Friedman > > > > > > On Wednesday, September 4, 2013 at 11:40 PM, Jun Rao wrote: > > > > > What createMessageStreams() returns should match what's in the input. Is > > it > > > possible for getTopicMap() to return an empty map? > > > > > > Thanks, > > > > > > Jun > > > > > > > > > On Wed, Sep 4, 2013 at 1:43 PM, Ian Friedman <i...@flurry.com > > > (mailto:i...@flurry.com) (mailto: > > i...@flurry.com (mailto:i...@flurry.com))> wrote: > > > > > > > Hey gang, I have a strange intermittent issue with our 0.7.2 consumers > > > > that is perpelxing me: > > > > > > > > It seems like, sometimes very infrequently, when restarting our > > Consumer > > > > fleet, some consumers threads just fail to start. The consumer > > > > > > > connector > > > > itself starts fine, and seems to be involved in subsequent rebalances, > > > > > > > but > > > > based on the logging I'm seeing, it looks like our one call to > > > > ConsumerConnector.createMessageStreams is returning an empty map. > > > > > > > > Is that something that should happen normally and should we be > > retrying in > > > > our calling code? Our consumer code looks like this: > > > > > > > > Map<String, List<KafkaStream<Message>>> topicMessageStreams = > > > > fConsumerConnector.createMessageStreams( getTopicMap() ); > > > > > > > > for( Map.Entry<String, List<KafkaStream<Message>>> entry : > > > > topicMessageStreams.entrySet() ) > > > > { > > > > final String topicName = entry.getKey(); > > > > log.info (http://log.info)( "starting stream for topic: " + topicName > > > > > > > > > > > ); > > > > final KafkaStream<Message> stream = entry.getValue().get(0); > > > > > > > > …. > > > > > > > > Are we doing anything dumb here? In the case I described, we're never > > > > getting the "starting stream for topic: " message in our log output, > > > > > > > > > > > so my > > > > only guess is that createMessageStreams is returning an empty map. > > > > > > > > Thanks in advance guys! > > > > > > > > -- > > > > Ian Friedman > > > > > > > > > > > > > >
