Agree that 0.9 is a few months out. 0.8.2, on the other hand, is probably less than a month away. On one hand, we are trying our best to minimize the set of changes to the existing consumer in the interest of saving time to work on the new consumer. Since the old consumer is very complex and is now stable for some time, the motivation is to make fewer changes to it to maintain that stability. On the other hand, if there are critical bug fixes, it makes sense to patch the old consumer and do a point release.
We would be happy to take a patch from you. How about we look at the size of the proposed changes and discuss a release timeline on the JIRA? Thanks, Neha On Wed, Jun 4, 2014 at 3:09 PM, Otis Gospodnetic <otis.gospodne...@gmail.com > wrote: > Hi Guozhang, > > Since the new consumer is 2-3 months out ... hm, no, looks like 4 months > out - October - > https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan > > Any way you could change this in 0.8.1.2 or 0.8.2? We can submit a patch. > > Thanks, > Otis > -- > Performance Monitoring * Log Analytics * Search Analytics > Solr & Elasticsearch Support * http://sematext.com/ > > > On Wed, Jun 4, 2014 at 4:20 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > > I think we will make this change in the new consumer, which may be > released > > in 0.9. > > > > Guozhang > > > > > > On Wed, Jun 4, 2014 at 12:13 PM, Vladimir Tretyakov < > > vladimir.tretya...@sematext.com> wrote: > > > > > Hi, thx Guozhang Wang, looking forward. > > > > > > When do you think this changes will available? 0.8.2? Jul 2014 > > > https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan > ? > > Or > > > later? > > > > > > Best regards, Vladimir. > > > > > > > > > On Wed, Jun 4, 2014 at 8:01 PM, Guozhang Wang <wangg...@gmail.com> > > wrote: > > > > > > > Hello Vladimir, comments in-lined. > > > > > > > > > > > > On Wed, Jun 4, 2014 at 6:08 AM, Vladimir Tretyakov < > > > > vladimir.tretya...@sematext.com> wrote: > > > > > > > > > Hi again, few more questions from me: > > > > > > > > > > *1.* > > > > > > > > > > What I see in JMX: > > > > > > > > > > > > > > > > > > > > > > > > > kafka.consumer:type="ZookeeperConsumerConnector",name="af_servers-af_servers-spm_new_cluster_topic-af_servers_wawanawna-Dell-1401353748289-fcaaea29-0-FetchQueueSize" > > > > > > > > > > From code: > > > > > > > > > > newGauge( > > > > > config.clientId + "-" + config.groupId + "-" + > > > topicThreadId._1 + > > > > > "-" + topicThreadId._2 + "-FetchQueueSize", > > > > > new Gauge[Int] { > > > > > def value = q.size > > > > > } > > > > > ) > > > > > > > > > > I've tried to parse part as I've understood they. > > > > > > > > > > config.clientId >> af_servers > > > > > topicThreadId._1 >> af_servers-spm_new_cluster_topic > > > > > topicThreadId._2 >> > > af_servers_wawanawna-Dell-1401353748289-fcaaea29-0 > > > > > > > > > > Yes I can suppose that this topicThreadId._1 will always looks like > > > > > GROUP_ID+TOPIC and topicThreadId._2 will contain CONSUMER HOST, but > > > will > > > > it > > > > > always true? > > > > > > > > > > > > > > > > With the new consumer coming soon, the metrics naming schemes would > > very > > > > likely to be refined. So I cannot say this will always be true. > > > > > > > > > > > > > > > > > > > > > > *2.* > > > > > > > > > > From code I see that sometimes Kafka uses "_" as separator, not > only > > > "-": > > > > > > > > > > val consumerIdString = { > > > > > var consumerUuid : String = null > > > > > config.consumerId match { > > > > > case Some(consumerId) // for testing only > > > > > => consumerUuid = consumerId > > > > > case None // generate unique consumerId automatically > > > > > => val uuid = UUID.randomUUID() > > > > > consumerUuid = "%s-%d-%s".format( > > > > > InetAddress.getLocalHost.getHostName, > > System.currentTimeMillis, > > > > > uuid.getMostSignificantBits().toHexString.substring(0,8)) > > > > > } > > > > > config.groupId + "_" + consumerUuid > > > > > } > > > > > > > > > > That means if user will use "_" as part of his host/topic/groupId > > name > > > it > > > > > maybe be a problem to parse string like: > > > > > > > > > > > > > > > > > > > > > > > > > kafka.consumer:type="ZookeeperConsumerConnector",name="af_servers-af_servers-spm_new_cluster_topic-af_servers_wawanawna-Dell-1401353748289-fcaaea29-0-FetchQueueSize" > > > > > > > > > > Look at part: "spm_new_cluster_topic-af_servers_wawanawna-Dell", > what > > > is > > > > > host name here "servers_wawanawna-Dell" or "wawanawna-Dell" ? > > > > > > > > > > So from one side if we want to be able parse name without any > > problems > > > we > > > > > have to avoid using "-" and "_" in host/topic/groupId/clientId, but > > at > > > > the > > > > > same time I see (from > > > > > > http://grokbase.com/t/kafka/users/133xfsnpdh/cant-use-in-client-name > > ): > > > > > > > > > > *"Client id is used for registering jmx beans for monitoring. > Because > > > of > > > > > the* > > > > > *restrictions in bean names, we limit the client id to be only > > > > > alpha-numeric* > > > > > *plus "-" and "_"."* > > > > > > > > > > Does that mean user can use only camelCase in his > > > > > host/topic/groupId/clientId for distinguish one part of name from > > > > another? > > > > > > > > > > Is this a problem? Or I didn't understand something? > > > > > > > > > > > > > Yeah I agree this is a problem, and we should fix it in the new > > consumer. > > > > > > > > > > > > > > > > > > Best regards from Sematext. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jun 3, 2014 at 3:24 AM, Otis Gospodnetic < > > > > > otis.gospodne...@gmail.com > > > > > > wrote: > > > > > > > > > > > Hi Guozhang, > > > > > > > > > > > > On Mon, Jun 2, 2014 at 7:18 PM, Guozhang Wang < > wangg...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > That is indeed a problem, for now, we recommend group name and > > > topic > > > > > > names > > > > > > > to use "_" when there is a need for "-", but this should be > fixed > > > > > > > systematically. > > > > > > > > > > > > > > > > > > > Right! > > > > > > > > > > > > For you use case, could you change your topic/group name using > "_"? > > > > > > > > > > > > > > > > > > Our own Kafka doesn't use topics with "-" characters, so we don't > > > have > > > > a > > > > > > problem. > > > > > > > > > > > > The problem, in our case, is that we have a general (Kafka) > > > monitoring > > > > > tool > > > > > > that other people use to monitor Kafka - see > > > http://sematext.com/spm/ > > > > . > > > > > > So > > > > > > we can't really tell people "hey, our tool will work but only if > > you > > > > > don't > > > > > > have a dash in your topic names and hosts and ... because if you > > use > > > > > dashes > > > > > > we won't know how to parse your Kafka's MBean names" :) > > > > > > > > > > > > > > > > > > > Also, do you mind to file a JIRA ticket to keep track of this > > > issue? > > > > > > > > > > > > > > > > > > Here it is: https://issues.apache.org/jira/browse/KAFKA-1481 > > > > > > > > > > > > Otis > > > > > > -- > > > > > > Performance Monitoring * Log Analytics * Search Analytics > > > > > > Solr & Elasticsearch Support * http://sematext.com/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Jun 2, 2014 at 5:18 AM, Vladimir Tretyakov < > > > > > > > vladimir.tretya...@sematext.com> wrote: > > > > > > > > > > > > > > > Hello everyone, > > > > > > > > > > > > > > > > We are adding Kafka 0.8.x monitoring support to SPM > > > > > > > > <http://sematext.com/spm/> here at Sematext. Unfortunately, > we > > > > > quickly > > > > > > > hit > > > > > > > > an issue caused by the new bean naming convention that embeds > > > > things > > > > > > like > > > > > > > > topic and host names in the beans along with metrics, > separated > > > by > > > > > > > dashes, > > > > > > > > making it hard to parse these beans. > > > > > > > > > > > > > > > > To put it simply: it is hard/impossible to automatically > figure > > > out > > > > > > which > > > > > > > > part of the bean name is e.g. consumer group, which is the > > topic, > > > > > which > > > > > > > is > > > > > > > > the host name, and which is the name of the metric. > > > > > > > > > > > > > > > > Let me show you what I mean: > > > > > > > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics", > > > > > > > > > > > > > > > > > name="af_servers-spm_topic-BytesPerSec" > > > > > > > > > > > > > > > > Here we actually CAN extract: > > > > > > > > > > > > > > > > * consumer group ('af_servers') > > > > > > > > > > > > > > > > * topic ('spm_topic') > > > > > > > > > > > > > > > > * metric (‘BytesPerSec’) > > > > > > > > > > > > > > > > BUT what if the consumer group id and/or topic name contain > > '-'? > > > > > > > > > > > > > > > > Then how would we extract consumer group and topic? > > > > > > > > > > > > > > > > Here is a concrete example of this problem: > > > > > > > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics", > > > > > > > > > > > > > > > > name="af-servers-spm-topic-BytesPerSec" > > > > > > > > > > > > > > > > How can we know what is group id or topic name here? > > > > > > > > > > > > > > > > This looks like a problem to me, but maybe I’m missing > > something? > > > > > > > > > > > > > > > > Is it possible to have all these values (group id, topic > name) > > as > > > > > > > separate > > > > > > > > attributes inside JMX bean? > > > > > > > > > > > > > > > > Or maybe the problem could be solved if a different delimiter > > was > > > > > used, > > > > > > > > such as the pipe (“I”)? > > > > > > > > > > > > > > > > It is really needed things and will be nice to have it to > build > > > > good > > > > > > tool > > > > > > > > for monitoring. > > > > > > > > > > > > > > > > Thx and best regards from Sematext. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > -- Guozhang > > > > > > > > > > > > > > > -- > > -- Guozhang > > >
