Hi Jun, I think your MBean renaming approach will work. I see https://issues.apache.org/jira/browse/KAFKA-1481 has Fix Version 0.8.2, but is not marked as a Blocker. We'd love to get the MBeans fixed so this makes it in 0.8.2 release. Do you know if this is on anyone's plate (the issue is currently Unassigned)? If not, should we provide a new patch that uses your approach?
Thanks, Otis -- Monitoring * Alerting * Anomaly Detection * Centralized Log Management Solr & Elasticsearch Support * http://sematext.com/ On Thu, Sep 18, 2014 at 4:49 PM, Jun Rao <jun...@gmail.com> wrote: > Otis, > > In kafka-1481, we will have to change the mbean names (at least the ones > with clientid and topic) anyway. Using the name/value pair in the mbean > name allows us to do this in a cleaner way. Yes, "," is not allowed in > clientid or topic. > > Bhavesh, > > Yes, I was thinking of making changes in the new metrics package. Something > like allowing the sensor names to have name/value pairs. The jmx names will > just follow accordingly. This is probably cleaner than doing the escaping. > Also, the metric names are more intuitive (otherwise, you have to know > which part is the clientid and which part is the topic). > > Thanks, > > Jun > > On Wed, Sep 17, 2014 at 2:32 PM, Otis Gospodnetic < > otis.gospodne...@gmail.com> wrote: > > > Hi Jun, > > > > On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao <jun...@gmail.com> wrote: > > > > > Bhavesh, > > > > > > Yes, allowing dot in clientId and topic makes it a bit harder to define > > the > > > JMX bean names. I see a couple of solutions here. > > > > > > 1. Disable dot in clientId and topic names. The issue is that dot may > > > already be used in existing deployment. > > > > > > 2. We can represent the JMX bean name differently in the new producer. > > > Instead of > > > kafka.producer.myclientid:type=mytopic > > > we could change it to > > > kafka.producer:clientId=myclientid,topic=mytopic > > > > > > I felt that option 2 is probably better since it doesn't affect > existing > > > users. > > > > > > > If it doesn't affect existing users, great. > > > > If you are saying that each "piece" of MBean name could be expressed as > > name=value pair, with something like "," (forbidden in host names, topic > > names, client IDs, etc. I assume?) then yes, I think this would be easier > > to parse and it would be easier for people to understand what is what. > > > > Otis > > -- > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management > > Solr & Elasticsearch Support * http://sematext.com/ > > > > > > > > > > > > Otis, > > > > > > We probably can also use option 2 to address KAFKA-1481. For > > topic/clientid > > > specific metrics, we could explicitly specify the metric name so that > it > > > contains "topic=mytopic,clientid=myclientid". That seems to be a much > > > cleaner way than having all parts included in a single string separated > > by > > > '|'. > > > > > > Thanks, > > > > > > Jun > > > > > > > > > > > > > > > On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry < > > > mistry.p.bhav...@gmail.com> > > > wrote: > > > > > > > HI Otis, > > > > > > > > What is migration path ? If topic with special chars exists already( > > > > ".","-","|" etc) in previous version of producer/consumer of Kafka, > > what > > > > happens after the upgrade new producer or consumer (kafka version) ? > > > Also, > > > > in new producer API (Kafka Trunk), does this enforce the rule about > > > client > > > > id as well ? > > > > > > > > Thanks, > > > > > > > > Bhavesh > > > > > > > > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic < > > > > otis.gospodne...@gmail.com> wrote: > > > > > > > > > Hi, > > > > > > > > > > So maybe I should I should have asked the Q explicitly: > > > > > Could we commit the patch from > > > > > https://issues.apache.org/jira/browse/KAFKA-1481 now that, I hope, > > > it's > > > > > clear what problems the current MBean names can cause? > > > > > > > > > > Thanks, > > > > > Otis > > > > > -- > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log > > Management > > > > > Solr & Elasticsearch Support * http://sematext.com/ > > > > > > > > > > > > > > > > > > > > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic < > > > > > otis.gospodne...@gmail.com> wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > *Problem:* > > > > > > Some Kafka 0.8.x MBeans have names composed of things like > > <consumer > > > > > > group>-<topic>-<metric name>. Note how dashes are used as > > > delimiters. > > > > > > When <consumer group> and <topic> don't contain the delimiter > > > > character > > > > > > all is good if you want to extract parts of this MBean name by > > simply > > > > > > splitting on the delimiter character. The problem is that dashes > > are > > > > > > allowed in topic and group names, so this splitting doesn't work. > > > > > > Moreover, underscores are also used as delimiters, and they can > > also > > > be > > > > > > used in things like topic names. > > > > > > > > > > > > *Example*: > > > > > > This MBean's name is composed of <consumer > > > group>-<topic>-BytesPerSec: > > > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics", > > > > name="*myGroup**-myTopic**-* > > > > > > BytesPerSec" > > > > > > > > > > > > Here we can actually split on "-" and extract all 3 parts from > the > > > > MBean > > > > > > name:: > > > > > > * consumer group ('*myGroup*') > > > > > > * topic ('*myTopic*') > > > > > > * metric (‘BytesPerSec’) > > > > > > > > > > > > All good! > > > > > > > > > > > > But imagine if I named the group: *my-Group* > > > > > > And if I named the topic: *my-Topic* > > > > > > > > > > > > Then we'd have: > > > > > > kafka.consumer:type="ConsumerTopicMetrics", > > > > > name="*my-Group**-my-Topic**-* > > > > > > BytesPerSec" > > > > > > > > > > > > Now splitting on "-" would no longer work! To extract "my-Group" > > and > > > > > > "my-Topic" and "BytesPerSec" parts I would have to know the > > specific > > > > > group > > > > > > name and topic name to look for and could not use generic > approach > > of > > > > > just > > > > > > splitting the MBean name on the delimiter. > > > > > > > > > > > > *Solution*: > > > > > > The patch in https://issues.apache.org/jira/browse/KAFKA-1481 > > > replaces > > > > > > all _ and - characters where they are used as delimiters in MBean > > > names > > > > > > with a "|" character. Because the "I" character is not allowed > in > > > > topic > > > > > > names, consumer groups, host names, splitting on this new and > > unified > > > > > > delimiter works. > > > > > > > > > > > > I hope this explains the problem, the solution, and that this can > > > make > > > > it > > > > > > in the next 0.8.x. > > > > > > > > > > > > Otis > > > > > > -- > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log > > > Management > > > > > > Solr & Elasticsearch Support * http://sematext.com/ > > > > > > > > > > > > > > > > > > > > > > > > > > >
