Otis, Yes, if you guys can help provide a patch in a few days, we can probably get it to the 0.8.2 release.
Thanks, Jun On Tue, Oct 7, 2014 at 12:10 PM, Otis Gospodnetic < otis.gospodne...@gmail.com> wrote: > Hi Jun, > > I think your MBean renaming approach will work. I see > https://issues.apache.org/jira/browse/KAFKA-1481 has Fix Version 0.8.2, > but > is not marked as a Blocker. We'd love to get the MBeans fixed so this > makes it in 0.8.2 release. Do you know if this is on anyone's plate (the > issue is currently Unassigned)? If not, should we provide a new patch that > uses your approach? > > Thanks, > Otis > -- > Monitoring * Alerting * Anomaly Detection * Centralized Log Management > Solr & Elasticsearch Support * http://sematext.com/ > > > On Thu, Sep 18, 2014 at 4:49 PM, Jun Rao <jun...@gmail.com> wrote: > > > Otis, > > > > In kafka-1481, we will have to change the mbean names (at least the ones > > with clientid and topic) anyway. Using the name/value pair in the mbean > > name allows us to do this in a cleaner way. Yes, "," is not allowed in > > clientid or topic. > > > > Bhavesh, > > > > Yes, I was thinking of making changes in the new metrics package. > Something > > like allowing the sensor names to have name/value pairs. The jmx names > will > > just follow accordingly. This is probably cleaner than doing the > escaping. > > Also, the metric names are more intuitive (otherwise, you have to know > > which part is the clientid and which part is the topic). > > > > Thanks, > > > > Jun > > > > On Wed, Sep 17, 2014 at 2:32 PM, Otis Gospodnetic < > > otis.gospodne...@gmail.com> wrote: > > > > > Hi Jun, > > > > > > On Wed, Sep 17, 2014 at 12:35 PM, Jun Rao <jun...@gmail.com> wrote: > > > > > > > Bhavesh, > > > > > > > > Yes, allowing dot in clientId and topic makes it a bit harder to > define > > > the > > > > JMX bean names. I see a couple of solutions here. > > > > > > > > 1. Disable dot in clientId and topic names. The issue is that dot may > > > > already be used in existing deployment. > > > > > > > > 2. We can represent the JMX bean name differently in the new > producer. > > > > Instead of > > > > kafka.producer.myclientid:type=mytopic > > > > we could change it to > > > > kafka.producer:clientId=myclientid,topic=mytopic > > > > > > > > I felt that option 2 is probably better since it doesn't affect > > existing > > > > users. > > > > > > > > > > If it doesn't affect existing users, great. > > > > > > If you are saying that each "piece" of MBean name could be expressed as > > > name=value pair, with something like "," (forbidden in host names, > topic > > > names, client IDs, etc. I assume?) then yes, I think this would be > easier > > > to parse and it would be easier for people to understand what is what. > > > > > > Otis > > > -- > > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management > > > Solr & Elasticsearch Support * http://sematext.com/ > > > > > > > > > > > > > > > > > Otis, > > > > > > > > We probably can also use option 2 to address KAFKA-1481. For > > > topic/clientid > > > > specific metrics, we could explicitly specify the metric name so that > > it > > > > contains "topic=mytopic,clientid=myclientid". That seems to be a much > > > > cleaner way than having all parts included in a single string > separated > > > by > > > > '|'. > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > On Tue, Sep 16, 2014 at 5:15 PM, Bhavesh Mistry < > > > > mistry.p.bhav...@gmail.com> > > > > wrote: > > > > > > > > > HI Otis, > > > > > > > > > > What is migration path ? If topic with special chars exists > already( > > > > > ".","-","|" etc) in previous version of producer/consumer of > Kafka, > > > what > > > > > happens after the upgrade new producer or consumer (kafka version) > ? > > > > Also, > > > > > in new producer API (Kafka Trunk), does this enforce the rule about > > > > client > > > > > id as well ? > > > > > > > > > > Thanks, > > > > > > > > > > Bhavesh > > > > > > > > > > On Tue, Sep 16, 2014 at 2:09 PM, Otis Gospodnetic < > > > > > otis.gospodne...@gmail.com> wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > So maybe I should I should have asked the Q explicitly: > > > > > > Could we commit the patch from > > > > > > https://issues.apache.org/jira/browse/KAFKA-1481 now that, I > hope, > > > > it's > > > > > > clear what problems the current MBean names can cause? > > > > > > > > > > > > Thanks, > > > > > > Otis > > > > > > -- > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log > > > Management > > > > > > Solr & Elasticsearch Support * http://sematext.com/ > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Sep 15, 2014 at 10:40 PM, Otis Gospodnetic < > > > > > > otis.gospodne...@gmail.com> wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > *Problem:* > > > > > > > Some Kafka 0.8.x MBeans have names composed of things like > > > <consumer > > > > > > > group>-<topic>-<metric name>. Note how dashes are used as > > > > delimiters. > > > > > > > When <consumer group> and <topic> don't contain the delimiter > > > > > character > > > > > > > all is good if you want to extract parts of this MBean name by > > > simply > > > > > > > splitting on the delimiter character. The problem is that > dashes > > > are > > > > > > > allowed in topic and group names, so this splitting doesn't > work. > > > > > > > Moreover, underscores are also used as delimiters, and they can > > > also > > > > be > > > > > > > used in things like topic names. > > > > > > > > > > > > > > *Example*: > > > > > > > This MBean's name is composed of <consumer > > > > group>-<topic>-BytesPerSec: > > > > > > > > > > > > > > kafka.consumer:type="ConsumerTopicMetrics", > > > > > name="*myGroup**-myTopic**-* > > > > > > > BytesPerSec" > > > > > > > > > > > > > > Here we can actually split on "-" and extract all 3 parts from > > the > > > > > MBean > > > > > > > name:: > > > > > > > * consumer group ('*myGroup*') > > > > > > > * topic ('*myTopic*') > > > > > > > * metric (‘BytesPerSec’) > > > > > > > > > > > > > > All good! > > > > > > > > > > > > > > But imagine if I named the group: *my-Group* > > > > > > > And if I named the topic: *my-Topic* > > > > > > > > > > > > > > Then we'd have: > > > > > > > kafka.consumer:type="ConsumerTopicMetrics", > > > > > > name="*my-Group**-my-Topic**-* > > > > > > > BytesPerSec" > > > > > > > > > > > > > > Now splitting on "-" would no longer work! To extract > "my-Group" > > > and > > > > > > > "my-Topic" and "BytesPerSec" parts I would have to know the > > > specific > > > > > > group > > > > > > > name and topic name to look for and could not use generic > > approach > > > of > > > > > > just > > > > > > > splitting the MBean name on the delimiter. > > > > > > > > > > > > > > *Solution*: > > > > > > > The patch in https://issues.apache.org/jira/browse/KAFKA-1481 > > > > replaces > > > > > > > all _ and - characters where they are used as delimiters in > MBean > > > > names > > > > > > > with a "|" character. Because the "I" character is not allowed > > in > > > > > topic > > > > > > > names, consumer groups, host names, splitting on this new and > > > unified > > > > > > > delimiter works. > > > > > > > > > > > > > > I hope this explains the problem, the solution, and that this > can > > > > make > > > > > it > > > > > > > in the next 0.8.x. > > > > > > > > > > > > > > Otis > > > > > > > -- > > > > > > > Monitoring * Alerting * Anomaly Detection * Centralized Log > > > > Management > > > > > > > Solr & Elasticsearch Support * http://sematext.com/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
