"." seems natural for grouping topic names. +1 for 2) going forward only without breaking previously created topics with "_" though that might require us to patch the code somewhat awkwardly till we phase it out a couple (purposely left vague to stay out of Ewen's wrath :-)) versions later.
On Fri, Jul 10, 2015 at 2:02 PM, Gwen Shapira <gshap...@cloudera.com> wrote: > I don't think we should break existing topics. Just disallow new > topics going forward. > > Agree that having both is horrible, but we should have a solution that > fails when you run "kafka_topics.sh --create", not when you configure > Ganglia. > > Gwen > > On Fri, Jul 10, 2015 at 1:53 PM, Jay Kreps <j...@confluent.io> wrote: > > Unfortunately '.' is pretty common too. I agree that it is perverse, but > > people seem to do it. Breaking all the topics with '.' in the name seems > > like it could be worse than combining metrics for people who have a > > 'foo_bar' AND 'foo.bar' (and after all, having both is DEEPLY perverse, > > no?). > > > > Where is our Dean of Compatibility, Ewen, on this? > > > > -Jay > > > > On Fri, Jul 10, 2015 at 1:32 PM, Todd Palino <tpal...@gmail.com> wrote: > > > >> My selfish point of view is that we do #1, as we use "_" extensively in > >> topic names here :) I also happen to think it's the right choice, > >> specifically because "." has more special meanings, as you noted. > >> > >> -Todd > >> > >> > >> On Fri, Jul 10, 2015 at 1:30 PM, Gwen Shapira <gshap...@cloudera.com> > >> wrote: > >> > >> > Unintentional side effect from allowing IP addresses in consumer > client > >> > IDs :) > >> > > >> > So the question is, what do we do now? > >> > > >> > 1) disallow "." > >> > 2) disallow "_" > >> > 3) find a reversible way to encode "." and "_" that won't break > existing > >> > metrics > >> > 4) all of the above? > >> > > >> > btw. it looks like "." and ".." are currently valid. Topic names are > >> > used for directories, right? this sounds like fun :) > >> > > >> > I vote for option #1, although if someone has a good idea for #3 it > >> > will be even better. > >> > > >> > Gwen > >> > > >> > > >> > > >> > On Fri, Jul 10, 2015 at 1:22 PM, Grant Henke <ghe...@cloudera.com> > >> wrote: > >> > > Found it was added here: > >> https://issues.apache.org/jira/browse/KAFKA-697 > >> > > > >> > > On Fri, Jul 10, 2015 at 3:18 PM, Todd Palino <tpal...@gmail.com> > >> wrote: > >> > > > >> > >> This was definitely changed at some point after KAFKA-495. The > >> question > >> > is > >> > >> when and why. > >> > >> > >> > >> Here's the relevant code from that patch: > >> > >> > >> > >> =================================================================== > >> > >> --- core/src/main/scala/kafka/utils/Topic.scala (revision 1390178) > >> > >> +++ core/src/main/scala/kafka/utils/Topic.scala (working copy) > >> > >> @@ -21,24 +21,21 @@ > >> > >> import util.matching.Regex > >> > >> > >> > >> object Topic { > >> > >> + val legalChars = "[a-zA-Z0-9_-]" > >> > >> > >> > >> > >> > >> > >> > >> -Todd > >> > >> > >> > >> > >> > >> On Fri, Jul 10, 2015 at 1:02 PM, Grant Henke <ghe...@cloudera.com> > >> > wrote: > >> > >> > >> > >> > kafka.common.Topic shows that currently period is a valid > character > >> > and I > >> > >> > have verified I can use kafka-topics.sh to create a new topic > with a > >> > >> > period. > >> > >> > > >> > >> > > >> > >> > AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK > currently > >> > uses > >> > >> > Topic.validate before writing to Zookeeper. > >> > >> > > >> > >> > Should period character support be removed? I was under the same > >> > >> impression > >> > >> > as Gwen, that a period was used by many as a way to "group" > topics. > >> > >> > > >> > >> > The code is pasted below since its small: > >> > >> > > >> > >> > object Topic { > >> > >> > val legalChars = "[a-zA-Z0-9\\._\\-]" > >> > >> > private val maxNameLength = 255 > >> > >> > private val rgx = new Regex(legalChars + "+") > >> > >> > > >> > >> > val InternalTopics = Set(OffsetManager.OffsetsTopicName) > >> > >> > > >> > >> > def validate(topic: String) { > >> > >> > if (topic.length <= 0) > >> > >> > throw new InvalidTopicException("topic name is illegal, > can't > >> be > >> > >> > empty") > >> > >> > else if (topic.equals(".") || topic.equals("..")) > >> > >> > throw new InvalidTopicException("topic name cannot be > \".\" or > >> > >> > \"..\"") > >> > >> > else if (topic.length > maxNameLength) > >> > >> > throw new InvalidTopicException("topic name is illegal, > can't > >> be > >> > >> > longer than " + maxNameLength + " characters") > >> > >> > > >> > >> > rgx.findFirstIn(topic) match { > >> > >> > case Some(t) => > >> > >> > if (!t.equals(topic)) > >> > >> > throw new InvalidTopicException("topic name " + topic > + " > >> is > >> > >> > illegal, contains a character other than ASCII alphanumerics, > '.', > >> '_' > >> > >> and > >> > >> > '-'") > >> > >> > case None => throw new InvalidTopicException("topic name " > + > >> > topic > >> > >> + > >> > >> > " is illegal, contains a character other than ASCII > alphanumerics, > >> > '.', > >> > >> > '_' and '-'") > >> > >> > } > >> > >> > } > >> > >> > } > >> > >> > > >> > >> > On Fri, Jul 10, 2015 at 2:50 PM, Todd Palino <tpal...@gmail.com> > >> > wrote: > >> > >> > > >> > >> > > I had to go look this one up again to make sure - > >> > >> > > https://issues.apache.org/jira/browse/KAFKA-495 > >> > >> > > > >> > >> > > The only valid character names for topics are alphanumeric, > >> > underscore, > >> > >> > and > >> > >> > > dash. A period is not supposed to be a valid character to use. > If > >> > >> you're > >> > >> > > seeing them, then one of two things have happened: > >> > >> > > > >> > >> > > 1) You have topic names that are grandfathered in from before > that > >> > >> patch > >> > >> > > 2) The patch is not working properly and there is somewhere in > the > >> > >> broker > >> > >> > > that the standard is not being enforced. > >> > >> > > > >> > >> > > -Todd > >> > >> > > > >> > >> > > > >> > >> > > On Fri, Jul 10, 2015 at 12:13 PM, Brock Noland < > br...@apache.org> > >> > >> wrote: > >> > >> > > > >> > >> > > > On Fri, Jul 10, 2015 at 11:34 AM, Gwen Shapira < > >> > >> gshap...@cloudera.com> > >> > >> > > > wrote: > >> > >> > > > > Hi Kafka Fans, > >> > >> > > > > > >> > >> > > > > If you have one topic named "kafka_lab_2" and the other > named > >> > >> > > > > "kafka.lab.2", the topic level metrics will be named > >> kafka_lab_2 > >> > >> for > >> > >> > > > > both, effectively making it impossible to monitor them > >> properly. > >> > >> > > > > > >> > >> > > > > The reason this happens is that using "." in topic names is > >> > pretty > >> > >> > > > > common, especially as a way to group topics into data > centers, > >> > >> > > > > relevant apps, etc - basically a work-around to our current > >> > lack of > >> > >> > > > > name spaces. However, most metric monitoring systems using > "." > >> > to > >> > >> > > > > annotate hierarchy, so to avoid issues around metric names, > >> > Kafka > >> > >> > > > > replaces the "." in the name with an underscore. > >> > >> > > > > > >> > >> > > > > This generates good metric names, but creates the problem > with > >> > name > >> > >> > > > collisions. > >> > >> > > > > > >> > >> > > > > I'm wondering if it makes sense to simply limit the range > of > >> > >> > > > > characters permitted in a topic name and disallow "_"? > >> Obviously > >> > >> > > > > existing topics will need to remain as is, which is a bit > >> > awkward. > >> > >> > > > > >> > >> > > > Interesting problem! Many if not most users I personally am > >> aware > >> > of > >> > >> > > > use "_" as a separator in topic names. I am sure that many > users > >> > >> would > >> > >> > > > be quite surprised by this limitation. With that said, I am > sure > >> > >> > > > they'd transition accordingly. > >> > >> > > > > >> > >> > > > > > >> > >> > > > > If anyone has better backward-compatible solutions to this, > >> I'm > >> > all > >> > >> > > ears > >> > >> > > > :) > >> > >> > > > > > >> > >> > > > > Gwen > >> > >> > > > > >> > >> > > > >> > >> > > >> > >> > > >> > >> > > >> > >> > -- > >> > >> > Grant Henke > >> > >> > Solutions Consultant | Cloudera > >> > >> > ghe...@cloudera.com | twitter.com/gchenke | > >> > linkedin.com/in/granthenke > >> > >> > > >> > >> > >> > > > >> > > > >> > > > >> > > -- > >> > > Grant Henke > >> > > Solutions Consultant | Cloudera > >> > > ghe...@cloudera.com | twitter.com/gchenke | > linkedin.com/in/granthenke > >> > > >> > -- Thanks, Neha
