Hi Jason, that is great to hear =8^) I just uploaded a patch against trunk for the 0.8.1 release. Once another committer gives it a review && +1 then I will commit it to trunk || make changes if need be.
Thanks! /******************************************* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> ********************************************/ On Fri, Jan 3, 2014 at 1:31 PM, Jason Rosenberg <j...@squareup.com> wrote: > Thanks Joe, > > I can confirm that your patch works for me, as applied to 0.8.0. > > Jason > > On Fri, Dec 20, 2013 at 6:28 PM, Jason Rosenberg <j...@squareup.com> wrote: > > Thanks Joe, > > > > I generally build locally, and upload to our maven proxy (using a custom > > pom). > > > > I haven't yet had luck using maven central (although, I might upgrade to > the > > 2.10 version, in which case I understand it to be in better shape?). > > > > I containerize the broker (and all the producers and consumers), so I use > > the kafka jar directly. > > > > I think if you do the patch against 0.8, I can apply and use. > Ultimately, > > I'll upgrade to 0.8.1, once that's in a beta release state. > > > > Thanks again, > > > > Jason > > > > > > > > > > On Fri, Dec 20, 2013 at 10:29 AM, Joe Stein <joe.st...@stealth.ly> > wrote: > >> > >> Hey Jason, I was able to reproduce the issue and have a fix in hand to > >> test > >> later today. If it looks good I will post the patch. I am going to-do > the > >> patch against 0.8 branch first. How do you deploy and use libraries? Is > >> it > >> download broker and use maven central? > >> > >> /******************************************* > >> Joe Stein > >> Founder, Principal Consultant > >> Big Data Open Source Security LLC > >> http://www.stealth.ly > >> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > >> ********************************************/ > >> > >> > >> On Wed, Dec 18, 2013 at 4:13 PM, Jason Rosenberg <j...@squareup.com> > wrote: > >> > >> > thanks Joe! > >> > > >> > > >> > On Wed, Dec 18, 2013 at 11:05 AM, Joe Stein <joe.st...@stealth.ly> > >> > wrote: > >> > > >> > > Hey Jason, I have someone looking into it now (they just started). > >> > > > >> > > I can look at it on Friday or if I finish up what I am working on > for > >> > > tomorrow then sooner. > >> > > > >> > > /******************************************* > >> > > Joe Stein > >> > > Founder, Principal Consultant > >> > > Big Data Open Source Security LLC > >> > > http://www.stealth.ly > >> > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > >> > > ********************************************/ > >> > > > >> > > > >> > > On Wed, Dec 18, 2013 at 8:15 AM, Jason Rosenberg <j...@squareup.com> > >> > wrote: > >> > > > >> > > > Joe, > >> > > > > >> > > > I think the java code I listed in the Jira ticket should reproduce > >> > > > the > >> > > > issue directly, does that not work? > >> > > > > >> > > > Jason > >> > > > > >> > > > > >> > > > On Tue, Dec 17, 2013 at 9:49 AM, Joe Stein <joe.st...@stealth.ly> > >> > wrote: > >> > > > > >> > > > > Hi Jason, I just replied on the ticket. If it is a bug the > update > >> > > > > to > >> > > > > create new filter or fix as bug, same. > >> > > > > > >> > > > > Can you post some code to help reproduce the problem? so apples > >> > > > > to > >> > > > apples > >> > > > > and such, thanks! > >> > > > > > >> > > > > /******************************************* > >> > > > > Joe Stein > >> > > > > Founder, Principal Consultant > >> > > > > Big Data Open Source Security LLC > >> > > > > http://www.stealth.ly > >> > > > > Twitter: @allthingshadoop > >> > > > > <http://www.twitter.com/allthingshadoop> > >> > > > > ********************************************/ > >> > > > > > >> > > > > > >> > > > > On Tue, Dec 17, 2013 at 1:16 AM, Jason Rosenberg > >> > > > > <j...@squareup.com> > >> > > > wrote: > >> > > > > > >> > > > > > Ping.... > >> > > > > > > >> > > > > > Any thoughts on this? > >> > > > > > > >> > > > > > Seems like a bug, but then again, we're not sure what the > >> > > > > > expected > >> > > > > behavior > >> > > > > > for regexes should be here (e.g. is there a way to whitelist > >> > > > > > topics > >> > > > with > >> > > > > a > >> > > > > > filter that looks for a leading substring, but then blocks > >> > subsequent > >> > > > > > substrings)? E.g. apply a blacklist to a whitelist :)..... > >> > > > > > > >> > > > > > Jason > >> > > > > > > >> > > > > > > >> > > > > > On Thu, Dec 12, 2013 at 1:01 PM, Jason Rosenberg > >> > > > > > <j...@squareup.com > >> > > > >> > > > > wrote: > >> > > > > > > >> > > > > > > All, I've filed: > >> > https://issues.apache.org/jira/browse/KAFKA-1180 > >> > > > > > > > >> > > > > > > We are needing to create a stream selector that essentially > >> > > combines > >> > > > > the > >> > > > > > > logic of the BlackList and WhiteList classes. That is, we > >> > > > > > > want > >> > to > >> > > > > > select a > >> > > > > > > topic that contains a certain prefix, as long as it doesn't > >> > > > > > > also > >> > > > > contain > >> > > > > > a > >> > > > > > > secondary string. > >> > > > > > > > >> > > > > > > This should be easy to do with ordinary java Regex's, but > >> > > > > > > we're > >> > > > running > >> > > > > > > into some issues, trying to do this with the WhiteList class > >> > only. > >> > > > > > > > >> > > > > > > We have a pattern that uses negative lookahead, like this: > >> > > > > > > > >> > > > > > > "test-(?!bad\\b)[\\w]+" > >> > > > > > > > >> > > > > > > So this should select a topic like: "test-good", but > exclude a > >> > > topic > >> > > > > like > >> > > > > > > "test-bad", and also exclude a topic without the "test" > >> > > > > > > prefix, > >> > > like > >> > > > > > > "foo-bar". > >> > > > > > > > >> > > > > > > Instead, what we see is a NullPointerException in the > >> > > > ConsumerIterator, > >> > > > > > > and the consumer just hangs, after sending a topic of > >> > 'test-topic' > >> > > > > > followed > >> > > > > > > by 'test-bad': > >> > > > > > > > >> > > > > > > 21700 > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > [ConsumerFetcherThread-group1_square-1a7ac0.local-1386869343370-dc19c7dc-0-1946108683] > >> > > > > > > ERROR kafka.consumer.ConsumerFetcherThread - > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > [ConsumerFetcherThread-group1_square-1a7ac0.local-1386869343370-dc19c7dc-0-1946108683], > >> > > > > > > Error due to > >> > > > > > > kafka.common.KafkaException: error processing data for > >> > > > > > > partition > >> > > > > > > [test-bad,0] offset 0 > >> > > > > > > at > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:137) > >> > > > > > > at > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:109) > >> > > > > > > at > scala.collection.immutable.Map$Map1.foreach(Map.scala:105) > >> > > > > > > at > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply$mcV$sp(AbstractFetcherThread.scala:109) > >> > > > > > > at > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:109) > >> > > > > > > at > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:109) > >> > > > > > > at kafka.utils.Utils$.inLock(Utils.scala:565) > >> > > > > > > at > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:108) > >> > > > > > > at > >> > > > > > > > >> > > > > > >> > > > >> > > > kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:86) > >> > > > > > > at > >> > kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) > >> > > > > > > Caused by: java.lang.NullPointerException > >> > > > > > > at > >> > > > > > >> > kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:60) > >> > > > > > > at > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:49) > >> > > > > > > at > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:128) > >> > > > > > > ... 9 more > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > >
