Re: [DISCUSS] Json libraries for Kafka

2015-07-15 Thread Ismael Juma
On Mon, Jul 13, 2015 at 10:05 PM, Ismael Juma wrote: > * It's not thread-safe (so external locks are needed to use it in a > concurrent environment) > I realise now that I didn't make the impact of this very clear. The current code is acquiring a JVM-wide lock for the duration of every Json pars

Re: [DISCUSS] Json libraries for Kafka

2015-07-14 Thread Jay Kreps
Yeah we really can't add any dependencies for that client jar. Maybe we could move the perf test to the tools jar though? -Jay On Tue, Jul 14, 2015 at 6:01 PM, Geoffrey Anderson wrote: > Hi all, my pull request here: > > https://github.com/apache/kafka/pull/70/files#diff-59f3fe36571d1eee9f923df

Re: [DISCUSS] Json libraries for Kafka

2015-07-14 Thread Geoffrey Anderson
Hi all, my pull request here: https://github.com/apache/kafka/pull/70/files#diff-59f3fe36571d1eee9f923df927a643eb would introduce a client-side dependency on the json-simple package. It is only used in a tool (VerifiableProducer.java), but it sounds like I should probably use Jackson instead? Fee

Re: [DISCUSS] Json libraries for Kafka

2015-07-14 Thread Jay Kreps
Ah, makes sense. Yes that addresses my concerns. -Jay On Tue, Jul 14, 2015 at 5:19 PM, Ismael Juma wrote: > Hi Jay, > > Comments inline. > > On Tue, Jul 14, 2015 at 11:04 PM, Jay Kreps wrote: > > > Is this going to become a dependency for core and then transitively for > the > > old clients? >

Re: [DISCUSS] Json libraries for Kafka

2015-07-14 Thread Ismael Juma
Hi Jay, Comments inline. On Tue, Jul 14, 2015 at 11:04 PM, Jay Kreps wrote: > Is this going to become a dependency for core and then transitively for the > old clients? That's right. > The current json library is definitely not great, but it does > parse json and it's not used in any contex

Re: [DISCUSS] Json libraries for Kafka

2015-07-14 Thread Ismael Juma
On Tue, Jul 14, 2015 at 10:46 PM, Joe Stein wrote: > Fasterxml/Jackson +1 to that. The scala databinds to case classes are gr8. > To be clear, case classes support would require the Scala module for Jackson and the Scala versions headache that goes with it (2.9 support is likely to be an issue,

Re: [DISCUSS] Json libraries for Kafka

2015-07-14 Thread Ismael Juma
Ewen, On Tue, Jul 14, 2015 at 10:41 PM, Ewen Cheslack-Postava wrote: > Currently the clients/server mismatch wouldn't be an issue since there are > no client-side uses of JSON, right? That said, if Copycat ends up included > in Kafka we'll need to provide at least one serializer which would be >

Re: [DISCUSS] Json libraries for Kafka

2015-07-14 Thread Joe Stein
Maybe after the existing scala clients are deprecated. ~ Joestein On Jul 14, 2015 6:04 PM, "Jay Kreps" wrote: > Is this going to become a dependency for core and then transitively for the > old clients? The current json library is definitely not great, but it does > parse json and it's not used

Re: [DISCUSS] Json libraries for Kafka

2015-07-14 Thread Jay Kreps
Is this going to become a dependency for core and then transitively for the old clients? The current json library is definitely not great, but it does parse json and it's not used in any context where performance is a concern. Because the older clients aren't well modularized, adding core dependen

Re: [DISCUSS] Json libraries for Kafka

2015-07-14 Thread Joe Stein
Fasterxml/Jackson +1 to that. The scala databinds to case classes are gr8. ~ Joestein On Jul 14, 2015 5:42 PM, "Ewen Cheslack-Postava" wrote: > Currently the clients/server mismatch wouldn't be an issue since there are > no client-side uses of JSON, right? That said, if Copycat ends up included

Re: [DISCUSS] Json libraries for Kafka

2015-07-14 Thread Ewen Cheslack-Postava
Currently the clients/server mismatch wouldn't be an issue since there are no client-side uses of JSON, right? That said, if Copycat ends up included in Kafka we'll need to provide at least one serializer which would be written in Java and I suspect some people would like JSON to be a candidate for

[DISCUSS] Json libraries for Kafka

2015-07-13 Thread Ismael Juma
Hi all, Kafka currently use scala.util.parsing.json.JSON as its json parser and it has a number of issues: * It encourages unsafe casts (returns `Option[Any]`) * It's slow (it relies on parser combinators under the hood) * It's not thread-safe (so external locks are needed to use it in a concurre