Re: [DISCUSS] KIP-128: Add ByteArrayConverter for Kafka Connect

2017-03-06 Thread Guozhang Wang
Sounds good. Thanks. On Wed, Mar 1, 2017 at 8:21 PM, Gwen Shapira wrote: > Yeah, you are right, it is best not to make converters actively break the > data structures :) > > On Wed, Mar 1, 2017 at 4:55 PM, Ewen Cheslack-Postava > wrote: > > > Guozhang, > > > > I'm fine w/ adjusting if people wa

Re: [DISCUSS] KIP-128: Add ByteArrayConverter for Kafka Connect

2017-03-01 Thread Gwen Shapira
Yeah, you are right, it is best not to make converters actively break the data structures :) On Wed, Mar 1, 2017 at 4:55 PM, Ewen Cheslack-Postava wrote: > Guozhang, > > I'm fine w/ adjusting if people want to, but it ends up being more code > since we also need to convert SerializationException

Re: [DISCUSS] KIP-128: Add ByteArrayConverter for Kafka Connect

2017-03-01 Thread Ewen Cheslack-Postava
Guozhang, I'm fine w/ adjusting if people want to, but it ends up being more code since we also need to convert SerializationExceptions to DataExceptions and the only thing the toConnectData method even does is specific to Connect (adding the SchemaAndValue). Gwen -- isn't that an SMT? ExtractFie

Re: [DISCUSS] KIP-128: Add ByteArrayConverter for Kafka Connect

2017-03-01 Thread Gwen Shapira
Hi Ewen, Thanks for the KIP, I think it will be useful :) I'm just wondering if we can add support not just for bytes schema, but also for a struct that contains bytes? I'm thinking of the scenario of using a connector to grab BLOBs out of a DB - I think you end up with this structure if you use

Re: [DISCUSS] KIP-128: Add ByteArrayConverter for Kafka Connect

2017-02-28 Thread Guozhang Wang
Generally I'd prefer not duplicating functional logic as sometimes you may miss to sync one of them when you change the other. I understand for this specific case such scenario may never happen as the logic is quite simple and static, but still sounds a good coding practice to me? Guozhang On T

Re: [DISCUSS] KIP-128: Add ByteArrayConverter for Kafka Connect

2017-02-28 Thread Ewen Cheslack-Postava
Guozhang, Did you look at the PR? I'm fine with doing that if we really think it's better, but since this is a config-less passthrough, it's actually just adding overhead to do that... -Ewen On Mon, Feb 27, 2017 at 11:47 AM, Guozhang Wang wrote: > Thanks Ewen, > > "use the corresponding serial

Re: [DISCUSS] KIP-128: Add ByteArrayConverter for Kafka Connect

2017-02-27 Thread Guozhang Wang
Thanks Ewen, "use the corresponding serializer internally and just add in the extra conversion steps for the data API" sounds good to me. Guozhang On Mon, Feb 27, 2017 at 8:24 AM, Ewen Cheslack-Postava wrote: > It's a different interface that's being implemented. The functionality is > the sa

Re: [DISCUSS] KIP-128: Add ByteArrayConverter for Kafka Connect

2017-02-27 Thread Ewen Cheslack-Postava
It's a different interface that's being implemented. The functionality is the same (since it's just a simple pass through), but we intentionally named Converters differently than Serializers since they do more work than Serializers (besides the normal serialization they also need to convert between

Re: [DISCUSS] KIP-128: Add ByteArrayConverter for Kafka Connect

2017-02-25 Thread Guozhang Wang
I'm wondering why we can't just use ByteArarySerde in o.a.k.common? Guozhang On Sat, Feb 25, 2017 at 2:25 PM, Ewen Cheslack-Postava wrote: > Hi all, > > I've added a pretty trivial KIP for adding a pass-through Converter for > Kafka Connect, similar to ByteArraySerializer/Deserializer. > > http

[DISCUSS] KIP-128: Add ByteArrayConverter for Kafka Connect

2017-02-25 Thread Ewen Cheslack-Postava
Hi all, I've added a pretty trivial KIP for adding a pass-through Converter for Kafka Connect, similar to ByteArraySerializer/Deserializer. https://cwiki.apache.org/confluence/display/KAFKA/KIP-128%3A+Add+ByteArrayConverter+for+Kafka+Connect This wasn't added with the framework originally becaus