Yes, that will be a separate release. Possibly an 0.8.2 beta-2, followed by
0.8.2 final.
Thanks,
Jun
On Wed, Nov 26, 2014 at 2:24 AM, Shlomi Hazan <shl...@viber.com> wrote:
> Jay, Jun,
> Thank you both for explaining. I understand this is important enough such
> that it must be done, and if so, the sooner the better.
> How will the change be released? a beta-2 or release candidate? I think
> that if possible, it should not overrun the already released version.
> Thank you guys for the hard work.
> Shlomi
>
> On Tue, Nov 25, 2014 at 7:37 PM, Jun Rao <jun...@gmail.com> wrote:
>
> > Bhavesh,
> >
> > This api change doesn't mean you need to change the format of the encoded
> > data. It simply moves the serialization logic from the application to a
> > pluggable serializer. As long as you preserve the serialization logic,
> the
> > consumer should still see the same bytes.
> >
> > If you are talking about how to evolve the data schema over time, that's
> a
> > separate story. Serialization libraries like Avro have better support on
> > schema evolution.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Nov 25, 2014 at 8:41 AM, Bhavesh Mistry <
> > mistry.p.bhav...@gmail.com>
> > wrote:
> >
> > > How will mix bag will work with Consumer side ? Entire site can not be
> > > rolled at once so Consumer will have to deals with New and Old
> Serialize
> > > Bytes ? This could be app team responsibility. Are you guys targeting
> > > 0.8.2 release, which may break customer who are already using new
> > producer
> > > API (beta version).
> > >
> > > Thanks,
> > >
> > > Bhavesh
> > >
> > > On Tue, Nov 25, 2014 at 8:29 AM, Manikumar Reddy <ku...@nmsworks.co.in
> >
> > > wrote:
> > >
> > > > +1 for this change.
> > > >
> > > > what about de-serializer class in 0.8.2? Say i am using new
> producer
> > > with
> > > > Avro and old consumer combination.
> > > > then i need to give custom Decoder implementation for Avro right?.
> > > >
> > > > On Tue, Nov 25, 2014 at 9:19 PM, Joe Stein <joe.st...@stealth.ly>
> > wrote:
> > > >
> > > > > The serializer is an expected use of the producer/consumer now and
> > > think
> > > > we
> > > > > should continue that support in the new client. As far as breaking
> > the
> > > > API
> > > > > it is why we released the 0.8.2-beta to help get through just these
> > > type
> > > > of
> > > > > blocking issues in a way that the community at large could be
> > involved
> > > in
> > > > > easier with a build/binaries to download and use from maven also.
> > > > >
> > > > > +1 on the change now prior to the 0.8.2 release.
> > > > >
> > > > > - Joe Stein
> > > > >
> > > > >
> > > > > On Mon, Nov 24, 2014 at 11:43 PM, Sriram Subramanian <
> > > > > srsubraman...@linkedin.com.invalid> wrote:
> > > > >
> > > > > > Looked at the patch. +1 from me.
> > > > > >
> > > > > > On 11/24/14 8:29 PM, "Gwen Shapira" <gshap...@cloudera.com>
> wrote:
> > > > > >
> > > > > > >As one of the people who spent too much time building Avro
> > > > repositories,
> > > > > > >+1
> > > > > > >on bringing serializer API back.
> > > > > > >
> > > > > > >I think it will make the new producer easier to work with.
> > > > > > >
> > > > > > >Gwen
> > > > > > >
> > > > > > >On Mon, Nov 24, 2014 at 6:13 PM, Jay Kreps <jay.kr...@gmail.com
> >
> > > > wrote:
> > > > > > >
> > > > > > >> This is admittedly late in the release cycle to make a change.
> > To
> > > > add
> > > > > to
> > > > > > >> Jun's description the motivation was that we felt it would be
> > > better
> > > > > to
> > > > > > >> change that interface now rather than after the release if it
> > > needed
> > > > > to
> > > > > > >> change.
> > > > > > >>
> > > > > > >> The motivation for wanting to make a change was the ability to
> > > > really
> > > > > be
> > > > > > >> able to develop support for Avro and other serialization
> > formats.
> > > > The
> > > > > > >> current status is pretty scattered--there is a schema
> repository
> > > on
> > > > an
> > > > > > >>Avro
> > > > > > >> JIRA and another fork of that on github, and a bunch of people
> > we
> > > > have
> > > > > > >> talked to have done similar things for other serialization
> > > systems.
> > > > It
> > > > > > >> would be nice if these things could be packaged in such a way
> > that
> > > > it
> > > > > > >>was
> > > > > > >> possible to just change a few configs in the producer and get
> > rich
> > > > > > >>metadata
> > > > > > >> support for messages.
> > > > > > >>
> > > > > > >> As we were thinking this through we realized that the new api
> we
> > > > were
> > > > > > >>about
> > > > > > >> to introduce was kind of not very compatable with this since
> it
> > > was
> > > > > just
> > > > > > >> byte[] oriented.
> > > > > > >>
> > > > > > >> You can always do this by adding some kind of wrapper api that
> > > wraps
> > > > > the
> > > > > > >> producer. But this puts us back in the position of trying to
> > > > document
> > > > > > >>and
> > > > > > >> support multiple interfaces.
> > > > > > >>
> > > > > > >> This also opens up the possibility of adding a
> MessageValidator
> > or
> > > > > > >> MessageInterceptor plug-in transparently so that you can do
> > other
> > > > > custom
> > > > > > >> validation on the messages you are sending which obviously
> > > requires
> > > > > > >>access
> > > > > > >> to the original object not the byte array.
> > > > > > >>
> > > > > > >> This api doesn't prevent using byte[] by configuring the
> > > > > > >> ByteArraySerializer it works as it currently does.
> > > > > > >>
> > > > > > >> -Jay
> > > > > > >>
> > > > > > >> On Mon, Nov 24, 2014 at 5:58 PM, Jun Rao <jun...@gmail.com>
> > > wrote:
> > > > > > >>
> > > > > > >> > Hi, Everyone,
> > > > > > >> >
> > > > > > >> > I'd like to start a discussion on whether it makes sense to
> > add
> > > > the
> > > > > > >> > serializer api back to the new java producer. Currently, the
> > new
> > > > > java
> > > > > > >> > producer takes a byte array for both the key and the value.
> > > While
> > > > > this
> > > > > > >> api
> > > > > > >> > is simple, it pushes the serialization logic into the
> > > application.
> > > > > > >>This
> > > > > > >> > makes it hard to reason about what type of data is being
> sent
> > to
> > > > > Kafka
> > > > > > >> and
> > > > > > >> > also makes it hard to share an implementation of the
> > serializer.
> > > > For
> > > > > > >> > example, to support Avro, the serialization logic could be
> > quite
> > > > > > >>involved
> > > > > > >> > since it might need to register the Avro schema in some
> remote
> > > > > > >>registry
> > > > > > >> and
> > > > > > >> > maintain a schema cache locally, etc. Without a
> serialization
> > > api,
> > > > > > >>it's
> > > > > > >> > impossible to share such an implementation so that people
> can
> > > > easily
> > > > > > >> reuse.
> > > > > > >> > We sort of overlooked this implication during the initial
> > > > discussion
> > > > > > >>of
> > > > > > >> the
> > > > > > >> > producer api.
> > > > > > >> >
> > > > > > >> > So, I'd like to propose an api change to the new producer by
> > > > adding
> > > > > > >>back
> > > > > > >> > the serializer api similar to what we had in the old
> producer.
> > > > > > >>Specially,
> > > > > > >> > the proposed api changes are the following.
> > > > > > >> >
> > > > > > >> > First, we change KafkaProducer to take generic types K and V
> > for
> > > > the
> > > > > > >>key
> > > > > > >> > and the value, respectively.
> > > > > > >> >
> > > > > > >> > public class KafkaProducer<K,V> implements Producer<K,V> {
> > > > > > >> >
> > > > > > >> > public Future<RecordMetadata> send(ProducerRecord<K,V>
> > > record,
> > > > > > >> Callback
> > > > > > >> > callback);
> > > > > > >> >
> > > > > > >> > public Future<RecordMetadata> send(ProducerRecord<K,V>
> > > > record);
> > > > > > >> > }
> > > > > > >> >
> > > > > > >> > Second, we add two new configs, one for the key serializer
> and
> > > > > another
> > > > > > >> for
> > > > > > >> > the value serializer. Both serializers will default to the
> > byte
> > > > > array
> > > > > > >> > implementation.
> > > > > > >> >
> > > > > > >> > public class ProducerConfig extends AbstractConfig {
> > > > > > >> >
> > > > > > >> > .define(KEY_SERIALIZER_CLASS_CONFIG, Type.CLASS,
> > > > > > >> > "org.apache.kafka.clients.producer.ByteArraySerializer",
> > > > > > >>Importance.HIGH,
> > > > > > >> > KEY_SERIALIZER_CLASS_DOC)
> > > > > > >> > .define(VALUE_SERIALIZER_CLASS_CONFIG, Type.CLASS,
> > > > > > >> > "org.apache.kafka.clients.producer.ByteArraySerializer",
> > > > > > >>Importance.HIGH,
> > > > > > >> > VALUE_SERIALIZER_CLASS_DOC);
> > > > > > >> > }
> > > > > > >> >
> > > > > > >> > Both serializers will implement the following interface.
> > > > > > >> >
> > > > > > >> > public interface Serializer<T> extends Configurable {
> > > > > > >> > public byte[] serialize(String topic, T data, boolean
> > > isKey);
> > > > > > >> >
> > > > > > >> > public void close();
> > > > > > >> > }
> > > > > > >> >
> > > > > > >> > This is more or less the same as what's in the old producer.
> > The
> > > > > > >>slight
> > > > > > >> > differences are (1) the serializer now only requires a
> > > > > parameter-less
> > > > > > >> > constructor; (2) the serializer has a configure() and a
> > close()
> > > > > method
> > > > > > >> for
> > > > > > >> > initialization and cleanup, respectively; (3) the
> serialize()
> > > > method
> > > > > > >> > additionally takes the topic and an isKey indicator, both of
> > > which
> > > > > are
> > > > > > >> > useful for things like schema registration.
> > > > > > >> >
> > > > > > >> > The detailed changes are included in KAFKA-1797. For
> > > > completeness, I
> > > > > > >>also
> > > > > > >> > made the corresponding changes for the new java consumer api
> > as
> > > > > well.
> > > > > > >> >
> > > > > > >> > Note that the proposed api changes are incompatible with
> > what's
> > > in
> > > > > the
> > > > > > >> > 0.8.2 branch. However, if those api changes are beneficial,
> > it's
> > > > > > >>probably
> > > > > > >> > better to include them now in the 0.8.2 release, rather than
> > > > later.
> > > > > > >> >
> > > > > > >> > I'd like to discuss mainly two things in this thread.
> > > > > > >> > 1. Do people feel that the proposed api changes are
> > reasonable?
> > > > > > >> > 2. Are there any concerns of including the api changes in
> the
> > > > 0.8.2
> > > > > > >>final
> > > > > > >> > release?
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> >
> > > > > > >> > Jun
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
