How will mix bag will work with Consumer side ? Entire site can not be
rolled at once so Consumer will have to deals with New and Old Serialize
Bytes ? This could be app team responsibility. Are you guys targeting
0.8.2 release, which may break customer who are already using new producer
API (beta version).
Thanks,
Bhavesh
On Tue, Nov 25, 2014 at 8:29 AM, Manikumar Reddy <ku...@nmsworks.co.in>
wrote:
> +1 for this change.
>
> what about de-serializer class in 0.8.2? Say i am using new producer with
> Avro and old consumer combination.
> then i need to give custom Decoder implementation for Avro right?.
>
> On Tue, Nov 25, 2014 at 9:19 PM, Joe Stein <joe.st...@stealth.ly> wrote:
>
> > The serializer is an expected use of the producer/consumer now and think
> we
> > should continue that support in the new client. As far as breaking the
> API
> > it is why we released the 0.8.2-beta to help get through just these type
> of
> > blocking issues in a way that the community at large could be involved in
> > easier with a build/binaries to download and use from maven also.
> >
> > +1 on the change now prior to the 0.8.2 release.
> >
> > - Joe Stein
> >
> >
> > On Mon, Nov 24, 2014 at 11:43 PM, Sriram Subramanian <
> > srsubraman...@linkedin.com.invalid> wrote:
> >
> > > Looked at the patch. +1 from me.
> > >
> > > On 11/24/14 8:29 PM, "Gwen Shapira" <gshap...@cloudera.com> wrote:
> > >
> > > >As one of the people who spent too much time building Avro
> repositories,
> > > >+1
> > > >on bringing serializer API back.
> > > >
> > > >I think it will make the new producer easier to work with.
> > > >
> > > >Gwen
> > > >
> > > >On Mon, Nov 24, 2014 at 6:13 PM, Jay Kreps <jay.kr...@gmail.com>
> wrote:
> > > >
> > > >> This is admittedly late in the release cycle to make a change. To
> add
> > to
> > > >> Jun's description the motivation was that we felt it would be better
> > to
> > > >> change that interface now rather than after the release if it needed
> > to
> > > >> change.
> > > >>
> > > >> The motivation for wanting to make a change was the ability to
> really
> > be
> > > >> able to develop support for Avro and other serialization formats.
> The
> > > >> current status is pretty scattered--there is a schema repository on
> an
> > > >>Avro
> > > >> JIRA and another fork of that on github, and a bunch of people we
> have
> > > >> talked to have done similar things for other serialization systems.
> It
> > > >> would be nice if these things could be packaged in such a way that
> it
> > > >>was
> > > >> possible to just change a few configs in the producer and get rich
> > > >>metadata
> > > >> support for messages.
> > > >>
> > > >> As we were thinking this through we realized that the new api we
> were
> > > >>about
> > > >> to introduce was kind of not very compatable with this since it was
> > just
> > > >> byte[] oriented.
> > > >>
> > > >> You can always do this by adding some kind of wrapper api that wraps
> > the
> > > >> producer. But this puts us back in the position of trying to
> document
> > > >>and
> > > >> support multiple interfaces.
> > > >>
> > > >> This also opens up the possibility of adding a MessageValidator or
> > > >> MessageInterceptor plug-in transparently so that you can do other
> > custom
> > > >> validation on the messages you are sending which obviously requires
> > > >>access
> > > >> to the original object not the byte array.
> > > >>
> > > >> This api doesn't prevent using byte[] by configuring the
> > > >> ByteArraySerializer it works as it currently does.
> > > >>
> > > >> -Jay
> > > >>
> > > >> On Mon, Nov 24, 2014 at 5:58 PM, Jun Rao <jun...@gmail.com> wrote:
> > > >>
> > > >> > Hi, Everyone,
> > > >> >
> > > >> > I'd like to start a discussion on whether it makes sense to add
> the
> > > >> > serializer api back to the new java producer. Currently, the new
> > java
> > > >> > producer takes a byte array for both the key and the value. While
> > this
> > > >> api
> > > >> > is simple, it pushes the serialization logic into the application.
> > > >>This
> > > >> > makes it hard to reason about what type of data is being sent to
> > Kafka
> > > >> and
> > > >> > also makes it hard to share an implementation of the serializer.
> For
> > > >> > example, to support Avro, the serialization logic could be quite
> > > >>involved
> > > >> > since it might need to register the Avro schema in some remote
> > > >>registry
> > > >> and
> > > >> > maintain a schema cache locally, etc. Without a serialization api,
> > > >>it's
> > > >> > impossible to share such an implementation so that people can
> easily
> > > >> reuse.
> > > >> > We sort of overlooked this implication during the initial
> discussion
> > > >>of
> > > >> the
> > > >> > producer api.
> > > >> >
> > > >> > So, I'd like to propose an api change to the new producer by
> adding
> > > >>back
> > > >> > the serializer api similar to what we had in the old producer.
> > > >>Specially,
> > > >> > the proposed api changes are the following.
> > > >> >
> > > >> > First, we change KafkaProducer to take generic types K and V for
> the
> > > >>key
> > > >> > and the value, respectively.
> > > >> >
> > > >> > public class KafkaProducer<K,V> implements Producer<K,V> {
> > > >> >
> > > >> > public Future<RecordMetadata> send(ProducerRecord<K,V> record,
> > > >> Callback
> > > >> > callback);
> > > >> >
> > > >> > public Future<RecordMetadata> send(ProducerRecord<K,V>
> record);
> > > >> > }
> > > >> >
> > > >> > Second, we add two new configs, one for the key serializer and
> > another
> > > >> for
> > > >> > the value serializer. Both serializers will default to the byte
> > array
> > > >> > implementation.
> > > >> >
> > > >> > public class ProducerConfig extends AbstractConfig {
> > > >> >
> > > >> > .define(KEY_SERIALIZER_CLASS_CONFIG, Type.CLASS,
> > > >> > "org.apache.kafka.clients.producer.ByteArraySerializer",
> > > >>Importance.HIGH,
> > > >> > KEY_SERIALIZER_CLASS_DOC)
> > > >> > .define(VALUE_SERIALIZER_CLASS_CONFIG, Type.CLASS,
> > > >> > "org.apache.kafka.clients.producer.ByteArraySerializer",
> > > >>Importance.HIGH,
> > > >> > VALUE_SERIALIZER_CLASS_DOC);
> > > >> > }
> > > >> >
> > > >> > Both serializers will implement the following interface.
> > > >> >
> > > >> > public interface Serializer<T> extends Configurable {
> > > >> > public byte[] serialize(String topic, T data, boolean isKey);
> > > >> >
> > > >> > public void close();
> > > >> > }
> > > >> >
> > > >> > This is more or less the same as what's in the old producer. The
> > > >>slight
> > > >> > differences are (1) the serializer now only requires a
> > parameter-less
> > > >> > constructor; (2) the serializer has a configure() and a close()
> > method
> > > >> for
> > > >> > initialization and cleanup, respectively; (3) the serialize()
> method
> > > >> > additionally takes the topic and an isKey indicator, both of which
> > are
> > > >> > useful for things like schema registration.
> > > >> >
> > > >> > The detailed changes are included in KAFKA-1797. For
> completeness, I
> > > >>also
> > > >> > made the corresponding changes for the new java consumer api as
> > well.
> > > >> >
> > > >> > Note that the proposed api changes are incompatible with what's in
> > the
> > > >> > 0.8.2 branch. However, if those api changes are beneficial, it's
> > > >>probably
> > > >> > better to include them now in the 0.8.2 release, rather than
> later.
> > > >> >
> > > >> > I'd like to discuss mainly two things in this thread.
> > > >> > 1. Do people feel that the proposed api changes are reasonable?
> > > >> > 2. Are there any concerns of including the api changes in the
> 0.8.2
> > > >>final
> > > >> > release?
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > Jun
> > > >> >
> > > >>
> > >
> > >
> >
>
