The magic byte is already in the Kafka Message data format, all you need to do is upon reading the magic byte for Avro, use your Avro resolver which treat the first 16 bytes as the schemaId to read the schema.
Guozhang On Tue, Aug 20, 2013 at 10:54 AM, Mark <static.void....@gmail.com> wrote: > So basically prepend the information in the header of each message? > Wouldn't this require changes in all readers/writers? > > On Aug 20, 2013, at 9:19 AM, Guozhang Wang <wangg...@gmail.com> wrote: > > > Hello Mark, > > > > The Kafka message format starts with a magic byte indicating what kind of > > serialization is used for this message. And if this byte indicates Avro, > > you can layout your message as starting with the schemaId and then > followed > > by message payload. Upon consumption, you can first get the schemaId, > query > > Avro for the schema given the id, and then use schema to deserialize the > > message. > > > > Guozhang > > > > > > On Tue, Aug 20, 2013 at 7:08 AM, Mark <static.void....@gmail.com> wrote: > > > >> Can someone break down how message serialization would work with Avro? > >> I've read instead of adding a schema to every single event it would be > wise > >> to add some sort of fingerprint with each message to identify which > schema > >> it should used. What I'm having trouble understanding is, how do we read > >> the fingerprint without a schema? Don't we need the schema to > deserialize? > >> Same question goes for working with Hadoop.. how does the input format > >> know which schema to use? > >> > >> Thanks > > > > > > > > > > -- > > -- Guozhang > > -- -- Guozhang
