There is a little nuance to this topic (hehe). When it comes down to it, yes, each unique avro schema has a unique id associated with it. That id can be used across multiple different topics. The enforcement of which schemas are allowed in a particular topic comes down to the combination of the subject (usually topic-name-key/value) and version (the version itself starts at 1 inside the subject, and itself has an id that ties to the globally unique schema id). So, yes, you can have multiple schemas within the same topic, and thats perfectly fine, so long as you're correctly configuring the schema registry.
Whether or not a schema is allowed to be registered for a particular subject is dependent upon the type of avro compatilibty enforced. There are 4 types: BACKWARD, FORWARD, FULL (combines forward and backward), and NONE. The schema registry is going to evaluate the schema being published to the history of schemas it knows about in the past for that subject + version combination. If the schema is evolved correctly according to the particular type configured in the schema registry, it will be allowed. So, if you select NONE as the compatibility type the schema registry will allow any schema to be registered, even if they are not compatible because you've informed the registry not to care. So, you should really choose amongst backward, forward, and full. I use FULL in production because the data being written is long lived, and will have multiple readers and writers of the data, and the data needs to be passively evolved. Backward and forward can be fine too, just depending upon the needs of the data being produced and consumed. On Thu, Aug 17, 2017 at 12:22 PM, Tauzell, Dave < [email protected]> wrote: > Hmm, I think you are right that you cannot have multiple schemas on the > same topic. > > -Dave > > > -----Original Message----- > From: Sreejith S [mailto:[email protected]] > Sent: Thursday, August 17, 2017 11:42 AM > To: [email protected] > Subject: RE: Different Schemas on same Kafka Topic > > Hi Dave, > > Would like to get a clarity on one thing. If i register more than one > schema for a topic, i am providing topic-key, topic-value to the schema > registry. > > Id is created by schema registry and it will create different version of > different schema. Still all schema have same id. Am i right ? > > If so, all avro messages holds same id. Then how multiple schemas on same > topic possble ? > > Please clarify > > Thanks, > Sreejith > > On 17-Aug-2017 9:49 pm, "Tauzell, Dave" <[email protected]> > wrote: > > > > How does consumer know A is the avro class when there could be > > > other > > classes like B,C and D denoting different schemas?. > > > > There isn't a good way. One option is to have an avro wrapper that > > contains type, version and data fields. Then you wrap everything. > > Another option is to do what Kafka is doing and prepend some sort of > > fixed length value to all messages that have the schema and version > > you are using for that message. > > > > -Dave > > > > -----Original Message----- > > From: Shajahan, Nishanth [mailto:[email protected]] > > Sent: Thursday, August 17, 2017 11:02 AM > > To: [email protected] > > Subject: RE: Different Schemas on same Kafka Topic > > > > Thanks Dave. We may not want to start using schema registery immediately > . > > We would have java producers and consumers . I might also go with > > using byte messages but when consumer de serilize how can they map > > the byte[] to the correct Avro object For example: > > > > KafkaConsumer<String,A> consumer = new > > KafkaConsumer<>(consumerConfig,new > > StringDeserializer(),new AvroDeserializer<>(A)); > > > > How does consumer know A is the avro class when there could be other > > classes like B,C and D denoting different schemas?. > > > > > > -Nishanth > > > > -----Original Message----- > > From: Tauzell, Dave [mailto:[email protected]] > > Sent: Thursday, August 17, 2017 8:30 AM > > To: [email protected] > > Subject: RE: Different Schemas on same Kafka Topic > > > > It does. The way it works is that the Avro serializer precedes each > > message with a two-byte integer that references a schema id in the > > confluent schema registry. The Avro deserializer looks at this value to > > determine which schema to de-serialize with. In order for this to work > > you need to use the java client on both ends and have the schema > > registry setup. > > > > We have some slightly different needs ( including non-java languages) > > so we are just using byte messages and then have our applications do > > the serialization and deserialization. > > > > -Dave > > > > -----Original Message----- > > From: Shajahan, Nishanth [mailto:[email protected]] > > Sent: Wednesday, August 16, 2017 5:13 PM > > To: [email protected] > > Subject: Different Schemas on same Kafka Topic > > > > Hello, > > > > Does kafka support writing different avro record types(very different > > schema) to the same topic . I guess we would have to write our own > > avro serializer and de serializer to do this ?. Is there a preferred > > way to do this ?.It would be great if some one can point me in the right > direction. > > > > Thanks, > > Nishanth > > > > This e-mail and any files transmitted with it are confidential, may > > contain sensitive information, and are intended solely for the use of > > the individual or entity to whom they are addressed. If you have > > received this e-mail in error, please notify the sender by reply > > e-mail immediately and destroy all copies of the e-mail and any > attachments. > > > > > > > This e-mail and any files transmitted with it are confidential, may > contain sensitive information, and are intended solely for the use of the > individual or entity to whom they are addressed. If you have received this > e-mail in error, please notify the sender by reply e-mail immediately and > destroy all copies of the e-mail and any attachments. >
