Re: KafkaAvroSerializer to produce to a single topic with different schemas used for records

2017-01-30 Thread Mike Cargal
If it comes to that we may consider it. However will will have a LOT of different schemas coming through and new ones added frequently. (Seems we’ve also seen issues that the Schema Registry doesn’t allow references to anything not in the same “file” for lack of a better term, that would become

Re: KafkaAvroSerializer to produce to a single topic with different schemas used for records

2017-01-30 Thread Andy Chambers
How about defining an avro union type containing all the schemas you wish to put on this topic (the schemas themselves could be defined independently and then bundled into an "uber-schema" at build time)? That means any messages you put on the topic must match one of the schemas defined in the uni

Re: KafkaAvroSerializer to produce to a single topic with different schemas used for records

2017-01-30 Thread Mike Cargal
This helps some. W’re planning to write a non-homogeneous set of records to a single topic (to preserve order). There would be no compatibility between records of different types. I assume that if I set the schema compatibility for this subject to “none” this would not be a problem. (can you

Re: KafkaAvroSerializer to produce to a single topic with different schemas used for records

2017-01-30 Thread Gerard Klijs
Not really, as you can update the schema, and have multiple of them at the same time. By default each schema has to backwards compatible, so you do have to exclude the specific topic you use with different schema's. With every write, the 'id' of the schema used is also written, so when you deserial

KafkaAvroSerializer to produce to a single topic with different schemas used for records

2017-01-29 Thread Mike Cargal
I was just looking into using KafkaAvroSerializer to produce records to a Kafka topic. We are interested because the wire format has a reference to the schema so we don’t have to schema lookup information independently. We plan to keep a single topic that contain records using many different s