Hey Devin, *"Kafka conforms to the JSON Schema specification"* Only when using Confluent's Schema Registry.
*"if a producer makes a change or omission, such as in a value used for tracking, it might not surface until way down the line"* So let me understand this: Although the producer has a schema, it does not use it for validation of JSON (as would implicitly occur for Avro? Is this correct? I agree that robust support for schema, certainly at the edges, is a cornerstone for a data system. I also agree that it would be better to adopt existing standards rather than implement them in a bespoke manner. I'd be interested to hear your thoughts on concrete improvements that you believe would be necessary - for example: * Producer validation of JSON occurs using "JSON Schema" * Evolutions of JSON Schema conform to ... * Users can declare topic schema using a JSON Schema document * Users can query topic schema and have a JSON schema document returned to them Thanks, Elliot. On Thu, 10 Nov 2022 at 16:51, Devin Bost <devin.b...@gmail.com> wrote: > One of the areas where Kafka has an advantage over Pulsar is around data > quality. Kafka conforms to the JSON Schema specification, which enables > integration with any technology that conforms to the standard, such as for > data validation, discoverability, lineage, versioning, etc. > Pulsar's implementation is non-compliant with the standard, and producers > and consumers have no built-in way in Pulsar to validate that values in > their messages match expectations. As a consequence, if a producer makes a > change or omission, such as in a value used for tracking, it might not > surface until way down the line, and then it can be very difficult to track > down the source of the problem, which kills the agility of teams > responsible for maintaining apps using Pulsar. It's also bad PR because > then incidents are associated with Pulsar, even though the business might > not understand that the data problem wasn't necessarily caused by Pulsar. > > What's the right way for us to address this problem? > > -- > Devin Bost > Sent from mobile > Cell: 801-400-4602 > -- Elliot West Senior Platform Engineer elliot.w...@streamnative.io streamnative.io <https://github.com/streamnative> <https://www.linkedin.com/company/streamnative> <https://twitter.com/streamnativeio>
