hi, Devin: the first Kafka doesn't support schema. `confluent `does. pulsar schema supports validation and versioning. Are you encountering a schema version caused by automatic registration, and the data source is not clear? I think you can turn off the producer's automatic registration schema, and control the schema changes through the management side. doc: https://pulsar.apache.org/docs/2.10.x/schema-manage#schema-autoupdate
Thanks, bo Elliot West <elliot.w...@streamnative.io.invalid> 于2022年11月14日周一 20:14写道： > > While we can get caught up in the specifics of exactly how JSON Schema is > supported in the Kafka ecosystem, it is ultimately possible if desired, and > is common, even if not part of open-source Apache Kafka. > > Devin's assertion is that JSON Schema compliant payload validation > and schema evolution are not currently supportable in the Pulsar ecosystem > and that perhaps they should be. > > Elliot. > > > On Fri, 11 Nov 2022 at 14:56, Elliot West <elliot.w...@streamnative.io> > wrote: > > > Hey Devin, > > > > *"Kafka conforms to the JSON Schema specification"* > > Only when using Confluent's Schema Registry. > > > > *"if a producer makes a change or omission, such as in a value used for > > tracking, it might not surface until way down the line"* > > So let me understand this: Although the producer has a schema, it does not > > use it for validation of JSON (as would implicitly occur for Avro? Is this > > correct? > > > > I agree that robust support for schema, certainly at the edges, is a > > cornerstone for a data system. I also agree that it would be better to > > adopt existing standards rather than implement them in a bespoke manner. > > > > I'd be interested to hear your thoughts on concrete improvements that you > > believe would be necessary - for example: > > > > * Producer validation of JSON occurs using "JSON Schema" > > * Evolutions of JSON Schema conform to ... > > * Users can declare topic schema using a JSON Schema document > > * Users can query topic schema and have a JSON schema document returned to > > them > > > > Thanks, > > > > Elliot. > > > > > > > > > > > > > > On Thu, 10 Nov 2022 at 16:51, Devin Bost <devin.b...@gmail.com> wrote: > > > >> One of the areas where Kafka has an advantage over Pulsar is around data > >> quality. Kafka conforms to the JSON Schema specification, which enables > >> integration with any technology that conforms to the standard, such as for > >> data validation, discoverability, lineage, versioning, etc. > >> Pulsar's implementation is non-compliant with the standard, and producers > >> and consumers have no built-in way in Pulsar to validate that values in > >> their messages match expectations. As a consequence, if a producer makes a > >> change or omission, such as in a value used for tracking, it might not > >> surface until way down the line, and then it can be very difficult to > >> track > >> down the source of the problem, which kills the agility of teams > >> responsible for maintaining apps using Pulsar. It's also bad PR because > >> then incidents are associated with Pulsar, even though the business might > >> not understand that the data problem wasn't necessarily caused by Pulsar. > >> > >> What's the right way for us to address this problem? > >> > >> -- > >> Devin Bost > >> Sent from mobile > >> Cell: 801-400-4602 > >> > > > > > > -- > > > > Elliot West > > > > Senior Platform Engineer > > > > elliot.w...@streamnative.io > > > > streamnative.io > > > > <https://github.com/streamnative> > > <https://www.linkedin.com/company/streamnative> > > <https://twitter.com/streamnativeio> > > > > > -- > > Elliot West > > Senior Platform Engineer > > elliot.w...@streamnative.io > > streamnative.io > > <https://github.com/streamnative> > <https://www.linkedin.com/company/streamnative> > <https://twitter.com/streamnativeio>
