Hi, Is this KIP focusing on Sink connectors? Meaning, is the idea to be able to decode JSON messages into Connect records using a schema read from disk?
If I look at this from the perspective of a source connector, not including the schema contents - or the ID of a schema from a schema registry - is exactly the same as not using schemas at all. Right? From: dev@kafka.apache.org At: 06/07/24 14:16:13 UTC-4:00To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-1054: Support external schemas in JSONConverter Hi Priyanka, Thanks for the KIP! I think you captured the motivation well: The Converter interface on its own implies a fairly large raw message size, and the ecosystem's strategies for deduplicating schema information are complex. I did have some questions/concerns. 1. Have you done a comparison between the external schemas approach and the current schemas.enable approach with compression enabled? I think that's currently the best alternative for users without an external schemas service, so I'd like to get an idea of how impactful this feature is in an example use-case. 2. This feature could be implemented without a KIP by creating and distributing a custom JsonConverter. Could you add that as a rejected alternative, with some justification for why it was rejected? 3. Recently we made an effort [1] to secure the FileConfigProvider and DirectoryConfigProvider plugins, which each have the capability of reading from the worker's disk. How do you think this could be made secure, such that a REST API user can't have unrestricted access to the filesystem via this plugin? And to synthesize points 2 and 3: It is very reasonable for "dangerous" features to be opted-in through installation of a custom Converter by operators that understand the risks. It is unreasonable to deliver a "dangerous" feature to operators without a simple way to opt-out, as the upstream plugins are somewhat difficult to remove. Thanks, Greg [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-993%3A+Allow+restricting+f iles+accessed+by+File+and+Directory+ConfigProviders On Fri, Jun 7, 2024 at 8:40 AM Fan Yang <fan...@hotmail.com> wrote: > Hi Priyanka, > > My suggestion is that the we can place schema file at any location, such > as file location, network location, etc. So, the new option could be just > schema.location > > > Best, > Fan > ________________________________ > From: Priyanka K U <priyanka....@ibm.com.INVALID> > Sent: Friday, June 7, 2024 16:22 > To: dev@kafka.apache.org <dev@kafka.apache.org> > Subject: [DISCUSS] KIP-1054: Support external schemas in JSONConverter > > Hi everyone, > > I'd like to start a discussion of KIP-1054, which aims to Support external > schemas in JSONConverter to Kafka Connect: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1054%3A+Support+external+s chemas+in+JSONConverter > > > > Looking forward for your feedback. > > Regards, > Priyanka >
