Hi Mario, Thanks for the KIP. My view is that it would be nice for Kafka to have a data model so that Kafka could do natively the things that people use schema registries for today. I'm not convinced that the Kafka Connect data classes are really up to the job, so I'm not sure that there's a lot of value in making a separate module. Having said that, it's a pretty small thing to do and it probably doesn't preclude someone coming along with a Kafka data model in the future.
Just my 2 cents. Thanks, Andrew ________________________________________ From: Mario Fiore Vitale <mvit...@redhat.com> Sent: 17 January 2025 11:20 To: dev@kafka.apache.org <dev@kafka.apache.org> Subject: Re: [DISCUSS] KIP-1122: Create a dedicated data module for Kafka Connect data classes Hi all, Any other feedback on this? I'll take the discussion open for another week and then start the vote. Thanks, Mario. On Fri, Jan 10, 2025 at 10:30 AM Mario Fiore Vitale <mvit...@redhat.com> wrote: > Hi Greg, thanks for giving a look. > > > It's a closed type system which cannot have new "physical" types added by > third-party developers > > Do you have some examples for this? In my experience I never had the need > to add a new type. > > > There are carve-outs in the implementation for framework logical types > (java.util.Date etc) > > Yeah, java.util.Date is quite old these days. > > > Compatibility between components isn't managed: custom types can be > misinterpreted as dumb physical types which is undesirable to users, who > may prefer to fail-fast instead. > > Any examples for this? > > > I think that we should evolve the Connect data model in the future to > resolve some or all of these problems. > > Yes, this for sure can have a positive impact. > > > This will be easier and less risky > if we change the data model first, before expanding it to non-Connect > users. > > understandable but I have then the feeling that the change to the data > model will be not so easy and fast with the risk to block the > expansion outside connect. > Maybe starting with the expansion and if there will be more traction also > outside the Connect community we will have more motivation to improve the > data model. > > > If there was a suitably general data model out there, we could promote > it within the Connect ecosystem and deprecate our current model. > > Honestly, I don't have anything in mind? Do you? > > Thanks, > Mario. > > On Thu, Jan 9, 2025 at 6:14 PM Greg Harris <greg.har...@aiven.io.invalid> > wrote: > >> Hi Mario, >> >> Thanks for the KIP! I think that using the same data model inside and >> outside of connect is valuable, and is a good motivation for this effort. >> >> However, I have seen many situations where the existing data model is >> insufficient for Connect plugin developers. >> * It's a closed type system which cannot have new "physical" types added >> by >> third-party developers >> * There are carve-outs in the implementation for framework logical types >> (java.util.Date etc) >> * Compatibility between components isn't managed: custom types can be >> misinterpreted as dumb physical types which is undesirable to users, who >> may prefer to fail-fast instead. >> >> I think that we should evolve the Connect data model in the future to >> resolve some or all of these problems. This will be easier and less risky >> if we change the data model first, before expanding it to non-Connect >> users. >> >> Another way to satisfy the requirements of your KIP would be to adopt an >> existing external data model in Connect. If there was a suitably general >> data model out there, we could promote it within the Connect ecosystem and >> deprecate our current model. >> >> Thanks, >> Greg >> >> >> On Thu, Jan 9, 2025 at 12:19 AM Mario Fiore Vitale <mvit...@redhat.com> >> wrote: >> >> > Hi all, >> > >> > I just want to bring the discussion up since it was open during the >> > Christmas holidays. >> > >> > Any other considerations? >> > >> > Thanks, >> > Mario. >> > >> > On Sat, Dec 28, 2024 at 3:04 PM Mario Fiore Vitale <mvital...@gmail.com >> > >> > wrote: >> > >> > > Hey Hector, >> > > >> > > the aim of this KIP is to separate the Connect data (Struct, Schema, >> > etc.) >> > > so it can be used without having a dependency con connect-api. >> > > >> > > So the proposal is more related to economics. >> > > >> > > As Kafka itself became an API during the years, maybe also the Connect >> > data >> > > can be used as a general data format not only for Connect. >> > > >> > > Is it more clear now? >> > > >> > > Il ven 20 dic 2024, 22:20 Hector Geraldino (BLOOMBERG/ 919 3RD A) < >> > > hgerald...@bloomberg.net> ha scritto: >> > > >> > > > Hey Mario, >> > > > >> > > > From a quick read of this KIP, it's not super clear to me what >> problem >> > it >> > > > is trying to solve. The motivation states that "current >> implementation >> > > > restricts the usage of data classes", and I'm not sure what you >> mean by >> > > > that. >> > > > >> > > > Can you maybe add one or two examples of things that are not >> possible >> > to >> > > > do today and could be achieved by this refactor. If this proposal is >> > more >> > > > about improving the ergonomics of users of these classes, that's >> also >> > > > valuable IMO. >> > > > >> > > > From: dev@kafka.apache.org At: 12/18/24 08:32:28 UTC-5:00To: >> > > > dev@kafka.apache.org >> > > > Subject: [DISCUSS] KIP-1122: Create a dedicated data module for >> Kafka >> > > > Connect data classes >> > > > >> > > > Hi Everyone, >> > > > >> > > > I would like to start a discussion on KIP-1122: Create a dedicated >> data >> > > > module for Kafka Connect data classes [1]. >> > > > >> > > > [1] >> > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1122%3A+Create+a+dedicated >> > > > +data+module+for+Kafka+Connect+data+classes >> > > > >> > > > Regards, >> > > > -- >> > > > Mario Fiore Vitale >> > > > >> > > > >> > > > >> > > >> > >> > >> > -- >> > >> > Mario Fiore Vitale >> > >> > Senior Software Engineer >> > >> > Red Hat <https://www.redhat.com/> >> > <https://www.redhat.com/> >> > >> > > > -- > > Mario Fiore Vitale > > Senior Software Engineer > > Red Hat <https://www.redhat.com/> > <https://www.redhat.com/> > -- Mario Fiore Vitale Senior Software Engineer Red Hat <https://www.redhat.com/> <https://www.redhat.com/>
