Hi Greg, thanks for giving a look. > It's a closed type system which cannot have new "physical" types added by third-party developers
Do you have some examples for this? In my experience I never had the need to add a new type. > There are carve-outs in the implementation for framework logical types (java.util.Date etc) Yeah, java.util.Date is quite old these days. > Compatibility between components isn't managed: custom types can be misinterpreted as dumb physical types which is undesirable to users, who may prefer to fail-fast instead. Any examples for this? > I think that we should evolve the Connect data model in the future to resolve some or all of these problems. Yes, this for sure can have a positive impact. > This will be easier and less risky if we change the data model first, before expanding it to non-Connect users. understandable but I have then the feeling that the change to the data model will be not so easy and fast with the risk to block the expansion outside connect. Maybe starting with the expansion and if there will be more traction also outside the Connect community we will have more motivation to improve the data model. > If there was a suitably general data model out there, we could promote it within the Connect ecosystem and deprecate our current model. Honestly, I don't have anything in mind? Do you? Thanks, Mario. On Thu, Jan 9, 2025 at 6:14 PM Greg Harris <greg.har...@aiven.io.invalid> wrote: > Hi Mario, > > Thanks for the KIP! I think that using the same data model inside and > outside of connect is valuable, and is a good motivation for this effort. > > However, I have seen many situations where the existing data model is > insufficient for Connect plugin developers. > * It's a closed type system which cannot have new "physical" types added by > third-party developers > * There are carve-outs in the implementation for framework logical types > (java.util.Date etc) > * Compatibility between components isn't managed: custom types can be > misinterpreted as dumb physical types which is undesirable to users, who > may prefer to fail-fast instead. > > I think that we should evolve the Connect data model in the future to > resolve some or all of these problems. This will be easier and less risky > if we change the data model first, before expanding it to non-Connect > users. > > Another way to satisfy the requirements of your KIP would be to adopt an > existing external data model in Connect. If there was a suitably general > data model out there, we could promote it within the Connect ecosystem and > deprecate our current model. > > Thanks, > Greg > > > On Thu, Jan 9, 2025 at 12:19 AM Mario Fiore Vitale <mvit...@redhat.com> > wrote: > > > Hi all, > > > > I just want to bring the discussion up since it was open during the > > Christmas holidays. > > > > Any other considerations? > > > > Thanks, > > Mario. > > > > On Sat, Dec 28, 2024 at 3:04 PM Mario Fiore Vitale <mvital...@gmail.com> > > wrote: > > > > > Hey Hector, > > > > > > the aim of this KIP is to separate the Connect data (Struct, Schema, > > etc.) > > > so it can be used without having a dependency con connect-api. > > > > > > So the proposal is more related to economics. > > > > > > As Kafka itself became an API during the years, maybe also the Connect > > data > > > can be used as a general data format not only for Connect. > > > > > > Is it more clear now? > > > > > > Il ven 20 dic 2024, 22:20 Hector Geraldino (BLOOMBERG/ 919 3RD A) < > > > hgerald...@bloomberg.net> ha scritto: > > > > > > > Hey Mario, > > > > > > > > From a quick read of this KIP, it's not super clear to me what > problem > > it > > > > is trying to solve. The motivation states that "current > implementation > > > > restricts the usage of data classes", and I'm not sure what you mean > by > > > > that. > > > > > > > > Can you maybe add one or two examples of things that are not possible > > to > > > > do today and could be achieved by this refactor. If this proposal is > > more > > > > about improving the ergonomics of users of these classes, that's also > > > > valuable IMO. > > > > > > > > From: dev@kafka.apache.org At: 12/18/24 08:32:28 UTC-5:00To: > > > > dev@kafka.apache.org > > > > Subject: [DISCUSS] KIP-1122: Create a dedicated data module for Kafka > > > > Connect data classes > > > > > > > > Hi Everyone, > > > > > > > > I would like to start a discussion on KIP-1122: Create a dedicated > data > > > > module for Kafka Connect data classes [1]. > > > > > > > > [1] > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1122%3A+Create+a+dedicated > > > > +data+module+for+Kafka+Connect+data+classes > > > > > > > > Regards, > > > > -- > > > > Mario Fiore Vitale > > > > > > > > > > > > > > > > > > > > > -- > > > > Mario Fiore Vitale > > > > Senior Software Engineer > > > > Red Hat <https://www.redhat.com/> > > <https://www.redhat.com/> > > > -- Mario Fiore Vitale Senior Software Engineer Red Hat <https://www.redhat.com/> <https://www.redhat.com/>
