Thanks Andrew and Greg for your opinion on that. Wondering if, for the future, Avro can be a good replacement for it.
Any other opinions on this KIP? Thanks, Mario. On Fri, Jan 17, 2025 at 7:43 PM Greg Harris <greg.har...@aiven.io.invalid> wrote: > Hi Mario, > > Thanks for your responses. > > > Do you have some examples for this? In my experience I never had the need > > to add a new type. > > Nobody has added a physical type because it's not possible. But we can see > the latent desire through very general types appearing as logical types > [1-2], and in unresolved type issues [3-9]. I picked the Confluent JDBC > connector out of familiarity, but these problems are not particular to that > connector. > Databases cause a lot of friction with our type system because database > type systems are much more mature and expressive. We should learn from the > decades that databases have spent building general data models and improve > our own data model before calling it general and promoting its common use. > > > > There are carve-outs in the implementation for framework logical types > (java.util.Date etc) > > Yeah, java.util.Date is quite old these days. > > My example of java.util.Date is not that it's old, it's that the framework > does things for its logical types which no third-party logical types can > [10]. > But it is indeed old, and if someone were to wish to replace it with > java.time.Instant, they could not do so. Only the framework could add the > required carve-outs. And realistically the framework couldn't even do so, > because introducing it would cause errors and exceptions in plugins which > aren't aware of it. > > > Any examples for this? > > Whenever you see someone complaining that they got some primitive type > (string or bytes most commonly) instead of their actual type, that's a > failure mode unique to logical types [11-15]. Again don't take this as an > indictment of the JDBC connector, it's particularly exposed to this problem > as a sink connector. > Also, arbitrary logical types being unsupported by the framework > SchemaProjector has been reported multiple times [16]. The schema converter > is being expected by users to handle logical types, which it has no way of > knowing about. > > Thanks, > Greg > > [1] > > https://github.com/confluentinc/schema-registry/blob/e461e9659ef1532a95b3f2dfd38e843d766aa955/schema-converter/src/main/java/io/confluent/connect/schema/ConnectEnum.java > [2] > > https://github.com/confluentinc/schema-registry/blob/e461e9659ef1532a95b3f2dfd38e843d766aa955/schema-converter/src/main/java/io/confluent/connect/schema/ConnectUnion.java > [3] https://github.com/confluentinc/kafka-connect-jdbc/issues/1378 > [4] https://github.com/confluentinc/kafka-connect-jdbc/issues/1006 > [5] https://github.com/confluentinc/kafka-connect-jdbc/issues/1002 > [6] https://github.com/confluentinc/kafka-connect-jdbc/issues/651 > [7] https://github.com/confluentinc/kafka-connect-jdbc/issues/410 > [8] https://github.com/confluentinc/kafka-connect-jdbc/issues/265 > [9] https://github.com/confluentinc/kafka-connect-jdbc/issues/81 > [10] > > https://github.com/apache/kafka/blob/a6faec179a3685d66698a2dc3c6a1823bd0c87f9/connect/api/src/main/java/org/apache/kafka/connect/data/ConnectSchema.java#L68-L71 > [11] https://github.com/confluentinc/kafka-connect-jdbc/issues/1155 > [12] https://github.com/confluentinc/kafka-connect-jdbc/issues/788 > [13] https://github.com/confluentinc/kafka-connect-jdbc/issues/749 > [14] https://github.com/confluentinc/kafka-connect-jdbc/issues/563 > [15] https://github.com/confluentinc/kafka-connect-jdbc/issues/393 > [16] https://issues.apache.org/jira/browse/KAFKA-16257 > > On Fri, Jan 17, 2025 at 5:57 AM Andrew Schofield < > andrew_schofield_j...@outlook.com> wrote: > > > Hi Mario, > > Thanks for the KIP. > > > > My view is that it would be nice for Kafka to have a data model so that > > Kafka could do natively the things that people use schema registries > > for today. I'm not convinced that the Kafka Connect data classes are > > really up to the job, so I'm not sure that there's a lot of value in > making > > a separate module. Having said that, it's a pretty small thing to do > > and it probably doesn't preclude someone coming along with a > > Kafka data model in the future. > > > > Just my 2 cents. > > > > Thanks, > > Andrew > > ________________________________________ > > From: Mario Fiore Vitale <mvit...@redhat.com> > > Sent: 17 January 2025 11:20 > > To: dev@kafka.apache.org <dev@kafka.apache.org> > > Subject: Re: [DISCUSS] KIP-1122: Create a dedicated data module for Kafka > > Connect data classes > > > > Hi all, > > > > Any other feedback on this? I'll take the discussion open for another > week > > and then start the vote. > > > > Thanks, > > Mario. > > > > On Fri, Jan 10, 2025 at 10:30 AM Mario Fiore Vitale <mvit...@redhat.com> > > wrote: > > > > > Hi Greg, thanks for giving a look. > > > > > > > It's a closed type system which cannot have new "physical" types > added > > by > > > third-party developers > > > > > > Do you have some examples for this? In my experience I never had the > need > > > to add a new type. > > > > > > > There are carve-outs in the implementation for framework logical > types > > > (java.util.Date etc) > > > > > > Yeah, java.util.Date is quite old these days. > > > > > > > Compatibility between components isn't managed: custom types can be > > > misinterpreted as dumb physical types which is undesirable to users, > who > > > may prefer to fail-fast instead. > > > > > > Any examples for this? > > > > > > > I think that we should evolve the Connect data model in the future to > > > resolve some or all of these problems. > > > > > > Yes, this for sure can have a positive impact. > > > > > > > This will be easier and less risky > > > if we change the data model first, before expanding it to non-Connect > > > users. > > > > > > understandable but I have then the feeling that the change to the data > > > model will be not so easy and fast with the risk to block the > > > expansion outside connect. > > > Maybe starting with the expansion and if there will be more traction > also > > > outside the Connect community we will have more motivation to improve > the > > > data model. > > > > > > > If there was a suitably general data model out there, we could > promote > > > it within the Connect ecosystem and deprecate our current model. > > > > > > Honestly, I don't have anything in mind? Do you? > > > > > > Thanks, > > > Mario. > > > > > > On Thu, Jan 9, 2025 at 6:14 PM Greg Harris > <greg.har...@aiven.io.invalid > > > > > > wrote: > > > > > >> Hi Mario, > > >> > > >> Thanks for the KIP! I think that using the same data model inside and > > >> outside of connect is valuable, and is a good motivation for this > > effort. > > >> > > >> However, I have seen many situations where the existing data model is > > >> insufficient for Connect plugin developers. > > >> * It's a closed type system which cannot have new "physical" types > added > > >> by > > >> third-party developers > > >> * There are carve-outs in the implementation for framework logical > types > > >> (java.util.Date etc) > > >> * Compatibility between components isn't managed: custom types can be > > >> misinterpreted as dumb physical types which is undesirable to users, > who > > >> may prefer to fail-fast instead. > > >> > > >> I think that we should evolve the Connect data model in the future to > > >> resolve some or all of these problems. This will be easier and less > > risky > > >> if we change the data model first, before expanding it to non-Connect > > >> users. > > >> > > >> Another way to satisfy the requirements of your KIP would be to adopt > an > > >> existing external data model in Connect. If there was a suitably > general > > >> data model out there, we could promote it within the Connect ecosystem > > and > > >> deprecate our current model. > > >> > > >> Thanks, > > >> Greg > > >> > > >> > > >> On Thu, Jan 9, 2025 at 12:19 AM Mario Fiore Vitale < > mvit...@redhat.com> > > >> wrote: > > >> > > >> > Hi all, > > >> > > > >> > I just want to bring the discussion up since it was open during the > > >> > Christmas holidays. > > >> > > > >> > Any other considerations? > > >> > > > >> > Thanks, > > >> > Mario. > > >> > > > >> > On Sat, Dec 28, 2024 at 3:04 PM Mario Fiore Vitale < > > mvital...@gmail.com > > >> > > > >> > wrote: > > >> > > > >> > > Hey Hector, > > >> > > > > >> > > the aim of this KIP is to separate the Connect data (Struct, > Schema, > > >> > etc.) > > >> > > so it can be used without having a dependency con connect-api. > > >> > > > > >> > > So the proposal is more related to economics. > > >> > > > > >> > > As Kafka itself became an API during the years, maybe also the > > Connect > > >> > data > > >> > > can be used as a general data format not only for Connect. > > >> > > > > >> > > Is it more clear now? > > >> > > > > >> > > Il ven 20 dic 2024, 22:20 Hector Geraldino (BLOOMBERG/ 919 3RD A) > < > > >> > > hgerald...@bloomberg.net> ha scritto: > > >> > > > > >> > > > Hey Mario, > > >> > > > > > >> > > > From a quick read of this KIP, it's not super clear to me what > > >> problem > > >> > it > > >> > > > is trying to solve. The motivation states that "current > > >> implementation > > >> > > > restricts the usage of data classes", and I'm not sure what you > > >> mean by > > >> > > > that. > > >> > > > > > >> > > > Can you maybe add one or two examples of things that are not > > >> possible > > >> > to > > >> > > > do today and could be achieved by this refactor. If this > proposal > > is > > >> > more > > >> > > > about improving the ergonomics of users of these classes, that's > > >> also > > >> > > > valuable IMO. > > >> > > > > > >> > > > From: dev@kafka.apache.org At: 12/18/24 08:32:28 UTC-5:00To: > > >> > > > dev@kafka.apache.org > > >> > > > Subject: [DISCUSS] KIP-1122: Create a dedicated data module for > > >> Kafka > > >> > > > Connect data classes > > >> > > > > > >> > > > Hi Everyone, > > >> > > > > > >> > > > I would like to start a discussion on KIP-1122: Create a > dedicated > > >> data > > >> > > > module for Kafka Connect data classes [1]. > > >> > > > > > >> > > > [1] > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1122%3A+Create+a+dedicated > > >> > > > +data+module+for+Kafka+Connect+data+classes > > >> > > > > > >> > > > Regards, > > >> > > > -- > > >> > > > Mario Fiore Vitale > > >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > >> > -- > > >> > > > >> > Mario Fiore Vitale > > >> > > > >> > Senior Software Engineer > > >> > > > >> > Red Hat <https://www.redhat.com/> > > >> > <https://www.redhat.com/> > > >> > > > >> > > > > > > > > > -- > > > > > > Mario Fiore Vitale > > > > > > Senior Software Engineer > > > > > > Red Hat <https://www.redhat.com/> > > > <https://www.redhat.com/> > > > > > > > > > -- > > > > Mario Fiore Vitale > > > > Senior Software Engineer > > > > Red Hat <https://www.redhat.com/> > > <https://www.redhat.com/> > > > -- Mario Fiore Vitale Senior Software Engineer Red Hat <https://www.redhat.com/> <https://www.redhat.com/>
