> was wondering why? In the confluent avro and existing debezium
>> formats,
>> > > there is no schema id in the config, but there is the ability to
>> specify
>> > a
>> > > complete schema. In the protobuf format there is no schema id.
>>
ed something like schemaIdForSink or
> > schemaIdForSerilization; to prevent confusion with the deserialization
> > case. We could have the schema as you suggest so we are compatible with
> the
> > confluent avro format.
> >
> >
> > WDYT?
> > Kind regards, Da
;
> > I would suggest aiming to copy what the confluent DeSer libraries do
> > rather than having a schema id hard coded in the config.
> >
> > WDYT?
> > Kind regards, David.
> >
> > From: Kevin Lam
> > Date: Tuesday, 26 March 2024 at 20:06
> > To: de
are compatible with the confluent
avro format.
WDYT?
Kind regards, David.
From: Anupam Aggarwal
Date: Saturday, 13 April 2024 at 16:08
To: dev@flink.apache.org
Subject: [EXTERNAL] Re: [DISCUSS] FLINK-34440 Support Debezium Protobuf
Confluent Format
Hi David,
Thank you for the suggestion
y what the confluent DeSer libraries do
> rather than having a schema id hard coded in the config.
>
> WDYT?
> Kind regards, David.
>
> From: Kevin Lam
> Date: Tuesday, 26 March 2024 at 20:06
> To: dev@flink.apache.org
> Subject: [EXTERNAL] Re: [DISCUSS] FLINK-
] FLINK-34440 Support Debezium Protobuf
Confluent Format
Thanks Anupam! Looking forward to it.
On Thu, Mar 14, 2024 at 1:50 AM Anupam Aggarwal
wrote:
> Hi Kevin,
>
> Thanks, these are some great points.
> Just to clarify, I do agree that the subject should be an option (like in
&
Thanks Kevin, I will pick this up more actively starting early next week.
On Wed, Mar 27, 2024 at 1:36 AM Kevin Lam
wrote:
> Thanks Anupam! Looking forward to it.
>
> On Thu, Mar 14, 2024 at 1:50 AM Anupam Aggarwal >
> wrote:
>
> > Hi Kevin,
> >
> > Thanks, these are some great points.
> > Just
Thanks Anupam! Looking forward to it.
On Thu, Mar 14, 2024 at 1:50 AM Anupam Aggarwal
wrote:
> Hi Kevin,
>
> Thanks, these are some great points.
> Just to clarify, I do agree that the subject should be an option (like in
> the case of RegistryAvroFormatFactory).
> We could fallback to subject a
Hi Kevin,
Thanks, these are some great points.
Just to clarify, I do agree that the subject should be an option (like in
the case of RegistryAvroFormatFactory).
We could fallback to subject and auto-register schemas, if schema-Id not
provided explicitly.
In general, I think it would be good to be
Hi Anupam,
Thanks again for your work on contributing this feature back.
Sounds good re: the refactoring/re-organizing.
Regarding the schema-id, in my opinion this should NOT be a configuration
option on the format. We should be able to deterministically map the Flink
type to the ProtoSchema and
Hi Kevin,
Thanks for starting the discussion on this.
I will be working on contributing this feature back (This was developed by
Dawid Wysakowicz and others at Confluent).
I have opened a (very initial) draft PR here
https://github.com/apache/flink/pull/24482 with our current implementation.
Than
Hey Robert,
Awesome thanks, that timeline works for me. Sounds good re: deciding on
FLIP once we have the PR, and thanks for looking into the field ids.
Looking forward to it!
On Thu, Feb 29, 2024 at 5:09 AM Robert Metzger wrote:
> Hey Kevin,
>
> Thanks a lot. Then let's contribute the Conflue
Hey Kevin,
Thanks a lot. Then let's contribute the Confluent implementation to
apache/flink. We can't start working on this immediately because of a team
event next week, but within the next two weeks, we will start working on
this.
It probably makes sense for us to open a pull request of what we
Hey Robert,
Thanks for your response. I have a partial implementation, just for the
decoding portion.
The code I have is pretty rough and doesn't do any of the refactors I
mentioned, but the decoder logic does pull the schema from the schema
registry and use that to deserialize the DynamicMessage
Hi all,
+1 to support the format in Flink.
@Kevin: Do you already have an implementation for this inhouse that you are
looking to upstream, or would you start from scratch?
I'm asking because my current employer, Confluent, has a Protobuf Schema
registry implementation for Flink, and I could help
Hi Kevin,
Some thoughts on this.
I suggested an Apicurio registry format in the dev list, and was advised to
raise a FLIP for this, I suggest the same would apply here (or the alternative
to FLIPs if you cannot raise one). I am prototyping an Avro Apicurio format,
prior to raising the Flip, and
16 matches
Mail list logo
