Re: Table schema and partition spec update

I hope to do even better. If the stream could provide information about the spec/specId for the record which we would like to write, then we could refresh and use a new writer immediately. Teaser - if the user could provide a converter which converts the input data to `DynamicData` then we can cre

Re: Table schema and partition spec update

Hi Péter, > I have seen requirements for accommodating partitioning scheme changes when > the Table has been changed. This is similar with request I received from users. It’s possible to update/refresh the table spec/schema in the next checkpoint without Flink Job restart. It requires some ex

Re: Table schema and partition spec update

Hi Fokko, Xianjin, Thanks for both proposals, I will take a deeper look soon! Both seems promising at the first glance. For the use cases, - I have seen requirements for converting incoming Avro records with evolving schema and writing them to a table. - I have seen requirements for creating new

Re: Table schema and partition spec update

Hey Péter, For evolving the schema, Spark has the ability to mergeSchema based into the new incoming Schema, you may want t

Re: Table schema and partition spec update

Hey Peter, Thanks for raising this since I recently ran into the same issue. The APIs that we have today nicely hide the field IDs from the user, which is great. I do think all the methods are in there to evolve the schema to the desired one, however, we don't have a way to control the field-IDs.

Table schema and partition spec update

Hi Team, I'm playing around with creating a Flink Dynamic Sink which would allow schema changes without the need for job restart. So when a record with an unknown schema arrives, then it would update the Iceberg table to the new schema and continue processing the records. Lets's say, I have the `