Thanks Peter and Max for kicking off this discussion, +1 for this proposal.

Dynamic Sink is a key improvement for Iceberg Sink, and it can be well 
integrated with Flink CDC’s  pipeline sink in the future as well.

Best,
Leonard


> 2024年11月13日 下午7:57,Maximilian Michels <m...@apache.org> 写道:
> 
> Thanks for driving this Peter! The Dynamic Iceberg Sink is meant to solve
> several pain points of the current Flink Iceberg sink. Note that the
> Iceberg sink lives in Apache Iceberg, not Apache Flink, although that may
> change one day when the interfaces are stable enough.
> 
> From the user perspective, there are three main advantages to using the
> Dynamic Iceberg sink:
> 
> 1. It supports writing to any number of tables (No more 1:1 sink/topic
> relationship).
> 2. It can dynamically create and update tables based on the user-supplied
> routing.
> 3. It can dynamically update the schema and partitioning of tables based on
> the user-supplied specification.
> 
> I'd say these features will have a big impact on Flink Iceberg users.
> 
> Cheers,
> Max
> 
> On Wed, Nov 13, 2024 at 12:27 PM Péter Váry <peter.vary.apa...@gmail.com>
> wrote:
> 
>> Hi Team, With Max Michels, we started to work on enhancing the current
>> Iceberg Sink to allow inserting evolving records into a changing table.
>> 
>> Created the design doc [1]
>> Created the Iceberg proposal [2]
>> Started the conversation on the Iceberg dev list [3]
>> 
>> From the abstract: --------- Flink Iceberg connector sink is the tool to
>> write data to an Iceberg table from a continuous Flink stream. The current
>> Sink implementations emphasize throughput over flexibility. The main
>> limiting factor is that the Iceberg Flink Sink requires static table
>> structure. The table, the schema, the partitioning specification need to be
>> constant. If one of the previous things changes the Flink Job needs to be
>> restarted. This allows using optimal record serialization and good
>> performance, but real life use-cases need to work around this limitation
>> when the underlying table has changed. We need to provide a tool to
>> accommodate these changes. [..] The following typical use cases are
>> considered during this design: - Incoming Avro records schema changes (new
>> columns are added, or other backward compatible changes happen). The Flink
>> job is expected to update the table schema dynamically, and continue to
>> ingest data with the new and the old schema without a job restart. -
>> Incoming records define the target Iceberg table dynamically. The Flink job
>> is expected to create the new table(s) and continue writing to them without
>> a job restart. - The partitioning schema of the table changes. The Flink
>> job is expected to update the specification and continue writing to the
>> target table without a job restart. --------- If you have any questions,
>> ideas, suggestions please let us know on any of the email threads, or in
>> comments on the document. Thanks, Peter
>> 
>> [1] -
>> 
>> https://docs.google.com/document/d/1R3NZmi65S4lwnmNjH4gLCuXZbgvZV5GNrQKJ5NYdO9s
>> [2] - https://github.com/orgs/apache/projects/429
>> [3] - https://lists.apache.org/thread/0sbosd8rzbl5rhqs9dtt99xr44kld3qp
>> 

Reply via email to