Re: Dynamic Iceberg Sink

2024-11-13 Thread Maximilian Michels
Thanks for driving this Peter! The Dynamic Iceberg Sink is meant to solve several pain points of the current Flink Iceberg sink. Note that the Iceberg sink lives in Apache Iceberg, not Apache Flink, although that may change one day when the interfaces are stable enough. >From the user perspective,

Re: [VOTE]FLIP-480: Support to deploy script in application mode

2024-11-13 Thread Shengkai Fang
+1(binding) I think I have received enough votes. Thanks for all the help and suggestions! Best, Shengkai

[jira] [Created] (FLINK-36705) Add ProcessTableFunction class and annotations

2024-11-13 Thread Timo Walther (Jira)
Timo Walther created FLINK-36705: Summary: Add ProcessTableFunction class and annotations Key: FLINK-36705 URL: https://issues.apache.org/jira/browse/FLINK-36705 Project: Flink Issue Type: Su

[jira] [Created] (FLINK-36706) Update TypeInferenceExtractor for PTFs

2024-11-13 Thread Timo Walther (Jira)
Timo Walther created FLINK-36706: Summary: Update TypeInferenceExtractor for PTFs Key: FLINK-36706 URL: https://issues.apache.org/jira/browse/FLINK-36706 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-36704) Update TypeInference with StaticArgument and StateTypeStrategy

2024-11-13 Thread Timo Walther (Jira)
Timo Walther created FLINK-36704: Summary: Update TypeInference with StaticArgument and StateTypeStrategy Key: FLINK-36704 URL: https://issues.apache.org/jira/browse/FLINK-36704 Project: Flink

[jira] [Created] (FLINK-36711) Add tips when System out and err are redirected to LOG or discarded

2024-11-13 Thread Vincent Woo (Jira)
Vincent Woo created FLINK-36711: --- Summary: Add tips when System out and err are redirected to LOG or discarded Key: FLINK-36711 URL: https://issues.apache.org/jira/browse/FLINK-36711 Project: Flink

[jira] [Created] (FLINK-36708) Optimize Table Collection Handling Logic to Improve Efficiency

2024-11-13 Thread wanghaofeng (Jira)
wanghaofeng created FLINK-36708: --- Summary: Optimize Table Collection Handling Logic to Improve Efficiency Key: FLINK-36708 URL: https://issues.apache.org/jira/browse/FLINK-36708 Project: Flink

FW: Re: Flink technical debt.

2024-11-13 Thread David Radley
Hi all, Yes Jim – 11:30 AM GMT. Sorry this is not very US friendly, I am trying to accommodate a few Indian colleagues who are interested. Would other people in the community want a USA time zone friendly meeting? I agree the build time is important. In the spirit of Apache to “scratch your ow

[jira] [Created] (FLINK-36709) Flink parquet will throw exception when reading nested row data with last column is array.

2024-11-13 Thread Wenchao Wu (Jira)
Wenchao Wu created FLINK-36709: -- Summary: Flink parquet will throw exception when reading nested row data with last column is array. Key: FLINK-36709 URL: https://issues.apache.org/jira/browse/FLINK-36709

[jira] [Created] (FLINK-36707) Integrate PTF TypeInference into Calcite validator

2024-11-13 Thread Timo Walther (Jira)
Timo Walther created FLINK-36707: Summary: Integrate PTF TypeInference into Calcite validator Key: FLINK-36707 URL: https://issues.apache.org/jira/browse/FLINK-36707 Project: Flink Issue Type

[jira] [Created] (FLINK-36702) FLIP-480: Support to deploy SQL script in application mode

2024-11-13 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-36702: - Summary: FLIP-480: Support to deploy SQL script in application mode Key: FLINK-36702 URL: https://issues.apache.org/jira/browse/FLINK-36702 Project: Flink

[jira] [Created] (FLINK-36703) FLIP-440: User-defined SQL operators / ProcessTableFunction (PTF)

2024-11-13 Thread Timo Walther (Jira)
Timo Walther created FLINK-36703: Summary: FLIP-440: User-defined SQL operators / ProcessTableFunction (PTF) Key: FLINK-36703 URL: https://issues.apache.org/jira/browse/FLINK-36703 Project: Flink

Re: [VOTE]FLIP-480: Support to deploy script in application mode

2024-11-13 Thread Yuepeng Pan
+1 (non-binding) Best, Yuepeng Pan 在 2024-11-13 02:37:40,"Ferenc Csaky" 写道: >+1 (binding) > >Best, >Ferenc > > > > >On Tuesday, November 12th, 2024 at 03:28, Xuyang wrote: > >> >> >> +1 (non-binding) >> >> >> -- >> >> Best! >> Xuyang >> >> >> >> >> >> 在 2024-11-12 10:02:27,"

[RESULT][VOTE] FLIP-480: Support to deploy script in application mode

2024-11-13 Thread Shengkai Fang
Hi, devs. I'm happy to announce that FLIP-480: Support to deploy script in application mode[1] has been accepted. There are 11 approving votes, 6 of which are binding: - Lincoln Lee(binding) - yuanfeng hu(non-binding) - Feng Jin(non-binding) - Gyula Fóra(binding) - Ron Liu(binding) - Xuyang(non-

[jira] [Created] (FLINK-36701) Pipeline failover again after handling a schema change event as the first event after a failover

2024-11-13 Thread zjjiang (Jira)
zjjiang created FLINK-36701: --- Summary: Pipeline failover again after handling a schema change event as the first event after a failover Key: FLINK-36701 URL: https://issues.apache.org/jira/browse/FLINK-36701

Dynamic Iceberg Sink

2024-11-13 Thread Péter Váry
Hi Team, With Max Michels, we started to work on enhancing the current Iceberg Sink to allow inserting evolving records into a changing table. Created the design doc [1] Created the Iceberg proposal [2] Started the conversation on the Iceberg dev list [3] >From the abstract: - Flink Icebe

[jira] [Created] (FLINK-36712) Autoscaler should skip resource check when new parallelism/memory is lower than current

2024-11-13 Thread Sai Sharath Dandi (Jira)
Sai Sharath Dandi created FLINK-36712: - Summary: Autoscaler should skip resource check when new parallelism/memory is lower than current Key: FLINK-36712 URL: https://issues.apache.org/jira/browse/FLINK-36712

[jira] [Created] (FLINK-36710) JobIDLoggingITCase fails on SourceCoordinator logs

2024-11-13 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-36710: - Summary: JobIDLoggingITCase fails on SourceCoordinator logs Key: FLINK-36710 URL: https://issues.apache.org/jira/browse/FLINK-36710 Project: Flink

Re: [VOTE] Release flink-connector-hbase v4.0.0, release candidate #1

2024-11-13 Thread Attila Kreiner
+1 (non-binding) Checked: - Built from source with jdk 11, maven 3.8.6 - Reviewed release notes - Verified hash & signature - Source code artifacts match the current release On Wed, Nov 13, 2024 at 6:42 AM Yanquan Lv wrote: > +1 (non-binding) > I checked: > - Review JIRA release notes > - Verify

RE: Re: [VOTE] FLIP-440: User-defined SQL operators / ProcessTableFunction (PTF)

2024-11-13 Thread David Radley
+1 (non-binding) kind regards, David From: Dawid Wysakowicz Date: Tuesday, 12 November 2024 at 10:47 To: dev@flink.apache.org Subject: [EXTERNAL] Re: Re: [VOTE] FLIP-440: User-defined SQL operators / ProcessTableFunction (PTF) +1 (binding) Best, Dawid On Tue, 12 Nov 2024 at 09:28, Fabian Hüske

Re: Dynamic Iceberg Sink

2024-11-13 Thread Leonard Xu
Thanks Peter and Max for kicking off this discussion, +1 for this proposal. Dynamic Sink is a key improvement for Iceberg Sink, and it can be well integrated with Flink CDC’s pipeline sink in the future as well. Best, Leonard > 2024年11月13日 下午7:57,Maximilian Michels 写道: > > Thanks for drivin

[jira] [Created] (FLINK-36713) NoSuchMethodError: PostgresConnection with PostgresIncrementalSource ar runtime

2024-11-13 Thread Mohsen Rezaei (Jira)
Mohsen Rezaei created FLINK-36713: - Summary: NoSuchMethodError: PostgresConnection with PostgresIncrementalSource ar runtime Key: FLINK-36713 URL: https://issues.apache.org/jira/browse/FLINK-36713 Pro

Re: Dynamic Iceberg Sink

2024-11-13 Thread Congxian Qiu
Thanks for kicking off this discussion, +1 for this proposal. Currently, we use FlinkCDC as the ingestion tool to write to Iceberg, and need to restart the job if there is any schema evolution, Look forward to seeing this integrated with FlinkCDC Best, Congxian Leonard Xu 于2024年11月14日周四 09:37写

Re: [VOTE] Release flink-connector-hbase v4.0.0, release candidate #1

2024-11-13 Thread Mate Czagany
+1 (non-binding) - Built from source with JDK 11 - Verified checksum and signatures - Verified source code does not contain binaries - Verified artifacts deployed to the maven repository - Verified no binaries in source release - Reviewed flink-web PR Thank you for driving this release Ferenc! B