snuyanzin commented on code in PR #871: URL: https://github.com/apache/flink-web/pull/871#discussion_r4124035305
########## docs/content/posts/2026-07-30-community-update.md: ########## @@ -0,0 +1,323 @@ +--- +authors: +- sofie204: null + name: Sophia Izokun +date: "2026-07-30T08:00:00Z" +excerpt: Flink Community Update for Q2 2026 +title: Flink Community Update for Q2 2026 +aliases: +- /news/2026/07/30/community-update.html +--- + +This is the Flink Community Update for Q2 2026, the first in a new quarterly format. Following a discussion with the dev community, the intention going forward is that these updates will appear quarterly rather than monthly, with the aim of highlighting higher-impact topics over a three-month window at a time. +This edition covers April through June 2026. + +<!--more--> + +Just like previous updates, we'll still cover releases, FLIPs, governance, and community announcements. However, we will begin with a few themes that stood out across the quarter. + +The changes and updates to Flink this quarter were quite impactful, with Flink SQL closing several gaps with the DataStream API through changelog conversion, and materialized tables gaining in-place evolution. With the 2.3 release branch cut on April 15, much of the quarter's engineering went into the Flink 2.4 cycle: Apache Calcite upgrades for the SQL planner, common subexpression elimination in generated SQL code, and groundwork for JDK 25. Flink Agents shipped a new 0.3.0 release spanning reusable skills, cross-language actions, reliability, and observability. The community also accepted an umbrella proposal defining a broader direction for AI-native, multimodal processing in Flink. + +The previous update is available in the [Flink Community Update for April 2026](https://flink.apache.org/2026/04/13/flink-community-update-for-april-2026/). + +## In this update + +- [Release of Apache Flink 2.3.0](#release-of-apache-flink-230) — Flink SQL becomes more capable, a native S3 filesystem, and what to check before you upgrade +- [Looking ahead to Flink 2.4](#looking-ahead-to-flink-24) — This cycle includes five Calcite upgrades, common subexpression elimination, and JDK 25 groundwork, all merged to the master branch this quarter +- [Flink Agents 0.3.0](#flink-agents-030) — the sub-project adopts the AI ecosystem's emerging standards on its road to 1.0 +- [New releases](#new-releases) — the quarter's releases at a glance; Flink 2.3.0, Agents 0.3.0, Kubernetes Operator 1.15.0, and maintenance and connector releases across every supported line +- [Governance and Community](#governance-and-community) — four new committers, a new PMC member, twelve accepted FLIPs, the StateFun sunset vote, and newly released guidelines for AI-assisted contributions +- [Staying up to date](#staying-up-to-date) — where to find the community and how to give feedback on these updates + +## Release of Apache Flink 2.3.0 +[Flink 2.3.0 was announced](https://flink.apache.org/2026/06/25/apache-flink-2.3.0-release-announcement/) on June 25th 2026, the product of a release cycle that ran end to end in this quarter. The [release branch was cut](https://lists.apache.org/thread/sr0zlz8nc5fkcxd7bqmblfwdv3j4h6rx) on April 15, an [open call for release testing](https://lists.apache.org/thread/t7jj2qvl093mto7jt6z3zkqo3jyo8zno) went out five days later, and after four release candidates the [vote concluded](https://lists.apache.org/thread/bqxlj4fm9hdp868xs95qw504kg231m9m) on June 18. The announcement blog covers all fifteen FLIPs and here we pick out the changes we think matter most to the Flink community. + +### Flink SQL becomes more capable +The [April community update](https://flink.apache.org/2026/04/13/flink-community-update-for-april-2026/) covered ongoing work around materialized tables, changelog conversion and process table functions. Several of those developments reached users as SQL updates with Flink 2.3.0 in June. + +These updates show that Flink SQL is expanding into use cases that previously required the DataStream API or custom workarounds. At the same time, the updates also brought fixes and deliberate changes that show Flink SQL becoming more explicit about behavior that can affect correctness, state growth, or the results written to a sink. + +#### Changelog conversion comes to SQL + +For the first time, SQL users can convert retract or upsert streams into append form with the new `FROM_CHANGELOG` and `TO_CHANGELOG` operators [FLIP-564](https://cwiki.apache.org/confluence/spaces/FLINK/pages/406620639/FLIP-564+Support+FROM_CHANGELOG+and+TO_CHANGELOG+built-in+PTFs), a class of operation that previously meant dropping down to the DataStream API. + +Worth knowing before you reach for it: +- 2.3 covers the basic use cases, with more (such as `PARTITION BY`, `invalid_op_handling`,`produces_full_deletes`) planned to follow in 2.4. + +#### Materialized tables can evolve with less re-processing + +Until now, changing a materialized table's query definition meant +dropping and recreating it and reprocessing all the historical data +that came with it. 2.3 removes that requirement ([FLIP-557](https://cwiki.apache.org/confluence/spaces/FLINK/pages/399279094/FLIP-557+Granular+Control+over+Data+Reprocessing+in+Materialized+Table+Evolution) and +[FLIP-550](https://cwiki.apache.org/confluence/spaces/FLINK/pages/387648095/FLIP-550+Add+similar+support+for+CREATE+ALTER+operations+for+MATERIALIZED+TABLEs+as+for+TABLEs)) such that definitions can evolve in place, without the unnecessary recompute. For users of Flink whose teams pay a cloud bill as a result of that recompute, skipping unnecessary reprocessing is a direct saving. + + +#### The follow-ups are already on `master` + +None of these SQL feature changes/additions stopped at 2.3. +- The `produces_full_deletes` option for `TO_CHANGELOG`, listed in the release notes as future work, [merged in late May and June](https://issues.apache.org/jira/browse/FLINK-39636). +- In-place conversion of an existing table into a materialized table, the subject of [FLIP-578](https://cwiki.apache.org/confluence/spaces/FLINK/pages/421958318/FLIP-578+In-place+Table+to+Materialized+Table+conversion?src=contextnavpagetreemode), [merged on June 15](https://issues.apache.org/jira/browse/FLINK-39847). +- The first building block of `LATERAL SNAPSHOT` joins ([FLIP-579](https://cwiki.apache.org/confluence/spaces/FLINK/pages/421958523/FLIP-579+LATERAL+SNAPSHOT+Join)), a `SNAPSHOT` built-in function definition, [landed on June 22](https://issues.apache.org/jira/browse/FLINK-39782). + +All of it targets Flink 2.4. + +#### Upsert sinks stop guessing + +When a query's upsert key differs from the sink's primary key, Flink +previously materialized state to resolve the conflict and that could result in a +source of unbounded state growth. 2.3 introduces an explicit +`ON CONFLICT` clause ([FLIP-558](https://cwiki.apache.org/confluence/spaces/FLINK/pages/399279158/FLIP-558+Improvements+to+SinkUpsertMaterializer+and+changelog+disorder)), and changes the default such that those queries now fail at planning time unless you state a conflict strategy. A query that planned fine on 2.2 can refuse to plan on 2.3 and that's a feature of the FLIP. + + +### Native S3 filesystem reduces the Hadoop dependency footprint + +Flink 2.3.0 ships an [experimental native S3 Filesystem](https://cwiki.apache.org/confluence/spaces/FLINK/pages/396790457/FLIP-555+Flink+Native+S3+FileSystem) built on the AWS SDK v2 with no Hadoop dependencies. This shows how Flink is moving towards a more cloud native foundation. The project has published [benchmarks](https://cwiki.apache.org/confluence/spaces/FLINK/pages/406620396/Benchmarking+Native+S3+FileSystem+flink-s3-fs-native+vs+Presto+S3+flink-s3-fs-presto) comparing the native implementation with the existing Presto-based S3 filesystem. + +It's important to note that the native filesystem is experimental in Flink 2.3.0. As such, users should review its documentation and limitations carefully before using it for production checkpoint, savepoint, or sink storage. + +An example of why Flink users should pay attention to the caution above surfaced in June. The native filesystem's recoverable writer [silently dropped the bytes written since the last complete part when resuming from a checkpoint](https://issues.apache.org/jira/browse/FLINK-39778), which breaks exactly once for streaming sinks. The fix merged on master on June 15 and is targeted at 2.4.0. + + + +- [Documentation](https://nightlies.apache.org/flink/flink-docs-release-2.3/docs/deployment/filesystems/s3/) + +### Before you upgrade + +Two items in 2.3.0 deserve attention regardless of which features you +care about. + +If you use mini-batch aggregation with `ONE_PHASE` aggregation, it would be worth reviewing the 2.3.0 upgrade. + +Flink 2.3.0 fixes a bug where a batch containing only retractions could cause all remaining keys in the bundle to be silently dropped +([FLINK-35661](https://issues.apache.org/jira/browse/FLINK-35661) — +[details in the announcement](https://flink.apache.org/2026/06/25/apache-flink-2.3.0-release-announcement/#critical-bug-fix-minibatch-aggregation-record-loss)). +Silent data loss fixes are a valid reason to upgrade. + +Watermark alignment was redesigned in [FLINK-37399](https://issues.apache.org/jira/browse/FLINK-37399) to remove a bottleneck that could limit how quickly jobs processed accumulated backlog. Starting in Flink 2.3, watermark alignment pauses sources a few seconds later than it did previously, which can modestly increase state size for windowed and temporal operators. Users who prefer the earlier behavior can set `pipeline.watermark-alignment.buffer-size` to `0`. + +### Also in 2.3.0 + +**Applications and rescaling become visible in the Web UI** + +A few more changes from the release worth knowing about, each covered +fully in the announcement: + +Application-level lifecycle management +([FLIP-549](https://cwiki.apache.org/confluence/spaces/FLINK/pages/386272238/FLIP-549+Support+Application+Management) / [FLIP-560](https://cwiki.apache.org/confluence/spaces/FLINK/pages/404160740/FLIP-560+Application+Capability+Enhancement)) replaces the cluster-job model with a cluster–application–job hierarchy, visible in a new Applications view in the Web UI. + +Rescaling gains a visible history with its own Web UI tab +([FLIP-487](https://cwiki.apache.org/confluence/spaces/FLINK/pages/327979197/FLIP-487+Show+history+of+rescales+in+Web+UI+for+AdaptiveScheduler#FLIP487:ShowhistoryofrescalesinWebUIforAdaptiveScheduler-RescaleOverviewUI) / [FLIP-495](https://cwiki.apache.org/confluence/spaces/FLINK/pages/334760525/FLIP-495+Support+AdaptiveScheduler+record+and+query+the+rescale+history)), so you can +see what the adaptive scheduler decided and when. + +**Checkpoints can now be triggered during recovery** + +([FLIP-547](https://cwiki.apache.org/confluence/spaces/FLINK/pages/384895569/FLIP-547+Support+checkpoint+during+recovery)) provides relief for large jobs recovering with unaligned checkpoints. + +## Looking ahead to Flink 2.4 + +The 2.3 release branch was cut on April 15, so most of what merged to master during the quarter targets Flink 2.4. Three threads in that work are described below. + +### The SQL planner catches up with Calcite + +Flink relies on [Apache Calcite](https://calcite.apache.org/) for parsing, query optimization and validating both regular and streaming SQL. Since [July 2025](https://issues.apache.org/jira/browse/FLINK-35855) the planner had been on Calcite 1.36 while upstream moved on. +Between May 21 and July 6, five consecutive upgrades merged, all targeting 2.4.0: [1.37](https://issues.apache.org/jira/browse/FLINK-35856), [1.38](https://issues.apache.org/jira/browse/FLINK-36602), [1.39](https://issues.apache.org/jira/browse/FLINK-39817), [1.40](https://issues.apache.org/jira/browse/FLINK-39828) and [1.41](https://issues.apache.org/jira/browse/FLINK-39859), four of them landed inside the quarter. + +There is an 18-month gap between the release of [Calcite 1.37](https://calcite.apache.org/news/2024/05/06/release-1.37.0/) (05/2024) and [1.41](https://calcite.apache.org/news/2025/11/01/release-1.41.0) (11/2025), which means that the SQL planner absorbed eighteen months of upstream releases in under two months. + + +Each of the Calcite release upgrades carries work Flink could not build on while pinned to the 1.36 version, for example Calcite 1.37 [added lambda expressions to SQL](https://calcite.apache.org/news/2024/05/06/release-1.37.0/), which the upgrade unblocks for Flink SQL, although [a FLIP is still needed](https://issues.apache.org/jira/browse/FLINK-35856) before they can be exposed to users. +The upgrades also let the planner shed code it had been carrying as private copies, forked Calcite classes were [removed](https://github.com/apache/flink/commit/3212758626d73dfefdf49317b9ef1da07c2d0c72), Flink's own empty relation pruning rules were [replaced by Calcite's](https://issues.apache.org/jira/browse/FLINK-39983), and Calcite's `COALESCE` is now being used to [apply simplifications from `RexSimplify`](https://issues.apache.org/jira/browse/FLINK-39577). + +[Preparation for Calcite 1.42 is underway](https://issues.apache.org/jira/browse/FLINK-40001). + +### Keeping it simple with less repeated work in generated code + +The three changes below, all targeting 2.4.0, stop Flink SQL from doing the same work more than once per row: + +- [Common subexpression elimination](https://issues.apache.org/jira/browse/FLINK-39268) is now applied in the code Flink generates for the Calc operator in both batch and streaming. When the same deterministic expression appears in several projections or filter conditions, it is computed once and reused. The ticket's example is an expensive UDF used in two output columns and the `WHERE` clause: evaluated three times per row before this change, once after. It was proposed in March by a community member whose company had already built the same optimization internally, and merged on May 7. One practical note for 2.4 upgraders: reuse relies on the existing `isDeterministic()` contract, the same one constant folding already uses, so a UDF with side effects must declare itself non-deterministic. +- `JSON_VALUE` and `JSON_QUERY` calls that read the same JSON document [used to parse it once per call](https://issues.apache.org/jira/browse/FLINK-39638); the generated code now parses once and reuses the result. +- `REGEXP_REPLACE` [no longer recompiles its pattern on every record](https://issues.apache.org/jira/browse/FLINK-39650). The patterns are cached, the error log line that an invalid pattern used to write for every record is gone, and an invalid literal pattern now fails at planning time instead of producing a `NULL` per row. + +### Groundwork for JDK 25 + +Java 25 is the current long-term-support release, and Flink's tracking ticket for it, [FLINK-37719](https://issues.apache.org/jira/browse/FLINK-37719), has been open since April 2025 with 34 subtasks. On May 19 a platform engineer [asked on the user list](https://lists.apache.org/thread/4psgwfsjkpz5m68wfylyvx8m46v8yj3q) whether support could be targeted for 2.4. The reply from the community mailing list named the main blocker as [JEP 486](https://openjdk.org/jeps/486), which permanently disabled the Security Manager in JDK 24. Hadoop depends on it, so anything that pulls Hadoop in, `flink-s3-fs-hadoop` included, needs a newer Hadoop with [its own fixes](https://issues.apache.org/jira/browse/HADOOP-19486). + + +Within days of that thread, subtasks started landing: nine merged between May 24 and June 15, from [build flags](https://issues.apache.org/jira/browse/FLINK-39806) and a [`FailsOnJava25` test marker](https://issues.apache.org/jira/browse/FLINK-39751) to [CI running on a JDK 25 image](https://issues.apache.org/jira/browse/FLINK-39933). Three subtasks remain open, and they are the hard ones: the Security Manager removal itself, the Hadoop tests that fail because of it, and a dedicated JDK 25 CI lane. The umbrella carries no fix version, so JDK 25 as a supported runtime is not yet a 2.4 promise. Review Comment: >FailsOnJava25 i'm not sure anyone outside of context can understand what it means, it is a pure Flink's internal tests, so may be can be just omitted. On the other side: currently we can say: if you don't use Hadoop and you don't need Security Manager features, you can use Flink with jdk25. Security Manager will be likely disabled for Flink. JDK people suggest several alternatives to implement. Contributions are welcome, however I wouldn't consider it as a blocker. We will probably have ML discussion about that however it's very likely we are moving this way -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
