Hi Gyula Thanks a lot for your reply and the +1!
On a conceptual level, lambdas and scalar functions / UDFs are distinct. A lambda isn't a stored/registered value. It's represented by the planning-only FUNCTION logical type and only exists as an argument to higher-order functions like ARRAY_FILTER and TRANSFORM. Essentially, it's the wrapper that introduces the element variable x. Within the lambda you can freely use any scalar function or UDF: ARRAY_FILTER(arr, x -> is_positive(x)) ARRAY_FILTER(arr, x -> is_positive(x) AND x < 100) What currently doesn't work is passing a bare UDF name in the lambda position (i.e. automatic expansion of is_positive into x -> is_positive(x)): ARRAY_FILTER(arr, is_positive) -- not supported We intentionally left this out of scope for now, but it's a straightforward follow-up if we see demand for it. Best, Dominik From: Gyula Fóra <[email protected]> Date: Thursday, 23 July 2026 at 16:07 To: [email protected] <[email protected]> Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL (TRANSFORM and ARRAY_FILTER) Be aware: This is an external email. Hey! Without a good background knowledge about the syntax in other systems, the ARRAY_FILTER, TRANSFORM functions seem really useful, especially with lambda functions. Overall +1 for this idea from a very high level but it would be great with someone with more deep context on SQL functions / runtime to chime in. One question I had is how do lambda functions relate to existing scalar functions / UDFs? Would the user be able to use boolean valued scalar functions where lambdas are shown in the proposal? Cheers Gyula On Wed, Jul 15, 2026 at 8:40 AM <[email protected]> wrote: > Hi everyone, > > I'd like to start a discussion on a FLIP that introduces higher-order > functions > (functions that take a lambda expression as an argument) to Flink SQL, > together > with the first two built-ins that use them: TRANSFORM and ARRAY_FILTER. > > FLIP: > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C1ad1fb4b5c76419d815308dee8c3bbc2%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204124551661418%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=S3%2BjqrnvtFjxTyC%2BaRTwRV3ynymQB6Jru1fcO4gw5Ew%3D&reserved=0<https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing> > > Motivation > > Users migrating to Flink SQL from Spark, Databricks, Snowflake, DuckDB, and > Presto/Trino expect to manipulate collections (arrays and maps) inline > with a > lambda instead of UNNEST + re-aggregate rewrites or bespoke UDFs. The > absence of > higher-order collection functions forces verbose query rewrites during > migration > and raises time-to-first-query. TRANSFORM(array, x -> x + 1) and > ARRAY_FILTER(array, x -> x > 0) are the two most requested entry points > and, > importantly, they can be built on lambda infrastructure that already > exists in > Calcite (CALCITE-3679), so the surface area we add on the Flink side is > rather > small. > > Summary > > • Introduce a new logical type FUNCTION that describes the type of a > lambda > (its argument types and its result type). This is the type-system > foundation > every higher-order function needs; it is a planning/translation helper > type > and is not a persisted column type. > • Add ARRAY_FILTER(array, element -> predicate), which returns a new > array > containing only the elements for which the predicate holds. > • Add TRANSFORM(collection, lambda), which applies a lambda to every > element of > an array (TRANSFORM(array, x -> expr)) or every entry of a map > (TRANSFORM(map, (k, v) -> expr)), returning a new array/map. > • Reuse Calcite's lambda parsing, validation and > RexLambda/FunctionSqlType > infrastructure (CALCITE-3679) rather than inventing a Flink-specific > lambda > syntax. The lambda arrow syntax x -> expr and (k, v) -> expr is already > parseable by the Calcite version Flink bundles (1.41.0). > > The functions are net-new syntax and are additive: no existing query > changes > behavior. There is no new configuration option — the functions are always > available once the release ships. > > Examples: > > SELECT ARRAY_FILTER(ARRAY[1, 2, 3, 4], x -> x > 2); -- > [3, 4] > SELECT TRANSFORM(ARRAY[1, 2, 3], x -> x * 10); -- > [10, 20, 30] > SELECT TRANSFORM(ARRAY['a', 'bb', 'ccc'], s -> CHAR_LENGTH(s)); -- > [1, 2, 3] > SELECT TRANSFORM(MAP['a', 1, 'b', 2], (k, v) -> v * 100); -- > {a=100, b=200} > > Thanks, > Dominik Bünzli > Data, Analytics & AI Engineer >
