Hey Dominik, Lambda functions are an important building block - not only for these built-in collection functions, but also for UDFs and PTFs whitin the whole archictecture. So thanks for the proposal and for wanting to contribute.
I'll take a look at the proposal soon. You mentioned a WIP branch a couple of times in the thread - could you share the link so we can take a look at that too? Kind regards, Gustavo On Thu, 30 Jul 2026 at 16:39, <[email protected]> wrote: > Hi all, > > Following up on this FLIP to see if there are any comments or concerns. > > Since the last mail I added tests for: > - views using ARRAY_FILTER / TRANSFORM > - materialized tables going through expand-and-replan > > Everything is green locally so far. > > Feedback on the planner integration, Calcite handling, missing edge cases, > or the overall direction would be appreciated. > > As I’m still new to the FLIP process, I’d also appreciate some guidance on > the next steps from here. Many thanks in advance. > > Best, > Dominik > > From: Bünzli Dominik, SCS-INI-DNA-INF <[email protected]> > Date: Friday, 24 July 2026 at 11:17 > To: [email protected] <[email protected]> > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL > (TRANSFORM and ARRAY_FILTER) > > Hi Sergey, > > Good catch. I didn’t have these cases in mind. Quickly added test cases > for them to my branch. > > The view test creates a view like "CREATE VIEW v AS SELECT id, > ARRAY_FILTER(vals, x -> x > 2) FROM ...", then selects from it and checks > that the arrays come back right. Same idea for TRANSFORM and the map lambda > (k, v) -> v * 100. > > The materialized table test does the equivalent: a CONTINUOUS table using > ARRAY_FILTER and TRANSFORM in its query, running on a mini-cluster, then a > SELECT to confirm the output matches. > > So, both should (to my still limited knowledge), go through the > expand-and-replan path and check real results. If Calcite mangles a lambda, > these tests should fail. Looks good so far, tests are green locally. > > Best, > Dominik > > From: Sergey Nuyanzin <[email protected]> > Date: Friday, 24 July 2026 at 09:59 > To: [email protected] <[email protected]> > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL > (TRANSFORM and ARRAY_FILTER) > > > Be aware: This is an external email. > > > > thanks for the proposal > > and working in this direction > > I haven't checked yet all the things > > however I see at least one missing piece here. > > In FlinkSQL we have views and materialized tables. Both are relying on > expanded ("rewritten" in Calcite terms) SQL. > And we have already faced a number of issues around that and fixed > them in Calcite (like just a few or them [1], [2], [3], [4], [5], [6], > [7], [8], [9], [10]) > > I'm pretty sure there will be something lambda related. For that > reason I would suggest also having test with views(or materialized > tables, does not matter) based on sql with lambda > and check that SELECT from them still able to produce output > > [1] > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519482673%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=wy%2FmMDx7Bc%2Blkbjdyr1rWiEpGyOFweCNjHKHMhEGDYo%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7660> > [2] > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519515485%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=jrSfxN8AhWe%2FY%2BeRtU7IWN9A5SIZrluz9a0jPCbrauE%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7575> > [3] > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519540885%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2XqM%2FzWVAUM9M9v2BvhF2rG1Gxo2y12hPNJVV2OMbtU%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7465> > [4] > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519565082%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=0MUvjzrXcdaphlgigoN0j6xW5m5JiGYJBCe0pXpehiE%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7480> > [5] > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519590890%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=53w4cH9008310XzR343RfJKSpD%2BuiPJXL5sB4GivfMg%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7471> > [6] > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519622955%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2FeYvSOwEryd6cE9JLIU%2Fyopj20WJzeu%2Fv8h2%2FSeGpUM%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7470> > [7] > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519653261%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=PS6dyyL3YaK6Sefib5icMJZweVcQ3192NZI1owUwSIM%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7466> > [8] > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519678467%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=56Zea5qfcDszsps%2B4VOdVCvn2UeVCVx2fOYaUWlX%2FDw%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7312> > [9] > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519700401%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=KJka%2Fs%2F8YPH1b%2F16FCwTgFtsb8vFyK9oY3DGeT%2BUzxU%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7217> > [10] > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519722120%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2B1HgC5SAvuBelfegMB0CLTCYdwcb6NxCBcTzdjC0Qlw%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-6944> > > On Fri, Jul 24, 2026 at 9:37 AM <[email protected]> wrote: > > > > Hi Gyula > > > > Thanks a lot for your reply and the +1! > > > > On a conceptual level, lambdas and scalar functions / UDFs are distinct. > A lambda isn't a stored/registered value. It's represented by the > planning-only FUNCTION logical type and only exists as an argument to > higher-order functions like ARRAY_FILTER and TRANSFORM. Essentially, it's > the wrapper that introduces the element variable x. > > > > Within the lambda you can freely use any scalar function or UDF: > > ARRAY_FILTER(arr, x -> is_positive(x)) > > ARRAY_FILTER(arr, x -> is_positive(x) AND x < 100) > > > > What currently doesn't work is passing a bare UDF name in the lambda > position (i.e. automatic expansion of is_positive into x -> is_positive(x)): > > ARRAY_FILTER(arr, is_positive) -- not supported > > > > We intentionally left this out of scope for now, but it's a > straightforward follow-up if we see demand for it. > > > > Best, > > Dominik > > > > From: Gyula Fóra <[email protected]> > > Date: Thursday, 23 July 2026 at 16:07 > > To: [email protected] <[email protected]> > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL > (TRANSFORM and ARRAY_FILTER) > > > > > > Be aware: This is an external email. > > > > > > > > Hey! > > > > Without a good background knowledge about the syntax in other systems, > the > > ARRAY_FILTER, TRANSFORM functions seem really useful, especially with > > lambda functions. > > Overall +1 for this idea from a very high level but it would be great > with > > someone with more deep context on SQL functions / runtime to chime in. > > > > One question I had is how do lambda functions relate to existing scalar > > functions / UDFs? Would the user be able to use boolean valued scalar > > functions where lambdas are shown in the proposal? > > > > Cheers > > Gyula > > > > On Wed, Jul 15, 2026 at 8:40 AM <[email protected]> wrote: > > > > > Hi everyone, > > > > > > I'd like to start a discussion on a FLIP that introduces higher-order > > > functions > > > (functions that take a lambda expression as an argument) to Flink SQL, > > > together > > > with the first two built-ins that use them: TRANSFORM and ARRAY_FILTER. > > > > > > FLIP: > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519743111%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=MZQgSpOw4rNqYRG3Tm0mUigSQGXQi4dDreRhyV%2F4wZY%3D&reserved=0 > < > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C9ceb773823ed4fe4adcf08dee9596eea%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639204767519772568%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XNwMzQrplv6Pt7nGCO%2F0Jnk2YFD9v%2FJtP7b7iiyeGGQ%3D&reserved=0 > >< > https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing > > > > > > > > Motivation > > > > > > Users migrating to Flink SQL from Spark, Databricks, Snowflake, > DuckDB, and > > > Presto/Trino expect to manipulate collections (arrays and maps) inline > > > with a > > > lambda instead of UNNEST + re-aggregate rewrites or bespoke UDFs. The > > > absence of > > > higher-order collection functions forces verbose query rewrites during > > > migration > > > and raises time-to-first-query. TRANSFORM(array, x -> x + 1) and > > > ARRAY_FILTER(array, x -> x > 0) are the two most requested entry points > > > and, > > > importantly, they can be built on lambda infrastructure that already > > > exists in > > > Calcite (CALCITE-3679), so the surface area we add on the Flink side is > > > rather > > > small. > > > > > > Summary > > > > > > • Introduce a new logical type FUNCTION that describes the type of a > > > lambda > > > (its argument types and its result type). This is the type-system > > > foundation > > > every higher-order function needs; it is a planning/translation > helper > > > type > > > and is not a persisted column type. > > > • Add ARRAY_FILTER(array, element -> predicate), which returns a new > > > array > > > containing only the elements for which the predicate holds. > > > • Add TRANSFORM(collection, lambda), which applies a lambda to every > > > element of > > > an array (TRANSFORM(array, x -> expr)) or every entry of a map > > > (TRANSFORM(map, (k, v) -> expr)), returning a new array/map. > > > • Reuse Calcite's lambda parsing, validation and > > > RexLambda/FunctionSqlType > > > infrastructure (CALCITE-3679) rather than inventing a > Flink-specific > > > lambda > > > syntax. The lambda arrow syntax x -> expr and (k, v) -> expr is > already > > > parseable by the Calcite version Flink bundles (1.41.0). > > > > > > The functions are net-new syntax and are additive: no existing query > > > changes > > > behavior. There is no new configuration option — the functions are > always > > > available once the release ships. > > > > > > Examples: > > > > > > SELECT ARRAY_FILTER(ARRAY[1, 2, 3, 4], x -> x > 2); -- > > > [3, 4] > > > SELECT TRANSFORM(ARRAY[1, 2, 3], x -> x * 10); -- > > > [10, 20, 30] > > > SELECT TRANSFORM(ARRAY['a', 'bb', 'ccc'], s -> CHAR_LENGTH(s)); -- > > > [1, 2, 3] > > > SELECT TRANSFORM(MAP['a', 1, 'b', 2], (k, v) -> v * 100); -- > > > {a=100, b=200} > > > > > > Thanks, > > > Dominik Bünzli > > > Data, Analytics & AI Engineer > > > > > > > -- > Best regards, > Sergey >
