Hi Michal and Dominik, Thanks for working on this - the new document is much more mature than the first one! In general, I think we're in a good direction, +1.
A few points: 1) Splitting the FLIP This is a lot of content for careful review, and each function really needs its own discussion. If we include so many in one FLIP, the review will probably take even longer. I'd suggest we focus on the foundation first and get that cleared, and have the function discussions in a second FLIP (and partially on the pull requests). Suggestion: - FLIP 1 (foundation): lambda syntax + FUNCTION type + capture lifting + codegen + Table API/Python etc. It can include a few example functions (e.g. ARRAY_TRANSFORM, ARRAY_FILTER, ARRAY_REDUCE) to exercise the design, but without committing to their shape. - FLIP 2+ (breadth): a second FLIP to discuss the first set of functions, their APIs and shapes. 2) Making it a FLIP document The document currently reads like a long decision log. To make it official as a FLIP, I think we should turn it into a concise, easy-to-read document: - Make it as short as possible with ofc still all the relevant information we need. Right now at 85 pages, it's really only consumable by AI. - Add an index at the top so it's easy to navigate. We often come back to a FLIP many times over the years, so this matters. - Add examples wherever we can - examples are in general super helpful to immediately grasp the content. There are some short ones in the "Composition" section - add more whetever you can. - As orientation, it helps to look at how recent FLIPs are structured. I'm happy to help make sure this lands on our Confluence when we're ready. 3) Content feedback (for later, not the first FLIP) Note: I don't think these belong in the first FLIP we should focus on, so feel free to ignore them for now - just sharing for the future and feel free to not reply directly to these here: - Predicates like exists/forall (Spark) and any_match/all_match/none_match (Trino). Very common, and hard to express well with FILTER + CARDINALITY. - An (element, index) variant of transform/filter, like Spark/Trino transform(a, (x, i) -> ...) with i the 0-based index. Pretty core and useful. - A custom comparator array_sort(array, comparator), as in Spark/Trino (2-arg comparator returning negative/0/positive). 4) EXPLAIN improvement For a capturing lambda, the plan shows the lifted form: the user writes x -> x + base, but the plan renders (x, cap$1) -> (x + cap$1), base. Could we de-lift at render time so EXPLAIN shows intent, not the optimization - i.e. Calc(select=[ARRAY_TRANSFORM(a, x -> (x + base)) AS EXPR$0])? The internal representation can stay as-is. Thanks again for driving this, Kind regards, Gustavo On Sun, 9 Aug 2026 at 10:05, Michal Stutzmann <[email protected]> wrote: > Hi Gustavo, > > Thanks for looking into this and the pointers. We have now included more > details in the FLIP draft (link shared by Dominik in the previous e-mail: > > https://docs.google.com/document/d/1vd-l922wRJGkbhsGaT9TxRTG2UKjj7nAtxC8J9RaG1U/edit?usp=sharing > ). > > In short: > - A UDF called from a lambda body — supported. > - A UDF passed as the lambda, e.g. `ARRAY_TRANSFORM(a, my_udf)`, is not > supported — this would need eta-expansion. The same call written as > `ARRAY_TRANSFORM(a, x -> my_udf(x))` is supported. > - Nesting: supported; arbitrary depth, any combination of built-in and > user-defined, in both directions, with each level capturing the > parameters of all enclosing lambdas plus outer columns. > > All three are covered by a new section in the FLIP draft called Public > Interfaces / 6. Composition, which lists the supported compositions with an > example for each. The mechanism behind it is in Proposed Changes / Lambda > captures. > > Every one of those is covered by an IT case that executes the query and > asserts the result. If you have other combinations in mind, we will add > them. > > Best, > Michal > > > On Fri, Aug 7, 2026 at 4:15 PM Gustavo de Morais <[email protected]> > wrote: > > > Hi Dominik, > > > > Thanks for the reply and for considering these. It's great that you're > both > > working on this. As mentioned, it's a very important feature, but there > are > > indeed some cases that might grow its scope - we should make sure we > design > > it in a complete way that fits the different parts of the project. > > > > I'll try to make some time next week to take a deeper look at what you > > shared. In the meantime, here are a few more pointers you can already > look > > into: you mentioned lambda arguments for user-defined functions - what > > about a UDF as an argument of a lambda function? Nesting is also worth > > considering: multiple nested lambda functions, lambda function -> UDF or > > regular function -> lambda function -> ... etc. > > > > Kind regards, > > Gustavo > > > > On Fri, 7 Aug 2026 at 11:31, <[email protected]> wrote: > > > > > Hi Gustavo, > > > > > > Sorry for the late response. I had "a few" discussions with my > colleague > > > Michal Stutzmann (you might remember him from the Flink Forward > > conference > > > in Barcelona) about the FLIP and the features it should cover. Michal > > > extended the proposal< > > > > > > https://docs.google.com/document/d/1vd-l922wRJGkbhsGaT9TxRTG2UKjj7nAtxC8J9RaG1U/edit?usp=sharing > > > > > > and implementation to also support the Table & Python APIs, as well as > > > lambda arguments for user-defined functions. Additionally, we now > support > > > using columns within lambda expressions. > > > > > > We pushed the WIP changes to a branch in our fork, which you can find > > here: > > > > > > > > > > > > https://github.com/swisscom-bigdata/flink/tree/flink-31207-higher-order-functions-and-lambdas > > > > > > The set of changes grew larger than expected, but in our opinion they > > > significantly simplify the user experience by reducing the need to work > > > with UDFs or custom code to handle collections. > > > > > > Thanks a lot in advance for taking a look! > > > > > > Best, > > > Dominik > > > > > > From: Gustavo de Morais <[email protected]> > > > Date: Friday, 31 July 2026 at 12:05 > > > To: [email protected] <[email protected]> > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL > > > (TRANSFORM and ARRAY_FILTER) > > > > > > > > > Be aware: This is an external email. > > > > > > > > > > > > Hey Dominik, > > > > > > Lambda functions are an important building block - not only for these > > > built-in collection functions, but also for UDFs and PTFs whitin the > > whole > > > archictecture. So thanks for the proposal and for wanting to > contribute. > > > > > > I'll take a look at the proposal soon. You mentioned a WIP branch a > > couple > > > of times in the thread - could you share the link so we can take a look > > at > > > that too? > > > > > > Kind regards, > > > Gustavo > > > > > > On Thu, 30 Jul 2026 at 16:39, <[email protected]> wrote: > > > > > > > Hi all, > > > > > > > > Following up on this FLIP to see if there are any comments or > concerns. > > > > > > > > Since the last mail I added tests for: > > > > - views using ARRAY_FILTER / TRANSFORM > > > > - materialized tables going through expand-and-replan > > > > > > > > Everything is green locally so far. > > > > > > > > Feedback on the planner integration, Calcite handling, missing edge > > > cases, > > > > or the overall direction would be appreciated. > > > > > > > > As I’m still new to the FLIP process, I’d also appreciate some > guidance > > > on > > > > the next steps from here. Many thanks in advance. > > > > > > > > Best, > > > > Dominik > > > > > > > > From: Bünzli Dominik, SCS-INI-DNA-INF <[email protected]> > > > > Date: Friday, 24 July 2026 at 11:17 > > > > To: [email protected] <[email protected]> > > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL > > > > (TRANSFORM and ARRAY_FILTER) > > > > > > > > Hi Sergey, > > > > > > > > Good catch. I didn’t have these cases in mind. Quickly added test > cases > > > > for them to my branch. > > > > > > > > The view test creates a view like "CREATE VIEW v AS SELECT id, > > > > ARRAY_FILTER(vals, x -> x > 2) FROM ...", then selects from it and > > checks > > > > that the arrays come back right. Same idea for TRANSFORM and the map > > > lambda > > > > (k, v) -> v * 100. > > > > > > > > The materialized table test does the equivalent: a CONTINUOUS table > > using > > > > ARRAY_FILTER and TRANSFORM in its query, running on a mini-cluster, > > then > > > a > > > > SELECT to confirm the output matches. > > > > > > > > So, both should (to my still limited knowledge), go through the > > > > expand-and-replan path and check real results. If Calcite mangles a > > > lambda, > > > > these tests should fail. Looks good so far, tests are green locally. > > > > > > > > Best, > > > > Dominik > > > > > > > > From: Sergey Nuyanzin <[email protected]> > > > > Date: Friday, 24 July 2026 at 09:59 > > > > To: [email protected] <[email protected]> > > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL > > > > (TRANSFORM and ARRAY_FILTER) > > > > > > > > > > > > Be aware: This is an external email. > > > > > > > > > > > > > > > > thanks for the proposal > > > > > > > > and working in this direction > > > > > > > > I haven't checked yet all the things > > > > > > > > however I see at least one missing piece here. > > > > > > > > In FlinkSQL we have views and materialized tables. Both are relying > on > > > > expanded ("rewritten" in Calcite terms) SQL. > > > > And we have already faced a number of issues around that and fixed > > > > them in Calcite (like just a few or them [1], [2], [3], [4], [5], > [6], > > > > [7], [8], [9], [10]) > > > > > > > > I'm pretty sure there will be something lambda related. For that > > > > reason I would suggest also having test with views(or materialized > > > > tables, does not matter) based on sql with lambda > > > > and check that SELECT from them still able to produce output > > > > > > > > [1] > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215787243%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=dhAIi9%2BVQbdypDURBS6hCR97oOt1Y%2BWDeYhcppgEH18%3D&reserved=0 > > > <https://issues.apache.org/jira/browse/CALCITE-7660> > > > > < > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215837165%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=WJ6rRpo1oeuVua%2BU6r65we7Q%2FyAkiVC3drjq4y%2F37ZM%3D&reserved=0 > > > <https://issues.apache.org/jira/browse/CALCITE-7660>> > > > > [2] > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215876426%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2yAtLVRqA5lHAT4dKoiNgOEZmCDrNLlq6XnXvidm9dk%3D&reserved=0 > > > <https://issues.apache.org/jira/browse/CALCITE-7575> > > > > < > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215913003%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=M%2Bv0m4nRXd%2B%2BbbvqPd8gB8PwbdSWqBG4eLVWMoUR4vo%3D&reserved=0 > > > <https://issues.apache.org/jira/browse/CALCITE-7575>> > > > > [3] > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215947484%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2F2pRc%2BwqNk80bEjsaexGT8uxTKn%2FGuw%2BBPSEYz3lGnk%3D&reserved=0 > > > <https://issues.apache.org/jira/browse/CALCITE-7465> > > > > < > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215983673%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=n14%2FEnyO%2F2WIcPudrEWwFaQ8KHqRIppuqRfJxK3NqtA%3D&reserved=0 > > > <https://issues.apache.org/jira/browse/CALCITE-7465>> > > > > [4] > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216022139%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=6Gyk1uF2SfLH0J7lQqhBYPDIU53NVYIzCrg0GrhMN7w%3D&reserved=0 > > > <https://issues.apache.org/jira/browse/CALCITE-7480> > > > > < > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216057890%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=f4gNZnMW6VBq12Yt8PkIzdFfUwdX8fYXTJOBOCxMOwg%3D&reserved=0 > > > <https://issues.apache.org/jira/browse/CALCITE-7480>> > > > > [5] > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216090334%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=d5YrH4adMbZBaEhqXEns9xR0IerRNTsjUZeC5DXQE50%3D&reserved=0 > > > <https://issues.apache.org/jira/browse/CALCITE-7471> > > > > < > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216116472%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2ZyKirj6%2FrAmUM77mq35uKuogcNVn3vXfQqT5B9mTEU%3D&reserved=0 > > > <https://issues.apache.org/jira/browse/CALCITE-7471>> > > > > [6] > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216150666%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2Fuu4GC7suv7WSI4HtZqxGtvTkV4HarcODd5Q8W7zDpQ%3D&reserved=0 > > > <https://issues.apache.org/jira/browse/CALCITE-7470> > > > > < > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216179872%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JS14KTB4YncuYb7dkjjQFu%2FVRvqjaHnPfTEjCn2kWDw%3D&reserved=0 > > > <https://issues.apache.org/jira/browse/CALCITE-7470>> > > > > [7] > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216199448%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=mrskfIM%2FHIKFxL1foGM9XiGiphFwbPxO9q0n%2FubrWS8%3D&reserved=0 > > > <https://issues.apache.org/jira/browse/CALCITE-7466> > > > > < > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216219518%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=r9y6sPdjB5UpVXSiVl8rq3XULGsl8SxlIWtY9LHlCkU%3D&reserved=0 > > > <https://issues.apache.org/jira/browse/CALCITE-7466>> > > > > [8] > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216239592%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Ra1qmGy%2Fnnxs98tuQZBMHMxceJ3NkYvyfKlwU1cHl84%3D&reserved=0 > > > <https://issues.apache.org/jira/browse/CALCITE-7312> > > > > < > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216258677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=GdHVoxqW6y2kiUi%2F3pckMK8szOLnGwqCtW4sconw4XQ%3D&reserved=0 > > > <https://issues.apache.org/jira/browse/CALCITE-7312>> > > > > [9] > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216284576%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=k%2FBsEShTSDCzRCfsgPZXz7cYJJnDsaeRnW45ldO5RiY%3D&reserved=0 > > > <https://issues.apache.org/jira/browse/CALCITE-7217> > > > > < > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216319677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XhpIIafvVURhq8tbckhdDytfZ9P%2FsJDoaEoIpYlOBW4%3D&reserved=0 > > > <https://issues.apache.org/jira/browse/CALCITE-7217>> > > > > [10] > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216350464%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ZiPFaESvPoORqz9KLJLh9I7z6IanGWhL2ZnCl2Ap9N4%3D&reserved=0 > > > <https://issues.apache.org/jira/browse/CALCITE-6944> > > > > < > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216379236%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Pqe2YCqIdEHIZbTSrLF6X6BAlGRej0vdDaOL%2F2J1%2BeM%3D&reserved=0 > > > <https://issues.apache.org/jira/browse/CALCITE-6944>> > > > > > > > > On Fri, Jul 24, 2026 at 9:37 AM <[email protected]> > wrote: > > > > > > > > > > Hi Gyula > > > > > > > > > > Thanks a lot for your reply and the +1! > > > > > > > > > > On a conceptual level, lambdas and scalar functions / UDFs are > > > distinct. > > > > A lambda isn't a stored/registered value. It's represented by the > > > > planning-only FUNCTION logical type and only exists as an argument to > > > > higher-order functions like ARRAY_FILTER and TRANSFORM. Essentially, > > it's > > > > the wrapper that introduces the element variable x. > > > > > > > > > > Within the lambda you can freely use any scalar function or UDF: > > > > > ARRAY_FILTER(arr, x -> is_positive(x)) > > > > > ARRAY_FILTER(arr, x -> is_positive(x) AND x < 100) > > > > > > > > > > What currently doesn't work is passing a bare UDF name in the > lambda > > > > position (i.e. automatic expansion of is_positive into x -> > > > is_positive(x)): > > > > > ARRAY_FILTER(arr, is_positive) -- not supported > > > > > > > > > > We intentionally left this out of scope for now, but it's a > > > > straightforward follow-up if we see demand for it. > > > > > > > > > > Best, > > > > > Dominik > > > > > > > > > > From: Gyula Fóra <[email protected]> > > > > > Date: Thursday, 23 July 2026 at 16:07 > > > > > To: [email protected] <[email protected]> > > > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink > SQL > > > > (TRANSFORM and ARRAY_FILTER) > > > > > > > > > > > > > > > Be aware: This is an external email. > > > > > > > > > > > > > > > > > > > > Hey! > > > > > > > > > > Without a good background knowledge about the syntax in other > > systems, > > > > the > > > > > ARRAY_FILTER, TRANSFORM functions seem really useful, especially > with > > > > > lambda functions. > > > > > Overall +1 for this idea from a very high level but it would be > great > > > > with > > > > > someone with more deep context on SQL functions / runtime to chime > > in. > > > > > > > > > > One question I had is how do lambda functions relate to existing > > scalar > > > > > functions / UDFs? Would the user be able to use boolean valued > scalar > > > > > functions where lambdas are shown in the proposal? > > > > > > > > > > Cheers > > > > > Gyula > > > > > > > > > > On Wed, Jul 15, 2026 at 8:40 AM <[email protected]> > > wrote: > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > I'd like to start a discussion on a FLIP that introduces > > higher-order > > > > > > functions > > > > > > (functions that take a lambda expression as an argument) to Flink > > > SQL, > > > > > > together > > > > > > with the first two built-ins that use them: TRANSFORM and > > > ARRAY_FILTER. > > > > > > > > > > > > FLIP: > > > > > > > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216408066%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=UuKdaeEmSd6hP87gokZhU9leFfw0BrrD2buEDctKjBk%3D&reserved=0 > > > < > > > > > > https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing > > > > > > > > < > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216440590%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=woLr6CpsHkDCs7y6yZaJ4%2FoCNhxCxuU9hXqh8AaRSVI%3D&reserved=0 > > > < > > > > > > https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing > > > > > > > > >< > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216476842%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=CuKWtmNDVEXK11ET%2BJDOqAp1b%2FHmHR5ionqmomI7cdE%3D&reserved=0 > > > < > > > > > > https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing > > > > > > > > > > > > > > > > > > > > > Motivation > > > > > > > > > > > > Users migrating to Flink SQL from Spark, Databricks, Snowflake, > > > > DuckDB, and > > > > > > Presto/Trino expect to manipulate collections (arrays and maps) > > > inline > > > > > > with a > > > > > > lambda instead of UNNEST + re-aggregate rewrites or bespoke UDFs. > > The > > > > > > absence of > > > > > > higher-order collection functions forces verbose query rewrites > > > during > > > > > > migration > > > > > > and raises time-to-first-query. TRANSFORM(array, x -> x + 1) and > > > > > > ARRAY_FILTER(array, x -> x > 0) are the two most requested entry > > > points > > > > > > and, > > > > > > importantly, they can be built on lambda infrastructure that > > already > > > > > > exists in > > > > > > Calcite (CALCITE-3679), so the surface area we add on the Flink > > side > > > is > > > > > > rather > > > > > > small. > > > > > > > > > > > > Summary > > > > > > > > > > > > • Introduce a new logical type FUNCTION that describes the type > > of > > > a > > > > > > lambda > > > > > > (its argument types and its result type). This is the > > type-system > > > > > > foundation > > > > > > every higher-order function needs; it is a > planning/translation > > > > helper > > > > > > type > > > > > > and is not a persisted column type. > > > > > > • Add ARRAY_FILTER(array, element -> predicate), which returns > a > > > new > > > > > > array > > > > > > containing only the elements for which the predicate holds. > > > > > > • Add TRANSFORM(collection, lambda), which applies a lambda to > > > every > > > > > > element of > > > > > > an array (TRANSFORM(array, x -> expr)) or every entry of a > map > > > > > > (TRANSFORM(map, (k, v) -> expr)), returning a new array/map. > > > > > > • Reuse Calcite's lambda parsing, validation and > > > > > > RexLambda/FunctionSqlType > > > > > > infrastructure (CALCITE-3679) rather than inventing a > > > > Flink-specific > > > > > > lambda > > > > > > syntax. The lambda arrow syntax x -> expr and (k, v) -> expr > is > > > > already > > > > > > parseable by the Calcite version Flink bundles (1.41.0). > > > > > > > > > > > > The functions are net-new syntax and are additive: no existing > > query > > > > > > changes > > > > > > behavior. There is no new configuration option — the functions > are > > > > always > > > > > > available once the release ships. > > > > > > > > > > > > Examples: > > > > > > > > > > > > SELECT ARRAY_FILTER(ARRAY[1, 2, 3, 4], x -> x > 2); > > > -- > > > > > > [3, 4] > > > > > > SELECT TRANSFORM(ARRAY[1, 2, 3], x -> x * 10); > > > -- > > > > > > [10, 20, 30] > > > > > > SELECT TRANSFORM(ARRAY['a', 'bb', 'ccc'], s -> > CHAR_LENGTH(s)); > > > -- > > > > > > [1, 2, 3] > > > > > > SELECT TRANSFORM(MAP['a', 1, 'b', 2], (k, v) -> v * 100); > > > -- > > > > > > {a=100, b=200} > > > > > > > > > > > > Thanks, > > > > > > Dominik Bünzli > > > > > > Data, Analytics & AI Engineer > > > > > > > > > > > > > > > > > > > > > > -- > > > > Best regards, > > > > Sergey > > > > > > > > > >
