+1 for the overall direction; we've needed this. It will significantly improve both usability and readability.
Also +1 for Gustavo's recommendations. The overall scope is just too much for one FLIP. On Fri, Aug 21, 2026 at 8:42 AM Gustavo de Morais <[email protected]> wrote: > Hi Michal and Dominik, > > Thanks for working on this - the new document is much more mature than the > first one! In general, I think we're in a good direction, +1. > > A few points: > > 1) Splitting the FLIP > > This is a lot of content for careful review, and each function really needs > its own discussion. If we include so many in one FLIP, the review will > probably take even longer. I'd suggest we focus on the foundation first and > get that cleared, and have the function discussions in a second FLIP (and > partially on the pull requests). > > Suggestion: > - FLIP 1 (foundation): lambda syntax + FUNCTION type + capture lifting + > codegen + Table API/Python etc. It can include a few example functions > (e.g. ARRAY_TRANSFORM, ARRAY_FILTER, ARRAY_REDUCE) to exercise the design, > but without committing to their shape. > - FLIP 2+ (breadth): a second FLIP to discuss the first set of functions, > their APIs and shapes. > > 2) Making it a FLIP document > > The document currently reads like a long decision log. To make it official > as a FLIP, I think we should turn it into a concise, easy-to-read document: > - Make it as short as possible with ofc still all the relevant information > we need. Right now at 85 pages, it's really only consumable by AI. > - Add an index at the top so it's easy to navigate. We often come back to a > FLIP many times over the years, so this matters. > - Add examples wherever we can - examples are in general super helpful to > immediately grasp the content. There are some short ones in the > "Composition" section - add more whetever you can. > - As orientation, it helps to look at how recent FLIPs are structured. I'm > happy to help make sure this lands on our Confluence when we're ready. > > 3) Content feedback (for later, not the first FLIP) > > Note: I don't think these belong in the first FLIP we should focus on, so > feel free to ignore them for now - just sharing for the future and feel > free to not reply directly to these here: > - Predicates like exists/forall (Spark) and any_match/all_match/none_match > (Trino). Very common, and hard to express well with FILTER + CARDINALITY. > - An (element, index) variant of transform/filter, like Spark/Trino > transform(a, (x, i) -> ...) with i the 0-based index. Pretty core and > useful. > - A custom comparator array_sort(array, comparator), as in Spark/Trino > (2-arg comparator returning negative/0/positive). > > 4) EXPLAIN improvement > > For a capturing lambda, the plan shows the lifted form: the user writes x > -> x + base, but the plan renders (x, cap$1) -> (x + cap$1), base. Could we > de-lift at render time so EXPLAIN shows intent, not the optimization - i.e. > Calc(select=[ARRAY_TRANSFORM(a, x -> (x + base)) AS EXPR$0])? The internal > representation can stay as-is. > > Thanks again for driving this, > > Kind regards, > Gustavo > > On Sun, 9 Aug 2026 at 10:05, Michal Stutzmann <[email protected]> > wrote: > > > Hi Gustavo, > > > > Thanks for looking into this and the pointers. We have now included more > > details in the FLIP draft (link shared by Dominik in the previous e-mail: > > > > > https://docs.google.com/document/d/1vd-l922wRJGkbhsGaT9TxRTG2UKjj7nAtxC8J9RaG1U/edit?usp=sharing > > ). > > > > In short: > > - A UDF called from a lambda body — supported. > > - A UDF passed as the lambda, e.g. `ARRAY_TRANSFORM(a, my_udf)`, is not > > supported — this would need eta-expansion. The same call written as > > `ARRAY_TRANSFORM(a, x -> my_udf(x))` is supported. > > - Nesting: supported; arbitrary depth, any combination of built-in and > > user-defined, in both directions, with each level capturing the > > parameters of all enclosing lambdas plus outer columns. > > > > All three are covered by a new section in the FLIP draft called Public > > Interfaces / 6. Composition, which lists the supported compositions with > an > > example for each. The mechanism behind it is in Proposed Changes / Lambda > > captures. > > > > Every one of those is covered by an IT case that executes the query and > > asserts the result. If you have other combinations in mind, we will add > > them. > > > > Best, > > Michal > > > > > > On Fri, Aug 7, 2026 at 4:15 PM Gustavo de Morais <[email protected] > > > > wrote: > > > > > Hi Dominik, > > > > > > Thanks for the reply and for considering these. It's great that you're > > both > > > working on this. As mentioned, it's a very important feature, but there > > are > > > indeed some cases that might grow its scope - we should make sure we > > design > > > it in a complete way that fits the different parts of the project. > > > > > > I'll try to make some time next week to take a deeper look at what you > > > shared. In the meantime, here are a few more pointers you can already > > look > > > into: you mentioned lambda arguments for user-defined functions - what > > > about a UDF as an argument of a lambda function? Nesting is also worth > > > considering: multiple nested lambda functions, lambda function -> UDF > or > > > regular function -> lambda function -> ... etc. > > > > > > Kind regards, > > > Gustavo > > > > > > On Fri, 7 Aug 2026 at 11:31, <[email protected]> wrote: > > > > > > > Hi Gustavo, > > > > > > > > Sorry for the late response. I had "a few" discussions with my > > colleague > > > > Michal Stutzmann (you might remember him from the Flink Forward > > > conference > > > > in Barcelona) about the FLIP and the features it should cover. Michal > > > > extended the proposal< > > > > > > > > > > https://docs.google.com/document/d/1vd-l922wRJGkbhsGaT9TxRTG2UKjj7nAtxC8J9RaG1U/edit?usp=sharing > > > > > > > > and implementation to also support the Table & Python APIs, as well > as > > > > lambda arguments for user-defined functions. Additionally, we now > > support > > > > using columns within lambda expressions. > > > > > > > > We pushed the WIP changes to a branch in our fork, which you can find > > > here: > > > > > > > > > > > > > > > > > > https://github.com/swisscom-bigdata/flink/tree/flink-31207-higher-order-functions-and-lambdas > > > > > > > > The set of changes grew larger than expected, but in our opinion they > > > > significantly simplify the user experience by reducing the need to > work > > > > with UDFs or custom code to handle collections. > > > > > > > > Thanks a lot in advance for taking a look! > > > > > > > > Best, > > > > Dominik > > > > > > > > From: Gustavo de Morais <[email protected]> > > > > Date: Friday, 31 July 2026 at 12:05 > > > > To: [email protected] <[email protected]> > > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL > > > > (TRANSFORM and ARRAY_FILTER) > > > > > > > > > > > > Be aware: This is an external email. > > > > > > > > > > > > > > > > Hey Dominik, > > > > > > > > Lambda functions are an important building block - not only for these > > > > built-in collection functions, but also for UDFs and PTFs whitin the > > > whole > > > > archictecture. So thanks for the proposal and for wanting to > > contribute. > > > > > > > > I'll take a look at the proposal soon. You mentioned a WIP branch a > > > couple > > > > of times in the thread - could you share the link so we can take a > look > > > at > > > > that too? > > > > > > > > Kind regards, > > > > Gustavo > > > > > > > > On Thu, 30 Jul 2026 at 16:39, <[email protected]> wrote: > > > > > > > > > Hi all, > > > > > > > > > > Following up on this FLIP to see if there are any comments or > > concerns. > > > > > > > > > > Since the last mail I added tests for: > > > > > - views using ARRAY_FILTER / TRANSFORM > > > > > - materialized tables going through expand-and-replan > > > > > > > > > > Everything is green locally so far. > > > > > > > > > > Feedback on the planner integration, Calcite handling, missing edge > > > > cases, > > > > > or the overall direction would be appreciated. > > > > > > > > > > As I’m still new to the FLIP process, I’d also appreciate some > > guidance > > > > on > > > > > the next steps from here. Many thanks in advance. > > > > > > > > > > Best, > > > > > Dominik > > > > > > > > > > From: Bünzli Dominik, SCS-INI-DNA-INF < > [email protected]> > > > > > Date: Friday, 24 July 2026 at 11:17 > > > > > To: [email protected] <[email protected]> > > > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink > SQL > > > > > (TRANSFORM and ARRAY_FILTER) > > > > > > > > > > Hi Sergey, > > > > > > > > > > Good catch. I didn’t have these cases in mind. Quickly added test > > cases > > > > > for them to my branch. > > > > > > > > > > The view test creates a view like "CREATE VIEW v AS SELECT id, > > > > > ARRAY_FILTER(vals, x -> x > 2) FROM ...", then selects from it and > > > checks > > > > > that the arrays come back right. Same idea for TRANSFORM and the > map > > > > lambda > > > > > (k, v) -> v * 100. > > > > > > > > > > The materialized table test does the equivalent: a CONTINUOUS table > > > using > > > > > ARRAY_FILTER and TRANSFORM in its query, running on a mini-cluster, > > > then > > > > a > > > > > SELECT to confirm the output matches. > > > > > > > > > > So, both should (to my still limited knowledge), go through the > > > > > expand-and-replan path and check real results. If Calcite mangles a > > > > lambda, > > > > > these tests should fail. Looks good so far, tests are green > locally. > > > > > > > > > > Best, > > > > > Dominik > > > > > > > > > > From: Sergey Nuyanzin <[email protected]> > > > > > Date: Friday, 24 July 2026 at 09:59 > > > > > To: [email protected] <[email protected]> > > > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink > SQL > > > > > (TRANSFORM and ARRAY_FILTER) > > > > > > > > > > > > > > > Be aware: This is an external email. > > > > > > > > > > > > > > > > > > > > thanks for the proposal > > > > > > > > > > and working in this direction > > > > > > > > > > I haven't checked yet all the things > > > > > > > > > > however I see at least one missing piece here. > > > > > > > > > > In FlinkSQL we have views and materialized tables. Both are relying > > on > > > > > expanded ("rewritten" in Calcite terms) SQL. > > > > > And we have already faced a number of issues around that and fixed > > > > > them in Calcite (like just a few or them [1], [2], [3], [4], [5], > > [6], > > > > > [7], [8], [9], [10]) > > > > > > > > > > I'm pretty sure there will be something lambda related. For that > > > > > reason I would suggest also having test with views(or materialized > > > > > tables, does not matter) based on sql with lambda > > > > > and check that SELECT from them still able to produce output > > > > > > > > > > [1] > > > > > > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215787243%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=dhAIi9%2BVQbdypDURBS6hCR97oOt1Y%2BWDeYhcppgEH18%3D&reserved=0 > > > > <https://issues.apache.org/jira/browse/CALCITE-7660> > > > > > < > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215837165%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=WJ6rRpo1oeuVua%2BU6r65we7Q%2FyAkiVC3drjq4y%2F37ZM%3D&reserved=0 > > > > <https://issues.apache.org/jira/browse/CALCITE-7660>> > > > > > [2] > > > > > > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215876426%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2yAtLVRqA5lHAT4dKoiNgOEZmCDrNLlq6XnXvidm9dk%3D&reserved=0 > > > > <https://issues.apache.org/jira/browse/CALCITE-7575> > > > > > < > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215913003%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=M%2Bv0m4nRXd%2B%2BbbvqPd8gB8PwbdSWqBG4eLVWMoUR4vo%3D&reserved=0 > > > > <https://issues.apache.org/jira/browse/CALCITE-7575>> > > > > > [3] > > > > > > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215947484%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2F2pRc%2BwqNk80bEjsaexGT8uxTKn%2FGuw%2BBPSEYz3lGnk%3D&reserved=0 > > > > <https://issues.apache.org/jira/browse/CALCITE-7465> > > > > > < > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215983673%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=n14%2FEnyO%2F2WIcPudrEWwFaQ8KHqRIppuqRfJxK3NqtA%3D&reserved=0 > > > > <https://issues.apache.org/jira/browse/CALCITE-7465>> > > > > > [4] > > > > > > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216022139%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=6Gyk1uF2SfLH0J7lQqhBYPDIU53NVYIzCrg0GrhMN7w%3D&reserved=0 > > > > <https://issues.apache.org/jira/browse/CALCITE-7480> > > > > > < > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216057890%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=f4gNZnMW6VBq12Yt8PkIzdFfUwdX8fYXTJOBOCxMOwg%3D&reserved=0 > > > > <https://issues.apache.org/jira/browse/CALCITE-7480>> > > > > > [5] > > > > > > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216090334%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=d5YrH4adMbZBaEhqXEns9xR0IerRNTsjUZeC5DXQE50%3D&reserved=0 > > > > <https://issues.apache.org/jira/browse/CALCITE-7471> > > > > > < > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216116472%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2ZyKirj6%2FrAmUM77mq35uKuogcNVn3vXfQqT5B9mTEU%3D&reserved=0 > > > > <https://issues.apache.org/jira/browse/CALCITE-7471>> > > > > > [6] > > > > > > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216150666%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2Fuu4GC7suv7WSI4HtZqxGtvTkV4HarcODd5Q8W7zDpQ%3D&reserved=0 > > > > <https://issues.apache.org/jira/browse/CALCITE-7470> > > > > > < > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216179872%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JS14KTB4YncuYb7dkjjQFu%2FVRvqjaHnPfTEjCn2kWDw%3D&reserved=0 > > > > <https://issues.apache.org/jira/browse/CALCITE-7470>> > > > > > [7] > > > > > > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216199448%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=mrskfIM%2FHIKFxL1foGM9XiGiphFwbPxO9q0n%2FubrWS8%3D&reserved=0 > > > > <https://issues.apache.org/jira/browse/CALCITE-7466> > > > > > < > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216219518%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=r9y6sPdjB5UpVXSiVl8rq3XULGsl8SxlIWtY9LHlCkU%3D&reserved=0 > > > > <https://issues.apache.org/jira/browse/CALCITE-7466>> > > > > > [8] > > > > > > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216239592%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Ra1qmGy%2Fnnxs98tuQZBMHMxceJ3NkYvyfKlwU1cHl84%3D&reserved=0 > > > > <https://issues.apache.org/jira/browse/CALCITE-7312> > > > > > < > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216258677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=GdHVoxqW6y2kiUi%2F3pckMK8szOLnGwqCtW4sconw4XQ%3D&reserved=0 > > > > <https://issues.apache.org/jira/browse/CALCITE-7312>> > > > > > [9] > > > > > > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216284576%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=k%2FBsEShTSDCzRCfsgPZXz7cYJJnDsaeRnW45ldO5RiY%3D&reserved=0 > > > > <https://issues.apache.org/jira/browse/CALCITE-7217> > > > > > < > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216319677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XhpIIafvVURhq8tbckhdDytfZ9P%2FsJDoaEoIpYlOBW4%3D&reserved=0 > > > > <https://issues.apache.org/jira/browse/CALCITE-7217>> > > > > > [10] > > > > > > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216350464%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ZiPFaESvPoORqz9KLJLh9I7z6IanGWhL2ZnCl2Ap9N4%3D&reserved=0 > > > > <https://issues.apache.org/jira/browse/CALCITE-6944> > > > > > < > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216379236%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Pqe2YCqIdEHIZbTSrLF6X6BAlGRej0vdDaOL%2F2J1%2BeM%3D&reserved=0 > > > > <https://issues.apache.org/jira/browse/CALCITE-6944>> > > > > > > > > > > On Fri, Jul 24, 2026 at 9:37 AM <[email protected]> > > wrote: > > > > > > > > > > > > Hi Gyula > > > > > > > > > > > > Thanks a lot for your reply and the +1! > > > > > > > > > > > > On a conceptual level, lambdas and scalar functions / UDFs are > > > > distinct. > > > > > A lambda isn't a stored/registered value. It's represented by the > > > > > planning-only FUNCTION logical type and only exists as an argument > to > > > > > higher-order functions like ARRAY_FILTER and TRANSFORM. > Essentially, > > > it's > > > > > the wrapper that introduces the element variable x. > > > > > > > > > > > > Within the lambda you can freely use any scalar function or UDF: > > > > > > ARRAY_FILTER(arr, x -> is_positive(x)) > > > > > > ARRAY_FILTER(arr, x -> is_positive(x) AND x < 100) > > > > > > > > > > > > What currently doesn't work is passing a bare UDF name in the > > lambda > > > > > position (i.e. automatic expansion of is_positive into x -> > > > > is_positive(x)): > > > > > > ARRAY_FILTER(arr, is_positive) -- not supported > > > > > > > > > > > > We intentionally left this out of scope for now, but it's a > > > > > straightforward follow-up if we see demand for it. > > > > > > > > > > > > Best, > > > > > > Dominik > > > > > > > > > > > > From: Gyula Fóra <[email protected]> > > > > > > Date: Thursday, 23 July 2026 at 16:07 > > > > > > To: [email protected] <[email protected]> > > > > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink > > SQL > > > > > (TRANSFORM and ARRAY_FILTER) > > > > > > > > > > > > > > > > > > Be aware: This is an external email. > > > > > > > > > > > > > > > > > > > > > > > > Hey! > > > > > > > > > > > > Without a good background knowledge about the syntax in other > > > systems, > > > > > the > > > > > > ARRAY_FILTER, TRANSFORM functions seem really useful, especially > > with > > > > > > lambda functions. > > > > > > Overall +1 for this idea from a very high level but it would be > > great > > > > > with > > > > > > someone with more deep context on SQL functions / runtime to > chime > > > in. > > > > > > > > > > > > One question I had is how do lambda functions relate to existing > > > scalar > > > > > > functions / UDFs? Would the user be able to use boolean valued > > scalar > > > > > > functions where lambdas are shown in the proposal? > > > > > > > > > > > > Cheers > > > > > > Gyula > > > > > > > > > > > > On Wed, Jul 15, 2026 at 8:40 AM <[email protected]> > > > wrote: > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > I'd like to start a discussion on a FLIP that introduces > > > higher-order > > > > > > > functions > > > > > > > (functions that take a lambda expression as an argument) to > Flink > > > > SQL, > > > > > > > together > > > > > > > with the first two built-ins that use them: TRANSFORM and > > > > ARRAY_FILTER. > > > > > > > > > > > > > > FLIP: > > > > > > > > > > > > > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216408066%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=UuKdaeEmSd6hP87gokZhU9leFfw0BrrD2buEDctKjBk%3D&reserved=0 > > > > < > > > > > > > > > > https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing > > > > > > > > > > < > > > > > > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216440590%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=woLr6CpsHkDCs7y6yZaJ4%2FoCNhxCxuU9hXqh8AaRSVI%3D&reserved=0 > > > > < > > > > > > > > > > https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing > > > > > > > > > > >< > > > > > > > > > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216476842%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=CuKWtmNDVEXK11ET%2BJDOqAp1b%2FHmHR5ionqmomI7cdE%3D&reserved=0 > > > > < > > > > > > > > > > https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing > > > > > > > > > > > > > > > > > > > > > > > > > Motivation > > > > > > > > > > > > > > Users migrating to Flink SQL from Spark, Databricks, Snowflake, > > > > > DuckDB, and > > > > > > > Presto/Trino expect to manipulate collections (arrays and maps) > > > > inline > > > > > > > with a > > > > > > > lambda instead of UNNEST + re-aggregate rewrites or bespoke > UDFs. > > > The > > > > > > > absence of > > > > > > > higher-order collection functions forces verbose query rewrites > > > > during > > > > > > > migration > > > > > > > and raises time-to-first-query. TRANSFORM(array, x -> x + 1) > and > > > > > > > ARRAY_FILTER(array, x -> x > 0) are the two most requested > entry > > > > points > > > > > > > and, > > > > > > > importantly, they can be built on lambda infrastructure that > > > already > > > > > > > exists in > > > > > > > Calcite (CALCITE-3679), so the surface area we add on the Flink > > > side > > > > is > > > > > > > rather > > > > > > > small. > > > > > > > > > > > > > > Summary > > > > > > > > > > > > > > • Introduce a new logical type FUNCTION that describes the > type > > > of > > > > a > > > > > > > lambda > > > > > > > (its argument types and its result type). This is the > > > type-system > > > > > > > foundation > > > > > > > every higher-order function needs; it is a > > planning/translation > > > > > helper > > > > > > > type > > > > > > > and is not a persisted column type. > > > > > > > • Add ARRAY_FILTER(array, element -> predicate), which > returns > > a > > > > new > > > > > > > array > > > > > > > containing only the elements for which the predicate holds. > > > > > > > • Add TRANSFORM(collection, lambda), which applies a lambda > to > > > > every > > > > > > > element of > > > > > > > an array (TRANSFORM(array, x -> expr)) or every entry of a > > map > > > > > > > (TRANSFORM(map, (k, v) -> expr)), returning a new > array/map. > > > > > > > • Reuse Calcite's lambda parsing, validation and > > > > > > > RexLambda/FunctionSqlType > > > > > > > infrastructure (CALCITE-3679) rather than inventing a > > > > > Flink-specific > > > > > > > lambda > > > > > > > syntax. The lambda arrow syntax x -> expr and (k, v) -> > expr > > is > > > > > already > > > > > > > parseable by the Calcite version Flink bundles (1.41.0). > > > > > > > > > > > > > > The functions are net-new syntax and are additive: no existing > > > query > > > > > > > changes > > > > > > > behavior. There is no new configuration option — the functions > > are > > > > > always > > > > > > > available once the release ships. > > > > > > > > > > > > > > Examples: > > > > > > > > > > > > > > SELECT ARRAY_FILTER(ARRAY[1, 2, 3, 4], x -> x > 2); > > > > -- > > > > > > > [3, 4] > > > > > > > SELECT TRANSFORM(ARRAY[1, 2, 3], x -> x * 10); > > > > -- > > > > > > > [10, 20, 30] > > > > > > > SELECT TRANSFORM(ARRAY['a', 'bb', 'ccc'], s -> > > CHAR_LENGTH(s)); > > > > -- > > > > > > > [1, 2, 3] > > > > > > > SELECT TRANSFORM(MAP['a', 1, 'b', 2], (k, v) -> v * 100); > > > > -- > > > > > > > {a=100, b=200} > > > > > > > > > > > > > > Thanks, > > > > > > > Dominik Bünzli > > > > > > > Data, Analytics & AI Engineer > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Best regards, > > > > > Sergey > > > > > > > > > > > > > > >
