Hi Dominik, Thanks for the reply and for considering these. It's great that you're both working on this. As mentioned, it's a very important feature, but there are indeed some cases that might grow its scope - we should make sure we design it in a complete way that fits the different parts of the project.
I'll try to make some time next week to take a deeper look at what you shared. In the meantime, here are a few more pointers you can already look into: you mentioned lambda arguments for user-defined functions - what about a UDF as an argument of a lambda function? Nesting is also worth considering: multiple nested lambda functions, lambda function -> UDF or regular function -> lambda function -> ... etc. Kind regards, Gustavo On Fri, 7 Aug 2026 at 11:31, <[email protected]> wrote: > Hi Gustavo, > > Sorry for the late response. I had "a few" discussions with my colleague > Michal Stutzmann (you might remember him from the Flink Forward conference > in Barcelona) about the FLIP and the features it should cover. Michal > extended the proposal< > https://docs.google.com/document/d/1vd-l922wRJGkbhsGaT9TxRTG2UKjj7nAtxC8J9RaG1U/edit?usp=sharing> > and implementation to also support the Table & Python APIs, as well as > lambda arguments for user-defined functions. Additionally, we now support > using columns within lambda expressions. > > We pushed the WIP changes to a branch in our fork, which you can find here: > > > https://github.com/swisscom-bigdata/flink/tree/flink-31207-higher-order-functions-and-lambdas > > The set of changes grew larger than expected, but in our opinion they > significantly simplify the user experience by reducing the need to work > with UDFs or custom code to handle collections. > > Thanks a lot in advance for taking a look! > > Best, > Dominik > > From: Gustavo de Morais <[email protected]> > Date: Friday, 31 July 2026 at 12:05 > To: [email protected] <[email protected]> > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL > (TRANSFORM and ARRAY_FILTER) > > > Be aware: This is an external email. > > > > Hey Dominik, > > Lambda functions are an important building block - not only for these > built-in collection functions, but also for UDFs and PTFs whitin the whole > archictecture. So thanks for the proposal and for wanting to contribute. > > I'll take a look at the proposal soon. You mentioned a WIP branch a couple > of times in the thread - could you share the link so we can take a look at > that too? > > Kind regards, > Gustavo > > On Thu, 30 Jul 2026 at 16:39, <[email protected]> wrote: > > > Hi all, > > > > Following up on this FLIP to see if there are any comments or concerns. > > > > Since the last mail I added tests for: > > - views using ARRAY_FILTER / TRANSFORM > > - materialized tables going through expand-and-replan > > > > Everything is green locally so far. > > > > Feedback on the planner integration, Calcite handling, missing edge > cases, > > or the overall direction would be appreciated. > > > > As I’m still new to the FLIP process, I’d also appreciate some guidance > on > > the next steps from here. Many thanks in advance. > > > > Best, > > Dominik > > > > From: Bünzli Dominik, SCS-INI-DNA-INF <[email protected]> > > Date: Friday, 24 July 2026 at 11:17 > > To: [email protected] <[email protected]> > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL > > (TRANSFORM and ARRAY_FILTER) > > > > Hi Sergey, > > > > Good catch. I didn’t have these cases in mind. Quickly added test cases > > for them to my branch. > > > > The view test creates a view like "CREATE VIEW v AS SELECT id, > > ARRAY_FILTER(vals, x -> x > 2) FROM ...", then selects from it and checks > > that the arrays come back right. Same idea for TRANSFORM and the map > lambda > > (k, v) -> v * 100. > > > > The materialized table test does the equivalent: a CONTINUOUS table using > > ARRAY_FILTER and TRANSFORM in its query, running on a mini-cluster, then > a > > SELECT to confirm the output matches. > > > > So, both should (to my still limited knowledge), go through the > > expand-and-replan path and check real results. If Calcite mangles a > lambda, > > these tests should fail. Looks good so far, tests are green locally. > > > > Best, > > Dominik > > > > From: Sergey Nuyanzin <[email protected]> > > Date: Friday, 24 July 2026 at 09:59 > > To: [email protected] <[email protected]> > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL > > (TRANSFORM and ARRAY_FILTER) > > > > > > Be aware: This is an external email. > > > > > > > > thanks for the proposal > > > > and working in this direction > > > > I haven't checked yet all the things > > > > however I see at least one missing piece here. > > > > In FlinkSQL we have views and materialized tables. Both are relying on > > expanded ("rewritten" in Calcite terms) SQL. > > And we have already faced a number of issues around that and fixed > > them in Calcite (like just a few or them [1], [2], [3], [4], [5], [6], > > [7], [8], [9], [10]) > > > > I'm pretty sure there will be something lambda related. For that > > reason I would suggest also having test with views(or materialized > > tables, does not matter) based on sql with lambda > > and check that SELECT from them still able to produce output > > > > [1] > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215787243%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=dhAIi9%2BVQbdypDURBS6hCR97oOt1Y%2BWDeYhcppgEH18%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7660> > > < > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215837165%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=WJ6rRpo1oeuVua%2BU6r65we7Q%2FyAkiVC3drjq4y%2F37ZM%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7660>> > > [2] > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215876426%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2yAtLVRqA5lHAT4dKoiNgOEZmCDrNLlq6XnXvidm9dk%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7575> > > < > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215913003%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=M%2Bv0m4nRXd%2B%2BbbvqPd8gB8PwbdSWqBG4eLVWMoUR4vo%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7575>> > > [3] > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215947484%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2F2pRc%2BwqNk80bEjsaexGT8uxTKn%2FGuw%2BBPSEYz3lGnk%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7465> > > < > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215983673%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=n14%2FEnyO%2F2WIcPudrEWwFaQ8KHqRIppuqRfJxK3NqtA%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7465>> > > [4] > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216022139%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=6Gyk1uF2SfLH0J7lQqhBYPDIU53NVYIzCrg0GrhMN7w%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7480> > > < > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216057890%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=f4gNZnMW6VBq12Yt8PkIzdFfUwdX8fYXTJOBOCxMOwg%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7480>> > > [5] > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216090334%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=d5YrH4adMbZBaEhqXEns9xR0IerRNTsjUZeC5DXQE50%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7471> > > < > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216116472%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2ZyKirj6%2FrAmUM77mq35uKuogcNVn3vXfQqT5B9mTEU%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7471>> > > [6] > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216150666%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2Fuu4GC7suv7WSI4HtZqxGtvTkV4HarcODd5Q8W7zDpQ%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7470> > > < > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216179872%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JS14KTB4YncuYb7dkjjQFu%2FVRvqjaHnPfTEjCn2kWDw%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7470>> > > [7] > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216199448%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=mrskfIM%2FHIKFxL1foGM9XiGiphFwbPxO9q0n%2FubrWS8%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7466> > > < > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216219518%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=r9y6sPdjB5UpVXSiVl8rq3XULGsl8SxlIWtY9LHlCkU%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7466>> > > [8] > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216239592%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Ra1qmGy%2Fnnxs98tuQZBMHMxceJ3NkYvyfKlwU1cHl84%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7312> > > < > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216258677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=GdHVoxqW6y2kiUi%2F3pckMK8szOLnGwqCtW4sconw4XQ%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7312>> > > [9] > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216284576%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=k%2FBsEShTSDCzRCfsgPZXz7cYJJnDsaeRnW45ldO5RiY%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7217> > > < > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216319677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XhpIIafvVURhq8tbckhdDytfZ9P%2FsJDoaEoIpYlOBW4%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-7217>> > > [10] > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216350464%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ZiPFaESvPoORqz9KLJLh9I7z6IanGWhL2ZnCl2Ap9N4%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-6944> > > < > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216379236%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Pqe2YCqIdEHIZbTSrLF6X6BAlGRej0vdDaOL%2F2J1%2BeM%3D&reserved=0 > <https://issues.apache.org/jira/browse/CALCITE-6944>> > > > > On Fri, Jul 24, 2026 at 9:37 AM <[email protected]> wrote: > > > > > > Hi Gyula > > > > > > Thanks a lot for your reply and the +1! > > > > > > On a conceptual level, lambdas and scalar functions / UDFs are > distinct. > > A lambda isn't a stored/registered value. It's represented by the > > planning-only FUNCTION logical type and only exists as an argument to > > higher-order functions like ARRAY_FILTER and TRANSFORM. Essentially, it's > > the wrapper that introduces the element variable x. > > > > > > Within the lambda you can freely use any scalar function or UDF: > > > ARRAY_FILTER(arr, x -> is_positive(x)) > > > ARRAY_FILTER(arr, x -> is_positive(x) AND x < 100) > > > > > > What currently doesn't work is passing a bare UDF name in the lambda > > position (i.e. automatic expansion of is_positive into x -> > is_positive(x)): > > > ARRAY_FILTER(arr, is_positive) -- not supported > > > > > > We intentionally left this out of scope for now, but it's a > > straightforward follow-up if we see demand for it. > > > > > > Best, > > > Dominik > > > > > > From: Gyula Fóra <[email protected]> > > > Date: Thursday, 23 July 2026 at 16:07 > > > To: [email protected] <[email protected]> > > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL > > (TRANSFORM and ARRAY_FILTER) > > > > > > > > > Be aware: This is an external email. > > > > > > > > > > > > Hey! > > > > > > Without a good background knowledge about the syntax in other systems, > > the > > > ARRAY_FILTER, TRANSFORM functions seem really useful, especially with > > > lambda functions. > > > Overall +1 for this idea from a very high level but it would be great > > with > > > someone with more deep context on SQL functions / runtime to chime in. > > > > > > One question I had is how do lambda functions relate to existing scalar > > > functions / UDFs? Would the user be able to use boolean valued scalar > > > functions where lambdas are shown in the proposal? > > > > > > Cheers > > > Gyula > > > > > > On Wed, Jul 15, 2026 at 8:40 AM <[email protected]> wrote: > > > > > > > Hi everyone, > > > > > > > > I'd like to start a discussion on a FLIP that introduces higher-order > > > > functions > > > > (functions that take a lambda expression as an argument) to Flink > SQL, > > > > together > > > > with the first two built-ins that use them: TRANSFORM and > ARRAY_FILTER. > > > > > > > > FLIP: > > > > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216408066%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=UuKdaeEmSd6hP87gokZhU9leFfw0BrrD2buEDctKjBk%3D&reserved=0 > < > https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing > > > > < > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216440590%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=woLr6CpsHkDCs7y6yZaJ4%2FoCNhxCxuU9hXqh8AaRSVI%3D&reserved=0 > < > https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing > > > > >< > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216476842%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=CuKWtmNDVEXK11ET%2BJDOqAp1b%2FHmHR5ionqmomI7cdE%3D&reserved=0 > < > https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing > > > > > > > > > > > > > Motivation > > > > > > > > Users migrating to Flink SQL from Spark, Databricks, Snowflake, > > DuckDB, and > > > > Presto/Trino expect to manipulate collections (arrays and maps) > inline > > > > with a > > > > lambda instead of UNNEST + re-aggregate rewrites or bespoke UDFs. The > > > > absence of > > > > higher-order collection functions forces verbose query rewrites > during > > > > migration > > > > and raises time-to-first-query. TRANSFORM(array, x -> x + 1) and > > > > ARRAY_FILTER(array, x -> x > 0) are the two most requested entry > points > > > > and, > > > > importantly, they can be built on lambda infrastructure that already > > > > exists in > > > > Calcite (CALCITE-3679), so the surface area we add on the Flink side > is > > > > rather > > > > small. > > > > > > > > Summary > > > > > > > > • Introduce a new logical type FUNCTION that describes the type of > a > > > > lambda > > > > (its argument types and its result type). This is the type-system > > > > foundation > > > > every higher-order function needs; it is a planning/translation > > helper > > > > type > > > > and is not a persisted column type. > > > > • Add ARRAY_FILTER(array, element -> predicate), which returns a > new > > > > array > > > > containing only the elements for which the predicate holds. > > > > • Add TRANSFORM(collection, lambda), which applies a lambda to > every > > > > element of > > > > an array (TRANSFORM(array, x -> expr)) or every entry of a map > > > > (TRANSFORM(map, (k, v) -> expr)), returning a new array/map. > > > > • Reuse Calcite's lambda parsing, validation and > > > > RexLambda/FunctionSqlType > > > > infrastructure (CALCITE-3679) rather than inventing a > > Flink-specific > > > > lambda > > > > syntax. The lambda arrow syntax x -> expr and (k, v) -> expr is > > already > > > > parseable by the Calcite version Flink bundles (1.41.0). > > > > > > > > The functions are net-new syntax and are additive: no existing query > > > > changes > > > > behavior. There is no new configuration option — the functions are > > always > > > > available once the release ships. > > > > > > > > Examples: > > > > > > > > SELECT ARRAY_FILTER(ARRAY[1, 2, 3, 4], x -> x > 2); > -- > > > > [3, 4] > > > > SELECT TRANSFORM(ARRAY[1, 2, 3], x -> x * 10); > -- > > > > [10, 20, 30] > > > > SELECT TRANSFORM(ARRAY['a', 'bb', 'ccc'], s -> CHAR_LENGTH(s)); > -- > > > > [1, 2, 3] > > > > SELECT TRANSFORM(MAP['a', 1, 'b', 2], (k, v) -> v * 100); > -- > > > > {a=100, b=200} > > > > > > > > Thanks, > > > > Dominik Bünzli > > > > Data, Analytics & AI Engineer > > > > > > > > > > > > -- > > Best regards, > > Sergey > > >
