Hi Gustavo, Sorry for the late response. I had "a few" discussions with my colleague Michal Stutzmann (you might remember him from the Flink Forward conference in Barcelona) about the FLIP and the features it should cover. Michal extended the proposal<https://docs.google.com/document/d/1vd-l922wRJGkbhsGaT9TxRTG2UKjj7nAtxC8J9RaG1U/edit?usp=sharing> and implementation to also support the Table & Python APIs, as well as lambda arguments for user-defined functions. Additionally, we now support using columns within lambda expressions.
We pushed the WIP changes to a branch in our fork, which you can find here: https://github.com/swisscom-bigdata/flink/tree/flink-31207-higher-order-functions-and-lambdas The set of changes grew larger than expected, but in our opinion they significantly simplify the user experience by reducing the need to work with UDFs or custom code to handle collections. Thanks a lot in advance for taking a look! Best, Dominik From: Gustavo de Morais <[email protected]> Date: Friday, 31 July 2026 at 12:05 To: [email protected] <[email protected]> Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL (TRANSFORM and ARRAY_FILTER) Be aware: This is an external email. Hey Dominik, Lambda functions are an important building block - not only for these built-in collection functions, but also for UDFs and PTFs whitin the whole archictecture. So thanks for the proposal and for wanting to contribute. I'll take a look at the proposal soon. You mentioned a WIP branch a couple of times in the thread - could you share the link so we can take a look at that too? Kind regards, Gustavo On Thu, 30 Jul 2026 at 16:39, <[email protected]> wrote: > Hi all, > > Following up on this FLIP to see if there are any comments or concerns. > > Since the last mail I added tests for: > - views using ARRAY_FILTER / TRANSFORM > - materialized tables going through expand-and-replan > > Everything is green locally so far. > > Feedback on the planner integration, Calcite handling, missing edge cases, > or the overall direction would be appreciated. > > As I’m still new to the FLIP process, I’d also appreciate some guidance on > the next steps from here. Many thanks in advance. > > Best, > Dominik > > From: Bünzli Dominik, SCS-INI-DNA-INF <[email protected]> > Date: Friday, 24 July 2026 at 11:17 > To: [email protected] <[email protected]> > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL > (TRANSFORM and ARRAY_FILTER) > > Hi Sergey, > > Good catch. I didn’t have these cases in mind. Quickly added test cases > for them to my branch. > > The view test creates a view like "CREATE VIEW v AS SELECT id, > ARRAY_FILTER(vals, x -> x > 2) FROM ...", then selects from it and checks > that the arrays come back right. Same idea for TRANSFORM and the map lambda > (k, v) -> v * 100. > > The materialized table test does the equivalent: a CONTINUOUS table using > ARRAY_FILTER and TRANSFORM in its query, running on a mini-cluster, then a > SELECT to confirm the output matches. > > So, both should (to my still limited knowledge), go through the > expand-and-replan path and check real results. If Calcite mangles a lambda, > these tests should fail. Looks good so far, tests are green locally. > > Best, > Dominik > > From: Sergey Nuyanzin <[email protected]> > Date: Friday, 24 July 2026 at 09:59 > To: [email protected] <[email protected]> > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL > (TRANSFORM and ARRAY_FILTER) > > > Be aware: This is an external email. > > > > thanks for the proposal > > and working in this direction > > I haven't checked yet all the things > > however I see at least one missing piece here. > > In FlinkSQL we have views and materialized tables. Both are relying on > expanded ("rewritten" in Calcite terms) SQL. > And we have already faced a number of issues around that and fixed > them in Calcite (like just a few or them [1], [2], [3], [4], [5], [6], > [7], [8], [9], [10]) > > I'm pretty sure there will be something lambda related. For that > reason I would suggest also having test with views(or materialized > tables, does not matter) based on sql with lambda > and check that SELECT from them still able to produce output > > [1] > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215787243%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=dhAIi9%2BVQbdypDURBS6hCR97oOt1Y%2BWDeYhcppgEH18%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7660> > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7660&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215837165%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=WJ6rRpo1oeuVua%2BU6r65we7Q%2FyAkiVC3drjq4y%2F37ZM%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7660>> > [2] > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215876426%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2yAtLVRqA5lHAT4dKoiNgOEZmCDrNLlq6XnXvidm9dk%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7575> > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7575&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215913003%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=M%2Bv0m4nRXd%2B%2BbbvqPd8gB8PwbdSWqBG4eLVWMoUR4vo%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7575>> > [3] > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215947484%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2F2pRc%2BwqNk80bEjsaexGT8uxTKn%2FGuw%2BBPSEYz3lGnk%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7465> > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7465&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891215983673%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=n14%2FEnyO%2F2WIcPudrEWwFaQ8KHqRIppuqRfJxK3NqtA%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7465>> > [4] > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216022139%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=6Gyk1uF2SfLH0J7lQqhBYPDIU53NVYIzCrg0GrhMN7w%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7480> > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7480&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216057890%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=f4gNZnMW6VBq12Yt8PkIzdFfUwdX8fYXTJOBOCxMOwg%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7480>> > [5] > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216090334%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=d5YrH4adMbZBaEhqXEns9xR0IerRNTsjUZeC5DXQE50%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7471> > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7471&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216116472%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2ZyKirj6%2FrAmUM77mq35uKuogcNVn3vXfQqT5B9mTEU%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7471>> > [6] > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216150666%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2Fuu4GC7suv7WSI4HtZqxGtvTkV4HarcODd5Q8W7zDpQ%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7470> > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7470&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216179872%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JS14KTB4YncuYb7dkjjQFu%2FVRvqjaHnPfTEjCn2kWDw%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7470>> > [7] > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216199448%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=mrskfIM%2FHIKFxL1foGM9XiGiphFwbPxO9q0n%2FubrWS8%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7466> > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7466&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216219518%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=r9y6sPdjB5UpVXSiVl8rq3XULGsl8SxlIWtY9LHlCkU%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7466>> > [8] > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216239592%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Ra1qmGy%2Fnnxs98tuQZBMHMxceJ3NkYvyfKlwU1cHl84%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7312> > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7312&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216258677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=GdHVoxqW6y2kiUi%2F3pckMK8szOLnGwqCtW4sconw4XQ%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7312>> > [9] > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216284576%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=k%2FBsEShTSDCzRCfsgPZXz7cYJJnDsaeRnW45ldO5RiY%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7217> > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-7217&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216319677%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XhpIIafvVURhq8tbckhdDytfZ9P%2FsJDoaEoIpYlOBW4%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-7217>> > [10] > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216350464%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ZiPFaESvPoORqz9KLJLh9I7z6IanGWhL2ZnCl2Ap9N4%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-6944> > <https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCALCITE-6944&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216379236%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Pqe2YCqIdEHIZbTSrLF6X6BAlGRej0vdDaOL%2F2J1%2BeM%3D&reserved=0<https://issues.apache.org/jira/browse/CALCITE-6944>> > > On Fri, Jul 24, 2026 at 9:37 AM <[email protected]> wrote: > > > > Hi Gyula > > > > Thanks a lot for your reply and the +1! > > > > On a conceptual level, lambdas and scalar functions / UDFs are distinct. > A lambda isn't a stored/registered value. It's represented by the > planning-only FUNCTION logical type and only exists as an argument to > higher-order functions like ARRAY_FILTER and TRANSFORM. Essentially, it's > the wrapper that introduces the element variable x. > > > > Within the lambda you can freely use any scalar function or UDF: > > ARRAY_FILTER(arr, x -> is_positive(x)) > > ARRAY_FILTER(arr, x -> is_positive(x) AND x < 100) > > > > What currently doesn't work is passing a bare UDF name in the lambda > position (i.e. automatic expansion of is_positive into x -> is_positive(x)): > > ARRAY_FILTER(arr, is_positive) -- not supported > > > > We intentionally left this out of scope for now, but it's a > straightforward follow-up if we see demand for it. > > > > Best, > > Dominik > > > > From: Gyula Fóra <[email protected]> > > Date: Thursday, 23 July 2026 at 16:07 > > To: [email protected] <[email protected]> > > Subject: Re: [DISCUSS] FLIP-XXXX: Higher-Order Functions in Flink SQL > (TRANSFORM and ARRAY_FILTER) > > > > > > Be aware: This is an external email. > > > > > > > > Hey! > > > > Without a good background knowledge about the syntax in other systems, > the > > ARRAY_FILTER, TRANSFORM functions seem really useful, especially with > > lambda functions. > > Overall +1 for this idea from a very high level but it would be great > with > > someone with more deep context on SQL functions / runtime to chime in. > > > > One question I had is how do lambda functions relate to existing scalar > > functions / UDFs? Would the user be able to use boolean valued scalar > > functions where lambdas are shown in the proposal? > > > > Cheers > > Gyula > > > > On Wed, Jul 15, 2026 at 8:40 AM <[email protected]> wrote: > > > > > Hi everyone, > > > > > > I'd like to start a discussion on a FLIP that introduces higher-order > > > functions > > > (functions that take a lambda expression as an argument) to Flink SQL, > > > together > > > with the first two built-ins that use them: TRANSFORM and ARRAY_FILTER. > > > > > > FLIP: > > > > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216408066%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=UuKdaeEmSd6hP87gokZhU9leFfw0BrrD2buEDctKjBk%3D&reserved=0<https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing> > < > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216440590%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=woLr6CpsHkDCs7y6yZaJ4%2FoCNhxCxuU9hXqh8AaRSVI%3D&reserved=0<https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing> > >< > https://che01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ%2Fedit%3Fusp%3Dsharing&data=05%7C02%7CDominik.Buenzli%40swisscom.com%7C8e16d7844e6f4dce86fc08deeeeb38fe%7C364e5b87c1c7420d9beec35d19b557a1%7C0%7C0%7C639210891216476842%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=CuKWtmNDVEXK11ET%2BJDOqAp1b%2FHmHR5ionqmomI7cdE%3D&reserved=0<https://docs.google.com/document/d/144P06vspNDwU3nevEluPeOsWBQsaaQUhe8DouiEFqeQ/edit?usp=sharing> > > > > > > > > Motivation > > > > > > Users migrating to Flink SQL from Spark, Databricks, Snowflake, > DuckDB, and > > > Presto/Trino expect to manipulate collections (arrays and maps) inline > > > with a > > > lambda instead of UNNEST + re-aggregate rewrites or bespoke UDFs. The > > > absence of > > > higher-order collection functions forces verbose query rewrites during > > > migration > > > and raises time-to-first-query. TRANSFORM(array, x -> x + 1) and > > > ARRAY_FILTER(array, x -> x > 0) are the two most requested entry points > > > and, > > > importantly, they can be built on lambda infrastructure that already > > > exists in > > > Calcite (CALCITE-3679), so the surface area we add on the Flink side is > > > rather > > > small. > > > > > > Summary > > > > > > • Introduce a new logical type FUNCTION that describes the type of a > > > lambda > > > (its argument types and its result type). This is the type-system > > > foundation > > > every higher-order function needs; it is a planning/translation > helper > > > type > > > and is not a persisted column type. > > > • Add ARRAY_FILTER(array, element -> predicate), which returns a new > > > array > > > containing only the elements for which the predicate holds. > > > • Add TRANSFORM(collection, lambda), which applies a lambda to every > > > element of > > > an array (TRANSFORM(array, x -> expr)) or every entry of a map > > > (TRANSFORM(map, (k, v) -> expr)), returning a new array/map. > > > • Reuse Calcite's lambda parsing, validation and > > > RexLambda/FunctionSqlType > > > infrastructure (CALCITE-3679) rather than inventing a > Flink-specific > > > lambda > > > syntax. The lambda arrow syntax x -> expr and (k, v) -> expr is > already > > > parseable by the Calcite version Flink bundles (1.41.0). > > > > > > The functions are net-new syntax and are additive: no existing query > > > changes > > > behavior. There is no new configuration option — the functions are > always > > > available once the release ships. > > > > > > Examples: > > > > > > SELECT ARRAY_FILTER(ARRAY[1, 2, 3, 4], x -> x > 2); -- > > > [3, 4] > > > SELECT TRANSFORM(ARRAY[1, 2, 3], x -> x * 10); -- > > > [10, 20, 30] > > > SELECT TRANSFORM(ARRAY['a', 'bb', 'ccc'], s -> CHAR_LENGTH(s)); -- > > > [1, 2, 3] > > > SELECT TRANSFORM(MAP['a', 1, 'b', 2], (k, v) -> v * 100); -- > > > {a=100, b=200} > > > > > > Thanks, > > > Dominik Bünzli > > > Data, Analytics & AI Engineer > > > > > > > -- > Best regards, > Sergey >
