Hi everyone,
I'm super excited to start a discussion about FLIP-440: User-defined SQL
operators / ProcessTableFunction (PTF) [1].
This FLIP has been in my head for many years and Flink 2.0 is a good
time to open up Flink SQL to a new category of use cases. Following the
principle "Make simple things easy, and complex ones possible", I would
like to propose a new UDF type "ProcessTableFunction" that gives users
access to Flink's primitives for advanced stream processing. This should
unblock people when hitting shortcomings in Flink SQL and expand the
scope of SQL from analytical to more event-driven applications.
This proposal is by no means a full replacement of DataStream API.
DataStream API will always provide the full power of Flink whereas PTFs
provide at least a necessary toolbox to cover ~80% of all use cases
without leaving the SQL ecosystem. The SQL ecosystem is a great
foundation with well-defined type system, catalog integration, CDC
support, and built-in functions/operators. PTFs complete it by offering
a standard compliant extension point.
Looking forward to your feedback.
Thanks,
Timo
[1] https://cwiki.apache.org/confluence/x/pQnPEQ
