fdolce opened a new pull request, #29146:
URL: https://github.com/apache/flink/pull/29146
## What is the purpose of the change
`pyflink.dataframe.sql()` currently binds only DataFrames. This PR extends
it to also bind UDFs created with `pyflink.dataframe.udf`, so a query can call
a Python UDF the same way it references a DataFrame: either by its Python
variable name via auto-bind, or under a chosen SQL name via an explicit keyword
binding. UDFs are registered as temporary system functions for the duration of
the call and dropped afterwards, mirroring how DataFrames are registered as
temporary views.
```python
@pf.udf
def add_one(value: int) -> int:
return value + 1
pf.sql("SELECT add_one(a) AS a1 FROM df1")
pf.sql("SELECT inc(a) FROM src", auto_bind=False, src=df1, inc=add_one)
```
## Brief change log
- `sql()` accepts `_DataFrameUDFWrapper` values in `**bindings` and picks
them up from the caller's scope when `auto_bind=True`; other types still raise
`TypeError`.
- UDFs are registered with `create_temporary_system_function` so they
resolve by bare name independently of the current catalog/database, and are
dropped in `finally`.
- Shadowing rules mirror the existing view rules: explicit bindings may
shadow built-in and permanent catalog functions but raise `ValueError` on
collision with an existing temporary function; auto-bound UDFs never shadow any
existing function and are skipped with a warning instead. Name collision checks
are case-insensitive, matching the function catalog.
- Registration of explicit bindings is all-or-nothing: if a later explicit
binding is rejected, views/functions registered earlier in the same call are
dropped before raising.
- UDF bindings do not take part in `TableEnvironment` resolution, which is
still driven by the bound DataFrames only.
- Refactored the internals of `sql.py` into `_register_views` /
`_register_functions` / `_drop_views` / `_drop_functions` helpers; updated
docstrings and `docs/reference/pyflink.dataframe/sql.rst`.
## Verifying this change
This change added tests and can be verified as follows:
- Extended `flink-python/pyflink/dataframe/tests/test_sql.py` with tests
covering:
- auto-bind of local and module-level UDFs, explicit bindings choosing
the SQL name, explicit bindings taking precedence over auto-bind,
`auto_bind=False` ignoring caller UDFs
- case-insensitive function names, pandas UDFs, composing the result
with the DataFrame API
- cleanup of registered functions after a failing query, and rollback of
earlier explicit bindings when a later one fails
- collision handling: auto-bind warns and skips on collision with
existing user-defined, built-in and permanent catalog functions; explicit
bindings shadow built-in and permanent functions but raise on temporary (system
and catalog) function collisions, case-insensitively
- invalid SQL identifiers as explicit UDF names raise `ValueError`
- UDF bindings do not influence `TableEnvironment` resolution
- Existing tests asserting that `pyflink.table.udf` objects are rejected /
ignored were kept (renamed to `*_table_udfs_*`).
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: yes (`pyflink.dataframe.sql`, `@PublicEvolving`, accepts
UDFs in `bindings`; backwards compatible)
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs
(`docs/reference/pyflink.dataframe/sql.rst`) and the `sql()` docstring
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Claude Fable 5.1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
