beetle0915 opened a new pull request, #29166: URL: https://github.com/apache/flink/pull/29166
## What is the purpose of the change Implement [FLINK-40420](https://issues.apache.org/jira/browse/FLINK-40420) by exposing set operations on the PyFlink DataFrame API. ## Brief change log - Add `union`, `union_all`, `intersect`, `intersect_all`, `minus`, and `minus_all` as `PublicEvolving` methods that delegate to the corresponding Table API operations and return new DataFrames. - Reject non-DataFrame operands while leaving schema, environment, and execution-mode validation to the underlying Table API. - Add API documentation and tests for row multiplicities, NULLs, empty inputs, positional column matching, validation, chaining, type conversion, and streaming `union_all`. ## Verifying this change Local verification uses Python 3.12.11, Java 17, and an existing Flink 2.4-SNAPSHOT Java distribution. Python sources are from this branch. - `python -m pytest -q pyflink/dataframe/tests`: 351 passed, including 12 new test methods and batch/streaming MiniCluster execution tests. Eight existing deprecation warnings remain. - `python -m flake8 --config=tox.ini pyflink/dataframe`: passed. - `python -m mypy --config-file tox.ini`: passed (83 source files). - Sphinx HTML build with `-W --keep-going`: passed. - `git diff --check`: passed. This PR is a draft: full-repository `./mvnw clean verify`, end-to-end tests, and the multi-version CI matrix have not been run locally. Java artifacts were reused rather than rebuilt for this Python-only patch. ### Existing Table API mixed-type limitation During execution testing, `intersect`, `intersect_all`, `minus`, and `minus_all` failed for mixed INT/BIGINT inputs even though the resolved result schema was BIGINT. This also reproduces by calling the original Table API directly from a clean Python checkout of the base commit, `e01bbcacd962189c726d8324a35d66cc162712e7`, without importing DataFrame. `union` and `union_all` succeed; explicitly casting the inputs to matching BIGINT types makes all six operations succeed. The clean-checkout comparison had 14 passing cases and four failing mixed-type cases across the six operations and three input scenarios. This PR does not change the planner or add implicit casts in the DataFrame wrappers. The four affected method docstrings advise explicit casts. Should the underlying Table API problem be tracked in a separate issue? <details> <summary>Minimal direct Table API reproduction (no DataFrame usage)</summary> ```python from pyflink.table import EnvironmentSettings, TableEnvironment t_env = TableEnvironment.create(EnvironmentSettings.in_batch_mode()) left = t_env.sql_query( "SELECT * FROM (VALUES (1), (2), (2), (3)) AS T(id)" ) right = t_env.sql_query( "SELECT * FROM (VALUES (CAST(2 AS BIGINT)), " "(CAST(2147483648 AS BIGINT))) AS T(other_id)" ) with left.intersect(right).execute().collect() as rows: print(list(rows)) ``` Expected: one row containing `2`. Observed: a type mismatch while applying `ReplaceIntersectWithSemiJoinRule`. `minus` fails in `ReplaceMinusWithAntiJoinRule`; the two ALL variants fail with a `UnionTransformation` input type mismatch. </details> ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: yes, adds six `PublicEvolving` DataFrame methods - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? Python API docstrings and the DataFrame reference autosummary index, with `versionadded:: 2.4.0`. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: OpenAI Codex (codex-cli 0.154.0-alpha.6.2) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
