autophagy opened a new pull request, #28657:
URL: https://github.com/apache/flink/pull/28657
## What is the purpose of the change
When testing the PTF Test Harness against the
`ProcessTableFunctionTestPrograms`, I found a bug that occurs when trying to
use a PTF that emits a non-row type in a mode where the harness adds extra
columns (`rowtime` with `on_time`, partition columns, etc). This exposed a bit
of a behaviour gap which this PR hopes to fill.
This PR redfines `getOutput` so that it returns only the emitted output from
the PTF itself, either the Pojo, atomic type, Row (without extra columns), etc.
It also introduces a `getSystemOutput` which always returns a `Row` and is more
like what the runtime would actually emit. The reason for this was that a user
is more likely to want to test the actual produced output of their PTF in
particular, than what the runtime also emits. There could be cases where the
user would like to test this, to see what partition keys are emitted for
example, so `getSystemOutput` would aim to cover this case.
I also fixed a bug that surfaced when implementing this, I also ran into a
bug where it was resetting the row kind of Rows to `INSERT` when appending the
rowtime column.
## Brief change log
- Changed `getOutput` in `ProcessTableFunctionTestHarness` to only return
the emitted object from the PTF based on the PTF's declared return type,
without extra columns.
- Added `getSystemOutput` in `ProcessTableFunctionTestHarness` to provide
users a way to examine what the runtime might actually emit, with passthrough
columns, partition columns, rowtime, etc.
## Verifying this change
This change added tests and can be verified by running
`ProcessTableFunctionTestHarnessTest`.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (docs / JavaDocs)
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change the checkbox below to `[X]` followed by the name of the tool, and
uncomment the
"Generated-by" line. See the ASF Generative Tooling Guidance for details:
https://www.apache.org/legal/generative-tooling.html
You are responsible for the quality and correctness of every change in this
PR
regardless of the tooling used. Low-effort AI-generated PRs will be closed.
See
AGENTS.md for the full guidance.
-->
- [x] Yes (2.1.201 (Claude Code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
