88fantasy opened a new pull request, #12210:
URL: https://github.com/apache/seatunnel/pull/12210
> **Stacked on #12208.** This branch is built on top of
`feature/openlineage-core` (PR #12208), so until #12208 merges, this diff also
shows #12208's changes (the `seatunnel-lineage`/`seatunnel-lineage-openlineage`
modules and the new `openlineage_*` options). It will shrink to just the
Flink-specific commits once #12208 merges into `dev`. Please review this PR's
own commits (`git log feature/openlineage-core..feature/openlineage-flink`);
#12208's commits are under separate review there.
### Purpose of this pull request
Implements #12206 (part of the umbrella #12207), stacked on #12208. Adds
`seatunnel-lineage-flink` (`LineageJobStatusHook`, the Flink-1.16+-only half)
and `FlinkLineageHooks`/`FlinkLineageSupport` in
`seatunnel-flink-20-starter`/`seatunnel-flink-starter-common`, so a job
submitted through the Flink 1.20 starter reports OpenLineage events when
`openlineage_enabled = true`. The Flink 13/15 starters are untouched and get no
lineage capability — documented as a known limitation (`JobStatusHook` doesn't
exist before Flink 1.16), not a bug to fix here.
### Does this PR introduce _any_ user-facing change?
Yes, but opt-in only, and only for the Flink 1.20 starter. Enabling it also
requires a one-time deployment step: a shaded jar
(`seatunnel-lineage-flink-*-shaded.jar`) must be placed in `$FLINK_HOME/lib` on
every JobManager and the JobManager restarted, because `JobGraph`
deserialization happens before the user classloader exists. Documented in this
PR's docs update, including the requirement that the client-side starter jar
and the cluster-side shaded jar be upgraded together.
### How was this patch tested?
- Unit tests: `LineageJobStatusHookTest`, `FlinkLineageRegistrationTest`,
`FlinkLineageHookNamingTest`, `FlinkLineageSupportTest`, and `ShadedArtifactIT`
(builds the actual shaded jar and asserts its OpenLineage facets survive
shading without corruption — gated behind `-DskipIT=false` like the rest of
this repo's IT suite; passes).
- End-to-end against a real Gravitino instance, using a shared sandbox Flink
1.20 standalone cluster: the shaded jar was placed in `lib/` on both the
JobManager and TaskManager, the cluster restarted, tested, then restored —
`conf/config.yaml`'s checksum was verified to match the pre-change baseline
exactly after restoring, and the lineage jar was removed from `lib/` again.
- **Detached** submission (`flink run -d`) of `FakeSource (10 rows) →
Doris`: Gravitino received exactly one `start` and one `complete` event — the
JobManager-side hook's own terminal report, carrying the Flink job ID but no
statistics, as designed for detached mode. Verified against the table: `SELECT
COUNT(*) = 10`.
- **Attached** submission (`flink run`) of the same job: exactly one
`start` and one `complete` event — this time the client-side report, carrying
`outputStatistics: {rowCount: 10, size: 170}` and an
`output_statistics_semantics: attempted` facet. Same table verification.
End-to-end client wall time was ~9.8s, confirming the JobManager-side terminal
send (asynchronous in this PR) does not block job completion.
- Both runs used this PR's client jar and cluster jar together, avoiding
the version-mismatch pitfall documented in the deployment doc.
- `./mvnw -q -DskipTests verify` across the whole reactor — 470 modules,
BUILD SUCCESS. This branch's affected-module unit tests all pass.
### Check list
* [ ] No new JAR beyond what #12208 already registers (this PR only adds a
shaded packaging of existing dependencies for cluster-side deployment)
* [x] Documentation updated (`docs/en`, `docs/zh` — Flink section of the
lineage concept page, including the `$FLINK_HOME/lib` deployment step)
* [ ] `incompatible-changes.md` — not applicable
* [ ] Connector-specific checklist items — not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
