88fantasy opened a new issue, #12206: URL: https://github.com/apache/seatunnel/issues/12206
Part of the umbrella #1, depends on #2 (the shared `seatunnel-lineage`/`seatunnel-lineage-openlineage` modules). Adds the Flink engine's call site — `seatunnel-flink-20-starter` only. ### What's needed - A `seatunnel-lineage-flink` module holding `LineageJobStatusHook`, which implements Flink's `org.apache.flink.core.execution.JobStatusHook` — a Flink 1.16+ API, which is why this is a separate module rather than living in `seatunnel-flink-starter-common`: a module compiled against Flink 1.13/1.15 can't even name the interface. The 13/15 starters simply don't get lineage reporting; this is a documented limitation, not a bug to fix here. - Client-side registration (`FlinkLineageHooks`/`FlinkLineageSupport` in the 20-starter and the common starter module) that builds the START event, registers the hook on the `StreamGraph`, and — for an **attached** submission (`flink run` without `-d`) — reports the COMPLETE event itself once `execute()` returns, because only the client can read Flink's job accumulators for the real row/byte counts. For a **detached** submission (`-d`), no client is left around by the time the job finishes, so the JobManager-side hook reports the terminal event itself, without statistics. Both paths must agree on exactly one terminal event; whichever one runs must tell the other to stay silent. - The hook's terminal-event send must not run inline on Flink's job-status-change callback — that callback is on a thread Flink's own scheduling depends on, and a slow or stuck HTTP call there is a self-inflicted latency problem. The send needs to be handed off to its own executor. - 🔒 The auth token must never be serialized into the `JobStatusHook` instance — that instance is serialized into the `JobGraph`, which lands in Flink's HA storage / BlobServer. The hook must re-resolve the token from the JobManager's own environment or cluster configuration at execution time. - Deployment documentation: because `JobGraph` deserialization on the JobManager happens *before* the user classloader exists, the hook's implementation class must already be on the JobManager's system classpath — a shaded uber-jar needs to be placed in `$FLINK_HOME/lib` and the JobManager restarted. The client-side starter jar and the cluster-side shaded jar are two different artifacts and must be upgraded together: a version mismatch doesn't error, it silently falls back to older behavior (Java serialization gives a missing field its default value rather than throwing). ### Backward compatibility Reporting is off unless a job explicitly sets `openlineage_enabled = true`. Nothing changes for Flink 1.13/1.15 users, since those starters don't carry this code at all. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
