88fantasy opened a new issue, #12205: URL: https://github.com/apache/seatunnel/issues/12205
Part of the umbrella #1, depends on #2 (the shared `seatunnel-lineage`/`seatunnel-lineage-openlineage` modules). Adds the Zeta engine's call site. ### What's needed A `ZetaLineageReporter`, wired from the job master (`JobMaster` / `PhysicalPlan` / `CheckpointCoordinator`), that when `openlineage_enabled = true`: - Sends a START event when a job's pipeline begins. - Sends a throttled RUNNING heartbeat on completed checkpoints, so long-running streaming jobs don't go stale in a lineage graph — throttled by `openlineage_heartbeat_min_interval_ms` (default 1 hour) since checkpoints can complete every few seconds and Zeta has no other natural place to hang a periodic callback. - Sends a terminal COMPLETE/FAIL/ABORT event when the pipeline ends, with output row-count/byte-size statistics read from the job's own `SinkCommittedCount`/`SinkWriteCount` metrics (falling back to the "attempted" counts if nothing has committed yet, flagged with an `output_statistics_semantics` facet so a consumer of the event knows which one it's looking at — Zeta and Flink end up reporting structurally different semantics here, and both need to say so rather than silently disagreeing). - Splits a multi-table sink's event into one event per underlying table, and derives each table's OpenLineage dataset name from its `CatalogTable` per the shared naming logic in #2. ### Backward compatibility Reporting is off unless a job explicitly sets `openlineage_enabled = true`. No change to checkpoint behavior, job scheduling, or existing metrics when disabled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
