88fantasy opened a new issue, #12204:
URL: https://github.com/apache/seatunnel/issues/12204
Title: [Feature][Core] Add the seatunnel-lineage /
seatunnel-lineage-openlineage modules and openlineage_* options
Body:
Part of the umbrella #1. This issue tracks the shared, engine-agnostic half
of table-level lineage reporting: two new modules and the configuration surface
both engines will use. No engine wiring here — see #3 (Zeta) and #4 (Flink).
### What's needed
- `seatunnel-lineage`: configuration resolution with job/environment/cluster
precedence, the `LineageEvent`/`LineageDataset`/`LineageEventType` model,
dataset naming per the OpenLineage naming spec, and a `LineageBackend` SPI
(`ServiceLoader`-based, not SeaTunnel's existing `EventHandler` SPI — that one
broadcasts to a shared event stream that already has a default subscriber,
which would leak custom lineage properties into unrelated logs).
- `seatunnel-lineage-openlineage`: the default `LineageBackend` —
OpenLineage 1.29.0 model classes (`io.openlineage:openlineage-java`, model
classes only) plus a hand-written `httpclient 4.5.13`-based transport.
- 11 new `env {}` options, all prefixed `openlineage_`
(`openlineage_enabled`, `_transport`, `_url`, `_namespace`, `_auth_token`,
`_timeout_ms`, `_retry_times`, `_run_facet`, `_run_properties`,
`_heartbeat_min_interval_ms`, `_producer`). All optional; `openlineage_enabled`
defaults to `false`.
### Notable constraints this issue needs to satisfy
- **Token handling**: `openlineage_auth_token` must be rejected if it
appears in a job's `env {}` block. Job configuration can be persisted to disk
or serialized into a Flink `JobGraph` (which lands in HA storage / the
BlobServer) — the token may only come from a process environment variable or
cluster-level configuration, resolved at execution time, never at parse time.
- **Serialization safety**: SeaTunnel's existing shaded `ObjectMapper`
cannot be reused to serialize OpenLineage's model classes — they rely on
`@JsonAnyGetter` for custom facet properties, and the shaded Jackson silently
produces a corrupted event (the facet gets wrapped in a literal
`additionalProperties` key instead of having its properties inlined) with no
error raised. The new modules therefore ship their own unshaded Jackson,
isolated by module boundary.
- **Dependency footprint**: pulling in `openlineage-java` unfiltered adds
~13 redundant transitive jars and a
`commons-logging`/`org.slf4j:jcl-over-slf4j` duplicate-class conflict. Needs 4
explicit Maven `<exclusion>`s (`httpclient5`, `micrometer-core`,
`commons-logging`, `commons-lang3`) while keeping the YAML/Jackson dependencies
OpenLineage's own static initializers need.
- **License/dependency bookkeeping**: register the new runtime jar in
`tools/dependencies/known-dependencies.txt` and
`seatunnel-dist/release-docs/LICENSE`; make sure `httpcore` doesn't end up at
two different versions in the distribution (pin to the version the rest of the
repo already ships).
### Backward compatibility
Purely additive. No existing option, module, or SPI is touched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
