88fantasy opened a new pull request, #12208:
URL: https://github.com/apache/seatunnel/pull/12208
### Purpose of this pull request
Implements #12204 (part of the umbrella #12207). Adds two new modules:
- `seatunnel-lineage`: engine-agnostic contract — job/environment/cluster
config resolution, the `LineageEvent`/`LineageDataset` model, OpenLineage-spec
dataset naming, and a pluggable `LineageBackend` SPI.
- `seatunnel-lineage-openlineage`: the default backend — OpenLineage 1.29.0
model classes plus a hand-written `httpclient 4.5.13` transport.
No engine wiring in this PR. The Zeta and Flink call sites are #12205 and
#12206, implemented as PR #{{PR2}} and PR #{{PR3}}, both built on top of this
branch.
### Does this PR introduce _any_ user-facing change?
Yes. 11 new optional `env {}` options, all prefixed `openlineage_`:
| Option | Type | Default |
|---|---|---|
| `openlineage_enabled` | Boolean | `false` |
| `openlineage_transport` | String | `"http"` |
| `openlineage_url` | String | (none) |
| `openlineage_namespace` | String | `"seatunnel"` |
| `openlineage_auth_token` | String | (none — env/cluster-level only, see
below) |
| `openlineage_timeout_ms` | Int | `10000` |
| `openlineage_retry_times` | Int | `3` |
| `openlineage_run_facet` | String | `"seatunnel_properties"` |
| `openlineage_run_properties` | Map | (none) |
| `openlineage_heartbeat_min_interval_ms` | Long | `3600000` (1h) |
| `openlineage_producer` | String | `https://seatunnel.apache.org/<version>`
|
`openlineage_enabled` defaults to `false`, so this PR changes nothing for
any existing job. Full description in
`docs/en(zh)/introduction/concepts/lineage.md`.
No existing option is renamed, removed, or has its default value changed.
### How was this patch tested?
- Unit tests for config resolution/precedence (`LineageConfigTest`), dataset
naming against the OpenLineage naming spec (`LineageDatasetNamingTest`,
`LineageDatasetFactoryTest`), event model serialization (`LineageEventTest`),
`LineageBackend` SPI discovery (`LineageBackendDiscoveryTest`), and the
OpenLineage backend/transport (`OpenLineageBackendTest`) — including a case
asserting that a configuration error (bad/missing URL) is logged without
leaking the raw endpoint (which can carry userinfo/query parameters).
- `EnvOptionRuleTest` covers that `openlineage_auth_token` is rejected when
present in a job's `env {}` — it may only come from a process environment
variable or cluster-level configuration, because job config can be persisted or
serialized into a `JobGraph`.
- Verified directly (not just asserted in a test) that SeaTunnel's existing
shaded `ObjectMapper` corrupts OpenLineage's facets: `@JsonAnyGetter` stops
working after shading, and a facet's custom properties get wrapped in a literal
`additionalProperties` key instead of being inlined, with no exception raised.
This is why the new modules bring their own unshaded Jackson rather than
reusing SeaTunnel's existing HTTP/JSON utilities.
- `./mvnw -q -DskipTests verify` across the whole reactor — 470 modules,
BUILD SUCCESS.
- Dependency footprint checked with `mvn dependency:list
-DincludeScope=runtime` on the new modules against `known-dependencies.txt`;
the only net-new runtime jar is `openlineage-java-1.29.0.jar`, now registered
there and in `seatunnel-dist/release-docs/LICENSE`. `httpcore` is pinned to
`4.4.16` (the version already used elsewhere in the distribution) to avoid a
second version landing in `seatunnel-dist`.
### Check list
* [x] New JAR binary package (`io.openlineage:openlineage-java:1.29.0`) —
License Notice added, see the `LICENSE` diff in this PR
* [x] Documentation updated (`docs/en`, `docs/zh`)
* [ ] `incompatible-changes.md` — not applicable, this PR only adds new
optional configuration
* [ ] Connector-specific checklist items — not applicable, this PR does not
touch a connector
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
