SEPURI-SAI-KRISHNA opened a new pull request, #13156:
URL: https://github.com/apache/seatunnel/pull/13156
Yes, glad to contribute to Apache SeaTunnel.
### Purpose of this pull request
Closes #13155.
On a failed task Spark calls `abort()`, which does not close the sink
writer. With `--master local` the tasks run in the driver JVM, so a writer that
holds non-daemon threads keeps `spark-submit` running after the job has failed,
and a script waiting on it never gets the exit code. The Cassandra sink does
this through its `CqlSession`, and on #13126 it used up the 5 hour CI limit.
`SeaTunnelSpark.main` (2.4 and 3.3) now catches the failure and, if
`spark.master` starts with `local`, prints the stack trace and calls
`System.exit(1)`. `SeaTunnel.run` has already logged the failure, and the
exception is printed because the JVM would print it only if it left `main`.
`SeaTunnelApplication` handles a failure from `SeaTunnel.run` the same way.
**Why not close the writer in `abort()`.** That also ends the hang, but
`close()` is not resource-only, as the reviews on #10060 explain: a failed
attempt can write its buffered rows. I measured it on a branch that did so:
JDBC went from 0 to 2 rows, and from 0 to 6 with `local[1,3]`. This PR leaves
the writer lifecycle as it is.
### Does this PR introduce _any_ user-facing change?
Yes, a failed job now exits by itself in that local mode case. Exit codes
and rows are otherwise unchanged. The stack trace no longer starts with
`Exception in thread "main"`, because the exception does not leave `main`. No
E2E test looks for that text. Other masters are not affected.
### How was this patch tested?
**Real jobs**, one run each, JDK 11, on real Spark built from this branch.
The Cassandra jobs ran against Cassandra 4.1.1 with `dev`'s connector classes.
A SQL transform fails on row 3, or a NULL `bigint` fails in the sink where
noted:
```
dev Spark 2.4.6
Spark 3.3.0
Cassandra, async_write = false no exit, 0 rows exit 1, 13 s,
0 exit 1, 15 s, 0
Cassandra, default async_write no exit, 2 rows exit 1, 13 s,
2 exit 1, 15 s, 2
Cassandra, NULL bigint, sync no exit, 0 rows exit 1, 13 s,
0 exit 1, 16 s, 0
Cassandra, NULL bigint, async no exit, 2 rows exit 1, 12 s,
2 exit 1, 15 s, 2
JDBC (SQLite) exit 1, 0 rows exit 1, 0
exit 1, 0
JDBC, local[*] not run exit 1, 0
exit 1, 0
JDBC, local[1,3] exit 1, 0 rows exit 1, 0
exit 1, 0
JDBC, local[1,3], batch_size = 1 exit 1, 6 rows exit 1, 6
exit 1, 6
LocalFile exit 1, 0 files exit 1, 0
files exit 1, 0 files
LocalFile, local[1,3] not run exit 1, 0
files exit 1, 0 files
Console, Assert exit 1 exit 1
exit 1
```
The `dev` column for the NULL `bigint` async row is from Spark 3.3.0 only,
and the others that ran are from both versions.
Also with this PR, on both versions:
- Jobs that succeed still exit 0 and close the writer: Console, Assert,
LocalFile and JDBC with 3 rows, and with the #13126 connector a Cassandra to
Cassandra job, a multi-table job and a one-row job.
- With the #13126 connector, its NULL key job exits 1 in 15 s on both
versions, leaves 0 rows, and prints the key error.
- A job that fails in `main` before any task runs exits 1. Compared with
`dev` on 3.3.0, stderr differs only in the `Exception in thread "main"` prefix
and one line number. On 2.4.6 the same two differences remain once values that
change per run, such as paths and ports, are ignored.
**Standalone cluster**, Spark 3.3.0, one worker: the failing Cassandra job
exits 1 in 21 s with 0 rows on both `dev` and this PR, so the hang is specific
to local mode and the new exit does not run. A job that succeeds exits 0.
**IT:** `CassandraIT.testFailedJobEndsWhileTheSinkWriterIsOpen` runs a job
whose SQL transform fails on row 3 while the sink writer holds its CQL session.
It asserts a non-zero exit and the transform's error code. It has a 5 minute
`@Timeout`, so if the hang returns it should fail the test instead of using up
the CI limit. I did not trigger that timeout. In [this fork
run](https://github.com/SEPURI-SAI-KRISHNA/seatunnel/actions/runs/42158589006)
it passes on JDK 11 and 17 on all 7 containers (4 Flink, Zeta, Spark 2.4 and
3.3), and each Spark case takes 16 to 20 s.
#13126 disables `testNullPrimaryKeyFailsTheJob` on Spark because of this
hang. Once both are merged that disable can go, and I will do that in a
follow-up. Both PRs add to `CassandraIT`, so the second to merge needs a small
conflict fix.
**Unit tests:** no new unit test, because the change is the JVM exit itself,
which the IT and the runs above cover. Existing tests in
`seatunnel-spark-starter-common` pass on JDK 11 and 17, 11 of 11. The tests of
the 2.4 and 3.3 starter modules do not start on my machine with
`NoClassDefFoundError: LauncherFactory`, and the same command fails the same
way on `dev`, so I am relying on CI for those. They pass in the fork run above.
**Not tested:** YARN and Kubernetes, where the check does not apply because
the master does not start with `local`, and streaming jobs.
**Limit of this fix:** the writers of failed attempts stay open until the
JVM exits, as they do today, so `local[1,3]` holds them across its retries.
Closing them safely needs an abort-safe cleanup hook on the sink writer API, as
suggested on #10060, which is a larger change for maintainers to decide on.
#11174 adds a Spark 4.1 starter with its own `SeaTunnelSpark`, which would
need the same change if it merges after this.
Also run: `spotless:check`, and `verify -DskipTests` on the Spark starter
and Cassandra E2E modules.
### Check list
* [x] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
(no new jar)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs (no option changes)
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR. (not needed, a job that failed still fails
with the same exit code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]