ryanmeowy opened a new pull request, #13143:
URL: https://github.com/apache/seatunnel/pull/13143
Yes — I am contributing this change to Apache SeaTunnel.
### Purpose of this pull request
Fixes #13142.
A File sink job fails with `NullPointerException` on the first row when a
`partition_by` column value is NULL. A nullable column is a legal schema:
`FileSinkConfig` only validates that the partition columns exist in the sink
columns, and the `partition_by` documentation does not require non-null values.
Today the job starts and then dies at write time; on restart it fails on the
same row.
Root cause: `AbstractWriteStrategy.generatorPartitionDir()` renders the
partition directory through `StringBuilder.append(Object)` — which already
turns a null into the text `null` — but takes the values that are carried into
the commit info via `.toString()` on the raw field, without a null check:
- `partition_dir_expression` unset: `AbstractWriteStrategy.java:529`
- `partition_dir_expression` set: the same `toString()` twice, for the
`${vN}` substitution map (`:538`) and for `vals` (`:539`)
Since the directory component already becomes the text `null`, this is an
unguarded dereference, not an unrepresentable value.
Fix: derive the partition value once with `String.valueOf(field)`. It keeps
the existing rendering (null -> `"null"`, anything else -> `toString()`) and
makes the directory name and the values in `FileCommitInfo` /
`FileAggregatedCommitInfo` consistent. The values list is only carried through
`toString()` and the committer's aggregation; no SQL is built from it.
Compatibility notes:
- Non-null partition values are unchanged.
- Multi-column partitions: only the NULL column renders as `null`.
- `partition_dir_expression`: `${vN}` substitutes the literal `null`,
consistent with the rest of the expression behavior.
- Hive-style `__HIVE_DEFAULT_PARTITION__` was intentionally not introduced
here: that would be a user-visible output path change and does not belong in an
NPE fix. Note that the Hive sink shares this writer through
`WriteStrategyFactory`, and its aggregated committer registers partitions from
the directory keys, so a NULL value becomes the Hive partition `col=null`
(literal string), not a NULL partition. If the Hive convention should win, that
is a separate decision that changes the directory name as well.
### Does this PR introduce _any_ user-facing change?
Yes. A row whose partition column is NULL used to fail the job with a
`NullPointerException`; it now writes to a partition directory `col=null` (the
same text the directory name already used) and the committed partition values
agree with it. Non-null values, directory layout for non-null values, and the
`partition_dir_expression` behavior for non-null inputs are unchanged.
The `partition_by` documentation does not currently state the NULL
semantics. I kept this PR to the two source files; I can add a sentence to
`docs/en` and `docs/zh` if maintainers prefer to document the behavior in the
same change.
### How was this patch tested?
`AbstractWriteStrategyTest` (extends the existing class and reuses its
config helpers):
- `generatorPartitionDirWithNullPartitionValue` — a null partition value
yields key `c_null=null` and values `["null"]`; this test throws NPE on the
parent commit.
- `generatorPartitionDirWithNullPartitionValueAndExpression` — same with
`partition_dir_expression = "${k0}/${v0}"`, asserting key `c_null/null`. The
expression is deliberately not `${k0}=${v0}`: that renders `c_null=null`,
identical to the non-expression branch, so the test could not tell whether the
expression path ran at all. With `${k0}/${v0}` an ignored expression fails the
assertion. Pre-fix this test errors at `:538`, which also proves the branch is
exercised.
- `generatorPartitionDirKeepsNonNullPartitionValue` — regression guard,
non-null output identical to today.
Commands:
```bash
JAVA_TOOL_OPTIONS="-Dnet.bytebuddy.experimental=true" \
./mvnw -pl seatunnel-connectors-v2/connector-file/connector-file-base test
\
-Dtest='AbstractWriteStrategyTest,FileSinkConfigTest'
-Dspotless.check.skip=true
# Tests run: 13, Failures: 0, Errors: 0
```
Also ran the whole `connector-file-base` module: 382 tests, 370 passed; the
12 errors are `xerial.snappy: no native library ... Mac/aarch64` in
`source/reader` Parquet tests (environmental on this machine, identical set on
the parent commit).
A standalone harness that drives the real `TextWriteStrategy` with a mocked
`FileSinkConfig` prints `partition dir = {c_null=null=[null]}` after the fix
(before: NPE at `:529`).
### Check list
- [x] If any new Jar binary package adding in your PR, please add License
Notice according [New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
— no new jar
- [x] If necessary, please update the documentation to describe the new
feature. — no new feature; documented NULL semantics offered above
- [x] If necessary, please update `incompatible-changes.md` — not an
incompatible change
- [x] Connector checklist (plugin-mapping, seatunnel-dist pom, labeler conf,
e2e, plugin_config) — not a new connector
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]