goutamadwant opened a new issue, #12835:
URL: https://github.com/apache/seatunnel/issues/12835
### What happened
CSV deserialization ignores `schema.columns[].defaultValue` when a field is
absent from the CSV header or its value matches `null_format`. The file reader
does not pass its `CatalogTable` into `CsvDeserializationSchema`, and the
parser assigns null without consulting the configured column metadata.
### Reproduction
Use a LocalFile CSV source with `csv_use_header_line = true`, `null_format =
"NULL"`, and these schema columns:
```hocon
schema {
columns = [
{ name = "id", type = "int", nullable = false, defaultValue = 7 }
{ name = "name", type = "string", nullable = false, defaultValue =
"unknown" }
{ name = "optional", type = "string", nullable = true }
]
}
```
Read a file containing:
```csv
id,optional
NULL,keep
```
Before the fix, `id` and the missing `name` are null despite their
configured defaults; an Assert sink expecting `(7, "unknown", "keep")` fails.
This was reproduced on `dev` at `c0d2398c8` using the LocalFile test path on
Java 11.0.19; the current `dev` has not changed the affected parser/reader
behavior.
### Expected behavior and proposed fix
Pass the existing `CatalogTable` to the CSV deserializer and apply each
column's configured default, converted to the declared type, for a missing
column or a field equal to `null_format`. Preserve the existing handling of
ordinary blank fields when they do not match `null_format`. Do not add a new
connector option or change jobs without configured defaults. Keep conversion
errors contextual without echoing configured values into exception messages.
### Before / after and compatibility
- Before: missing/header-absent and configured-null CSV fields become null
even when `defaultValue` is set.
- After: those fields receive the typed configured default; fields with no
default remain null.
- This is an intentional behavior change for jobs that configured a default
but relied on null downstream. Removing that default preserves the previous
null result. The EN/ZH incompatible-changes pages document this boundary.
### Validation
The CSV-format tests (19) and file-base tests (341) passed.
`LocalFileIT#testLocalFileCsv` passed six Zeta, Flink, and Spark invocations (0
failures) after missing local starter artifacts were built. Full repository CI
remains for the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
