eugenegujing opened a new pull request, #6323:
URL: https://github.com/apache/texera/pull/6323
### What changes were proposed in this PR?
CSV/JSONL scan sources infer each column's type from only the first
`INFER_READ_LIMIT` (= 100) rows. When a later row held a value that did not
parse as the inferred type, the exec caught the error, mapped the row to
`null`/`None`, and filtered it out — **silently dropping the entire row** with
no error, warning, skipped-row count, or log. Every downstream count,
aggregate, and join then ran on a silently truncated dataset. The bug is the
silence, not the rejection; no comparable engine (Spark, Polars, DuckDB, Arrow)
silently drops whole rows.
```
Before: row fails to parse -> caught -> null/None -> filtered out
(silent data loss)
After: row fails to parse -> RuntimeException -> console error +
run stops
```
Changes:
- Throw a `RuntimeException` (original exception kept as cause) instead of
silently dropping an unparsable row, in `CSVScanSourceOpExec`,
`JSONLScanSourceOpExec`, `ParallelCSVScanSourceOpExec`, and
`CSVOldScanSourceOpExec`. The throw surfaces through
`DataProcessor.handleExecutorException` as a console error and stops the run
(fail-fast, matching the default of Spark FAILFAST / Polars / DuckDB / Arrow).
- Add a shared `ScanRowParseError` helper that identifies the offending
column by re-parsing each field and builds a message that leads with the
essential facts, so it stays useful even when the console title line truncates,
e.g.:
> Row 150: value '55.5' in column 'age' cannot be read as INTEGER. Column
types were inferred from the first 100 rows of the file, and this value does
not match. Fix the value in the file, or clean the data before scanning.
A generic fallback covers rows with no single identifiable column (e.g. a
malformed JSON line).
- Remove the now-dead `null` filter in CSV and csvOld; `ParallelCSV` keeps
its legitimate `null`-skip paths (exhausted block, all-null/blank line) and its
filter — only parse failures abort.
Inference logic is unchanged (the 100-row sampling is untouched). Scala only
— no UI change.
**Behavior-change note for reviewers:** workflows that previously ran to
completion while silently discarding malformed rows will now fail on the first
such row. This is the intended fix (loud over silent) and matches the
modern-engine default, but it is a behavior change — see the discussion on
#6279. Sample photo is attached below.
### Any related issues, documentation, discussions?
Closes #6279
### How was this PR tested?
Scan-operator unit tests (updated to assert the thrown error and message;
added a JSONL malformed-line fallback test):
```
sbt "WorkflowOperator/testOnly \
org.apache.texera.amber.operator.source.scan.csv.CSVScanSourceOpExecSpec \
org.apache.texera.amber.operator.source.scan.json.JSONLScanSourceOpExecSpec \
org.apache.texera.amber.operator.source.scan.csv.ParallelCSVScanSourceOpExecSpec
\
org.apache.texera.amber.operator.source.scan.csvOld.CSVOldScanSourceOpExecSpec"
# => Tests: succeeded 28, failed 0
```
Full backend gate in one clean session (lint + format + whole suite):
```
sbt clean "scalafixAll --check" scalafmtCheckAll test
# scalafixAll --check: pass
# scalafmtCheckAll: pass
# test => Total number of tests run: 1721; succeeded 1721, failed 0, aborted
0 (pending 2)
```
Manual, in the Texera UI: scanned a CSV whose column is integer for the
first 100 rows with a non-integer value (`55.5`) at data row 150. The CSV File
Scan operator turned red and the console showed the message above; the run
stopped at row 150 instead of silently emitting a short result.
<img width="1342" height="838" alt="Screenshot 2026-07-10 at 3 48 20 PM"
src="https://github.com/user-attachments/assets/22df6a04-e97d-4818-80f3-e054afa8e7d6";
/>
### Was this PR authored or co-authored using generative AI tooling?
Co-authored by: Claude Code (Fable 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
