eugenegujing opened a new pull request, #6214: URL: https://github.com/apache/texera/pull/6214
### What changes were proposed in this PR? Manual backport of #6053 to the `release/v1.2` branch. **Problem.** Pandas-based Python operators (e.g. Sort via `TableOperator`) build a DataFrame from input tuples. When an INT/LONG column contains nulls, pandas promotes the whole column to float64 because a numpy-backed int column cannot hold NaN, so `119` becomes `119.0`. On output, the worker's strict schema validation in `Tuple.finalize()` then fails with `TypeError: Unmatched type for field 'weight', expected AttributeType.INT, got 119.0 (<class 'float'>) instead.` This crashed every workflow whose CSV had an integer column with at least one missing value, and the only workaround was manually inserting a Type Casting operator for each affected column. **The fix** (in `Tuple.cast_to_schema()` only; `validate_schema()` is unchanged): when the target type is INT or LONG and the value is a float (including `np.float64`) with a zero fractional part, cast it back to int — but only when the result is provably the original integer: - INT window: Arrow int32 capacity `[-2^31, 2^31 - 1]`. int32 values are always exactly representable in float64, so capacity is the only constraint. - LONG window: the float64 exact-integer range `[-(2^53) + 1, 2^53 - 1]` instead of int64 capacity. Above 2^53, float64 rounds, so the received float may already be a corrupted rendition of the original integer; coercing it would turn a loud validation error into silent data corruption. The endpoint 2^53 itself is excluded because it is ambiguous (`2^53 + 1` also rounds to float `2^53`). - The range check compares the converted int rather than floats, to avoid float rounding at the window endpoints. - Non-integral, infinite, and out-of-window floats are left untouched so `validate_schema()` still rejects them: lossy coercion must never happen silently. An out-of-window integral float additionally logs an actionable warning suggesting a cast to STRING or DOUBLE (or LONG for large integers in an INT field). Restructuring the if-chain in `cast_to_schema()` also fixes a pre-existing stale-variable bug: a NaN destined for a BINARY field was first set to None and then re-pickled from the stale local variable, producing pickled-NaN bytes instead of None. NaN in a BINARY field now correctly finalizes to None (guarded by a dedicated test). `INTEGRAL_TYPE_RANGES` lives next to the other per-`AttributeType` maps in `models/schema/attribute_type.py`. ### Any related issues, documentation, discussions? Backport of #6053 to `release/v1.2`, as requested in that PR's review discussion. Fixes #5935 on the release branch. ### How was this PR tested? Same test suite as #6053, re-verified on the `release/v1.2` baseline: - `test_tuple.py` on this branch: **57 passed** (the coercion cases including int32 and float64-exact-window boundaries and `np.float64`; rejection of non-integral / infinite / out-of-window floats; the out-of-window warning; NaN/None handling; DOUBLE and STRING fields staying untouched; the coercion pinned into `cast_to_schema` rather than `validate_schema`; and an integration-style test reproducing the full pipeline). The count is 57 (vs 59 on main) only because `release/v1.2`'s pre-existing `test_tuple.py` has two fewer baseline tests; the fix's own test additions are identical. - `ruff check` and `ruff format --check` clean on all three changed files. - `sbt "scalafixAll --check"` and `sbt scalafmtCheckAll`: pass (no Scala changed). - Backend `AMBER_TEST_FILTER=skip-integration sbt test`: **2034 passed, 0 failed, 0 aborted** across 195 suites, run against a clean isolated iceberg catalog (matching how CI provisions one per run). ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Fable 5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
