fatmanverse opened a new pull request, #12919:
URL: https://github.com/apache/seatunnel/pull/12919
Fixes #12914
### Purpose of this pull request
`numeric_special_value_as_null` (#12792) did not cover PostgreSQL `Infinity`
/ `-Infinity` once pgjdbc uses the binary protocol. There the driver cannot
decode the sign word of the infinities and throws `IllegalArgumentException:
invalid sign in "numeric" value` from `getBigDecimal`, and `getString` on the
same column fails the same way. `JdbcFieldTypeUtils#getBigDecimal` only caught
`SQLException` / `ClassCastException`, and `isNumericSpecialValue` relied on
`getString`, so the job failed. pgjdbc switches to binary for server-prepared
statements, which happens with the default URL after 5 executions on one
connection, for example from the 6th split of a partitioned read.
Changes:
- `getBigDecimal` also catches `IllegalArgumentException` and, as before,
returns null only when the value is a special value; any other error is
rethrown, and nothing changes when the option is off.
- When `getString` fails, `isNumericSpecialValue` reads the raw value with
`getBytes` and checks the binary numeric header: no digit words and the sign
word `0xC000` (NaN), `0xD000` (Infinity) or `0xF000` (-Infinity). Text-mode
values are still recognised from `getString` and never reach this check.
- `jdbc_postgres_source_and_sink_numeric_special_value_binary.conf` only set
`binaryTransfer=true`, which is the driver default, so the binary e2e case ran
in text mode. It now also sets `prepareThreshold=-1` so the statement is
server-prepared before its first execution and the case really runs in binary.
The header layout was checked against pgjdbc 42.4.3 (the version in the
distribution) on PostgreSQL 16: `getBytes` returns `00000000d0000020` for
Infinity, `00000000f0000020` for -Infinity, `00000000c0000000` for NaN and
`0002000000000002007b1194` for 123.45.
### Does this PR introduce _any_ user-facing change?
Yes, as a bug fix: with `numeric_special_value_as_null = true`, `Infinity`
and `-Infinity` are now read as null in binary mode, as they already were in
text mode. No option, default or behaviour with the option off changes.
### How was this patch tested?
- `JdbcFieldTypeUtilsTest`: binary Infinity/-Infinity read as null, the
error is kept with the option off, a failure on a header that has digit words
is rethrown, and a header with digits is not treated as a special value.
`connector-jdbc` unit tests pass on JDK 17, except
`JdbcOptionRuleCompletenessTest#testSinkRuleDeclaresAllSinkOptions`, which
fails on dev too (#12874, fixed by #12875).
- On a Zeta cluster with PostgreSQL 16, with `numeric_special_value_as_null
= true`:
- `prepareThreshold=-1` with `table_path` and with `query`: failed on the
Infinity row before, now NaN/Infinity/-Infinity are null and ordinary values
are unchanged.
- Default URL, `partition_column` with 25 splits, Infinity and -Infinity
in later splits: wrote 51 of 100 rows and failed before, now writes 100 rows
with nulls exactly at the special values.
- With the option off the job still fails, in text and in binary mode.
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [ ] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
