fatmanverse opened a new pull request, #12802:
URL: https://github.com/apache/seatunnel/pull/12802
Fixes #12798
### Purpose of this pull request
DOUBLE and DECIMAL values changed in an Excel write/read round trip. There
were three causes:
1. `ExcelGenerator` wrote a DECIMAL as
`Double.parseDouble(value.toString())`.
2. `ExcelReadStrategy#getCellValue` (POI engine) returned
`DataFormatter#formatCellValue` for numeric cells. With the "General" format
this keeps about 10 significant digits (`10.285714285` -> `10.28571428`).
3. `ExcelCellUtils#convert` built a DECIMAL with
`BigDecimal.valueOf(Double.parseDouble(text))`.
Changes:
- Sink: a DECIMAL that a `double` represents exactly is still written as a
number cell. Any other DECIMAL is written as a text cell with
`toPlainString()`, so no digit is lost.
- POI reader: for numeric target types (TINYINT ... DECIMAL), a numeric cell
returns `getNumericCellValue()`. Date-formatted cells, STRING targets and every
other type keep using `DataFormatter`, so reading a numeric cell into a string
column is unchanged.
- `ExcelCellUtils`: a DECIMAL is parsed with `new BigDecimal(text)`.
Impact checked:
- For a number read from a cell, `new BigDecimal(Double.toString(d))` gives
the same value and scale as the previous `BigDecimal.valueOf(d)`, so the scale
of DECIMAL values read from number cells is unchanged.
- The EasyExcel engine only benefits from the third change. Its text values
are now parsed exactly as well.
- The Excel e2e cases read fixture files and assert on `NOT_NULL` and row
counts only. `ExcelReadStrategyTest` and `ExcelGeneratorTest` pass unchanged.
### Does this PR introduce _any_ user-facing change?
Yes:
- Numeric columns read from Excel keep their full stored value.
- A DECIMAL with more significant digits than a `double` holds is written as
a text cell, so spreadsheet formulas such as `SUM` skip it.
Both are documented in `incompatible-changes.md` (en/zh).
### How was this patch tested?
- Added `ExcelGeneratorTest#testDecimalKeepsAllDigits` and
`ExcelReadStrategyTest#testPoiReadKeepsNumericPrecision`.
- `./mvnw -pl seatunnel-connectors-v2/connector-file/connector-file-base
test` on JDK 17: all tests pass.
- On a deployed Zeta cluster, a 3000-row round trip MySQL `DOUBLE` /
`DECIMAL(20,4)` -> LocalFile excel -> LocalFile excel source -> MySQL had 2512
DOUBLE and 2241 DECIMAL mismatches before this change and 0 after it.
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
