[
https://issues.apache.org/jira/browse/SPARK-60200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60200:
-----------------------------------
Labels: pull-request-available (was: )
> Casting a decimal with a huge negative scale to string or showing it builds
> an unbounded plain string
> -----------------------------------------------------------------------------------------------------
>
> Key: SPARK-60200
> URL: https://issues.apache.org/jira/browse/SPARK-60200
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.8, 4.2.0, 4.3.0, 4.1.3, 4.0.4, 4.4.0
> Reporter: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
>
> With `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, `DecimalType` has
> no lower bound on a negative scale. Casting such a decimal to string under
> ANSI (`useDecimalPlainString`, `ToStringBase`) and `Dataset.show`
> (`ToPrettyString`) call `Decimal.toPlainString`, which builds the full plain
> string. For `1E+2147483647` it fails with `OutOfMemoryError: too large to fit
> in a String`:
> {code:scala}
> lit("1e2147483647").cast(DecimalType(1, -2147483647)).cast("string")
> spark.range(1).select(lit(new java.math.BigDecimal("1e2147483647"))).show()
> {code}
> The inner cast succeeds because the precision and scale already match, so
> `Decimal.changePrecision` takes its fast path and the value reaches the
> string conversion.
> SPARK-60119 bounds this only for the `NUMERIC_VALUE_OUT_OF_RANGE` error
> message (scientific notation below scale -1000). The threshold could move to
> a `Decimal`-level helper shared by these callers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
