[
https://issues.apache.org/jira/browse/SPARK-40351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-40351.
----------------------------------
Resolution: Invalid
Resolving as Invalid — this is a usage/how-to question rather than a specific
Spark defect or actionable change. Usage questions are best directed to
[email protected] (https://spark.apache.org/community.html) or Stack
Overflow (tag apache-spark). Findings from triage: Verified against
apache/master. Sum.scala:75-77 maps DecimalType.Fixed(precision, scale) ->
DecimalType.bounded(precision + 10, scale), and Average.scala:76 does the same
for its sum accumulator; both confirmed by direct read. git log -L 75,77 on
Sum.scala shows the "precision + 10" constant present continuously (only
modifier changes: private/protected/lazy), never altered — stable ~10 years,
consistent with the cited history. DecimalType.bounded caps at
MAX_PRECISION=38, so the widening cannot overflow the max. No SQLConf toggles
this. The ticket is a design-rationale/how-to question ("why 1
Please reopen with a concrete reproducer or a specific proposed change if this
is actually a bug or an actionable improvement.
> Spark Sum increases the precision of DecimalType arguments by 10
> ----------------------------------------------------------------
>
> Key: SPARK-40351
> URL: https://issues.apache.org/jira/browse/SPARK-40351
> Project: Spark
> Issue Type: Question
> Components: Optimizer
> Affects Versions: 3.2.0
> Reporter: Tymofii
> Priority: Minor
>
> Currently in Spark automatically increases Decimal field by 10 (hard coded
> value) after SUM aggregate operation -
> [https://github.com/apache/spark/blob/branch-3.2/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala#L1877.]
> There are a couple of questions:
> # Why was 10 chosen as default one?
> # Does it make sense to allow the user to override this value via
> configuration?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
