[
https://issues.apache.org/jira/browse/SPARK-59443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59443:
-----------------------------------
Labels: pull-request-available (was: )
> Reduce allocations when reading small-precision decimals from
> ArrowColumnVector
> -------------------------------------------------------------------------------
>
> Key: SPARK-59443
> URL: https://issues.apache.org/jira/browse/SPARK-59443
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Peifeng Li
> Priority: Minor
> Labels: pull-request-available
>
> ArrowColumnVector.DecimalAccessor converts each non-null Decimal128 value
> through Decimal.apply(vector.getObject(rowId), precision, scale).
> Arrow Java 19.0.0 constructs a temporary byte array and BigInteger when
> materializing the BigDecimal. This also happens for source precision 1–18,
> whose valid unscaled integers fit in a signed long.
> A small-precision accessor can read the native Decimal128 words, check
> sign extension, and construct BigDecimal from the unscaled long and source
> scale. Values that do not fit in a long retain the full-width conversion.
> Spark still applies the requested precision and scale, and the result
> retains BigDecimal-backed Decimal behavior for checked integer casts.
> ArrowCachedBatchSerializer already reads native long words for compact
> decimals when writing directly to UnsafeRow. This change applies the same
> decoding approach to the general ArrowColumnVector interface.
> A prototype based on master 15ec0e55b5d5faa9b7844978eddfec696f95f778,
> using Arrow Java 19.0.0 and Corretto Java 17.0.16, measured:
> - decimal(18,2) -> decimal(18,2): 192 -> 96 allocated bytes per read.
> - decimal(18,2) -> decimal(18,1): 256 -> 160 allocated bytes per read.
> - decimal(38,2) -> decimal(38,2): 200 -> 200 allocated bytes per read.
> The allocation comparison used the original and modified ArrowColumnVector
> classes in separate JVMs, with two runs per version and five measured
> samples per scenario. Input batches contained 4,096 alternating positive
> and negative values; returned Decimal objects were retained in an array.
> Related Vortex connector discussion:
> https://github.com/vortex-data/vortex/issues/9837
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
