[
https://issues.apache.org/jira/browse/SPARK-59615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59615:
-----------------------------------
Labels: pull-request-available (was: )
> Avoid an intermediate buffer in VectorizedPlainValuesReader.readGeoData
> -----------------------------------------------------------------------
>
> Key: SPARK-59615
> URL: https://issues.apache.org/jira/browse/SPARK-59615
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.1.0
> Reporter: David Mollitor
> Priority: Minor
> Labels: pull-request-available
>
> h3. Problem
> {{VectorizedPlainValuesReader.readGeoData}} (the vectorized PLAIN-encoding
> read path for GEOMETRY/GEOGRAPHY columns) buffers every converted value for
> the whole page into a {{ByteBufferOutputStream}} created with no initial
> capacity, then copies it out into the column vector:
> {code:java}
> int base = v.arrayData().getElementsAppended();
> int dataLen = 0;
> final int intSize = 4;
> ByteBuffer lenBuffer = ByteBuffer.allocate(intSize);
> ByteBufferOutputStream out = new ByteBufferOutputStream();
> for (int i = 0; i < total; i++) {
> int len = readInteger();
> byte[] physicalValue = converter.convert(in.readNBytes(len), srid);
> v.putArray(rowId + i, base + dataLen + intSize, physicalValue.length);
> lenBuffer.putInt(0, physicalValue.length);
> out.write(lenBuffer.array());
> out.write(physicalValue);
> dataLen += intSize + physicalValue.length;
> }
> out.close();
> v.arrayData().appendBytes(dataLen, out.toByteArray(), 0);
> {code}
> This has two costs:
> * The no-arg {{ByteBufferOutputStream}} starts at the JDK default of 32 bytes
> and reallocates and copies its backing array on the order of {{log2(pageBytes
> / 32)}} times as it accumulates the page; the data is then copied twice more
> ({{toByteArray()}} and the final {{appendBytes}}).
> * A redundant 4-byte length prefix is written before each value. The element
> length is already recorded by {{putArray}}'s third argument, and {{putArray}}
> points _past_ the prefix, so those bytes are never read.
> The sibling reader {{VectorizedDeltaByteArrayReader.readGeoData}} already
> avoids both: it appends each converted value directly to the column vector's
> {{arrayData()}} with no intermediate buffer and no length prefix. The two
> implementations have diverged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
