jsve opened a new issue, #484:
URL: https://github.com/apache/arrow-js/issues/484
### Describe the bug, including details regarding any error messages,
version, and platform.
We hit an issue downstream where picking returned the wrong row, but only
for batches with an even number of rows. I traced it to our code iterating over
`Data.valueOffsets`. After an IPC round trip, the array includes an extra
trailing zero for these batches.
Is valueOffsets intended to expose the full backing buffer, including
padding, or only the logical offsets? Data.slice() trims it to length + 1,
which made the difference surprising.
### Repro
Reproduced on: apache-arrow 21.1.0, Node v22.20.0
```ts
import { Table, Utf8, tableFromIPC, tableToIPC, vectorFromArray } from
"apache-arrow";
function check(n) {
const values = Array.from({ length: n }, (_, i) => `v${i}`);
const table = new Table({ s: vectorFromArray(values, new Utf8()) });
const after = tableFromIPC(tableToIPC(table)).getChild("s").data[0];
console.log({
rows: n,
expected: n + 1,
actual: after.valueOffsets.length,
tail: Array.from(after.valueOffsets.slice(-3)),
afterSlice: after.slice(0, n).valueOffsets.length,
});
}
check(4);
check(5);
check(6);
```
Gives:
```
{ rows: 4, expected: 5, actual: 6, tail: [ 6, 8, 0 ], afterSlice: 5 }
{ rows: 5, expected: 6, actual: 6, tail: [ 6, 8, 10 ], afterSlice: 6 }
{ rows: 6, expected: 7, actual: 8, tail: [ 10, 12, 0 ], afterSlice: 7 }
```
List behaves the same. Using `[[0, 1], [2, 3], [4, 5], [6, 7]]` as the
values and `new List(new Field("item", new Int32(), true))` as the type
produces `valueOffsets` of `[0, 2, 4, 6, 8, 0]` after the IPC round trip.
An even row count needs an odd number of 32 bit offsets, which is 4 bytes
short of an 8 byte boundary, so one zero offset gets appended. Odd row counts
come out at exactly length + 1.
### Traces
`addBuffer` in
[`visitor/vectorassembler.ts`](https://github.com/apache/arrow-js/blob/5d2519f14c461dea55b1f12d9b42438877ee7e12/src/visitor/vectorassembler.ts#L126-L131)
rounds the declared region length up to 8 bytes:
```ts
const byteLength = (values.byteLength + 7) & ~7;
```
so the padding sits inside the `BufferRegion` the writer declares, and [the
loader](https://github.com/apache/arrow-js/blob/5d2519f14c461dea55b1f12d9b42438877ee7e12/src/visitor/vectorloader.ts#L140-L143)
hands the whole region back as the typed array. The format allows this. The
`Buffer.length` docs in
[format/Schema.fbs](https://github.com/apache/arrow/blob/e2524aa088cd6a391b40635245d757542e2b2810/format/Schema.fbs#L548-L559)
say padding bytes "do not need to be accounted for in the size here", so a
reader cannot assume the declared length excludes padding.
### Observations
`Data.slice()` already clamps. `_sliceBuffers` in `data.ts` does
`arr.subarray(offset, offset + length + 1)` for the offsets buffer, which
explains the `afterSlice` behaviour above. I couldn’t find documentation
explaining whether callers should expect this difference between loaded and
sliced data.
### Suggested fix
Would it make sense to trim the offsets view to `length + 1` when loading
IPC data? That would make it consistent with `Data.slice()` without copying the
buffer. I’m unsure whether this belongs in the loader or whether exposing the
full buffer is intentional.
When length is omitted, makeData defaults to valueOffsets.length - 1 for
[Utf8, LargeUtf8, Binary and
LargeBinary](https://github.com/apache/arrow-js/blob/5d2519f14c461dea55b1f12d9b42438877ee7e12/src/data.ts#L306-L336),
and
[List](https://github.com/apache/arrow-js/blob/5d2519f14c461dea55b1f12d9b42438877ee7e12/src/data.ts#L373-L378).
Reconstructing the loaded Utf8 data from its buffers without explicitly
passing length therefore reports one extra row for the even-row examples above.
Builders also over-allocate: `vectorFromArray` with four `Utf8` rows
produces valueOffsets.length === 16 before any IPC. I’m unsure whether trimming
should be specific to IPC loading or apply more broadly to these
variable-length types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
