ianmcook opened a new issue, #496:
URL: https://github.com/apache/arrow-js/issues/496
> [!NOTE]
> I discovered this issue and wrote it up with help from Claude Opus 5.5.
### Describe the bug, including details regarding any error messages,
version, and platform.
Since #377 (released in 21.2.0), `instanceof Table` accepts a `Table`
created by another copy of apache-arrow. `tableToIPC` then serializes such a
table without any error, but the Arrow IPC stream it writes is unusable: field
types are missing or wrong, and the dictionary and record batch messages have
empty bodies. None of the data is written.
An easy way to get two copies is the package's own CommonJS and ES module
builds. Node loads them separately when, for example, an ES module application
passes a table to a CommonJS library. Bundlers can do the same through the
`import` and `require` conditions.
#### Reproduction
apache-arrow 21.2.0, Node.js 24.8.0, macOS:
```js
// repro.mjs
import { createRequire } from "node:module";
import * as esm from "apache-arrow";
const cjs = createRequire(import.meta.url)("apache-arrow");
const table = esm.tableFromArrays({ id: Int32Array.from([1, 2]), text: ["a",
"b"] });
console.log(esm.Table === cjs.Table, table instanceof cjs.Table); // false
true
const crossBuild = cjs.tableToIPC(table, "stream"); // No error
const sameBuild = esm.tableToIPC(table, "stream");
console.log(crossBuild.length, sameBuild.length); // 440 648
esm.tableFromIPC(crossBuild);
// TypeError: Cannot destructure property 'length' of '(intermediate
value)(intermediate value)(intermediate value)' as it is undefined.
// at .../apache-arrow/visitor/vectorloader.mjs:154
```
PyArrow 25.0.1 rejects the same `crossBuild` bytes with `OSError: Invalid
flatbuffers message.` The `sameBuild` stream reads correctly in both.
Single-column tables fail in different ways. In each case the cross-build
stream has empty message bodies:
| Column type | Same-build bodies | Cross-build bodies | PyArrow reading the
cross-build stream |
| --- | --- | --- | --- |
| `Int32` | batch: 8 bytes | batch: 0 bytes | `ArrowNotImplementedError:
Integers with less than 8 bits not implemented` |
| `Float64` | batch: 16 bytes | batch: 0 bytes | `OSError: buffer_index out
of range.` |
| `Utf8` | batch: 24 bytes | batch: 0 bytes | `OSError: buffer_index out of
range.` |
| `Dictionary<Utf8, Int32>` | dictionary: 24, batch: 16 bytes | dictionary:
0, batch: 0 bytes | `OSError: Unexpected null field
DictionaryEncoding.indexType in flatbuffer-encoded metadata` |
#### Cause
#377 added
[`DataType.isDataType()`](https://github.com/apache/arrow-js/blob/c94c7709a34c13faee2b8e10c7a4b86dcdec0849/src/type.ts#L58-L69)
for checks across copies, and deliberately did not give `DataType` a
`Symbol.hasInstance`. But
[`getVisitFn`](https://github.com/apache/arrow-js/blob/c94c7709a34c13faee2b8e10c7a4b86dcdec0849/src/visitor.ts#L61-L76)
still dispatches on `node instanceof DataType` and `node.type instanceof
DataType`. A type from another copy fails both checks and falls through to
`Type.NONE`. As a result, `TypeAssembler` writes no type, and no `indexType`
for a dictionary, and the writer emits no buffers. Nothing reports an error.
#### Suggested fix
Use the cross-copy check in `getVisitFn`:
```diff
- if (node && (node instanceof DataType)) {
+ if (node && DataType.isDataType(node)) {
return getVisitFnByTypeId(visitor, inferDType(node),
throwIfNotFound);
}
- if (node?.type && (node.type instanceof DataType)) {
+ if (node?.type && DataType.isDataType(node.type)) {
return getVisitFnByTypeId(visitor, inferDType(node.type),
throwIfNotFound);
}
```
I applied the equivalent change to `visitor.js` in the CommonJS build of
21.2.0. With it, the cross-build write in the reproduction round-trips, and its
output is byte-identical to a same-build write. I also checked a table with an
`Int32` column and a `Dictionary<Utf8, Int16>` column: identical bytes, and
PyArrow reads the result.
It may also be worth making the writer throw when a type can't be visited,
rather than writing an empty body.
### Related issues
#61, #377, #56
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
