ianmcook opened a new issue, #956: URL: https://github.com/apache/arrow-nanoarrow/issues/956
> [!NOTE] > I discovered this issue and wrote it up with help from Claude Opus 5.5. ### Describe the bug, including details regarding any error messages, version, and platform. The nanoarrow IPC decoder segfaults when a dictionary-encoded field's `DictionaryEncoding` omits `indexType`. The format allows that: [`format/Schema.fbs`](https://github.com/apache/arrow/blob/e85e181188ac3c5e8407b2160331faf5e78fd429/format/Schema.fbs#L500-L505) says: > If this field is null, the indices must be signed int32. The same stream with `indexType` present reads correctly. #### Reproduction nanoarrow 0.9.0 (the latest release), Python 3.11, macOS. The script embeds two Arrow IPC streams that PyArrow 25.0.1 wrote, each holding one `dictionary<values=string, indices=int32>` column with the values `["a", "b", "a"]`. The second stream omits `DictionaryEncoding.indexType`; nothing else about the data differs. The script that made them is below. ```python """nanoarrow reading Arrow IPC streams with and without DictionaryEncoding.indexType.""" import base64 import sys import nanoarrow as na import nanoarrow.ipc as ipc # The same stream, written by PyArrow 25.0.1: one dictionary<values=string, indices=int32> # column with values ["a", "b", "a"]. The second copy omits DictionaryEncoding.indexType, # which Schema.fbs allows: "If this field is null, the indices must be signed int32." WITH_INDEX_TYPE = base64.b64decode( "/////5AAAAAQAAAAAAAKAAwABgAFAAgACgAAAAABBAAEAAAAuP///wQAAAABAAAAFAAAABAAGAAI" "AAYABwAMABAAFAAQAAAAAAABBRQAAABEAAAAIAAAAAQAAAAAAAAABAAAAHRleHQAAAAACAAIAAAA" "BAAIAAAADAAAAAgADAAIAAcACAAAAAAAAAEgAAAABAAEAAQAAAD/////qAAAABQAAAAAAAAADAAU" "AAYABQAIAAwADAAAAAACBAAUAAAAGAAAAAAAAAAIAAoAAAAEAAgAAAAQAAAAAAAKABgADAAEAAgA" "CgAAAEwAAAAQAAAAAgAAAAAAAAAAAAAAAwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAA" "AAAAEAAAAAAAAAACAAAAAAAAAAAAAAABAAAAAgAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAgAAAAAA" "AABhYgAAAAAAAP////+IAAAAFAAAAAAAAAAMABYABgAFAAgADAAMAAAAAAMEABgAAAAQAAAAAAAA" "AAAACgAYAAwABAAIAAoAAAA8AAAAEAAAAAMAAAAAAAAAAAAAAAIAAAAAAAAAAAAAAAAAAAAAAAAA" "AAAAAAAAAAAMAAAAAAAAAAAAAAABAAAAAwAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAAAAAD/" "////AAAAAA==" ) WITHOUT_INDEX_TYPE = base64.b64decode( "/////5gAAAAQAAAAAAAKAAwABgAFAAgACgAAAAABBAAEAAAAuP///wQAAAABAAAAFAAAABAAGAAI" "AAYABwAMABAAFAAQAAAAAAABBRQAAABEAAAAIAAAAAQAAAAAAAAABAAAAHRleHQAAAAACAAIAAAA" "BADc////DAAAAAgADAAIAAcACAAAAAAAAAEgAAAABAAEAAQAAAAIAAgAAAAAAP////+oAAAAFAAA" "AAAAAAAMABQABgAFAAgADAAMAAAAAAIEABQAAAAYAAAAAAAAAAgACgAAAAQACAAAABAAAAAAAAoA" "GAAMAAQACAAKAAAATAAAABAAAAACAAAAAAAAAAAAAAADAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" "AAAADAAAAAAAAAAQAAAAAAAAAAIAAAAAAAAAAAAAAAEAAAACAAAAAAAAAAAAAAAAAAAAAAAAAAEA" "AAACAAAAAAAAAGFiAAAAAAAA/////4gAAAAUAAAAAAAAAAwAFgAGAAUACAAMAAwAAAAAAwQAGAAA" "ABAAAAAAAAAAAAAKABgADAAEAAgACgAAADwAAAAQAAAAAwAAAAAAAAAAAAAAAgAAAAAAAAAAAAAA" "AAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAAAAAAAAEAAAADAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAA" "AAAAAAAAAP////8AAAAA" ) stream = WITH_INDEX_TYPE if sys.argv[1:] == ["with"] else WITHOUT_INDEX_TYPE with ipc.InputStream.from_readable(stream) as source: array = na.ArrayStream(source).read_all() print(array.schema, list(array.iter_py())) ``` With `indexType` present (`python repro.py with`): ``` <Schema> non-nullable struct<text: dictionary(int32)<string>> [{'text': 'a'}, {'text': 'b'}, {'text': 'a'}] ``` Without it (`python -X faulthandler repro.py`): ``` Fatal Python error: Segmentation fault Current thread 0x00000001fc8a2180 (most recent call first): File ".../site-packages/nanoarrow/ipc.py", line 77 in __arrow_c_stream__ File ".../site-packages/nanoarrow/c_array_stream.py", line 73 in c_array_stream File ".../site-packages/nanoarrow/array_stream.py", line 60 in __init__ ``` **Expected:** the same output as the stream with `indexType`, since a missing `indexType` means signed int32. At a minimum, an error rather than a crash. <details> <summary>How the streams were made</summary> This script writes the stream with PyArrow, then makes a copy with `indexType` omitted from its schema message. With PyArrow 25.0.1 it prints exactly the base64 above. ```python """Write the streams above: one from PyArrow, and a copy with indexType omitted.""" import base64 import struct import pyarrow as pa def table_at(buf, pos): """Return the (table, vtable) positions for the table referenced at pos.""" table = pos + struct.unpack_from("<I", buf, pos)[0] return table, table - struct.unpack_from("<i", buf, table)[0] def field_offset(buf, table, vtable, slot): """Return a field's offset within its table, or 0 if the field is absent.""" entry = 4 + 2 * slot present = entry < struct.unpack_from("<H", buf, vtable)[0] return struct.unpack_from("<H", buf, vtable + entry)[0] if present else 0 def without_index_type(stream): """Omit DictionaryEncoding.indexType from the first field of the schema message. Flatbuffers share identical vtables between tables (here the Schema's and the DictionaryEncoding's), so the encoding gets its own copy of its vtable, appended to the message, with indexType cleared. """ buf = bytearray(stream) length = struct.unpack_from("<i", buf, 4)[0] # After the 0xFFFFFFFF continuation marker meta = 8 message, message_vt = table_at(buf, meta) schema, schema_vt = table_at(buf, message + field_offset(buf, message, message_vt, 2)) # Message.header fields = schema + field_offset(buf, schema, schema_vt, 1) # Schema.fields field, field_vt = table_at(buf, fields + struct.unpack_from("<I", buf, fields)[0] + 4) # fields[0] encoding, encoding_vt = table_at(buf, field + field_offset(buf, field, field_vt, 4)) # Field.dictionary vtable = bytearray(buf[encoding_vt:encoding_vt + struct.unpack_from("<H", buf, encoding_vt)[0]]) struct.pack_into("<H", vtable, 4 + 2 * 1, 0) # DictionaryEncoding.indexType: absent struct.pack_into("<i", buf, encoding, encoding - (meta + length)) metadata = bytes(buf[meta:meta + length]) + bytes(vtable) metadata += bytes(-len(metadata) % 8) return struct.pack("<Ii", 0xFFFFFFFF, len(metadata)) + metadata + bytes(buf[meta + length:]) table = pa.table({"text": pa.array(["a", "b", "a"]).dictionary_encode()}) # int32 indices sink = pa.BufferOutputStream() with pa.ipc.new_stream(sink, table.schema) as writer: writer.write_table(table) data = sink.getvalue().to_pybytes() print("WITH_INDEX_TYPE:", base64.b64encode(data).decode()) print("WITHOUT_INDEX_TYPE:", base64.b64encode(without_index_type(data)).decode()) ``` </details> #### Cause [`ArrowIpcSetDictionaryEncoding`](https://github.com/apache/arrow-nanoarrow/blob/ecba475603d82755856fd1b89bbaaa7fa43d8393/src/nanoarrow/ipc/decoder.c#L1249-L1250) passes `DictionaryEncoding_indexType_get(dictionary_encoding)` straight to [`ArrowIpcDecoderSetTypeInt`](https://github.com/apache/arrow-nanoarrow/blob/ecba475603d82755856fd1b89bbaaa7fa43d8393/src/nanoarrow/ipc/decoder.c#L747-L753). When the field is absent that table is NULL, and `ArrowIpcDecoderSetTypeInt` reads `is_signed` and `bitWidth` from it. #### Suggested fix ```c ns(Int_table_t) index_type = ns(DictionaryEncoding_indexType_get(dictionary_encoding)); if (index_type == NULL) { // Schema.fbs: "If this field is null, the indices must be signed int32." NANOARROW_RETURN_NOT_OK( ArrowIpcDecoderSetTypeSimple(schema, NANOARROW_TYPE_INT32, error)); } else { NANOARROW_RETURN_NOT_OK(ArrowIpcDecoderSetTypeInt(schema, index_type, error)); } ``` Arrow C++ has a related bug with the same stream: it rejects it with an error instead of crashing. That is reported separately as https://github.com/apache/arrow/issues/51779. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
