CurtHagenlocher opened a new issue, #443:
URL: https://github.com/apache/arrow-dotnet/issues/443
`ArrowArrayConcatenator.Concatenate` has three problems with two families of
types. Seen on Apache.Arrow 23.0.0 (.NET 10, Windows 11). The code involved is
unchanged on `main` as of ebdd8a0.
1. **Extension arrays are refused.** Concatenating two `GuidArray`s throws
instead of concatenating the storage and wrapping the result:
```
ArgumentException: Specified array type <Extension> does not match
expected type(s) <FixedSizedBinary> (Parameter 'TypeId')
at Apache.Arrow.ArrayDataExtensions.EnsureDataType(ArrayData data,
ArrowTypeId id)
at
Apache.Arrow.ArrayDataConcatenator.ArrayDataConcatenationVisitor.CheckData(IArrowType
type, Int32 expectedBufferCount)
at
Apache.Arrow.ArrayDataConcatenator.ArrayDataConcatenationVisitor.Visit(FixedWidthType
type)
```
2. **A concatenated view array does not own its data buffers.**
`ConcatenateBinaryViewArrayData` builds a new view buffer, but it puts the
inputs' own variadic data buffers into the result without `Retain()`-ing them
(`ArrayDataConcatenator.cs:704-711`). The result reads correctly until an input
is disposed. After that, every non-inlined value throws:
```
NullReferenceException
at Apache.Arrow.Memory.SharedMemoryHandle.get_Memory()
at Apache.Arrow.ArrowBuffer.get_Memory()
at Apache.Arrow.BinaryViewArray.GetBytes(Int32 index, Boolean& isNull)
```
This also happens when each input holds its own reference
(`SliceShared`), so the caller cannot work around it. Every other type's result
has buffers of its own, so disposing the inputs is safe for them.
3. **A zero-length view input with variadic buffers makes concatenation
throw.** `ConcatenateViewBuffer` skips zero-length inputs (`:989`), so their
buffers are not counted in `variadicBufferCount` (`:1007`).
`ConcatenateBinaryViewArrayData` sizes the result's buffer array from that
count (`:701`), but then copies every input's variadic buffers into it, empty
inputs included (`:709`):
```
IndexOutOfRangeException: Index was outside the bounds of the array.
at
Apache.Arrow.ArrayDataConcatenator.ArrayDataConcatenationVisitor.ConcatenateBinaryViewArrayData(IArrowType
type)
```
An empty slice of a view array keeps its parent's data buffers, so it is
enough to trigger this.
**Expected:** each concatenates, and the result stays valid after the inputs
are disposed. For extension types, that means concatenating the storage arrays
and calling `ExtensionType.CreateArray`. For views, it means retaining (or
copying) the data buffers the result references, and counting and copying the
same set of inputs' buffers.
**Repro** (console app referencing Apache.Arrow 23.0.0):
```csharp
using Apache.Arrow;
static void Run(string name, Func<string> body)
{
try { Console.WriteLine($"{name}: {body()}"); }
catch (Exception ex) { Console.WriteLine($"{name}: {ex.GetType().Name}:
{ex.Message}"); }
}
static StringViewArray Strings(params string[] values)
{
var b = new StringViewArray.Builder();
foreach (var v in values) b.Append(v);
return b.Build();
}
Run("1. extension", () =>
{
var a = new GuidArray.Builder().Append(Guid.NewGuid()).Build();
var b = new GuidArray.Builder().Append(Guid.NewGuid()).Build();
return ArrowArrayConcatenator.Concatenate(new IArrowArray[] { a, b
}).Length.ToString();
});
Run("2. views, inputs disposed", () =>
{
var a = Strings("a string longer than twelve bytes");
var b = Strings("another string longer than twelve");
var c = (StringViewArray)ArrowArrayConcatenator.Concatenate(new
IArrowArray[] { a, b });
string before = c.GetString(0) + " | " + c.GetString(1); // fine
a.Dispose();
b.Dispose();
return $"before: [{before}]; after: [{c.GetString(0)} |
{c.GetString(1)}]"; // throws
});
Run("2b. views, SliceShared inputs disposed", () =>
{
var source = Strings("a string longer than twelve bytes", "another
string longer than twelve");
var a = new StringViewArray(source.Data.SliceShared(0, 1));
var b = new StringViewArray(source.Data.SliceShared(1, 1));
source.Dispose();
var c = (StringViewArray)ArrowArrayConcatenator.Concatenate(new
IArrowArray[] { a, b });
a.Dispose();
b.Dispose();
return $"[{c.GetString(0)} | {c.GetString(1)}]"; // throws
});
Run("3. views, empty input", () =>
{
var empty = new
StringViewArray(Strings("zzzzzzzzzzzzzzzzzzzzzzzzzzzzzz").Data.Slice(0, 0));
var b = Strings("another string longer than twelve");
var c = (StringViewArray)ArrowArrayConcatenator.Concatenate(new
IArrowArray[] { empty, b }); // throws
return c.GetString(0);
});
```
Output:
```
1. extension: ArgumentException: Specified array type <Extension> does not
match expected type(s) <FixedSizedBinary> (Parameter 'TypeId')
2. views, inputs disposed: NullReferenceException: Object reference not set
to an instance of an object.
2b. views, SliceShared inputs disposed: NullReferenceException: Object
reference not set to an instance of an object.
3. views, empty input: IndexOutOfRangeException: Index was outside the
bounds of the array.
```
Found while reading a Parquet row group in batches, where a batch that spans
two decoded runs is concatenated and each batch may be disposed independently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
