CurtHagenlocher opened a new issue, #418: URL: https://github.com/apache/arrow-dotnet/issues/418
`NativeBuffer<TItem, TTracker>.Grow` doubles the current length in a `checked` context and does not saturate, so once a buffer passes half of the addressable maximum its **next** grow throws `OverflowException` — however small the requested increase, and even though the requested size still fits comfortably. ```csharp // src/Apache.Arrow/Memory/NativeBuffer.cs // Exponential growth (2x) to amortise repeated grows // TODO: There might be a size that's big enough to work for this case but not too big to overflow. // We could use that instead of blindly doubling. int newCount = Math.Max(newElementCount, checked(Length * 2)); ``` The existing `TODO` on those lines describes exactly this. For a `NativeBuffer<byte, …>` that means a hard ceiling near **1 GiB**: at `Length` above `int.MaxValue / 2`, `checked(Length * 2)` overflows, so a buffer cannot grow to 1.2 GiB even though `int.MaxValue` bytes are addressable and `Memory<byte>` could hold them. ## Impact Any caller that grows a buffer incrementally past that point fails, and fails misleadingly: the exception is arithmetic overflow, which reads as an internal error rather than as a size limit the caller was approaching. There is no way for a caller to avoid it — asking for a smaller increment does not help, because the overflow is in the doubling and not in the request. I hit this decoding a large Parquet BYTE_ARRAY column, where the buffer grows once per page. A column with roughly 1.5 GiB of string data — comfortably inside what a `StringArray` can address — failed with `OverflowException` from inside the allocator rather than reading successfully. ## Expected Growth saturates at the largest addressable element count instead of overflowing, so a request that fits succeeds. A request that genuinely cannot be addressed should still fail, as it does today, at the byte-size calculation. ## Note on the neighbouring limit This is separate from the 2 GiB ceiling on `ArrowBuffer` itself (`ReadOnlyMemory<byte>`, `int Length`). Fixing this does not raise that ceiling; it just stops buffers failing at half of it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
