[
https://issues.apache.org/jira/browse/ARROW-16958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608453#comment-17608453
]
David Li commented on ARROW-16958:
----------------------------------
I've sort of wanted to explore this anyways in any case - can we match up the
chunks that gRPC gives us to the buffers in the IPC payload, and avoid
concatenating? I suppose that optimization would bring little benefit if we
ended up having to align all of them anyways, though. And as mentioned, it
might be possible to insert padding data to align the buffer on the wire, but
then you can't count on the client being polite like that.
> [C++][FlightRPC] Flight generates misaligned buffers
> ----------------------------------------------------
>
> Key: ARROW-16958
> URL: https://issues.apache.org/jira/browse/ARROW-16958
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++, FlightRPC
> Reporter: David Li
> Assignee: David Li
> Priority: Major
>
> Protobuf's wire format design + our zero-copy serializer/deserializer mean
> that buffers can end up misaligned. On some Arrow versions, this can cause
> segfaults in kernels assuming alignment (and generally violates
> expectations).
> We should:
> * Possibly include buffer alignment in array validation
> * See if we can adjust the serializer to somehow pad things properly
> * See if we can do anything about this in the deserializer
> Example:
> {code:python}
> import pyarrow as pa
> import pyarrow.flight as flight
> class TestServer(flight.FlightServerBase):
> def do_get(self, context, ticket):
> schema = pa.schema(
> [
> ("index", pa.int64()),
> ("int8", pa.float64()),
> ("int16", pa.float64()),
> ("int32", pa.float64()),
> ]
> )
> return flight.RecordBatchStream(pa.table([
> [0, 1, 2, 3],
> [0, 1, None, 3],
> [0, 1, 2, None],
> [0, None, 2, 3],
> ], schema=schema))
> with TestServer() as server:
> client = flight.connect(f"grpc://localhost:{server.port}")
> table = client.do_get(flight.Ticket(b"")).read_all()
> for col in table:
> print(col.type)
> for chunk in col.chunks:
> for buf in chunk.buffers():
> if not buf: continue
> print("buffer is 8-byte aligned?", buf.address % 8)
> chunk.cast(pa.float32())
> {code}
> On Arrow 8
> {noformat}
> int64
> buffer is 8-byte aligned? 1
> double
> buffer is 8-byte aligned? 1
> buffer is 8-byte aligned? 1
> double
> buffer is 8-byte aligned? 1
> buffer is 8-byte aligned? 1
> double
> buffer is 8-byte aligned? 1
> buffer is 8-byte aligned? 1
> {noformat}
> On Arrow 7
> {noformat}
> int64
> buffer is 8-byte aligned? 4
> double
> buffer is 8-byte aligned? 4
> buffer is 8-byte aligned? 4
> fish: Job 1, 'python ../test.py' terminated by signal SIGSEGV (Address
> boundary error)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
