andygrove commented on PR #6371: URL: https://github.com/apache/datafusion-comet/pull/6371#issuecomment-6263756468
Thanks @viirya. On where the narrowing lives: I would keep it at serialization. That is where mixed widths break broadcast coalescing and the native stream schema, and it is the one place that needs a copy anyway. Normalizing where `CometArrowPythonRunnerBase` wraps the worker's vectors would add a copy per batch on the row-transition path, which today reads 64-bit offsets without one, so I am not convinced it is better either. You are right that `NativeUtil.exportBatch` still rejects these vectors, through `getFieldVector(valueVector, "export")`, and that a native operator above `CometMapInBatchExec` may be the likelier consumer. I filed #7109 for that edge. `narrowOffsets` is private for now and can be shared when it is picked up. The smaller points are fixed in a06a59a05b, with a reply on each thread. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
