GitHub user carloea2 added a comment to the discussion: Support Arrow batches 
as an explicit Python UDF input and output mode

Thanks @sshiv012 and @Ma77Ball. For a prototype, I propose comparing the 
current pandas UDF path, a pure Arrow UDF, and an Arrow UDF that converts to 
pandas. I would use numeric-heavy, string-heavy, and mixed columns with nulls 
and timestamps, across several batch sizes.

The end-to-end runs would use identical scan, filter, aggregation, and sink 
workflows, measuring total time, throughput, and peak memory, with output 
equivalence checks. Surya, I will coordinate with #8914 on explicit output 
ports, per-port schema validation, matching partition hashes, and completion 
flushing.

The 77x remains a conversion-only result, not an expected workflow speedup.

GitHub link: 
https://github.com/apache/texera/discussions/8477#discussioncomment-19189603

----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Reply via email to