GitHub user singhpratech closed the discussion with a comment: How DataFusion could support other compute engines (libcudf, velox)
> I personally think the idea of implementing `ExecutionPlans` and an > associated rewrite rule that could take advantage of a GPU's hardware sounds > like a pretty neat idea and would be interested in seeing some sort of POC That is the shape [ArrowMetal](https://github.com/singhpratech/ArrowMetal)'s DataFusion crate takes, on Apple silicon rather than CUDA. ArrowMetal runs Apache Arrow compute on the Apple silicon GPU through Metal; `datafusion-arrowmetal` is a `PhysicalOptimizerRule` plus its own `ExecutionPlan`, the granular approach in the opening post applied per node rather than through a `PhysicalPlanner`: registered on the `SessionContext`, the rule replaces a `SortExec` or an `AggregateExec` of a measured shape with its own `ExecutionPlan`, which runs that node on the GPU through Metal and returns the `RecordBatch`es DataFusion expects; every other node in the plan stays DataFusion's. On these machines the CPU and the GPU share one memory, so the batch is scanned where it already is, with no copy across a bus. Measured against DataFusion 55.1 on an M4 Max: full `ORDER BY` sorts 6.9x to 28.8x faster from 250,000 to 50,000,000 rows; the ten group-by series the default takes (count, DISTINCT, integer MIN/MAX over 50M-row MemTables) 2.31x to 4.27x; top-k (0.14x to 0.41x) and filters (0.31x to 0.79x) stay with DataFusion, which is faster there. 13,632 query pairs run with and without the rule give the same answers, and `rule.report()` lists each node, taken or left, with the reason. The numbers, their result files and the decision table are in https://github.com/singhpratech/ArrowMetal/blob/main/docs/DATAFUSION.md; the crate is `datafusion-arrowmetal` on crates.io, pinned to `datafusion = "=55.1.0"` because the physical-plan API moves between releases. What made the rule approach workable: an `ExecutionPlan` is enough to own one node, and nothing in DataFusion had to change. The one cost is the version pin; a stable subset of the physical-plan API would let a crate like this follow releases. GitHub link: https://github.com/apache/datafusion/discussions/8498#discussioncomment-18726848 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
