zero-element commented on issue #2703: URL: https://github.com/apache/datafusion/issues/2703#issuecomment-2692570732
I agree with @faucct that this approach could be beneficial for building Online ML Feature Store engines. This is because such systems often involve numerous element-wise calculations, which can be memory bound (considering low flops/bytes ratio with result of each operation needs to be written to memory). JIT is not inherently in conflict with vectorized code. In fact, the paper referenced earlier also mentioned: > it is sometimes beneficial to fuse the loops of two (or more) VectorWise-style primitives into a single loop—saving materialization step This concept may be similar to Whole-Stage Code Generation in Apache Spark. However, I am not entirely certain about this similarity without inspecting the actual implementation details. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
