Youngwb opened a new issue #2722: Spark on doris query performance improvement URL: https://github.com/apache/incubator-doris/issues/2722 Spark on doris transmit result by using Arrow format, Arrow specifies columnar memory format for flat and hierarchical data, and columnar layout is optimized for data locality for better performance on modern hardware like CPUs and GPUs. But spark-doris-connector converts from Arrow to Spark internal Row in row form, it not use the feature of columnar layout. it wold improve query performance a lot if result batch are processed by column.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
