xingnailu commented on issue #1450: URL: https://github.com/apache/datafusion-comet/issues/1450#issuecomment-2689580180
> Thanks for the info [@xingnailu](https://github.com/xingnailu). It does look like Comet is working as expected. I wonder if the bottleneck is reading data from OSS. Do you have the ability to run the benchmarks with Parquet data stored locally on each node to confirm this? spark version is 3.4.3. cargo 1.83.0 (5ffbef321 2024-10-29) rustc 1.83.0 (90b35a623 2024-11-26) Is it because we are using an ARM architecture processor? We understand that when Rust compiles, it only utilizes the NENO instruction set and does not yet support the SVE instruction set, whereas in the x86 architecture, AVX512 is already supported. This could be why the efficiency of vectorized processing is not as good as that provided by the Comet official website. We have also tested this on local storage before, and the performance was similar to that on OSS. ``` Architecture: aarch64 Byte Order: Little Endian CPU(s): 64 On-line CPU(s) list: 0-63 Thread(s) per core: 1 Core(s) per socket: 64 Socket(s): 1 NUMA node(s): 1 Vendor ID: ARM BIOS Vendor ID: Alibaba Cloud Model: 0 Model name: Neoverse-N2 BIOS Model name: virt-rhel7.6.0 Stepping: r0p0 CPU MHz: 3000.000 CPU max MHz: 3000.0000 CPU min MHz: 3000.0000 BogoMIPS: 100.00 L1d cache: 64K L1i cache: 64K L2 cache: 1024K L3 cache: 65536K NUMA node0 CPU(s): 0-63 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs sb dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
