Hi Liya, Thanks for opening this discuss. +1 for this, vectorization makes sense for Flink especially for batch work loads, I think Flink should look into supporting it progressively.
Thanks, Ji Liu ------------------------------------------------------------------ From:Jeff Zhang <zjf...@gmail.com> Send Time:2019年7月2日(星期二) 15:50 To:dev <dev@flink.apache.org> Subject:Re: [DISCUSS] Vectorization Support in Flink Hi Liya, Displaying image is not supported in apache mail list, you need to put it elsewhere and post link in mail list. Fan Liya <liya.fa...@gmail.com> 于2019年7月2日周二 下午3:40写道： > Performance chart. FYI. > > Best, > Liya Fan > [image: image.png] > > On Tue, Jul 2, 2019 at 3:37 PM Fan Liya <liya.fa...@gmail.com> wrote: > >> Hi all, >> >> We have opened an issue about vectorization in Flink (FLINK-13053 >> <https://issues.apache.org/jira/browse/FLINK-13053>). Would you please >> give your valuable feedback? Thank you in advance. >> >> Vectorization is a popular technique in SQL engines today. Compared with >> traditional row-based approach, it has some distinct advantages, for >> example: >> >> >> >> 1) Better use of CPU resources (e.g. SIMD) >> >> 2) More compact memory layout >> >> 3) More friendly to compressed data format. >> >> >> >> Currently, Flink is based on a row-based SQL engine for both stream and >> batch workloads. To enjoy the above benefits, we want to bring >> vectorization to Flink. This involves substantial changes to the existing >> code base. Therefore, we give a plan to carry out such changes in small, >> incremental steps, in order not to affect existing features. We want to >> apply it to batch workload first. The details can be found in our proposal. >> >> >> >> For the past months, we have developed an initial implementation of the >> above ideas. Initial performance evaluations on TPC-H benchmarks show that >> substantial performance improvements can be obtained by vectorization (see >> the figure below). More details can be found in our proposal. >> >> >> >> [image: >> https://lh5.googleusercontent.com/hjXkXGImWOjaiB8zF0SKIMoItY6VCBm-BmJWWEXRo0ZPHdwLgKzCmIoNKef1YPCaAA7NXN6RvO-nwBBXBee52KeAtBjyIvh_NcAuChvW3BEtQuZGL5GPddqxL_iMV7HvEVCC6k-m] >> >> >> >> Special thanks to @Kurt Young’s team for all the kind help. >> >> Special thanks to @Piotr Nowojski for all the valuable feedback and help >> suggestions. >> >> >> Best, >> >> Liya Fan >> > -- Best Regards Jeff Zhang
