Thanks Ron for the proposal. Do you have some benchmark results for the performance improvement? I am more concerned about the improvement on Flink than the data in other papers.
Best, Jingsong On Mon, May 29, 2023 at 2:16 PM liu ron <ron9....@gmail.com> wrote: > > Hi, dev > > I'd like to start a discussion about FLIP-315: Support Operator Fusion > Codegen for Flink SQL[1] > > As main memory grows, query performance is more and more determined by the > raw CPU costs of query processing itself, this is due to the query > processing techniques based on interpreted execution shows poor performance > on modern CPUs due to lack of locality and frequent instruction > mis-prediction. Therefore, the industry is also researching how to improve > engine performance by increasing operator execution efficiency. In > addition, during the process of optimizing Flink's performance for TPC-DS > queries, we found that a significant amount of CPU time was spent on > virtual function calls, framework collector calls, and invalid > calculations, which can be optimized to improve the overall engine > performance. After some investigation, we found Operator Fusion Codegen > which is proposed by Thomas Neumann in the paper[2] can address these > problems. I have finished a PoC[3] to verify its feasibility and validity. > > Looking forward to your feedback. > > [1]: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-315+Support+Operator+Fusion+Codegen+for+Flink+SQL > [2]: http://www.vldb.org/pvldb/vol4/p539-neumann.pdf > [3]: https://github.com/lsyldliu/flink/tree/OFCG > > Best, > Ron
