Hi, dev I'd like to start a discussion about FLIP-315: Support Operator Fusion Codegen for Flink SQL[1]
As main memory grows, query performance is more and more determined by the raw CPU costs of query processing itself, this is due to the query processing techniques based on interpreted execution shows poor performance on modern CPUs due to lack of locality and frequent instruction mis-prediction. Therefore, the industry is also researching how to improve engine performance by increasing operator execution efficiency. In addition, during the process of optimizing Flink's performance for TPC-DS queries, we found that a significant amount of CPU time was spent on virtual function calls, framework collector calls, and invalid calculations, which can be optimized to improve the overall engine performance. After some investigation, we found Operator Fusion Codegen which is proposed by Thomas Neumann in the paper[2] can address these problems. I have finished a PoC[3] to verify its feasibility and validity. Looking forward to your feedback. [1]: https://cwiki.apache.org/confluence/display/FLINK/FLIP-315+Support+Operator+Fusion+Codegen+for+Flink+SQL [2]: http://www.vldb.org/pvldb/vol4/p539-neumann.pdf [3]: https://github.com/lsyldliu/flink/tree/OFCG Best, Ron
