Hi, dev

I'd like to start a discussion about FLIP-315: Support Operator Fusion
Codegen for Flink SQL[1]

As main memory grows, query performance is more and more determined by the
raw CPU costs of query processing itself, this is due to the query
processing techniques based on interpreted execution shows poor performance
on modern CPUs due to lack of locality and frequent instruction
mis-prediction. Therefore, the industry is also researching how to improve
engine performance by increasing operator execution efficiency. In
addition, during the process of optimizing Flink's performance for TPC-DS
queries, we found that a significant amount of CPU time was spent on
virtual function calls, framework collector calls, and invalid
calculations, which can be optimized to improve the overall engine
performance. After some investigation, we found Operator Fusion Codegen
which is proposed by Thomas Neumann in the paper[2] can address these
problems. I have finished a PoC[3] to verify its feasibility and validity.

Looking forward to your feedback.

[1]:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-315+Support+Operator+Fusion+Codegen+for+Flink+SQL
[2]: http://www.vldb.org/pvldb/vol4/p539-neumann.pdf
[3]: https://github.com/lsyldliu/flink/tree/OFCG

Best,
Ron

Reply via email to