Hi everyone, In the world of big data, Flink session cluster is mostly used for ETL, OLTP, ML and other scenarios, but rarely used for OLAP. In my company, due to business needs, we rely on the Flink session cluster for querying and analysis in OLAP scenarios. But after conducting some experiments, we found two phenomenons: 1. The query or analysis latency is very high. 2. When the QPS of the query increases, the latency of the query becomes higher.
So to perform well in OLAP scenarios, it is necessary to optimize the latency of individual queries and the maximum query QPS that Flink session cluster can withstand. This FLIP only optimizes the latency of individual queries, and optimizing query QPS of Flink session cluster will be discussed in the next FLIP. Many users have been asking for some optimizations for it. Therefore, I'd like to revive the discussion of FLIP-534 to optimize this problem. Zhu Zhu, Mao Li and I had some offline discussions to refine the design[1]. Looking forward to your feedback! Thanks, Weijun Wang [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=371296797
