Thanks for bringing this up, Shammon. In general, I'd be +1 to improve Flink's ability to serve as an OLAP engine.
I see a great value in Flink becoming a unified Large-scale Data Processing / Analysis tool. Through my interactions with users (Alibaba internal users, external users on Alibaba Cloud, developers from other companies via conferences / meetups), it's commonly complained how complicated and costly it is to build a data processing platform out of a bunch of different tools. That usually means higher learning / developing / operation and maintenance cost. In addition, I also see a trend that many projects and products are going along the same direction. I personally would not be concerned about losing focus. Unlike Apache Paimon (Flink Table Store) which tries to solve a completely different problem other than data processing, OLAP querying is just a special case of batch SQL data processing, where typically you have massive concurrent short-lived queries. As Shammon mentioned, Flink already has most of the essential building blocks: batch SQL processing, session mode, sql-gateway, etc. IMHO, the missing piece is mostly about improving the performance in the specific OLAP scenarios. That sounds like a reasonable extension to me. I'd consider improving the OLAP capability as nice-to-have improvements. That is to say, it must not come at the price of sacrificing the experiences in other streaming / batch scenarios, nor significantly complicate the system. I think one of the reasons that FLINK-25318 became stale was that some of the proposed solutions are too dedicated for the OLAP scenarios and require extra efforts to carefully re-design in order to not affect other scenarios. I'd be glad to see such efforts being revived. Regarding whether to include this in the roadmap, I'd suggest including it but maybe not as a top-level section. I think this is a good initiative. But to be considered as one of the major directions, I think we need some more clear and concrete, and commonly agreed plans for it. Alternatively, we may consider putting it under the "Towards Streaming Warehouses" or "Performance" sections, and using tones like "the community is trying to explore the possibility of ..." to indicate its exploratory stage. Best, Xintong On Tue, Aug 8, 2023 at 11:16 AM Yangze Guo <karma...@gmail.com> wrote: > Thanks for driving this, Shammon! And thanks for the valuable comments > from Matthias. > > As one of the contributors to FLINK-25318, I would also like to share > my thoughts. > > 1. Regarding using Flink as an OLAP engine: > > I understand the value of having a unified engine for streaming, > batch, and OLAP, as described. From the user's perspective, using the > same engine throughout different stages of their daily workflow can > definitely save a lot of learning costs. It also gives us the > opportunity to leverage fragment resources in a batch standalone > cluster, thereby improving resource utilization efficiency. > @Matthias I agree with the Unix philosophy you mentioned, but I would > consider streaming/batch ETL and OLAP queries as a complete story. > Just as the unified stream-batch engine of Flink frees users from > maintaining two sets of business code, expanding Flink's use as an > OLAP query engine can further reduce costs for users aiming to build > an end-to-end streaming warehouse. AFAIK, within the ByteDance, users > are already benefiting from this. > > > 2. Regarding adding OLAP support to Flink 2.0 roadmap: > > Given the increasing user expectations for Flink's performance and > functionality in OLAP scenarios, it seems reasonable to add OLAP > support to the roadmap. However, I agree that introducing too many > objectives can lead to scattered attention. Therefore, I'm partially > in favor (+0.5) of it. > > BTW, in addressing concerns about the growing complexity of Flink's > codebase, we can mitigate this issue through careful design choices > and thorough code reviews. Another principle we need to follow is all > of those improvements should not negatively impact stream processing. > AFAICS, these enhancements will also benefit streaming users. > > Second to @Matthias opinion, at least in a first step, we can consider > OLAP queries as short-lived batch processing and focus on driving > optimizations specifically for these tasks. > > 3. Regarding the performance tests: > > I'm big +1 for adding a specific performance tests for OLAP or > short-lived queries. In addition to add some micro benchmarks in [1], > we can also develop an end-to-end benchmark similar to Nexmark[2]. I'm > enthusiastic about participating in this project. > > [1] https://github.com/apache/flink-benchmarks > [2] https://github.com/nexmark/nexmark > > > Best, > Yangze Guo > > On Mon, Aug 7, 2023 at 8:20 PM Matthias Pohl > <matthias.p...@aiven.io.invalid> wrote: > > > > Thanks Shammon FY for starting this discussion. > > > > I'm not sure whether we have to expand the focus of Apache Flink in a way > > you're describing it with OLAP being a topic next to Batch and Stream > > Processing. You've rightfully pointed out that there are already > solutions > > to cover OLAP. In this sense, I like the Unix philosophy: One tool for > one > > job [1]. My concern is that extending the scope of Flink would guide us > > into a direction where we lose focus: I see the risk in Flink's codebase > > becoming even more complex with the proposed change of the roadmap. That > > was one reason why a project like Flink Table Store ended up becoming an > > independent project Apache Paimon. > > > > That said, I still see value in some of the ideas you've collected in > > FLINK-25318 [2]. Can't we just see OLAP queries as a specific form of > Batch > > Processing? The idea of the session cluster was to have the ability to > run > > multiple short-lived jobs, anyway. And there are definitely ways to > improve > > the execution of those queries as you pointed out. > > > > Therefore, I'm not rejecting the ideas you're suggesting in FLINK-25318 > > [2]. I'm just skeptical about expanding the scope of Apache Flink by > > mentioning OLAP explicitly in Flink's roadmap. > > > > Independent of the roadmap discussion: If the community decides to put > more > > focus on short-lived queries as you suggested, I would like to have that > > reflected in the performance tests (like it was suggested by Piotr in his > > FLINK-25318 comment [3] and that's already covered by FLINK-25356 [4]). > We > > should provide a performance test set before we proceed with implementing > > improvements to cover those scenarios. This would enable us to measure > the > > changes that were collected in FLINK-25318 [2]. > > > > I'm looking forward to other views on that topic. > > > > Best, > > Matthias > > > > [1] http://www.catb.org/~esr/writings/taoup/html/ch01s06.html > > [2] https://issues.apache.org/jira/browse/FLINK-25318 > > [3] > > > https://issues.apache.org/jira/browse/FLINK-25318?focusedCommentId=17460615&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17460615 > > [4] https://issues.apache.org/jira/browse/FLINK-25356 > > > > On Mon, Aug 7, 2023 at 11:53 AM Shammon FY <zjur...@gmail.com> wrote: > > > > > Hi everyone, > > > > > > According to the discussion between @Matthias and me in FLINK-32667 > [1], I > > > would like to initiate a discussion about Flink OLAP. Let me introduce > > > ourselves first, we are the Streaming Compute team in ByteDance, and > about > > > half a year ago @Songxintong and we had an offline discussion on Flink > OLAP > > > and created FLINK-25318 [2] to track the improvements to Flink OLAP. We > > > also have some simple online discussion with @Piotr on the issue. Here > I > > > would like to share my thoughts on Flink OLAP and our practice in the > past, > > > and initiate a broader discussion to get more input and thoughts about > > > Flink OLAP in this thread. I hope if possible that OLAP could be the > main > > > capability besides Streaming and Batch for Flink and the community > could > > > consider adding it to Flink 2.0 Roadmap, which can attract more > developers > > > and users to promote and improve Flink OLAP in the future. > > > > > > 1. Why we want to support OLAP in Flink > > > > > > Currently, there are many excellent open-source OLAP engines, so why > do we > > > want to support OLAP in Flink? There are three main types in the > current > > > data processing progress: streaming, batch and OLAP. As an excellent > > > streaming and batch processing engine, flink processes data and writes > > > results to storage such as lakehouse, key-value stores, databases and > ect. > > > After that, users need an OLAP engine to analyze their data. > > > > > > 1.1) As we all know, OLAP is a very important scenario and users need > an > > > OLAP engine indeed. > > > > > > 1.2) Flink OLAP can reduce team costs. We have encountered many small > or > > > medium-sized tech teams who are using Flink for streaming and batch > > > processing. However, they have to introduce an additional OLAP engine > for > > > their users to analyze data, which increases the technical costs and > > > operation costs for multiple engines and clusters. > > > > > > 1.3) Flink OLAP can reduce user usage costs. Many of our users are > > > familiar with Flink and use Flink streaming and batch SQL to process > data. > > > After that, they are more focused on business. They are not very > interested > > > in introducing an additional OLAP engine which requires a new > familiarity > > > with the new engine and increases usage costs. > > > > > > 2. Why Flink can be an OLAP engine. > > > > > > So, technically speaking, can Flink support OLAP well? My answer is: OF > > > COURSE! > > > > > > 2.1) From Architecture, Flink supports jdbc driver, Sql-Gateway and > > > session mode. Flink session cluster is a typical MPP architecture and > each > > > query does not need to require new resources. Users can easily submit > > > SELECT statements by the jdbc driver and fetch results on the second or > > > even sub-second level. > > > > > > 2.2) Powerful batch processing power. Flink OLAP can take many batch > > > operators and optimizations. At the same time, there are also large > queries > > > in OLAP and Flink can support them based on the ability of Flink batch > > > without the need to introduce an external batch engine like other OLAP > > > engines. > > > > > > 2.3) Flink supports standard SQL syntax such as QUERY/INSERT/UPDATE > which > > > meets the interaction requirements of OLAP users. > > > > > > 2.4) Powerful connector ecosystem. Flink has defined comprehensive > > > interfaces for input and output and implemented many embedding > connectors > > > such as database, lakehouse. Users can easily implement customized > > > connectors based on these interfaces too. > > > > > > 3. The main issues for Flink OLAP. > > > > > > We found there are two main types of issues that need to be addressed > for > > > Flink OLAP in our practice: Latency and QPS. Unlike streaming and > batch, > > > OLAP has the following two characteristics: > > > > > > a) OLAP jobs are usually short-lived and return results in seconds or > > > sub-seconds which will have higher requirements for data processing > latency. > > > > > > b) Users have QPS requirements for OLAP such as users often want to > > > perform hundreds of simple queries or tens of complex queries in a > Flink > > > OLAP cluster concurrently in our practice. > > > > > > We need to make some improvements in Flink to support OLAP better. > Based > > > on our previously created FLINK-25318 [2] and our practical > experience, I > > > classify the improvements as follows: > > > > > > a) Improve the interaction between Flink and external components such > as > > > zk and filesystem. For example, Flink stores relevant information in > zk to > > > support failover and this is not needed by OLAP but rather causes > delays > > > due to the network jitter. > > > > > > b) Improvements in Flink internal progress, such as job submission > > > progress, data fetch optimization, excessive number of interactive > events > > > among components(JM/TM/RM) etc. These will cause Flink to increase job > > > latency from a few milliseconds to tens of seconds when running several > > > small queries concurrently. > > > > > > c) Support OLAP specific features. Due to the OLAP characteristics, > Flink > > > may need to add some features only for OLAP. For example, we need to > > > isolate resources between jobs, and ensure that large queries do not > affect > > > small queries. > > > > > > 4. Out practice about Flink OLAP. > > > > > > In addition to using Flink to support massive streaming businesses in > > > ByteDance, we have also conducted many big improvements on Flink to > support > > > OLAP well. This can not only be seen as a POC, but we also provide > Flink > > > OLAP clusters in production for our users. > > > > > > After our improvements, we built a 500 core Flink OLAP cluster to test > > > Flink OLAP e2e latency and QPS. There will be nearly a thousand QPS for > > > simple queries of 128 subtasks with tens of milliseconds latency; for > > > complex jobs with two JOIN operators and 700 subtasks, there will be > about > > > sixty QPS with hundreds of milliseconds latency. > > > > > > Our Flink OLAP in production supports more than ten internal businesses > > > and there are more than 6000 cores in all clusters and 500 thousand > queries > > > will be performed each day. > > > > > > 5. My Proposal > > > > > > Above all, I personally think it will provide great value for users > and is > > > technically feasible if Flink supports OLAP and this is also what Flink > > > engine can achieve through some improvements although this is not a > simple > > > matter. I hope the community could consider supporting OLAP as one of > Flink > > > technical directions and add it to the roadmap. As I mentioned at the > > > beginning, this can attract more developers and users to participate > and > > > promote the improvement of Flink OLAP and build Flink as a unified > engine > > > for streaming, batch and OLAP. > > > > > > Looking forward to your feedback, thanks! > > > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-32667 > > > [2] https://issues.apache.org/jira/browse/FLINK-25318 > > > > > > Best, > > > Shammon FY > > > > > > >
