Hi yu, Thanks for your reply. I have some inline comment.
Yu Li <l...@apache.org> 于2019年4月28日周日 下午12:24写道： > Glad to see discussions around QueryableState in mailing list, and it seems > we have included a bigger scope in the discussion, that what's the data > model in Flink and how to (or is it possible to) use Flink as a database. I > suggest to open another thread for this bigger topic and personally I think > the first question should be answered is what's the relationship between > Flink ledger and QueryableState. > *About the scope, yes, it seems it's big. Actually, I think the questions you provided make it bigger than I have done.* *Here I think we don't need to answer the two questions(we can discuss in another thread, or answer it later).* *My original thought is that we found the queryable state is hard to use and it may cause few users to use this function. We may think the reason and the result affect each other. And IMO, currently, the queryable state's architecture caused this problem. So I opened a thread to see how to improve them. * *We mentioned these keywords e.g. "state、database" is to emphasize the queryable state is very important. The data model and use Flink as a database is not this thread's main topic (as Elias's reply said, many issues cause the road to this goal is so long). This thread I assume we do not change the state's core design and the goal is to bring a better query solution.* *About the relationship between ledger and Queryable State, I also think it is out of this thread.* > > Back to the user scenario itself, I'd like to post two open questions about > QueryableState for ad-hoc query: > 1. Currently the isolation level of QueryableState is *Read Uncommitted* > since failover might happen and cause data rollback. Although the > "uncommitted" data will be replayed again and get final consistency, > application will see unstable query result. Probably some kind of > applications could bare such drawback but what exactly? > *Yes, the QueryableState's isolation level is *Read Uncommitted*. I think if we need a higher isolation level, may need other mechanisms to guarantee this. I am sorry, I can not give the solution.* *However, I think it would not affect we discuss how to improve the queryable state's architecture, right?* > > 2. Currently in Flink sink is more commonly regarded as the "result > partition" and state of operators in the pipeline more like "intermediate > data". Used for debugging purpose is easy to understand but not for ad-hoc > query. Or in another word, what makes user prefer querying the state data > instead of sink? Or why we need to query the intermediate data instead of > the result? > > *About the opinion that state of operators in the pipeline more like "intermediate data". Yes, you are right. It's intermediate data, and we need it in some scene.* *The valuable is that it represents "real-time". When querying a state, we need its current value, we can not wait for sink. The intermediate data is also valuable, for example, we just need a partitioned data stream's real-time measure value.* > Further back to the original topic proposed in this thread about > introducing a QueryableStateProxy, I could see some careful consideration > on query load on the proxy. However, under heavy load the pressure is not > only on query serving but also on meta requesting, which is handled by JM > for now. So to release JM pressure, we should also extract the meta serving > task out, and my suggestion is to introduce a new component like > *StateMetaServer* and take over both query and meta serving > responsibilities. > *I think the opinion of metadata's pressure and *StateMetaServer* are good. We need to care about them when we design.* *I mentioned the meta info(registry) in the two option's simple architecture picture. Although, I just emphasized the query proxy server, because it is the main component.* *Your worry is reasonable. The proxy server's architecture is good for processing this, such as the mechanisms of request flow control, pressure transfer to a single entry point(for opt2 and opt3, we can serve meta-query in a single process).* *Anyway, it just opened a discussion to listen to the community's opinion.* > > Best Regards, > Yu > > > On Sat, 27 Apr 2019 at 11:58, vino yang <yanghua1...@gmail.com> wrote: > > > Hi Elias, > > > > I agree with your opinion that "*Flink jobs don't sufficiently meet these > > requirements to work as a replacement for a data store.*". Actually, I > > think it's obviously not Flink's goal. If we think that the database > > contains the main two parts(inexactitude): data query and data store. > What > > I and Paul mean is the former. > > > > Yes, you have mentioned it's major value: ad hoc and debugging(IMO, > > especially for the former). To give a real-time calculation result is > very > > import for some scene(such as real-time measure for real-time OLAP) in a > > long-term (no-window or large window). > > > > So, my opinion: Queryable state is not dedicated to replacing data > stores. > > However, if we could query state more conveniently, it makes the > streaming > > works more like DB in query aspect. > > > > Best, > > Vino. > > > > Elias Levy <fearsome.lucid...@gmail.com> 于2019年4月27日周六 上午1:30写道： > > > > > On Fri, Apr 26, 2019 at 1:41 AM vino yang <yanghua1...@gmail.com> > wrote: > > > > > > > You are right, currently, the queryable state has few users. And I > > > totally > > > > agree with you, it makes the streaming works more like a DB. > > > > > > > > > > Alas, I don't think queryable state will really be used much in > > production > > > other than for ad hoc queries or debugging. Real data stores at scale > > are > > > resilient, replicated, and with very low downtime. In my opinion, > Flink > > > jobs don't sufficiently meet these requirements to work as a > replacement > > > for a data store. Jobs too frequently fail and restart because of > > > checkpoint failures, particularly ones with large state. And when a > job > > > does restart, all too often local restore can't be used (e.g. if you > > loose > > > a node). And since there is no fine grained job recovery and there is > no > > > hot replicas of the data, all the state will need to be restored from > the > > > DFS, which for something with large state can take a while. It's a > nice > > > idea, just not realistic in practice. > > > > > >
