TL;DR: IMO a more complete solution is to cover both query and meta request
serving in a new component. We could use the proposal here as step one but
we should have a global plan. And before improving a seemingly not widely
used feature, we'd better weigh the gain and efforts.
Let me clarify the purpose of my previous questions, that before we start
detailed design and code development, it's better to get consensus on:
1. What's the value of the work?
- As noticed, the queryable state feature has been implemented for some
while but not widely used in production (AFAIK), why? If it did been used
in critical scenarios, what those scenarios are?
- I think it's a good time discussing about this (since raised in this
thread by others) and confirm the value of efforts improving this feature.
2. Would there be duplicated work?
- This is the main reason I asked about the relationship between ledger
and queryable-state.
And some answers to the inline comments:
bq. About the relationship between ledger and Queryable State, I also think
it is out of this thread
True, that's why I suggested to open another thread. But as mentioned
above, the question is relative if we think about the whole.
bq. Yes, the QueryableState's isolation level is *Read Uncommitted*...
However, I think it would not affect we discuss how to improve the
queryable state's architecture, right?
Correct, but my real question here is what kind of application could bear
the changing query result.
bq. The intermediate data is also valuable, for example, we just need a
partitioned data stream's real-time measure value.
In this case there must be some complicated operation in the pipeline which
causes long latency at sink? Could you talk more about the real-world case?
Thanks.
bq. Your worry is reasonable.
Then I suggest to think it as a whole. We could split the implementation
into steps, but better to have a global plan, to make it really applicable
in production (under heavy load).
Best Regards,
Yu
On Sun, 28 Apr 2019 at 14:48, vino yang <yanghua1...@gmail.com> wrote:
> Hi yu,
>
> Thanks for your reply. I have some inline comment.
>
> Yu Li <l...@apache.org> 于2019年4月28日周日 下午12:24写道：
>
> > Glad to see discussions around QueryableState in mailing list, and it
> seems
> > we have included a bigger scope in the discussion, that what's the data
> > model in Flink and how to (or is it possible to) use Flink as a
> database. I
> > suggest to open another thread for this bigger topic and personally I
> think
> > the first question should be answered is what's the relationship between
> > Flink ledger and QueryableState.
> >
>
> *About the scope, yes, it seems it's big. Actually, I think the questions
> you provided make it bigger than I have done.*
> *Here I think we don't need to answer the two questions(we can discuss in
> another thread, or answer it later).*
>
> *My original thought is that we found the queryable state is hard to use
> and it may cause few users to use this function. We may think the reason
> and the result affect each other. And IMO, currently, the queryable state's
> architecture caused this problem. So I opened a thread to see how to
> improve them. *
>
> *We mentioned these keywords e.g. "state、database" is to emphasize the
> queryable state is very important. The data model and use Flink as a
> database is not this thread's main topic (as Elias's reply said, many
> issues cause the road to this goal is so long). This thread I assume we do
> not change the state's core design and the goal is to bring a better query
> solution.*
>
> *About the relationship between ledger and Queryable State, I also think it
> is out of this thread.*
>
>
> >
> > Back to the user scenario itself, I'd like to post two open questions
> about
> > QueryableState for ad-hoc query:
> > 1. Currently the isolation level of QueryableState is *Read Uncommitted*
> > since failover might happen and cause data rollback. Although the
> > "uncommitted" data will be replayed again and get final consistency,
> > application will see unstable query result. Probably some kind of
> > applications could bare such drawback but what exactly?
> >
>
> *Yes, the QueryableState's isolation level is *Read Uncommitted*. I think
> if we need a higher isolation level, may need other mechanisms to guarantee
> this. I am sorry, I can not give the solution.*
> *However, I think it would not affect we discuss how to improve the
> queryable state's architecture, right?*
>
>
> >
> > 2. Currently in Flink sink is more commonly regarded as the "result
> > partition" and state of operators in the pipeline more like "intermediate
> > data". Used for debugging purpose is easy to understand but not for
> ad-hoc
> > query. Or in another word, what makes user prefer querying the state data
> > instead of sink? Or why we need to query the intermediate data instead of
> > the result?
> >
> >
> *About the opinion that state of operators in the pipeline more like
> "intermediate data". Yes, you are right. It's intermediate data, and we
> need it in some scene.*
> *The valuable is that it represents "real-time". When querying a state, we
> need its current value, we can not wait for sink. The intermediate data is
> also valuable, for example, we just need a partitioned data stream's
> real-time measure value.*
>
>
> > Further back to the original topic proposed in this thread about
> > introducing a QueryableStateProxy, I could see some careful consideration
> > on query load on the proxy. However, under heavy load the pressure is not
> > only on query serving but also on meta requesting, which is handled by JM
> > for now. So to release JM pressure, we should also extract the meta
> serving
> > task out, and my suggestion is to introduce a new component like
> > *StateMetaServer* and take over both query and meta serving
> > responsibilities.
> >
>
> *I think the opinion of metadata's pressure and *StateMetaServer* are good.
> We need to care about them when we design.*
> *I mentioned the meta info(registry) in the two option's simple
> architecture picture. Although, I just emphasized the query proxy server,
> because it is the main component.*
>
> *Your worry is reasonable. The proxy server's architecture is good for
> processing this, such as the mechanisms of request flow control, pressure
> transfer to a single entry point(for opt2 and opt3, we can serve meta-query
> in a single process).*
>
> *Anyway, it just opened a discussion to listen to the community's opinion.*
>
>
> >
> > Best Regards,
> > Yu
> >
> >
> > On Sat, 27 Apr 2019 at 11:58, vino yang <yanghua1...@gmail.com> wrote:
> >
> > > Hi Elias,
> > >
> > > I agree with your opinion that "*Flink jobs don't sufficiently meet
> these
> > > requirements to work as a replacement for a data store.*". Actually, I
> > > think it's obviously not Flink's goal. If we think that the database
> > > contains the main two parts(inexactitude): data query and data store.
> > What
> > > I and Paul mean is the former.
> > >
> > > Yes, you have mentioned it's major value: ad hoc and debugging(IMO,
> > > especially for the former). To give a real-time calculation result is
> > very
> > > import for some scene(such as real-time measure for real-time OLAP) in
> a
> > > long-term (no-window or large window).
> > >
> > > So, my opinion: Queryable state is not dedicated to replacing data
> > stores.
> > > However, if we could query state more conveniently, it makes the
> > streaming
> > > works more like DB in query aspect.
> > >
> > > Best,
> > > Vino.
> > >
> > > Elias Levy <fearsome.lucid...@gmail.com> 于2019年4月27日周六 上午1:30写道：
> > >
> > > > On Fri, Apr 26, 2019 at 1:41 AM vino yang <yanghua1...@gmail.com>
> > wrote:
> > > >
> > > > > You are right, currently, the queryable state has few users. And I
> > > > totally
> > > > > agree with you, it makes the streaming works more like a DB.
> > > > >
> > > >
> > > > Alas, I don't think queryable state will really be used much in
> > > production
> > > > other than for ad hoc queries or debugging. Real data stores at
> scale
> > > are
> > > > resilient, replicated, and with very low downtime. In my opinion,
> > Flink
> > > > jobs don't sufficiently meet these requirements to work as a
> > replacement
> > > > for a data store. Jobs too frequently fail and restart because of
> > > > checkpoint failures, particularly ones with large state. And when a
> > job
> > > > does restart, all too often local restore can't be used (e.g. if you
> > > loose
> > > > a node). And since there is no fine grained job recovery and there
> is
> > no
> > > > hot replicas of the data, all the state will need to be restored from
> > the
> > > > DFS, which for something with large state can take a while. It's a
> > nice
> > > > idea, just not realistic in practice.
> > > >
> > >
> >
>
