Re: States split over to external storage

2017-01-25 Thread Chen Qin
l policy should be consided. > > > > Sent from HUAWEI AnyOffice > > *发件人：*Stephan Ewen > > *收件人：*dev@flink.apache.org, > > *抄送：*iuxinc...@huawei.com,Aljoscha Krettek,时金魁, > > *时间：*2017-01-20 18:35:46 > > *主题:*Re: States split over to external storage > > &g

Re: States split over to external storage

2017-01-20 Thread Stephan Ewen
.apache.org, > *抄送：*iuxinc...@huawei.com,Aljoscha Krettek,时金魁, > *时间：*2017-01-20 18:35:46 > *主题:*Re: States split over to external storage > > Hi! > > This is an interesting suggestion. > Just to make sure I understand it correctly: Do you design this for cases > where the state per

答复: States split over to external storage

2017-01-20 Thread liuxinchun
What's more I make a little change in WindowOperator for ListState in https://issues.apache.org/jira/browse/FLINK-5572 发件人：Stephan Ewen 收件人：dev 抄送：iuxinc...@huawei.com,Aljoscha Krettek,时金魁 时间：2017-01-20 18:35:46 主题:Re: States split over to external storage Hi! This is an intere

Re: States split over to external storage

2017-01-20 Thread liuxinchun
oscha Krettek,时金魁, 时间：2017-01-20 18:35:46 主题:Re: States split over to external storage Hi! This is an interesting suggestion. Just to make sure I understand it correctly: Do you design this for cases where the state per machine is larger than that machines memory/disk? And in that case, you cannot solv

Re: States split over to external storage

2017-01-20 Thread Fabian Hueske
If I got it correctly, part of the motivation is to move rarely used / cold state to an external storage (please correct me if I'm wrong). 2017-01-20 11:35 GMT+01:00 Stephan Ewen : > Hi! > > This is an interesting suggestion. > Just to make sure I understand it correctly: Do you design this for c

Re: States split over to external storage

2017-01-20 Thread Stephan Ewen
Hi! This is an interesting suggestion. Just to make sure I understand it correctly: Do you design this for cases where the state per machine is larger than that machines memory/disk? And in that case, you cannot solve the problem by scaling out (having more machines)? Stephan On Tue, Jan 17, 20

Re: 答复: States split over to external storage

2017-01-17 Thread Chen Qin
of Chen Qin maybe a candidate strategy. We can keep > in touch and exchange our respective strategy. > -邮件原件- > 发件人: Chen Qin [mailto:c...@uber.com] > 发送时间: 2017年1月17日 13:30 > 收件人: dev@flink.apache.org > 抄送: iuxinc...@huawei.com; Aljoscha Krettek; shijinkui > 主题: Sta

答复: States split over to external storage

2017-01-16 Thread liuxinchun
ategy. -邮件原件- 发件人: Chen Qin [mailto:c...@uber.com] 发送时间: 2017年1月17日 13:30 收件人: dev@flink.apache.org 抄送: iuxinc...@huawei.com; Aljoscha Krettek; shijinkui 主题: States split over to external storage Hi there, I would like to discuss split over local states to external storage. The use case

States split over to external storage

2017-01-16 Thread Chen Qin
Hi there, I would like to discuss split over local states to external storage. The use case is NOT another external state backend like HDFS, rather just to expand beyond what local disk/ memory can hold when large key space exceeds what task managers could handle. Realizing FLINK-4266 might be har