Re: [DISCUSS] Proposal to support disk spilling in HeapKeyedStateBackend

2019-05-30 Thread Yu Li
Thanks all for the feedbacks here. And thanks for the reminder about the plan of reworking abstractions of snapshot strategies @Gordon, will definitely watch it and make sure of the collaboration of the two works. Since this discussion thread has been open for some time and we all get a consensus,

Re: [DISCUSS] Proposal to support disk spilling in HeapKeyedStateBackend

2019-05-30 Thread Tzu-Li (Gordon) Tai
Hi Yu, +1 to move forward with efforts to add this feature. As mentioned in the document as well as some offline discussions, from my side the only comments I have are related to how we snapshot the off-heap key groups. I think a recent discussion I posted about savepoint format unification for k

Re: [DISCUSS] Proposal to support disk spilling in HeapKeyedStateBackend

2019-05-29 Thread Yuzhao Chen
+1, thanks for you nice work, Yu Li ! Best, Danny Chan 在 2019年5月24日 +0800 PM8:51,Yu Li ,写道: > Hi All, > > As mentioned in our speak[1] given in FlinkForwardChina2018, we have > improved HeapKeyedStateBackend to support disk spilling and put it in > production here in Alibaba for last year's Single

Re: [DISCUSS] Proposal to support disk spilling in HeapKeyedStateBackend

2019-05-29 Thread Stefan Richter
Hi Yu, Sorry for the late reaction. As already discussed internally, I think this is a very good proposal and design that can help to improve a major limitation of the current state backend. I think that most discussion is happening in the design doc and I left my comments there. Looking forwar