Hi everyone, I would like to start a discussion on FLIP-301: Hybrid Shuffle supports Remote Storage[1].
In the cloud-native environment, it is difficult to determine the appropriate disk space for Batch shuffle, which will affect job stability. This FLIP is to support Remote Storage for Hybrid Shuffle to improve the Batch job stability in the cloud-native environment. The goals of this FLIP are as follows. 1. By default, use the local memory and disk to ensure high shuffle performance if the local storage space is sufficient. 2. When the local storage space is insufficient, use remote storage as a supplement to avoid large-scale Batch job failure. Looking forward to hearing from you. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-301%3A+Hybrid+Shuffle+supports+Remote+Storage Best, Yuxin
