Thanks for your start this discuss
Here I am a bit confused about the memory layer definition. This refers to local memory. Should we define a data acceleration layer like Alluxio [1] in remote storage? Let me cite a scenario: If I use Fluid [2] to mount an AlluxioRuntime [3] on K8S, it looks like a local disk (but it is actually a remote memory storage), Have we specified this behavior or optimized it for this scenario? [1] What is alluxio : https://docs.alluxio.io/os/user/stable/en/Overview.html [2] Fluid: https://fluid-cloudnative.github.io/ [3] Fluid Alluxio Runtime: https://fluid-cloudnative.github.io/samples/tieredstore_config.html#prerequisites liu ron <ron9....@gmail.com> 于2023年3月17日周五 10:39写道： > Hi, Yuxin, > > Thanks for creating this FLIP. Adding remote storage capability to Flink's > Hybrid Shuffle is a significant improvement that addresses the issue of > local disk storage limitations, this also can improve the stability of > Flink Batch Job. > I just have one question: can the Hybrid Shuffle replace the RSS in the > future? Due to the Hybrid Shuffle having remote storage ability, I think > maybe we don't need to maintain a standalone RSS, it will simplify our > operation work. > -- Best ConradJam
