Alex, We are working on the same thing, for the same exact reason. We are trying to avoid the complexities of running HDFS just for the file storage. We are also okay with the S3 limitations it introduces. We'll try and update the group if we find solutions for parallelizing the files consumption.
Scott, What problems did you encounter trying to save flink state and savepoint data? Flink checkpoint & savepoint data. Like you, I tried using S3 to persist > Flink state, but encountered AWS SDK issues and felt like I was going down > an ill-advised path -Steve -- Steve Morin | Hacker, Entrepreneur, Startup Advisor twitter.com/SteveMorin | stevemorin.com *Live the dream start a startup. Make the world ... a better place.*
