Re: rocksdb backend on s3 window operator checkpoint issue

2016-05-17 Thread Robert Metzger
I tried reproducing the issue using the org.apache.hadoop.fs.s3a.S3AFileSystem and it worked. I had some dependency issues with the S3AFileSystem so I didn't follow that path for now. If you've used the S3AFileSystem, I can try to get that one working as well. On Tue, May 17, 2016 at 11:59 AM, Rob

Re: rocksdb backend on s3 window operator checkpoint issue

2016-05-17 Thread Robert Metzger
Hi, from the code you've provided, everything seems to look okay. I'm currently trying to reproduce the issue. Which Flink version are you using? Which s3 implementation did you configure in the hadoop configuration? Regards, Robert On Mon, May 16, 2016 at 11:52 PM, Chen Qin wrote: > Hi ther

rocksdb backend on s3 window operator checkpoint issue

2016-05-16 Thread Chen Qin
Hi there, I have been testing checkpointing on rocksdb backed by s3. Checkpoints seems successful except snapshot states of timeWindow operator on keyedStream. Here is the env setting I used env.setStateBackend(new RocksDBStateBackend(new URI("s3://backupdir/"))) The checkpoint for always fail co