Hi Vinay,
We found that our problems were not with RocksDb, but rather what we were
throwing at it. We were working with more complex data types (e.g. Collections)
and found that nearly 80% of the time was spent in serialization, so optimizing
that helped a lot. But if your state is more primit
Testing externalized checkpoints in a YARN-based cluster, configured with:
env.getCheckpointConfig.enableExternalizedCheckpoints(ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
I can confirm that checkpoint is retained between cancelled jobs, however it’s
deleted when the Job Manager sess
CDH5. I'll
verify and submit the original PR tomorrow morning EST.
On Jul 22, 2016 12:26 AM, Clifford Resnick wrote:
I have a fix and test for a recursive HDFSCopyToLocal. I also added similar
code to Yarn application staging. However, even though all files and resources
now copy correctly,
coming 1.1 RC (probably end of this week or beginning of next).
Feel free to ping me if you need any feed back or pointers.
– Ufuk
On Mon, Jul 18, 2016 at 9:52 PM, Clifford Resnick
wrote:
> In 1.1, AbstractYarnClusterDescriptor pushes contents of flink/lib (lo
ve File copy in Flink (in HDFSCopyToLocal) and
contribute this via a pull request.
– Ufuk
On Mon, Jul 18, 2016 at 7:22 PM, Clifford Resnick
wrote:
> Hi Ufuk,
>
> My mail was down, so I missed this response. Thanks for that.
>
> On
, Jul 16, 2016 at 6:26 PM, Clifford Resnick
wrote:
> Using Flink 1.1-SNAPSHOT, Hadoop-aws 2.6.4
>
>
>
> The error I’m getting is :
>
>
>
> 11:05:44,425 ERROR org.apache.flink.streaming.runtime.tasks.StreamTask
> - Caught
ion of Hadoop and Flink, and was there anything you did
other than configure core-site.xml?
-Cliff
From: Clifford Resnick
Reply-To: "user@flink.apache.org"
Date: Saturday, July 16, 2016 at 12:26 PM
To: "user@flink.apache.org"
Subject: Error using S3a State Backend: Window O
Using Flink 1.1-SNAPSHOT, Hadoop-aws 2.6.4
The error I’m getting is :
11:05:44,425 ERROR org.apache.flink.streaming.runtime.tasks.StreamTask
- Caught exception while materializing asynchronous checkpoints.
com.amazonaws.AmazonClientException: Unable to calculate MD5 hash:
/var/folders
