Hi,
they are actually using different interfaces and dependencies. Checkpointing
uses Flink FileSystem and the shaded Hadoop Filesystem is a special
implementation of this based on the Hadoop S3 FileSystem that has all
dependencies bundled in. The BucketingSink uses HDFS/Hadoop FileSystem,
the
Hi Andrey,
Yes, we followed the guide. Our checkpoints/savepoints are already being
saved on S3/Ceph, using the ShadedHadoop/S3AFileSystem (because it's the
one we managed to completely override the AWS address to point to our Ceph
cluster).
I suppose I can add the package with the AmazonClientEx
Hi Julio,
What's the Flink version for this setup?
--
Thanks,
Amit
On Wed, Oct 3, 2018 at 4:22 PM Andrey Zagrebin
wrote:
> Hi Julio,
>
> Looks like some problem with dependencies.
> Have you followed the recommended s3 configuration guide [1]?
> Is it correct that your job already created chec
Hi Julio,
Looks like some problem with dependencies.
Have you followed the recommended s3 configuration guide [1]?
Is it correct that your job already created checkpoints/savepoints on s3 before?
I think if you manually create file system using FileSystem.get(path), it
should be configured the s
Hey guys,
I've setup a BucketingSink as a dead letter queue into our Ceph
cluster using S3, but when I start the job, I get this error:
java.lang.NoClassDefFoundError: com/amazonaws/AmazonClientException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.
