Thanks Jungtaek!
It makes sense, we are currently changing to an HDFS-Compatible FS, I was
wondering how this change would impact the checkpoint, but after what you
said it is more clear now.
On Thu, 3 Dec 2020 at 00:23, Jungtaek Lim
wrote:
> In theory it would work, but works very inefficien
In theory it would work, but works very inefficiently on checkpointing. If
I understand correctly, it will write the content to the temp file on s3,
and rename the file which actually gets the temp file from s3 and write the
content of temp file to the final path on s3. Compared to checkpoint with
Hello!
@Gabor Somogyi I wonder that now that s3
is *strongly
consistent* , would work fine.
Regards!
https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-read-after-write-consistency/
On Thu, 17 Sep 2020 at 11:55, German Schiavon
wrote:
> Hi Gabor,
>
> Makes sense, thanks a lot!
>
> On
Hi Gabor,
Makes sense, thanks a lot!
On Thu, 17 Sep 2020 at 11:51, Gabor Somogyi
wrote:
> Hi,
>
> Structured Streaming is simply not working when checkpoint location is on
> S3 due to it's read-after-write consistency.
> Please choose an HDFS compliant filesystem and it will work like a charm.
Hi,
Structured Streaming is simply not working when checkpoint location is on
S3 due to it's read-after-write consistency.
Please choose an HDFS compliant filesystem and it will work like a charm.
BR,
G
On Wed, Sep 16, 2020 at 4:12 PM German Schiavon
wrote:
> Hi!
>
> I have an Structured Stre
