>
> *From: *Jing Ge
> *Date: *Tuesday, October 10, 2023 at 16:04
> *To: *"dev@flink.apache.org"
> *Cc: *"Zhao, Kevin" , "Josephraj, Prabhu" <
> jopra...@amazon.com>, emr-flink-team
> *Subject: *RE: [EXTERNAL] Support AWS SDK V2 for
+1 for the s3 file consolidation. We already have many issues with internal
communication and talking to customers. Different file schemas are not very
user friendly, btw.
Best regards,
Jing
On Mon, Oct 9, 2023 at 6:49 PM Matthias Pohl
wrote:
> I would agree with David's proposal as well.
>
>
at 15:35
To: "dev@flink.apache.org"
Cc: "Zhao, Kevin" , "Josephraj, Prabhu"
, emr-flink-team
Subject: RE: [EXTERNAL] Support AWS SDK V2 for Flink's S3 FileSystem
CAUTION: This email originated from outside of the organization. Do not click
links or open atta
Just to add a bit more context to the performance test question: What I had
in mind was the exists call on a (non-existing) directories in a bucket
with a lot of objects. A comment from one of the SDK contributors about
that call was that it could be an expensive call in an object store if
implemen
I would agree with David's proposal as well.
Would it make sense to come up with some performance comparisons for the
different S3 implementations in the end? ...just to ensure that we're
improving things or (at least) don't make things worse. Or is there
something like that already somewhere?
A
+1 for David's suggestion. We should get away from the current
approach with two abstractions and get to one rock solid one.
On Mon, Oct 2, 2023 at 11:13 PM David Morávek wrote:
>
> Hi Maomao,
>
> I wonder whether it would make sense to take a stab at consolidating the S3
> filesystems instead an
Hi Maomao,
I wonder whether it would make sense to take a stab at consolidating the S3
filesystems instead and introduce a native one. The whole Hadoop wrapper
around the S3 client exists for legacy reasons, and it adds complexity and
probably an unnecessary performance penalty.
If you take a loo
Hi Flink Dev,
I’m Maomao, a developer from AWS EMR.
Recently, our team is working on adding AWS SDK V2 support for Flink’s S3
Filesystem. During development, we found out that our work was blocked by
Presto. This is because that Presto still uses AWS SDK V1 and won’t add support
for AWS SDK V2
Maomao Min created FLINK-33157:
--
Summary: Support AWS SDK V2 for Flink's S3 FileSystem
Key: FLINK-33157
URL: https://issues.apache.org/jira/browse/FLINK-33157
Project: Flink
Issue
