Hi Peter,

I am coming from the Flink side, but at Cloudera we also use
Iceberg as well.

Utilizing the Flink delegation token fw via the Iceberg Java API
would be great. I think that simplifying the configuration for
Flink related cases also has value on its own, and could help to
eliminate some confusion regarding when/where set properties.

Regarding the naming, maybe it would worth to be more explicit and
call the class `FlinkFSFileIO`? Just to emphasize that "Flink" in
this context is referred as the FS abstraction layer, not as a
processing engine. WDYT?

Looking forward to this change!

Best,
Ferenc

On 2024/04/19 17:08:23 Péter Váry wrote:
> Hi Iceberg Team,
>
> Flink has its own FileSystem implementation. See:
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/overview/
> .
> This FileSystem already has several implementations:
>
>    - Hadoop
>    - Azure
>    - S3
>    - Google Cloud Storage
>    - ...
>
> As a general rule in Flink, one should use this FileSystem to consume and
> persistently store data.
> If these FileSystems are configured, then Flink makes sure that the
> configurations are consistent and available for the JM/TM.
> Also as an added benefit, delegation tokens are handled and distributed for
> these FileSystems automatically. See:
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/security/security-delegation-token/
>
> In house, some of our new users are struggling with parametrizing
> HadooFileIO, and S3FileIO for Iceberg, trying to wrap their head around
> that they have to provide different configurations for the checkpointing
> and for the Iceberg table storage (even if they are stored in the same
> bucket, or on the same HDFS cluster)
>
> I have created a PR, which provides a FileIO implementation which uses
> FlinkFileSystem. Very imaginatively I have named it FlinkFileIO. See:
> https://github.com/apache/iceberg/pull/10151
>
> This would allow the users to configure the FileSystem only once, and use
> this FileSystem to access Iceberg tables. Also, if for whatever reason the
> global nature of flink file system config is limiting, the users still
> could revert back using the other FileIO implementations.
>
> What do you think? Would this be a useful addition to the Iceberg-Flink
> integration?
>
> Thanks,
> Peter
>

Reply via email to