Re: Re: Re: Re: Refactor the code of HadoopTableOptions

2024-07-14 Thread Jean-Baptiste Onofré
Hi I agree with Ryan and it was my comment in a previous message: "About FileIO, we can always extend it, but as it's used in different Iceberg layers (like ResolvedFileIO for instance), we have to be careful adding new operations here, especially if it's specific for HadoopCatalog table/view ope

Re:Re: Re: Re: Re: Refactor the code of HadoopTableOptions

2024-07-14 Thread lisoda
Regarding HadoopTableOptions, if the filesystem supports rename operations that do not overwrite the target file, the entire HadoopTableOptions does not need to use lockManager. One of the reasons for keeping LockManager is simply because it was used in the code for the original implementation.

Re: Re: Re: Re: Refactor the code of HadoopTableOptions

2024-07-12 Thread Ryan Blue
FileIO purposely does not support a rename operation because we wanted to keep a minimal API that handled object stores correctly rather than using a FileSystem concept. While we may need some extensions outside of what the core provides for reading and writing tables, I think we still need to be c

Re: Re: Re: Re: Refactor the code of HadoopTableOptions

2024-07-11 Thread Jean-Baptiste Onofré
Thanks ! I will take a look asap (I would like to complete the 1.6.0 release soon). Regards JB On Fri, Jul 12, 2024 at 8:07 AM lisoda wrote: > > Hi,Sir. > I've finished extending the usual distributed locks.I think we'll no need to > extend distributed locks for a long time. > > PR:https://git

Re: Re: Re: Refactor the code of HadoopTableOptions

2024-07-05 Thread Jean-Baptiste Onofré
Hi, Actually the JDBC catalog relies on the RDBMS backend of the lock. That's one of the reasons why we are using a single RDBMS table for both tables and views. So, I don't think we would need a lock mechanism for JDBC, the RDBMS one is OK for now. About FileIO, we can always extend it, but as it