, we have achieved reliable catalog management
on object storage such as S3. If possible, after refining our prototype, we
would like to contribute it to Iceberg.
Tks.
Lisoda.
在 2024-11-14 02:24:50，"Marc Cenac" 写道：
Thanks for the proposal Ashvin!
I see value in addi
On Tue, Jul 23, 2024 at 8:46 AM Ryan Blue wrote:
Thanks for the context, lisoda. I agree that it's good to understand the issues
you're facing with the HadoopCatalog. One follow up question that I have is
what the underlying storage is. Are you using HDFS for those 30,000 custome
ers facing similar issues, can we at
least retain a module similar to iceberg-hadoop to extend hadoop_catalog? If it
is removed, we won't be able to continue providing services to customers. So,
if possible, please consider this option.
Thank you all.
Kind regards,
lisoda
At 2024
ity wishes to stop supporting FileSystemCatalog,
I will respect the community's choice.
I'm glad to hear from you.
Regards
lisoda
在 2024-07-16 23:18:42，"Steven Wu" 写道：
Lisoda, HadoopCatalog has many issues for production usage like Dan said. It
has never been
g server implementation.
>
>Regards
>JB
>
>On Mon, Jul 15, 2024 at 8:13 AM lisoda wrote:
>>
>> Sir. Even if the entire hadoopCatalog can be used without lockManager,
>> should we delete it?
>>
>>
>>
>>
>>
>>
>&
party lock
>mechanism: for instance, JDBC Catalog uses the RDBMS table/row
>locking, REST Catalog uses implementation lock.
>I would rather remove HadoopCatalog and the lock manager in favor of
>the REST catalog and implementation lock mechanism.
>
>Just my $0.01 :)
>
>Regards
ystem catalog to the test codebase like the
in-memory catalog.
-Dan
On Sun, Jul 14, 2024 at 4:12 AM lisoda wrote:
At present, the file system based catalogues have the following problems (this
is what I can think of at the moment, perhaps not comprehensive).
1. does not support renaming
ative to a real catalog
implementation.
I don't think we should be adding functionality here and we should probably
deprecate or relocate the file system catalog to the test codebase like the
in-memory catalog.
-Dan
On Sun, Jul 14, 2024 at 4:12 AM lisoda wrote:
At present, the fi
I don't think we should be adding functionality here and we should probably
deprecate or relocate the file system catalog to the test codebase like the
in-memory catalog.
-Dan
On Sun, Jul 14, 2024 at 4:12 AM lisoda wrote:
At present, the file system based catalogues have the fol
ck implementations.
Ryan
On Thu, Jul 11, 2024 at 10:42 PM lisoda wrote:
Currently, the only lockManager implementation in iceberg-core is
InMemoryLockManager. This PR extends two LockManager implementations, one based
on the Redis, and another based on the Rest-API.
In general, most users use r
cies to core, like
Redis for lock implementations.
Ryan
On Thu, Jul 11, 2024 at 10:42 PM lisoda wrote:
Currently, the only lockManager implementation in iceberg-core is
InMemoryLockManager. This PR extends two LockManager implementations, one based
on the Redis, and another based on the Res
s for lock implementations.
Ryan
On Thu, Jul 11, 2024 at 10:42 PM lisoda wrote:
Currently, the only lockManager implementation in iceberg-core is
InMemoryLockManager. This PR extends two LockManager implementations, one based
on the Redis, and another based on the Rest-API.
In general, most us
the use of HadoopTableOperations for several
years now. Maybe updating it to use locks and moving it to a separate module is
a good compromise, but my strong preference is for removing it.
On Thu, Jul 11, 2024 at 11:08 PM lisoda wrote:
Hi,Sir.
I've finished extending the usual distribute
at if you could give me your opinion on this.
Also, please let me know if there is anything I can do to support the creation
of views.
Tks.
Regards
lisoda
At 2024-07-05 16:09:54, "Jean-Baptiste Onofré" wrote:
>Hi,
>
>Actually the JDBC catalog relies on the RDBMS backe
Currently, the only lockManager implementation in iceberg-core is
InMemoryLockManager. This PR extends two LockManager implementations, one based
on the Redis, and another based on the Rest-API.
In general, most users use redisLockManager is sufficient to cope with most of
the scenarios, for red
Sir, I have added some comments in PR and I hope it will help you. If you have
some other suggestions, please let me know, thanks.
在 2024-07-04 23:49:34，"lisoda" 写道：
yea.If I'm not mistaken, the jdbc catalog has the same problem with concurrent
commits.It doesn'
t FileIO, we can always extend it, but as it's used in different
>Iceberg layers (like ResolvedFileIO for instance), we have to be
>careful adding new operations here, especially if it's specific for
>HadoopCatalog table/view operations. I will take a look.
>
>Thanks !
>Rega
gt;
>For the view support, I can take a look (as I worked on the JDBC
>catalog view support).
>
>Anyway, I'm gonna take a look at your PR. Thanks again for your contribution !
>
>Regards
>JB
>
>On Thu, Jul 4, 2024 at 4:05 PM lisoda wrote:
>>
>> Hello.
be production ready (I'm thinking on
>scalability, or view support). What's your thoughts?
>By the way, I'm happy to take a look at adding view support if it helps.
>
>Regards
>JB
>
>On Thu, Jul 4, 2024 at 8:27 AM lisoda wrote:
>>
>> Hi Team.
>>
adding view support if it helps.
>
>Regards
>JB
>
>On Thu, Jul 4, 2024 at 8:27 AM lisoda wrote:
>>
>> Hi Team.
>> I've refactored the logic of the commit method in HadoopTableOptions.With
>> this refactoring, I believe that hadoopCatalog is ready to b
Hi Team.
I've refactored the logic of the commit method in HadoopTableOptions.With this
refactoring, I believe that hadoopCatalog is ready to be used in a production
environment. Now HadoopTableOptions can implement atomic commits while being
compatible with the differences in behaviour between
21 matches
Mail list logo
