lisoda, I don't think there is a good way to fix the HadoopCatalog
implementation. That's why we recommend not using it.
In the quickstart, the assumption is that you're using a Hive catalog. The
HadoopCatalog example shows how to add additional catalogs (in this case, a
local one for testing). I
Hello steven.
HadoopCatalog does have many problems, but because the community added it to
the QuickStart chapter in the first place, many users have actually stayed with
hadoopCatalog. There is a huge cost to switching catalogs. In addition, HIVE
even uses HadoopCatalog as an implementation o
Lisoda, HadoopCatalog has many issues for production usage like Dan said.
It has never been recommended in production. It was widely used in unit
test code, which is also slowly moving toward InMemoryCatalog. As the
community is aligned behind the REST catalog, it is preferable to limit the
work re
Again, it's my "vision": if the community wants to maintain and move
forward on HadoopCatalog, that's fine (not sure it would be a good
idea regarding the "limitations" of filesystem based catalog :)).
Let's see what the others are thinking.
Regards
JB
On Mon, Jul 15, 2024 at 8:29 AM lisoda wro
Okay. I see..
I‘m so sad. :(
But anyway, thanks for answering all my questions.
在 2024-07-15 14:25:16，"Jean-Baptiste Onofré" 写道：
>Hi
>
>HadoopCatalog is not a "recommended" catalog for production (at least
>up to now). So, we should consider either to move it in a separate
>repo (i
Hi
HadoopCatalog is not a "recommended" catalog for production (at least
up to now). So, we should consider either to move it in a separate
repo (if we have the guarantee that it's gonna be maintained, else it
doesn't make sense) or remove it to avoid confusion. My take here is
the same (for sever
Hi
My understanding is that lock manager is mostly used on the
HadoopCatalog. The other catalogs relays on a third party lock
mechanism: for instance, JDBC Catalog uses the RDBMS table/row
locking, REST Catalog uses implementation lock.
I would rather remove HadoopCatalog and the lock manager in f
Sir. Following this PR, we can modify hadoopTableOptions to support atomic
commits based on filesystem catalogues without distributed locks..core:Refactor
the code of HadoopTableOptions by BsoBird · Pull Request #10623 ·
apache/iceberg (github.com)
在 2024-07-15 00:08:26，"Daniel Weeks"
.
Replied Message
| From | Daniel Weeks |
| Date | 07/15/2024 00:08 |
| To | dev@iceberg.apache.org |
| Cc | |
| Subject | Re: Re: Core:support redis and http lock-manager |
Iisoda,
Unfortunately, I don't agree with your assessment. The problems with file
system based catalog implementa
|
| Date | 07/15/2024 00:08 |
| To | dev@iceberg.apache.org |
| Cc | |
| Subject | Re: Re: Core:support redis and http lock-manager |
Iisoda,
Unfortunately, I don't agree with your assessment. The problems with file
system based catalog implementations are inherent and steps taken to ad
Iisoda,
Unfortunately, I don't agree with your assessment. The problems with file
system based catalog implementations are inherent and steps taken to
address them are not adequate to have confidence in the implementation.
Commit atomicity is not solved as it relies on locking, which has a numbe
I have pretty similar concerns as Ryan. I don't think we should be adding
any new locking implementations to the project since at the moment the only
catalog which requires it for atomic operations is HadoopCatalog (and it
doesn't even completely address all the cases). Every locking
implementation
I think one of the main questions is whether we want to support locking
strategies moving forward. These were needed in early catalogs that didn't
have support for atomic operations (HadoopCatalog and GlueCatalog). Now,
Glue supports atomic commits and we have been discouraging the use of
HadoopCat
13 matches
Mail list logo
