Hi Jack,
I agree and it's my fault :)
You are right, we need a dev ml thread on this topic (I sent a message
a couple of weeks ago to start the discussion about Iceberg 2.0, the
idea was to start a thread on each topic). Definitely it has to be
discussed on the mailing list to include anyone in t
Could we have a separate devlist thread dedicated for this discussion? It
is a bit awkward to continue this critical Iceberg 2.0 catalog
consolidation topic under this Gravitino thread, although I understand it
is related. At least I have overlooked all these discussions until now, and
I feel there
To clarify, what I meant was that Iceberg has, so far, avoided providing
runtime services that are ready to be deployed and used. I still think that
is a good choice, and I agree with the reasons that Renjie listed.
I disagree that this is inconsistent. We don't supply any of the other
services th
Hey Brian
Thanks for the summary ! Good one !
I would just add the "REST ref impl" discussion.
Regarding the anti patterns, I agree with the lists, imho, some are
more "opinionated implementation", so definitely not in the API scope.
+1
Thanks again !
Regards
JB
On Fri, Mar 1, 2024 at 1:15 P
My attempt to consolidate a list of goals, anti patterns , and impl details
mentioned since this discussion was brought up at the last Iceberg sync.
Tried to roughly capture who mentioned these things so we can follow up if
needed. Hopefully this can serve as a basis for the design discussion.
Goa
Hi Renjie,
maybe I wasn't clear, sorry about that: the target is really both ref
impl (where we can test different Iceberg parts like we do with the
InMemoryCatalog, JdbcCatalog, etc) and ready to go service for users
(simple but to start with).
But we can't prevent the community from working on
Hi:
I think one thing missing in the discussion is that, if the iceberg
community wants to maintain a rest catalog service, what's the target use
case? Different target use cases may lead to different directions.
If it's mainly designed for first time users to play or experience with
rest catalog
Hi Fokko
If service means the actual runtime service, I partially agree.
I would love to see REST Catalog API the "central cornerstone" used in
iceberg-java, pyiceberg, etc. So I think we should provide the
resources for an user to bootstrap a REST Catalog ref impl.
A lot of Apache projects provi
Hey everyone,
Thanks for raising this. I think a test-jar would be a great first step.
We already maintain "service" considering JDBC, Hive, etc catalogs. REST
Catalog ref impl in Iceberg would be the sam.
What I think Ryan means by a service is having to maintain Postgres (JDBC
backend), Hive
Hi Ryan
If we plan to reduce the number of catalogs (and I think it makes
sense and I'm with you on that), we will need a impl/service in
Iceberg for the REST Catalog API, else the users won't be able to use
Iceberg "out of the box".
We already maintain "service" considering JDBC, Hive, etc catalo
Ryan, thank you for the clarifications, they're truly appreciated.
I'd like to expand on the last point I raised.
> From the perspective of open-source users, the absence of an open-source
> implementation for the REST catalog within Iceberg may be inconvenient or
> frustrating.
REST catalog wa
There is a reference implementation in the project, in the CatalogHandlers
class. That implements REST requests using a catalog and returns REST
responses. I believe this is what Gravatno relies on and I mentioned it
above in the discussion about whether we should have a catalog service.
Catalog t
By the way, I think it's a good time to think about REST Catalog API v2.
Actually, I would name this the Catalog RFC containing:
- the RFC description itself (documentation)
- the improved OpenAPI 3.0 spec
- possible OpenAPI extensions (allowing extra features, vendor
specific, etc) https://swagge
Hi Ajantha,
Thanks for sharing your thoughts.
It makes sense for Gravitino to be a TLP (after the incubation period)
because Gravitino is "more" than an Iceberg catalog. It implements the
Iceberg REST Catalog API, but it's also a metadata catalog/repo with
additional features.
That said, I agree
I apologize for the delay in responding.
I'm pleased to see the development of an open-source REST catalog
implementation, and the potential transition of Gravitino to an ASF project
is certainly promising.
But REST catalog server implementation will be a small part of Gravitino
ASF project. Which
+1 for using test-jar!
-Jack
On Fri, Jan 26, 2024 at 10:48 AM Ryan Blue wrote:
> I think I'd be fine exposing this through a test Jar, but it seems to me
> that if we were to put it into a normal package it would turn into the
> situation we want to avoid. People would use it for unintended pur
I think I'd be fine exposing this through a test Jar, but it seems to me
that if we were to put it into a normal package it would turn into the
situation we want to avoid. People would use it for unintended purposes and
it would become a distraction.
What do you think about using the tests Jar for
Yes, sorry I did not make it clear, I also agree it is not the right
direction to invest a lot of community effort. I am more talking about
casual use cases like importing a server for unit tests outside Iceberg,
running some local debugging, etc. I think it would be valuable to provide
a server in
> I know we have the RESTCatalogAdapter and RESTCatalogSevlet for unit
tests, and technically we have a very similar Jetty server implementation
in TestRESTCatalog. Should we think about making those components out of
the tests into an iceberg-rest-server module for this use case, and merge
with th
Really cool project!
I browsed a bit of the codebase, and see this implementation of the REST
service backend:
-
https://github.com/datastrato/gravitino/blob/main/catalogs/catalog-lakehouse-iceberg/src/main/java/com/datastrato/gravitino/catalog/lakehouse/iceberg/IcebergRESTService.java#L39
-
https
Thanks Justin for the sharing.
It's pretty cool to see an open source REST catalog implementation in
action. Having dabbled a bit in the early development of Gravitino myself,
I'm really excited about its potential with the Iceberg REST catalog.
The idea of Gravitino moving to an ASF project is p
Hi Justin,
I talked with Junping a couple of months ago about Gravitino. Thanks
for sharing !
Regards
JB
On Thu, Jan 25, 2024 at 12:15 AM Justin Mclean wrote:
>
> Hi,
>
> We open-sourced a new project, Gravitino, in December and have been working
> on growing the community and adding new funct
22 matches
Mail list logo
