Re: [DISCUSS] Drop In-memory MetaStore in favour of H2

2025-01-15 Thread Alex Dutra
I agree that it has some advantages, like debugging. How about we keep it, but rename its identifier to "test" to clearly specify its intent? It would also align with other "test" components such as the token service and the token broker. Ideally, we would eventually put all these "test" component

Re: [VOTE] Release Apache Polaris 0.9.0-incubating (rc2)

2025-01-15 Thread Jean-Baptiste Onofré
Ok. I disagree but I don’t want to argue. I will cancel the vote and do the changes. Regards JB Le mer. 15 janv. 2025 à 01:20, rdb...@gmail.com a écrit : > Hey JB, I just sent a reply to the list with additional details. I don't > think that this release applies license policy correctly even in

[CANCEL][VOTE] Release Apache Polaris 0.9.0-incubating (rc2)

2025-01-15 Thread Jean-Baptiste Onofré
Hi folks I cancel this vote to fix concerns expressed by Ryan. I will have a PR later today and I will cut rc3 asap. Thanks Regards JB Le mer. 8 janv. 2025 à 17:00, Jean-Baptiste Onofré a écrit : > Hi folks, > > As mentioned in another thread, I submit Apache Polaris > 0.9.0-incubating rc2 t

Re: Table maintenance in Polaris @ Thu, Nov 7, 2024 9:00am – 10:00am (GMT-08)

2025-01-15 Thread Yufei Gu
I think "when" is really part of the "how," so we should try to avoid requiring "when" unless absolutely necessary. For example, compaction could be based on time or metrics, but ideally, users would just turn it on, and it runs automatically in the background. They wouldn’t need to specify "when"

Re: Table maintenance in Polaris @ Thu, Nov 7, 2024 9:00am – 10:00am (GMT-08)

2025-01-15 Thread Jean-Baptiste Onofré
Hi Yufei Don't we risk a kind of inconsistency ? Let me take the example of compaction. Does it make sense to have two engines doing compaction with different properties/recipes ? It sounds very risky to me (even if Polaris "manages" the compaction orchestration). If the use case is to have, on on

Re: Table maintenance in Polaris @ Thu, Nov 7, 2024 9:00am – 10:00am (GMT-08)

2025-01-15 Thread Eric Maynard
One quick point: "engines" do not by themselves do compaction. Apache Spark, off the shelf, does not know how to compact an Iceberg table with certain properties. You need some kind of maintenance service built on top of one or more engines. These *services* may support different properties or hav

Re: Table maintenance in Polaris @ Thu, Nov 7, 2024 9:00am – 10:00am (GMT-08)

2025-01-15 Thread Omar Al-Safi
> > Ultimately, tasks like table maintenance are indeed application-specific. > One user may wish to build a service that Z-Orders their table, while > another might wish to ensure all Parquet rowgroups in their table are 16MB. > This is all in the data plane, and it doesn't seem like the job of th

Re: Docker images and distributions for Polaris on Quarkus

2025-01-15 Thread Jean-Baptiste Onofré
Hi Alex I agree with Docker changes, it sounds good to me. Removing the root Dockerfile in favor of the "Quarkus" approach is way better imho. Regarding EclipseLink and drivers, it makes sense to start with PostgreSQL and H2. Depending on how we move forward regarding EclipseLink (if we keep it o

Re: [VOTE] Release Apache Polaris 0.9.0-incubating (rc2)

2025-01-15 Thread Jean-Baptiste Onofré
Hi Ryan Thanks for the details. I will do a PR to address your comments, I will gently ask you to review before I merge it :) Thanks again, Regards JB On Wed, Jan 15, 2025 at 1:20 AM rdb...@gmail.com wrote: > > Hey JB, I just sent a reply to the list with additional details. I don't > think tha

Re: Next steps for design discussion of Apache Polaris Catalog Federation

2025-01-15 Thread Dennis Huo
In case anyone wasn't on the googlegroups list but wanted to join the meeting tomorrow, here's the meeting info for tomorrow: Apache Polaris - Catalog Federation Meeting Thursday, January 16 · 9:00 – 10:00am Time zone: America/Los_Angeles Google Meet joining info Video call link: https://meet.goog