What I'm getting from this conversation is that if one want to know what to
do with a generic table, just check the spark code which is not what I was
hoping for as the standard for new entities to be added to the
specification. If you compare with iceberg itself, how to access iceberg
tables and i
Thanks, Laurent, for bringing up spec "readiness" and, I guess, by
extension backward compatibility concerns.
Regardless of how deep current spec is in Polaris, I believe it is
important to have it written down as an artifact in the Polaris repo. I
know we had a design doc at some point, but the p
I cut the 1.0.x branch yesterday morning.
As a matter of best practice, given the previous related discussion thread
[1], it would have been nice to conclude it with a message about starting
the 1.0 release process before actually cutting the branch.
[1] https://lists.apache.org/thread/8kx1mjg7h
Hi Laurent,
I do agree that the generic table spec needs to be evolved to provide
richer standardization. However,
the current support of generic table does provides good amount of values:
1) Polaris can be a centralized catalog service for discovering all tables,
not just Iceberg tables. (Of cour
Thanks everyone for the contribution. We've finally resolved all
blockers[1]. I cut the 1.0.x branch yesterday morning. Will only cherry
pick bug fixes and license related commits to this branch starting now.
[1]. PR1695 is labeled with 1.0 blocker, but we agreed that it's a
best-to-have instead o
> Can you explain what is a proper plain English spec for this feature?
I mean a doc page similar to [1] that explains what Generic Tables are, how
to use them in Spark, how to use them is some other query engine, and most
importantly the planned evolution for the Generic Tables API and
specificat
> I cut the 1.0.x branch yesterday morning
I do not see a "release/1.0.0" branch (which is a pattern used for previous
releases).
[1]. PR1695 is labeled with 1.0 blocker, but we agreed that it's a
best-to-have instead of a blocker per offline discussion,
Where is this agreement recorded?
As fo
There are solid use cases for adding generic-table support with the Spark
plugin:
- Single Catalog, Many Formats – Keep Delta, CSV, Parquet (and future
formats) side-by-side in one place instead of juggling separate catalogs.
- Seamless Migrations – Let teams move data from one format to
> I mean a doc page similar to [1] that explains what Generic Tables are,
how
to use them in Spark, how to use them is some other query engine, and most
importantly the planned evolution for the Generic Tables API and
specification.
Yes, we can definitely add a webpage to describe the current guar
As for the evolution, I do think it is a good strage to evolve step by
step, instead of trying to standardize
everything in one shot.
This approach makes sense to me, but we need to be explicit about it in the
spec.
Cheers,
Dmitri.
On Wed, Jun 11, 2025 at 5:45 PM yun zou wrote:
> > I mean a d
The branch name is "1.0.x".
Where is this agreement recorded?
Discussed multiple times with JB last Thursday(6/5/2025) and this
Monday(6/9/2025), we agreed to consider it as a nice-to-have instead of a
blocker.
As a matter of best practice, given the previous related discussion thread
> [1], it
Discussed multiple times with JB last Thursday(6/5/2025) and this
Monday(6/9/2025), we agreed to consider it as a nice-to-have instead of a
blocker.
I'm sure none of JB's statements (even though I was not part of those
meetings, IIRC) implied that the agreement is community-wide.
The Apache Way
Yes. I think we have agreed that we will make sure things are described
clearly in both the spec and website for
the critical fields added.
We are currently trying to get a webpage out for the Generic Table support
in Polaris.
Best Regards,
Yun
On Wed, Jun 11, 2025 at 3:09 PM Dmitri Bourlatchkov
Also the
last PPMC member's agreement on thread[1] happened 5 days ago, which passed
the lazy consensus window. But I agreed it's nice to conclude a thread.
The consensus in that thread was to skip the 0.10.0 release.
>From my POV an agreement to skip 0.10.0 does not mean that the scope for
1.0
I added a 1.0 blocker label to [1857].
I believe it is a critical usability fix for users of Generic Tables in
Spark.
[1857] https://github.com/apache/polaris/pull/1857
Cheers,
Dmitri.
On Thu, May 15, 2025 at 5:39 PM Yufei Gu wrote:
> Hi folks,
>
> Many users have been asking about the Polari
What I was trying to say is that i'm sure there's plenty of value for
spark, but in it's current state the value is little from a Polaris point
of view as an open catalog service?
Of course we can follow-up on that but is the current spec still considered
wip or when 1.0 will be released, we would
> I don't think there's a lot of value where the specification of a table
format is left to the client
Considering that you currently can use non-Iceberg tables in Polaris with
the Spark client and it works end-to-end, I'd have a hard time agreeing
that there is no value.
But I think this discussi
Yufei, could you explain the decision to merge [1543] within 2 hours of new
replies to this email thread that was specifically intended to discuss the
proposed SPEC change?
I do not think that the new points got enough time for consideration by all
community members, therefore one can hardly assum
18 matches
Mail list logo
