gt; > AuthorDate: Wed Jul 31 16:12:03 2024 +0200
> > > Commit: JB Onofré
> > > CommitDate: Wed Jul 31 16:12:03 2024 +0200
> > >
> > > Force user.language as some tests are locale dependent
> > >
> > > commit fe0ef3d50f9f133ca1998ab8bba33ca407be75f1
I'd prefer to avoid having the APIs reflect the scope of the implementation
beans under the hood. On the other hand, I don't like forcing methods at
the bottom of the call stack requiring parameters because they're necessary
later on. I liked passing CallContext around because it didn't really
matt
ightly different. The
> > > > question we're trying to reach consensus on, is about what we should
> > > > annotate as @Nullable in Polaris code.
> > > >
> > > > I'd say if a value is received from external code, which is not
> > > a
I generally assume the other way around. Many (most) libraries don't
annotate the return values of their methods, so I assume everything is
nullable unless specifically told otherwise. I would prefer everything be
non-nullable unless specifically stated (the ? is the one thing I would
steal from ty
I've been doing some experimenting with storing table metadata in the
persistence, and I think I'm a fan, though I haven't yet collected real
numbers. Personally, though I think storing as separate entities is the
right approach rather than putting it in table properties. I appreciate
that we don't
y to demonstrate that by running a small test [1]. On my
> > >>> machine, the first 2 iterations (10 and 100 realms) complete, but
> the 3rd
> > >>> iteration (1000 realms) runs for about 1 minute then ends up in
> > >>> java.lang.OutOfMemoryError: Java he
ork on the disk buffering now
> and
> >> wait for the Events API to finalize before working on the API side of
> the
> >> end-to-end implementation.
> >>
> >> Best,
> >> Adnan Hemani
> >>
> >>> On May 20, 2025, at 10:30
Hmm, we do use the realm tag in our metric publishing. I understand the
concern re: cardinality. Maybe we can support filtering metrics that have
realm and support another metric without realm?
On Mon, May 19, 2025 at 12:24 PM Dmitri Bourlatchkov
wrote:
> Removing realm_id from metrics tags make
That’s super interesting. Glad this is being worked on. Personally, I don’t
know that the latency for writing events to a persistent storage is really
all that concerning. Looking at the enum of supported operations, only
write operations seem to trigger the event. It’s not like every read
request
It doesn't really matter. Robert, if I remove the flag, will you remove the
-1 on the PR?
On Wed, May 7, 2025 at 10:43 AM Dmitri Bourlatchkov
wrote:
> On Wed, May 7, 2025 at 11:29 AM Robert Stupp wrote:
>
> > If federated principals cannot be created, it doesn't make sense to me
> > to even hav
Congrats, folks!
> On May 5, 2025, at 9:23 AM, Russell Spitzer wrote:
>
> Hurrah!
>
> On Mon, May 5, 2025 at 11:11 AM Neelesh Salian
> wrote:
>
>> Congratulations to all of you!
>>
>> Regards,
>> Neelesh S. Salian
>>
>>
>>
>> On Mon, May 5, 2025 at 09:08 Yufei Gu wrote:
>>
>>> Congrats
a blocker from my POV because the behaviour of the current
> builder code is hard to predict without reading the implementation, which
> IMHO developers do not have to do on a regular basis as builders have a
> pretty simple API, where a set call should result in the argument becoming
>
ipals from the API in the PR for the
> sake of clarity?
>
> From my POV, we can reconsider that if/when SCIM (or something similar)
> comes.
>
> Thanks,
> Dmitri.
>
> [1] https://github.com/apache/polaris/pull/1353
>
> On Wed, Apr 30, 2025 at 11:50 AM Michael Collado
> SecurityContext doesn't expose the principal roles. I guess, the problem
> > now is how to move forward with this suggestion – I imagine a change
> > in AuthenticatedPolarisPrincipal or in the Resolver would create
> > repercussions in a lot of places in the code base as we
t
> task (build) right ? We should include it.
>
> I agree the tests have to pass.
>
> I will cancel this vote and we have to fix the tests on main and on the
> release branch.
>
> If no volunteer I will fix these tests.
>
> Thanks !
> Regards
> JB
>
>
-1 (non-binding)
There are test failures in the regression tests. A couple of them seem small -
the spark_sql_basic.sh.ref file reports an incorrect response from the catalog
creation.
However, the AWS test_spark_sql_s3_with_privileges.py test also fails with this
assertion:
2025-04-24 11:00:
t; in sync with local data (and the other way around too).
> >
> > That could be resolved by Polaris owning access tokens (via token
> exchange)
> > and assuming control over role membership based on local data and access
> > time (I commented on this in the doc too).
> >
We do the newcomer "Hi, I'm here to learn/talk about..." in the OpenLineage
syncs and I've always found it useful to hear what users are interested in
discussing. Definitely +1 to that idea.
On Tue, Apr 15, 2025 at 12:22 PM Jean-Baptiste Onofré
wrote:
> Great points, Russell !
>
> Let’s give pri
My main
> concern here is avoiding a maze of if/else statements throughout the
> Polaris codebase for supporting Federation.
>
> I guess our approach to this depends largely on the outcome of topic 2
> above.
>
> WDYT?
>
> Thanks,
> Dmitri.
>
> [1]
>
> ht
use more than one IDP per realm, since Quarkus OIDC is
> > multi-tenant and has the ability to select tenants based on various
> > criteria, such as the token issuer URL. This is what Realm E illustrates
> in
> > my example above.
> >
> > Is that OK for you?
> &
persisted. Specific authorizer implementations
> (including the default one) can certainly expect persisted principals, but
> the API should require that for the sake of flexibility of possible AuthN/Z
> extensions. WDYT?
>
> Cheers,
> Dmitri.
>
> On Thu, Nov 14, 2024 at 7:43 P
Hi Alex
I'm going through the PR now and I think the Quarkus security approach
seems fine. I was actually thinking of working on this previously myself.
> This shall be done by implementing a new HttpAuthenticationMechanism
that will pick the right authentication mechanism (internal token broker
My $.02 is that Option 1 is entirely possible using a DataSource that
dynamically creates Connections as needed. Option 1 is nice because, as
Pierre said, it gives admins the ability to dynamically allocate resources
to different clients as needed.
Personally, I'm less inclined to option 3 just be
I think a simple metrics API makes a lot of sense. Decoupling this from
events makes sense, as this would just be useful to query periodically for
a variety of reasons not tied to event triggering.
Mike
On Thu, Apr 10, 2025 at 3:00 PM Jean-Baptiste Onofré
wrote:
> Hi Eric
>
> Thanks a lot for y
Wow! Awesome news. Congrats folks!
Mike
On Wed, Mar 26, 2025 at 2:47 PM Russell Spitzer
wrote:
> Hi y'all!
>
> I'm excited to let the Polaris Community know that the PPMC has added three
> new members. Dmitri Bourlatchkov, Dennis Huo and Yufei Gu are all now
> members of the Polaris PPMC.
>
>
Personally, I don’t mind if have to maintain a bit of Scala code - I like
Scala, though every time the question of using comes up, I see the same
concerns that Russell brought up.
I will say that if the alternative is to introduce JMeter into the repo,
I’m a hard -1. I’ll write Scala all day long
-1 on changing the path for the existing table APIs. People already have
production systems pointing to Polaris endpoints and I don't think there's
a strong reason to force them to migrate. I think adding new prefixes for
new functionality is fine, but adding new features shouldn't break existing
b
Yeah, we should remove the “bug” label from anything that’s not a bug.
Improving code quality and improving code and api abstractions are good
goals, but they’re not bug fixes.
Mike
On Mon, Feb 17, 2025 at 8:12 AM Eric Maynard
wrote:
> There are still many issues labelled as "bug" that are not
base for openapi. I'd okay
>> to remove that, because the code generated by that generator is IMHO not
>> great.
>>
>> Overall we should focus on the "real stuff" and not bother about
>> manually writing builders and really immutable types.
>>
>> On 1
I prefer native Java constructs over third party libraries and compiler
plugins, whenever possible. I’m a fan of Java records.
Mike
On Mon, Feb 17, 2025 at 6:55 AM Jean-Baptiste Onofré
wrote:
> Hi Robert,
>
> Yes, my intention about Java records (e.g.
> https://openjdk.org/jeps/395) is to lever
Personally, I already use slack every day, so it’s easier for me to see and
respond to messages as they come in. I’d be more amenable to Zulip if I
used it for anything else, but I don’t, so for me, Slack is a clear winner.
Mike
On Fri, Jan 31, 2025 at 11:31 AM Dmitri Bourlatchkov
wrote:
> Good
Mustache templates for server and client code are copied and modified from
https://github.com/OpenAPITools/openapi-generator/tree/master/modules/openapi-generator/src/main/resources/JavaJaxRS/resteasy
Mike
On Mon, Jan 27, 2025 at 2:36 PM rdb...@gmail.com wrote:
> -1 because I think I’ve found c
They are leftover from a previous Authz design. They don’t do anything
anymore, so I don’t see a reason to keep them.
Mike
On Fri, Jan 24, 2025 at 6:57 AM Alex Dutra
wrote:
> Hi Dmitri,
>
> I think it would make sense to remove these annotations. While convenient,
> such annotations freeze the
I agree with Russell 100%. I think the intent here is to codify a spirit of
collaboration. That’s always going to be imperfect. But clarifying
expectations is important, even if it’s imperfect.
For some of the requirements (e.g., waiting an appropriate period or
waiting merging with unresolved com
> It is possible for some Polaris APIs to be developed first to unblock
support of certain use cases, and later contributed back to the IRC spec,
depending on the appetite of both communities.
This was exactly the intention with the notification API. It was never
expected to live long-term in the
I agree that breaking changes are inevitable on the main branch, but to
Russell's point there are people with deployments of Polaris in production
and I think it's unfair to them to simply say "there can be no breaking
changes" because there hasn't been a GA release. Breaking changes are
inevitable
ug from a personal standpoint
> > > but not related to project history)
> > > - we should use "new feature" for new functionality we want to
> > > implement in the project
> > > - we should use "proposal" for design/MVP
> > >
> >
Hi JB
This is a great topic and worth spending time on. Naturally, we want to
move fast and get things out quickly, but it’s also important that everyone
has a chance to weigh in on changes. This is easy to miss when we’re spread
across so many time zones and some groups are working while others a
Hey folks
There are over 40 issues with the "bug" label in github right now, many of
which are not actually bugs, but seem to me like personal preferences or
possible improvements. A lot of these issues seem like reasonable or good
changes to me, but I think we should reserve the "bug" label for t
I also noticed Yufei's comment re: the health checker endpoint, which is
now gone. Is there an issue to address that as well? I think we ought to
treat the admin endpoints (/healthcheck and /metrics) just as we would
treat the management or catalog APIs - that is to say, we wouldn't simply
remove o
Personally, I find the current in-memory store invaluable for debugging.
It's really easy to crack open the tree map and see what entities exist
when tracing down a problem. I'd hate for that to go away and rely on H2 or
sqllite or something.
I also haven't yet heard any user championing EclipseLi
+1 (non-binding). The source distro looks good to me. There is a lot of
work to do before we're ready for the binary distribution, but I don't
think that should block this first source distro.
Mike
On Tue, Jan 14, 2025 at 1:19 AM Jean-Baptiste Onofré
wrote:
> +1 (binding)
>
> As this release in
Woohoo! Awesome to hear! Congrats!
Mike
On Tue, Jan 7, 2025 at 10:46 AM Yufei Gu wrote:
> Congrats Dmitri! Thanks for all the contributions!
>
> Yufei
>
>
> On Tue, Jan 7, 2025 at 10:44 AM Eric Maynard
> wrote:
>
> > Congrats Dmitri, and thanks for all your contributions so far!
> >
> > --EM
>
Hey folks
I think keeping the momentum moving forward is important, so I don’t want
to delay unnecessarily. However, a lot of the US folks have been out of the
office due to the Christmas holiday and many won’t be back until next week.
Personally, I take this time to be able to focus on family and
This is a great idea. It’s currently a bit ambiguous when an idea has
consensus to move forward.
What’s the process for adding the label to issues? Do we add this only when
we’ve had a vote on the mailing list?
Mike
On Wed, Dec 18, 2024 at 2:08 AM Jean-Baptiste Onofré
wrote:
> Hi folks,
>
> FY
; does
>>>>> not require an invitation (hello Slack). Another chat systems just
>> adds
>>>>> (IMHO mostly unneeded) feature over feature. Zulip is stable on Linux
>>> and
>>>>> macOS and offers a web interface for those that don’t want to install
&
I agree. And I especially agree with the point regarding features over
technical discussions.
I feel like there’s been a bit of bike-shedding (I can bike-shed with the best
of them, I’m afraid) on details that are somewhat inconsequential. Things like
support for specific NoSQL database (mongo
Hey folks
We've talked about breaking up the metastore interface and better
encapsulating the various areas of responsibility. I think there are some
competing thoughts around what should be handled by the different
interfaces, so I created https://github.com/apache/polaris/issues/526 to
try to ca
ommunication channels :)
>
> Are you suggesting to drop Zulip and use the polaris channel in the Iceberg
> slack workspace?
>
> Thanks,
> Dmitri.
>
> On Fri, Dec 6, 2024 at 3:42 PM Michael Collado
> wrote:
>
> > Currently, the communication channels for Polari
er in a way that
> services do not 'see' cache APIs directly, but interact with some other
> interface, which then can have multiple implementations: with and without
> caching, with and without "remote" caching, etc. WDYT?
>
> Thanks,
> Dmitri.
>
> On Fri,
Hey folks
Someone pinged about https://github.com/apache/polaris/pull/389 yesterday
and I thought it was worth bringing up for discussion.
On-prem s3 compat sounds like a super useful feature and I'm fully on board
with supporting it, but I think we need to make a decision about whether we
suppor
Currently, the communication channels for Polaris include
Mailing list
Iceberg Slack
GitHub
Zulip
These are in addition to the regular comm channels I have to keep up with
in a given day. I think something’s gotta give.
We can’t get rid of the mailing list or GitHub and I’m already in slack all
My intention when splitting up the PolarisMetaStoreManager interface was
always to cut the ties between the persistence manager and the other
responsibilities. For me, the first step seemed to break up the interface,
then change the consumers to depend on the most specific interface needed
to accom
ssary. Would it be able to find `@Produces` annotations automatically?
>
> Let's proceed with the migration path doc (as discussed in other emails in
> this thread).
>
> Cheers,
> Dmitri.
>
>
> On Tue, Nov 26, 2024 at 12:07 AM Michael Collado
>
t; created (let's use the Polaris proposal process ;)).
>
> Thanks !
> Regards
> JB
>
> On Tue, Nov 26, 2024 at 8:24 AM Michael Collado
> wrote:
> >
> > I’m happy to consider the rest of the ecosystem plugins, though I would
> > like to decouple tha
eady provided by Quarkus (thanks to
> the extensions). I think it's urgent to have a consensus here to know
> in which direction we are going.
>
> Let's chat together about that.
>
> Thanks !
> Regards
> JB
>
> On Tue, Nov 26, 2024 at 6:06 AM Michael Collado
>
implementations).
Mike
On Tue, Nov 19, 2024 at 8:37 PM Michael Collado
wrote:
> Hey thanks for calling that out. It’s strictly a convenience change and
> actually isn’t necessary. I added that in because, at the end of the day,
> most components need a PolarisMetaStoreManag
ns, without having to
> extend/implement framework interfaces.
>
> Cheers,
> Dmitri.
>
> On Thu, Nov 7, 2024 at 8:37 PM Michael Collado
> wrote:
>
> > FYI, I took a stab at seeing how Polaris would work with HK2 as a CDI
> impl.
> > I only spent yesterday and toda
+1
On Sat, Nov 16, 2024 at 10:00 PM Jean-Baptiste Onofré
wrote:
> Hi everyone,
>
> I propose that we release the following RC as the official Apache
> Polaris 0.9.0-incubating release.
>
> * This corresponds to the tag: apache-polaris-0.9.0-incubating-rc1
> *
> https://github.com/apache/polaris/
uld propose to create an abstract Authenticator in
> Polaris that we can decorate in the Polaris runtime framework.
>
> I will comment in the doc.
>
> Thanks again!
> Regards
> JB
>
> On Fri, Nov 15, 2024 at 1:43 AM Michael Collado
> wrote:
> >
> > Hey f
Hey folks
As discussed during the community sync, I've put together some thoughts on
how we'd add support for federated identities in Polaris. I copied over
some of what I had in the issue at
https://github.com/apache/polaris/issues/441 and put it into the doc here:
https://docs.google.com/docume
;s appropriate
> > for DW).
> >
> > If everybody is on board with the plan above, how about we create GH
> issues
> > for each step, and maybe even a GH Project to track progress on this
> front
> > and split the effort?
> >
> > Thanks,
> >
&g
7cf923ffe8...collado-mike:polaris:mcollado-hk2-di
and let me know what you think.
Mike
On Tue, Nov 5, 2024 at 9:42 AM Michael Collado
wrote:
> I added a table to the README with the differences that were called out. I
> added some details that I think are worth understanding better. E.g., the
Congrats, folks! Glad to have you on board
Mike
On Wed, Nov 6, 2024 at 10:12 AM Eric Maynard
wrote:
> Congrats Alex and Yong!
>
> On Wed, Nov 6, 2024 at 9:47 AM Yufei Gu wrote:
>
> > Congrats, Alex and Yong! Thanks for all the contributions!
> >
> > Yufei
> >
> >
> > On Wed, Nov 6, 2024 at 9:3
I added a table to the README with the differences that were called out. I
added some details that I think are worth understanding better. E.g., the
Json layout we added has specific custom functionality we wanted for
supporting key/value pairs and the micrometer annotation was added for some
custo
One Lake that don't have a REST endpoint,
> this will be a requirement.
>
> On Thu, Oct 31, 2024 at 12:15 PM Michael Collado
> wrote:
>
> > IMO, anything that's not directly managed by Polaris source code is
> > external. If someone can create a table directly in the o
IMO, anything that's not directly managed by Polaris source code is
external. If someone can create a table directly in the other catalog and
it shows up in Polaris, it's external.
As of now, all of the direct manipulation APIs are blocked for external
catalogs - createTable, updateTable, etc. I t
Agree on all points. I’m hoping to get pluggable authentication in Polaris
to support SSO/SCIM so we can have better interoperability with IdPs.
But really, I think the main focus for interoperability should be
compatibility with query engines, as JB said. We can support Iceberg
clients in any lan
ms
> 3.1).
>
> So, regarding your example, I would say that Polaris 2.0 should not
> necessary support v1 APIs, but focus on vX APIs and important changes.
> We can still maintain Polaris 1.x for v1 API (with dependency updates
> and bug fixes).
>
> Regards
> JB
>
>
pate that catalog will have tables in wacky locations, too. I set
> the same two flags mentioned above the same way. I get the same behavior.
>
> Does the above workflow make sense to you? I like the simplicity of this
> approach.
>
> On Tue, Oct 22, 2024 at 5:08 PM Michael Collado
e #264
> <https://github.com/apache/polaris/pull/264> we allow overrides of this on
> the catalog level, so admins that wanted to allow unsecure overlapping
> table locations in external catalogs they could do so.
>
> On Tue, Oct 22, 2024 at 4:38 PM Michael Collado
> wrot
o allows for finer-grained control if these
> overrides are ever respected for namespaces or tables.
>
> Other flags like the ones to allow location overlap or to disable
> subscoping are not specific to a catalog type.
>
> On Tue, Oct 22, 2024 at 3:15 PM Michael Collado
> wrote:
&
Hey folks
I opened a PR at https://github.com/apache/polaris/pull/395 to support
disabling credential vending for external catalogs.
Currently, some remote catalogs don't control the table location for
Iceberg tables, allowing people to create tables in overlapping directories
in storage. A perso
What's the support policy for API versions? Do we increment API versions
for breaking API changes (this seems reasonable and likely)? And do we
support previous API versions for some period of time?
If the answer to both of those questions is "yes" , then what is the
purpose of fast major verison
I know there’s a rebranding effort underway. I’m sure the docs are being
addressed as part of that. But yeah, it’s not the kind of thing that
happens overnight. Marketing has their own schedules for these things.
Whatever it is, they’ll work with ASF for timing.
On Wed, Oct 16, 2024 at 7:25 AM Ric
+1
I like face to face meetings even virtual faces) very once in a while. 9AM
PST works for me as well
Mike
On Fri, Oct 11, 2024 at 2:36 PM Yufei Gu wrote:
> Thanks Russell for setting this up. Thursday 9pm PST works for me.
>
> Yufei
>
>
> On Fri, Oct 11, 2024 at 1:37 PM Russell Spitzer >
> w
fresh project?
>
> Thanks,
> Dmitri.
>
> On Tue, Sep 24, 2024 at 10:22 AM Michael Collado
> wrote:
>
> > Agree with Yufei 100%. Making it as deprecated, as the Iceberg spec does,
> > makes total sense. We should remove it when Iceberg 2.0 is released.
> >
> &
hould
> be a
> > > > separate discussion thread.
> > > >
> > > > My point about the first release is that such a concern already
> exists
> > > and
> > > > has been discussed in the Iceberg community. I think we ought to
> &
+1 on Russell’s comment.
Re: the OAuth endpoint, it seems to me that compatibility with Iceberg
clients needs to be considered. I think that prior to Iceberg 1.5 or so,
there was not support for an external oauth tokens endpoint. If we remove
the endpoint in Polaris, clients prior to that release
Currently, we can set table defaults at the catalog level - e.g., we do
this to set the s3 endpoint in tests:
https://github.com/apache/polaris/blob/main/polaris-service/src/test/java/org/apache/polaris/service/catalog/PolarisSparkIntegrationTest.java#L151-L158
The plan for StorageConfigurationInf
My opinion is that it should be a pretty big deal to make a change to the
REST spec and _not_ push it upstream. Polaris is not intended to be a fork
of the Iceberg REST spec, but an implementation. We don't own clients that
aren't Iceberg clients. Compatibility with those clients should be a
primar
81 matches
Mail list logo
