A lot of the concerns seem to arise from the fact that Polaris essentially
has two distinct deployment modes:
1. Provide a highly isolated multi-tenant (multi-realm) Iceberg
REST service with a tightly curated set of secure integrations to external
dependencies
2. Provide a turnkey Iceberg REST se
Sure, I will add it to the community sync agrenda.
My concern here is that the other IRC will still need to re-implement
significant portions of Polaris logic, some important blocks of the
top of the list: Auth token management, entity persistence, principals
management, entity access API; which i
I think this deserves a discussion in the next community sync, feel
free to add it to the agenda.
>From what I read in this thread: nobody is against the idea of being
able to federate to HMS.
However, technical concerns should really be considered, and I think
that it is a much safer option for
Hi everyone,
I believe Hive Metastore (HMS) federation is a critical feature for the success
and widespread
adoption of the project. The reality of the current data lake ecosystem is that
a vast number of
organizations have significant footprint in HMS. For Polaris to be the Iceberg
Rest Catal
It’s a valid point that Polaris needs to support multi-tenancy, or even
across different external catalogs (such as remote HMS) within a single
realm.
Unfortunately, Kerberos isn’t compatible with this model, as it requires
global configuration per JVM, making it inherently single-tenant. So I’d
s
Following up on my email:
Polaris would really benefit from supporting HMS and other catalog
types. And the way I see to get there is to have a "HMS only" IRC
service, which can be legibly built on Java 11, use Kerberos, etc.
Polaris can then federate to that HMS catalog.
AFAIU clients can authent
Let's recap what Polaris offers:
1. Multi tenancy via realms
2. Multiple catalogs per realm
3. OAuth/OIDC
Adding Kerberos is global per JVM, making #1 impossible and likely
also not suitable for #2, plus adding another complicated and complex
auth mechanism.
If Kerberos is a strong concern, I prop
HMS integration is a key step toward one of Polaris’s critical missions:
helping users move off HMS. It brings clear value by aligning with our
long-term direction.
I’m not too concerned about hive.xml, most of its configurations can be
dynamically injected at runtime. The real challenge lies in K
I think having some integration with HMS is definitely a good idea. We've
already seen
users build this in the wild on top of Polaris showing that there is
definitely a demand.
I'm still a strong believer that we should be helping users get to Polaris
from whatever systems
they are currently using
1. We (Polaris) can provide end users a way to migrate off of these
catalogs that the Iceberg project no longer wants to invest into.
Implementing HMS federation is in service to the goal of removing
non-Iceberg catalogs, not in contradiction to it.
2. This does not seem like a user-centered conce
I'd really prefer to not add "anything Hive" to Polaris itself, and I'd
really like to see Hadoop being removed entirely from the Polaris code base.
There are multiple reasons for this:
1. The Iceberg project would rather like to remove all catalogs except
the REST catalog. (That's at least wh
I think this is orthogonal to Dmitri's note above but my understanding is
the following:
When the auth type is NONE, the config doesn't tell Polaris which (or any)
hive-site.xml to pick up. This is essentially just a test mode or a mode
for when the Polaris service is deployed as a wrapper around
Hey Eric,
The part about other auth types has a few aspects we should consider:
1. We can reliably support HMS instances that have (Simple, ie. NO_AUTH) and
LDAP (with some implementation of LDAPAuthenticationType).
2. We can't support Kerberos because it requires that the user via the Polaris
IMO 2a should be off the table; the person creating an EXTERNAL catalog
does not necessarily have permission to access the path that a
hive-site.xml is as (whether local to the Polaris catalog service or in
object storage) or to even know what paths the catalog has access to. It's
the same problem
14 matches
Mail list logo
