Just to reiterate my points discussed in the community sync here:
the more I think about it the more I agree the OAuth endpoint
*should be removed from the REST spec*. Even though the endpoint is
optional, and even if we do not care about the security concerns, it
still provides users an impression that the endpoint "should" be
implemented, or "is the preferred authentication mechanism". And as
we have found out, the server capability proposal does not cover
this case since this is the first endpoint to hit before the
GetConfig endpoint.
As Ryan said, if we want to do that we need an alternative plan. I
don't have anything concrete, but here is my line of thought:
1. remove OAuth2 endpoint from the "REST OpenAPI spec"
2. create a client-side interface (in each language) that different
authentication mechanisms can be plugged in to talk to the REST catalog
3. refactor and make OAuth2 an implementation of that interface. I
can also help with doing the same for AWS Sigv4, and the community
can further support some additional ones like Kerberos, SAML, Google
SSO, etc. based on the individual use cases.
4. turn 2 + 3 into a "REST catalog authentication spec" that
documents about all the supported authentication mechanisms and
their defaults. For OAuth2, the default is to have the auth server
at the same endpoint as the resource server for backwards
compatibility, but that is a configurable property, and we could
recommend not to do that based on security concerns.
Best,
Jack Ye
On Wed, May 29, 2024 at 10:28 AM Steven Wu <stevenz...@gmail.com> wrote:
Wondering if the auth endpoints can be separated out to a
separate OpenAPI spec file. Then we still have some reference
for interactions with auth server and make it clear it is not
required as part of the REST catalog server. In most enterprise
environments, auth server is likely a separate server.
On Tue, May 28, 2024 at 1:25 PM Alex Dutra
<alex.du...@dremio.com.invalid> wrote:
Hi,
On point 4, isn't that possible today, Can't that be
achieved with the current token exchange approach, and
the internal implementation of the endpoint?
Unfortunately, no. Token exchange is not widely adopted yet:
for example, Keycloak has only partial support for it, and
Authelia, or Authentik, have no support for it at all.
This, and a few other technical issues with the current
internals of the REST client, makes it nearly impossible to
achieve a good integration of Iceberg REST with the majority
of popular OSS authorization servers.
I am planning to start another email thread to discuss these
practicalities, but let's first reach consensus on the
broader security issues voiced here, before we tackle the
details.
Thanks,
Alex Dutra
On Tue, May 28, 2024 at 8:41 PM Amogh Jahagirdar
<am...@tabular.io> wrote:
I disagree with removing "/v1/oauth/tokens" and I think
I also disagree with the premise that implementing that
endpoint is required, but I can understand how that's
not clear in the spec. I think we can address the
required vs non-required discussion with the
capabilities PR.
<https://github.com/apache/iceberg/pull/9940>
It seems like another part of what's driving this
discussion is some concern around how do we enforce REST
catalog implementations which do implement this endpoint
to make sure that the implementation is secure (for
example to avoid the MITM example that was brought up).
This is ultimately a runtime detail. To me it seems like
if we make it clear that such an endpoint should be
implemented respecting OAuth2 standards, and we know
that OAuth2 compliance requires avoiding that MITM
situation, then runtime implementations should just
follow the spec there
>3. Enable flexibility for Iceberg REST servers to opt
for other
authorization mechanisms than OAuth 2.0.
>4. Enable REST servers to opt for integrating with any
standard OAuth2 /
OIDC provider (e.g. Okta, Keycloak, Authelia).
I agree with both of these points; again I don't think
the intention is Oauth2 is the only way, but I think the
capabilities PR will make that even more clear.
On point 4, isn't that possible today, Can't that be
achieved with the current token exchange approach, and
the internal implementation of the endpoint? Sorry if I
missed that explanation.
Thanks,
Amogh Jahagirdar
On Tue, May 28, 2024 at 11:13 AM Yufei Gu
<flyrain...@gmail.com> wrote:
Not an expert on authentication, but reading from
the context, I agree that it’s not a good practice
to use a resource server as a token server. The
resource server would need to securely handle and
store credentials or tokens, increasing the risk of
credential theft or leakage. Making the token
endpoint optional will mitigate the issue a bit. But
if we want to disable it completely, it's better to
do it now to prevent any issues and migration costs
in the future. Can we have a consensus on it?
I would prefer to deprecate it to prevent any
intentional and unintentional misuse. We will also
need to change the clients since it connects to the
endpoint by default.
Yufei
On Tue, May 28, 2024 at 8:27 AM Jack Ye
<yezhao...@gmail.com> wrote:
Sounds like we should try to finalize a
consensus around
https://github.com/apache/iceberg/pull/9940, so
that we make it very clear what APIs/features
are optional.
-Jack
On Tue, May 28, 2024 at 7:25 AM Fokko Driesprong
<fo...@apache.org> wrote:
Hey Robert,
Sorry for the late reply as I was out last
week. I'm not an OAuth guru either, but some
context from my end.
* Credentials (for example
username/password) must _never_ be sent to
the resource server, only to the
authorization server.
In an earlier discussion
<https://github.com/apache/iceberg/pull/8976>,
it was agreed that the resource server can
also function as the authorization server.
But the roles can also be separate.
1.2. As long as OAuth2 is the only
mechanism supported by the Iceberg
client, make the existing client
parameter “oauth2-server-uri”
mandatory. The Iceberg REST catalog must
fail to initialize if the
“oauth2-server-uri” parameter is not
defined.
It can also be that there is no
authentication in the case of an internal
REST catalog. For example, the
iceberg-rest-image
<https://github.com/tabular-io/iceberg-rest-image>
that we use for integration tests in PyIceberg.
We think that Apache Iceberg REST
Catalog spec should not mandate that a
catalog implementation responds to
requests to produce Auth Tokens
(since the REST spec v1 defines a
/v1/tokens endpoint, current
implementations have to take deliberate
actions when responding to those
requests, whether with successful token
responses or with “access
denied” or “unsupported” responses).
The `/v1/tokens` endpoint is optional
<https://github.com/apache/iceberg-python/blob/756ae625a2ea0f9c12df78430512ce991f6a1976/pyiceberg/catalog/rest.py#L488-L489>.
* Credentials (for example
username/password) must _never_ be sent to
the resource server, only to the
authorization server.
I fully agree!
Even if an Iceberg REST server does not
implement the ‘/v1/oauth/tokens’
endpoint, it can still receive requests
to ‘/v1/oauth/tokens’ containing
clear text credentials, if clients are
misconfigured (humans do make
mistakes) - it’s a non-zero risk - bad
actors can implement/intercept
that ‘/v1/oauth/tokens’ endpoint and
just wait for misconfigured
clients to send credentials.
I think the wording is chosen badly. It
should not send any credentials, but the
code (as in this example
<https://developers.google.com/identity/protocols/oauth2#installed> by
GCS).
I think Jack makes a good point with AWS
SigV4 Authentication. I suppose, in REST
Catalog implementations that support
that auth method, the /v1/oauth/token
Catalog REST endpoint is redundant.
There are other cloud providers next to AWS.
Kind regards,
Fokko
Op do 23 mei 2024 om 15:49 schreef Dmitri
Bourlatchkov
<dmitri.bourlatch...@dremio.com.invalid>:
I think Jack makes a good point with AWS
SigV4 Authentication. I suppose, in REST
Catalog implementations that support
that auth method, the /v1/oauth/token
Catalog REST endpoint is redundant.
Cheers,
Dmitri.
On Thu, May 23, 2024 at 9:20 AM Jack Ye
<yezhao...@gmail.com> wrote:
I do not know enough details about
OAuth to make comments about this
issue, but just regarding the
statement "OAuth2 is the only
mechanism supported by the Iceberg
client", AWS Sigv4 auth is also
supported, at least in the Java
client implementation
<https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/HTTPClient.java#L72>.
It would be nice if we formalize
that in the spec, at least define it
as a generic authorization header.
Best,
Jack Ye
On Thu, May 23, 2024 at 2:51 AM
Robert Stupp <sn...@snazy.de> wrote:
Hi all,
Iceberg REST implementations,
either accessible on the public
internet
or inside an organization, are
usually being secured using
appropriate
authorization mechanisms. The
Nessie team is looking at
implementing the
Iceberg REST specification and
have some questions around the
security
endpoint(s) defined in the spec.
TL;DR we have questions
(potentially concerns) about
having the
‘/v1/oauth/tokens’ endpoint, for
the reasons explained below. We
think
that ‘/v1/oauth/tokens’ poses
potential security and OAuth2
compliance
issues, and imposes how
authorization should be implemented.
* As an open table format, it
would be good for Iceberg to
focus on the
table format / catalog and not
how authorization is
implemented. The
existence of an OAuth endpoint
pushes implementations to adopt
authorization using only OAuth,
whereas the implementers might
choose
several other ways to implement
authorization (e.g. SAML). In our
opinion the spec should leave it
open to the implementation to
decide
how authorization will be
implemented.
* The existence of that endpoint
also pushes operators of Iceberg
REST
endpoints into the authorization
service business.
* Clients might expose their
clear-text credentials to the wrong
service, if the (correct) OAuth
endpoint is not configured
(humans do
make mistakes).
* (Naive) Iceberg REST servers
may proxy requests received for
‘/v1/oauth/tokens’ - and
effectively become a
“man-in-the-middle”, which
is not fully compliant with the
OAuth 2.0 specification.
Our goals with this discussion are:
1. Secure the Iceberg REST
specification by preventing
accidental
misuse/misimplementation.
2. Prevent that Iceberg REST to
get into dictating the
“authorization
server specifics”.
3. Enable flexibility for
Iceberg REST servers to opt for
other
authorization mechanisms than
OAuth 2.0.
4. Enable REST servers to opt
for integrating with any
standard OAuth2 /
OIDC provider (e.g. Okta,
Keycloak, Authelia).
OAuth 2.0 [1] is one of the
common standards accepted in the
industry.
It defines a secure mechanism to
access resources (here: Iceberg
REST
endpoints). The most important
aspect for OAuth 2.0 resources
is that
(Iceberg REST) servers do not
(have to) support password
authentication,
especially considering the
security weaknesses inherent in
passwords.
Compromised passwords result in
compromised data protected by
that password.
Therefore OAuth 2.0 defines a
set of strict rules. Some of
these are:
* Credentials (for example
username/password) must _never_
be sent to
the resource server, only to the
authorization server.
* OAuth 2.0 refresh tokens must
_never_ be sent to the resource
server,
only to the authorization
server. (“Unlike access tokens,
refresh tokens
are intended for use only with
authorization servers and are
never sent
to resource servers.”, cite from
section 1.5 of the OAuth RFC 6749.)
* While the OAuth RFC states
"The authorization server may be
the same
server as the resource server or
a separate entity", this should
not be
mandated. i.e the spec should be
open to supporting
implementations that
have the authorization server
and resource server co-located
as well as
separate.
The Iceberg PR 4771 [2] added
the OpenAPI path
‘/v1/oauth/tokens’,
intentionally marked to “To
exchange client credentials
(client ID and
secret) for an access token.
This uses the client credentials
flow.”
[3]. Technically: client ID and
secret are submitted using a
HTTP POST
request to that Iceberg REST
endpoint.
Having ‘/v1/oauth/tokens’ in the
Iceberg REST specification can
easily
be seen as a hard requirement.
In order to implement this in
compliance
with the OAuth 2.0 spec, that
‘/v1/oauth/tokens’ MUST be the
authorization server. If users
do not (want to) implement an
authorization server, the only
way to implement this
‘/v1/oauth/tokens’
endpoint would be to proxy
‘/v1/oauth/tokens’ to the actual
authorization server, which
means, that this proxy
technically becomes a
“man in the middle” - knowing
both all credentials and all
involved tokens.
Even if an Iceberg REST server
does not implement the
‘/v1/oauth/tokens’
endpoint, it can still receive
requests to ‘/v1/oauth/tokens’
containing
clear text credentials, if
clients are misconfigured
(humans do make
mistakes) - it’s a non-zero risk
- bad actors can
implement/intercept
that ‘/v1/oauth/tokens’ endpoint
and just wait for misconfigured
clients to send credentials.
Further usages of the REST
Catalog API path
‘/v1/oauth/tokens’ are “To
exchange a client token and an
identity token for a more
specific access
token. This uses the token
exchange flow.” and “To exchange
an access
token for one with the same
claims and a refreshed
expiration period
This uses the token exchange
flow.” Both usages should and
can be
implemented differently.
Apache Iceberg, as a table
format project, should recommend
protecting
sensitive information. But
Iceberg should not mandate _how_
that
protection is implemented - but
the Iceberg REST specification does
effectively mandate OAuth 2.0,
because other Iceberg REST
endpoints do
refer/require OAuth 2.0
specifics. Users that want to
use other
mechanisms, because they are
forced to do so by their
organization,
would be locked out of Iceberg REST.
Apache Iceberg should not
mandate OAuth 2.0 as the only
option - for the
sake of openness for the project
and flexibility for the server
implementations.
We think that Apache Iceberg
REST Catalog spec should not
mandate that a
catalog implementation responds
to requests to produce Auth Tokens
(since the REST spec v1 defines
a /v1/tokens endpoint, current
implementations have to take
deliberate actions when
responding to those
requests, whether with
successful token responses or
with “access
denied” or “unsupported” responses).
We propose the following actions:
1. Immediate mitigation:
1.1. Remove the
‘/v1/oauth/tokens’ endpoint
entirely from the Iceberg’s
OpenAPI spec w/o replacement.
1.2. As long as OAuth2 is the
only mechanism supported by the
Iceberg
client, make the existing client
parameter “oauth2-server-uri”
mandatory. The Iceberg REST
catalog must fail to initialize
if the
“oauth2-server-uri” parameter is
not defined.
1.3. Remove all fallbacks to the
‘/v1/oauth/tokens’ endpoint.
1.4. Forbid or discourage the
communication of tokens from any
Iceberg
REST Catalog endpoint, both via
the "token" property or with any
of the
"urn:ietf:params:oauth:token-type:*"
properties.
2. As a follow up: We’d propose
a couple of implementation fixes
and
changes and test improvements.
3. As a follow up: Define a
discovery mechanism for both the
Iceberg
REST base URI and OAuth 2.0
endpoints/discovery, which
allows users to
use a single URI to securely
access Iceberg REST endpoints.
4. As a follow up: Not new, but
we also want to improve the
Iceberg REST
specification via the “new” REST
proposal.
We do not think that adding
recommendations to
inline-documentation is
enough to fully mitigate the
above concerns.
References:
[1] RFC 6749 - The OAuth 2.0
Authorization Framework,
https://datatracker.ietf.org/doc/html/rfc6749
[2] Iceberg pull request 4771 -
Core: Add OAuth2 to REST catalog
spec -
https://github.com/apache/iceberg/pull/4771
[3] Iceberg pull request 4843 -
Spec: Add more context about
OAuth2 to
the REST spec -
https://github.com/apache/iceberg/pull/4843
--
Robert Stupp
@snazy
