Hey Robert, Sorry for the late reply as I was out last week. I'm not an OAuth guru either, but some context from my end.
* Credentials (for example username/password) must _never_ be sent to > the resource server, only to the authorization server. In an earlier discussion <https://github.com/apache/iceberg/pull/8976>, it was agreed that the resource server can also function as the authorization server. But the roles can also be separate. 1.2. As long as OAuth2 is the only mechanism supported by the Iceberg > client, make the existing client parameter “oauth2-server-uri” > mandatory. The Iceberg REST catalog must fail to initialize if the > “oauth2-server-uri” parameter is not defined. It can also be that there is no authentication in the case of an internal REST catalog. For example, the iceberg-rest-image <https://github.com/tabular-io/iceberg-rest-image> that we use for integration tests in PyIceberg. We think that Apache Iceberg REST Catalog spec should not mandate that a > catalog implementation responds to requests to produce Auth Tokens > (since the REST spec v1 defines a /v1/tokens endpoint, current > implementations have to take deliberate actions when responding to those > requests, whether with successful token responses or with “access > denied” or “unsupported” responses). The `/v1/tokens` endpoint is optional <https://github.com/apache/iceberg-python/blob/756ae625a2ea0f9c12df78430512ce991f6a1976/pyiceberg/catalog/rest.py#L488-L489> . * Credentials (for example username/password) must _never_ be sent to > the resource server, only to the authorization server. I fully agree! Even if an Iceberg REST server does not implement the ‘/v1/oauth/tokens’ > endpoint, it can still receive requests to ‘/v1/oauth/tokens’ containing > clear text credentials, if clients are misconfigured (humans do make > mistakes) - it’s a non-zero risk - bad actors can implement/intercept > that ‘/v1/oauth/tokens’ endpoint and just wait for misconfigured > clients to send credentials. I think the wording is chosen badly. It should not send any credentials, but the code (as in this example <https://developers.google.com/identity/protocols/oauth2#installed> by GCS). I think Jack makes a good point with AWS SigV4 Authentication. I suppose, > in REST Catalog implementations that support that auth method, the > /v1/oauth/token Catalog REST endpoint is redundant. > There are other cloud providers next to AWS. Kind regards, Fokko Op do 23 mei 2024 om 15:49 schreef Dmitri Bourlatchkov <dmitri.bourlatch...@dremio.com.invalid>: > I think Jack makes a good point with AWS SigV4 Authentication. I suppose, > in REST Catalog implementations that support that auth method, the > /v1/oauth/token Catalog REST endpoint is redundant. > > Cheers, > Dmitri. > > On Thu, May 23, 2024 at 9:20 AM Jack Ye <yezhao...@gmail.com> wrote: > >> I do not know enough details about OAuth to make comments about this >> issue, but just regarding the statement "OAuth2 is the only mechanism >> supported by the Iceberg client", AWS Sigv4 auth is also supported, at >> least in the Java client implementation >> <https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/HTTPClient.java#L72>. >> It would be nice if we formalize that in the spec, at least define it as a >> generic authorization header. >> >> Best, >> Jack Ye >> >> >> >> On Thu, May 23, 2024 at 2:51 AM Robert Stupp <sn...@snazy.de> wrote: >> >>> Hi all, >>> >>> Iceberg REST implementations, either accessible on the public internet >>> or inside an organization, are usually being secured using appropriate >>> authorization mechanisms. The Nessie team is looking at implementing the >>> Iceberg REST specification and have some questions around the security >>> endpoint(s) defined in the spec. >>> >>> TL;DR we have questions (potentially concerns) about having the >>> ‘/v1/oauth/tokens’ endpoint, for the reasons explained below. We think >>> that ‘/v1/oauth/tokens’ poses potential security and OAuth2 compliance >>> issues, and imposes how authorization should be implemented. >>> * As an open table format, it would be good for Iceberg to focus on the >>> table format / catalog and not how authorization is implemented. The >>> existence of an OAuth endpoint pushes implementations to adopt >>> authorization using only OAuth, whereas the implementers might choose >>> several other ways to implement authorization (e.g. SAML). In our >>> opinion the spec should leave it open to the implementation to decide >>> how authorization will be implemented. >>> * The existence of that endpoint also pushes operators of Iceberg REST >>> endpoints into the authorization service business. >>> * Clients might expose their clear-text credentials to the wrong >>> service, if the (correct) OAuth endpoint is not configured (humans do >>> make mistakes). >>> * (Naive) Iceberg REST servers may proxy requests received for >>> ‘/v1/oauth/tokens’ - and effectively become a “man-in-the-middle”, which >>> is not fully compliant with the OAuth 2.0 specification. >>> >>> Our goals with this discussion are: >>> 1. Secure the Iceberg REST specification by preventing accidental >>> misuse/misimplementation. >>> 2. Prevent that Iceberg REST to get into dictating the “authorization >>> server specifics”. >>> 3. Enable flexibility for Iceberg REST servers to opt for other >>> authorization mechanisms than OAuth 2.0. >>> 4. Enable REST servers to opt for integrating with any standard OAuth2 / >>> OIDC provider (e.g. Okta, Keycloak, Authelia). >>> >>> OAuth 2.0 [1] is one of the common standards accepted in the industry. >>> It defines a secure mechanism to access resources (here: Iceberg REST >>> endpoints). The most important aspect for OAuth 2.0 resources is that >>> (Iceberg REST) servers do not (have to) support password authentication, >>> especially considering the security weaknesses inherent in passwords. >>> Compromised passwords result in compromised data protected by that >>> password. >>> >>> Therefore OAuth 2.0 defines a set of strict rules. Some of these are: >>> * Credentials (for example username/password) must _never_ be sent to >>> the resource server, only to the authorization server. >>> * OAuth 2.0 refresh tokens must _never_ be sent to the resource server, >>> only to the authorization server. (“Unlike access tokens, refresh tokens >>> are intended for use only with authorization servers and are never sent >>> to resource servers.”, cite from section 1.5 of the OAuth RFC 6749.) >>> * While the OAuth RFC states "The authorization server may be the same >>> server as the resource server or a separate entity", this should not be >>> mandated. i.e the spec should be open to supporting implementations that >>> have the authorization server and resource server co-located as well as >>> separate. >>> >>> The Iceberg PR 4771 [2] added the OpenAPI path ‘/v1/oauth/tokens’, >>> intentionally marked to “To exchange client credentials (client ID and >>> secret) for an access token. This uses the client credentials flow.” >>> [3]. Technically: client ID and secret are submitted using a HTTP POST >>> request to that Iceberg REST endpoint. >>> >>> Having ‘/v1/oauth/tokens’ in the Iceberg REST specification can easily >>> be seen as a hard requirement. In order to implement this in compliance >>> with the OAuth 2.0 spec, that ‘/v1/oauth/tokens’ MUST be the >>> authorization server. If users do not (want to) implement an >>> authorization server, the only way to implement this ‘/v1/oauth/tokens’ >>> endpoint would be to proxy ‘/v1/oauth/tokens’ to the actual >>> authorization server, which means, that this proxy technically becomes a >>> “man in the middle” - knowing both all credentials and all involved >>> tokens. >>> >>> Even if an Iceberg REST server does not implement the ‘/v1/oauth/tokens’ >>> endpoint, it can still receive requests to ‘/v1/oauth/tokens’ containing >>> clear text credentials, if clients are misconfigured (humans do make >>> mistakes) - it’s a non-zero risk - bad actors can implement/intercept >>> that ‘/v1/oauth/tokens’ endpoint and just wait for misconfigured >>> clients to send credentials. >>> >>> Further usages of the REST Catalog API path ‘/v1/oauth/tokens’ are “To >>> exchange a client token and an identity token for a more specific access >>> token. This uses the token exchange flow.” and “To exchange an access >>> token for one with the same claims and a refreshed expiration period >>> This uses the token exchange flow.” Both usages should and can be >>> implemented differently. >>> >>> Apache Iceberg, as a table format project, should recommend protecting >>> sensitive information. But Iceberg should not mandate _how_ that >>> protection is implemented - but the Iceberg REST specification does >>> effectively mandate OAuth 2.0, because other Iceberg REST endpoints do >>> refer/require OAuth 2.0 specifics. Users that want to use other >>> mechanisms, because they are forced to do so by their organization, >>> would be locked out of Iceberg REST. >>> >>> Apache Iceberg should not mandate OAuth 2.0 as the only option - for the >>> sake of openness for the project and flexibility for the server >>> implementations. >>> >>> We think that Apache Iceberg REST Catalog spec should not mandate that a >>> catalog implementation responds to requests to produce Auth Tokens >>> (since the REST spec v1 defines a /v1/tokens endpoint, current >>> implementations have to take deliberate actions when responding to those >>> requests, whether with successful token responses or with “access >>> denied” or “unsupported” responses). >>> >>> We propose the following actions: >>> 1. Immediate mitigation: >>> 1.1. Remove the ‘/v1/oauth/tokens’ endpoint entirely from the Iceberg’s >>> OpenAPI spec w/o replacement. >>> 1.2. As long as OAuth2 is the only mechanism supported by the Iceberg >>> client, make the existing client parameter “oauth2-server-uri” >>> mandatory. The Iceberg REST catalog must fail to initialize if the >>> “oauth2-server-uri” parameter is not defined. >>> 1.3. Remove all fallbacks to the ‘/v1/oauth/tokens’ endpoint. >>> 1.4. Forbid or discourage the communication of tokens from any Iceberg >>> REST Catalog endpoint, both via the "token" property or with any of the >>> "urn:ietf:params:oauth:token-type:*" properties. >>> 2. As a follow up: We’d propose a couple of implementation fixes and >>> changes and test improvements. >>> 3. As a follow up: Define a discovery mechanism for both the Iceberg >>> REST base URI and OAuth 2.0 endpoints/discovery, which allows users to >>> use a single URI to securely access Iceberg REST endpoints. >>> 4. As a follow up: Not new, but we also want to improve the Iceberg REST >>> specification via the “new” REST proposal. >>> >>> We do not think that adding recommendations to inline-documentation is >>> enough to fully mitigate the above concerns. >>> >>> >>> References: >>> >>> [1] RFC 6749 - The OAuth 2.0 Authorization Framework, >>> https://datatracker.ietf.org/doc/html/rfc6749 >>> [2] Iceberg pull request 4771 - Core: Add OAuth2 to REST catalog spec - >>> https://github.com/apache/iceberg/pull/4771 >>> [3] Iceberg pull request 4843 - Spec: Add more context about OAuth2 to >>> the REST spec - https://github.com/apache/iceberg/pull/4843 >>> >>> -- >>> Robert Stupp >>> @snazy >>> >>>
