jerryshao commented on issue #12690:
URL: https://github.com/apache/gravitino/issues/12690#issuecomment-5618632000
I cross-checked the draft 1.3.1 connector release notes against the actual
code/docs already merged into `branch-1.3` and found the three behavior changes
from this issue's checklist were present in the code but not yet called out as
behavior changes (with upgrade actions and escape hatches) in the notes. I also
found one more undocumented behavior change that landed in `branch-1.3` today
(#13040): an unconstrained PostgreSQL `NUMERIC` column is now read as a string
instead of a decimal type in the Trino, Spark, and Flink connectors.
Below is a draft rewrite of the connector sections with a "Behavior changes"
callout up front in each, plus issue references. Posting here for review before
it goes into the actual release notes doc.
---
## Trino connector 1.3.1 changes
### Behavior changes (read before upgrading)
- **Non-REST `lakehouse-iceberg` catalogs are now routed through the
Gravitino Iceberg REST service (IRC) by default.** (#12554) A metalake with
`catalog-backend=jdbc` or `hive` Iceberg catalogs changes behavior on upgrade:
the connector now builds an internal Iceberg REST catalog against Gravitino's
IRC instead of translating `catalog-backend` into Trino's
`jdbc`/`hive_metastore` catalog type. Set
`gravitino.iceberg.rest-routing-enabled=false` to keep the previous behavior.
- **If the IRC has authentication enabled, it now requires its own
credential, configured separately on every Trino node.** (#12554) This
credential is not derived from Gravitino catalog properties and does not travel
with the generated `CREATE CATALOG` statement — it must be set locally in
`catalog.properties` (or via Trino's `${ENV:...}` secret substitution) on the
**coordinator and every worker** before upgrading:
```properties
gravitino.iceberg.rest-catalog.security=OAUTH2
gravitino.iceberg.rest-catalog.oauth2.credential=${ENV:IRC_OAUTH_CREDENTIAL}
gravitino.iceberg.rest-catalog.oauth2.server-uri=http://your-idp/token
gravitino.iceberg.rest-catalog.oauth2.scope=email
```
Missing this on any node does not fail `CREATE CATALOG` — the catalog
registers and lists normally. It only surfaces as an authentication error the
first time a query on that node reaches the IRC, which is exactly the "looks
healthy, fails later" pattern this issue is about.
- **The internal JDBC connection credential (`trino.jdbc.password`) is no
longer propagated into the catalogs the connector creates.** (#12542) It no
longer reaches connector logs or the generated catalog properties file.
- **An unconstrained PostgreSQL `NUMERIC` column (no precision/scale) is now
read as a string** (#13040) instead of `Decimal(38, 18)`. Queries doing
arithmetic, aggregation, or comparison on such a column will need to cast it
explicitly after upgrading. (Same change applies to the Spark and Flink
connectors.)
### Functional changes
- Added Starburst compatibility. (#12526)
- Added TLS support for the connector's internal JDBC connection. (#12542)
- Added support for activating Trino roles on the internal JDBC connection.
(#12542)
- Added `trino.jdbc.properties.*` passthrough. (#12542)
- Extended session user forwarding to OAuth2, with per-user credential
vending for IRC-backed Iceberg. (PR #12045, tagged `[MINOR]` — no separate
tracking issue)
- Added automatic discovery of the Iceberg REST service endpoint. (#12554)
- Added `system.catalog_status`. (#12546)
- Added `system.load_status`. (#12546)
- Fixed projection pushdown to preserve engine-assigned column types.
(#12518)
### Supporting changes
- Replaced SLF4J with Airlift Logger. (#12634)
- Upgraded the PostgreSQL JDBC driver from 42.7.0 to 42.7.11. (#11795)
- Added an end-to-end integration test for TLS + OAuth2 + IRC credential
vending. (#12712)
- Updated the Trino Iceberg REST documentation. (#12554)
---
## Spark connector 1.3.1 changes
### Behavior changes (read before upgrading)
- **Non-REST `lakehouse-iceberg` catalogs are now routed through the IRC by
default** (#12709), matching Trino. Set
`spark.sql.gravitino.iceberg.rest-routing-enabled=false` to keep the previous
behavior.
- **When no IRC endpoint can be discovered, the connector now fails with an
actionable error instead of silently falling back to Hive/JDBC translation.**
(#12709) Set `spark.sql.gravitino.iceberg.rest-routing-enabled=false` to
restore the old fallback.
- **If the Gravitino client uses OAuth2, the connector reuses that
configuration for the IRC connection by default**
(`spark.sql.gravitino.iceberg.reuseOAuth2=true`, #12709). Set
`reuseOAuth2=false` to configure the IRC's credential separately.
- **An unconstrained PostgreSQL `NUMERIC` column is now read as a string**
(#13040) — same as the Trino entry above.
---
Related: #12687 (Spark connector IRC routing items, closed) covers the
underlying fixes for the Spark behavior changes above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
