mengw15 opened a new issue, #6150:
URL: https://github.com/apache/texera/issues/6150
### Task Summary
`storage.iceberg.catalog.type` accepts `hadoop`, `rest`, or `postgres`.
Production uses `rest` (Lakekeeper); `postgres` is the local/CI backend. The
`hadoop` type — an Iceberg `HadoopCatalog` — appears unused and is worth
dropping:
- **Not configured / not selected anywhere.** `storage.conf` has `rest { …
}` and `postgres { … }` blocks but no `hadoop { … }` block; `hadoop` only
appears in the `# either hadoop, rest, or postgres` comment. No deployment
(prod / local-dev / single-node) selects it.
- **Only safe on HDFS, which we don't use.** The [Iceberg
docs](https://iceberg.apache.org/docs/latest/java-api-quickstart/) state a
Hadoop catalog "can only be used with HDFS or similar file systems that support
atomic rename." Texera stores Iceberg data on S3/MinIO and never on HDFS —
exactly the object-store case that lacks atomic rename — so `hadoop` was never
a viable backend here. (Not formally `@Deprecated` in Iceberg 1.7.1, just
documented as unsuitable.)
- **Confirmed in practice.** Evaluating `hadoop` as the unit-job catalog in
#6049 flooded `CommitFailedException: Version N already exists` / `Unable to
acquire lock` under amber's concurrent result-port commits.
- **Less to maintain** — one fewer catalog backend.
**Seeking input:** does anyone rely on the `hadoop` catalog type? If not,
we'll remove it.
### Scope
- `IcebergCatalogInstance`: drop the `case "hadoop" =>
IcebergUtil.createHadoopCatalog(…)` branch (the `case _` already rejects
unsupported types).
- `IcebergUtil`: remove `createHadoopCatalog`; narrow the import
`{HadoopCatalog, HadoopFileIO}` → `HadoopFileIO` (still used by the
`rest`/`postgres` catalogs).
- `storage.conf`: update the comment to `# either rest or postgres`.
- No tests or docs reference the type (confirmed by grep).
Suggested by @Yicong-Huang in #6049.
### Task Type
- [x] Refactor / Cleanup
- [ ] DevOps / Deployment / CI
- [ ] Testing / QA
- [ ] Documentation
- [ ] Performance
- [ ] Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
