cshuo opened a new pull request, #20243: URL: https://github.com/apache/hudi/pull/20243
### Describe the issue this Pull Request addresses Closes #20231. Add a Helm-deployable Spark Connect endpoint under `hudi-lakehouse`, following [discussion #19843](https://github.com/apache/hudi/discussions/19843). Direct ADBC clients can read and write Hudi tables against a shared catalog and object store without Spark Thrift Server. Gateway and BI adapters are separate work. ### Summary and Changelog - Add a release-pinned image: Spark 4.1.3, Hudi 1.2.0, Scala 2.13, and Java 17, with S3A dependencies and the Parquet variant jar needed by Hudi writes. - Add the `hudi-spark-connect` chart with external Hive Metastore/S3 configuration, existing Secret references, resources, ClusterIP service, probes, and config-driven rollouts. Preserve backslashes and decimal integer formatting in Spark properties. - Make `quickstart.sh --spark-connect` deploy MinIO, HMS, and Connect independently of the legacy Trino/writer build chain; no host Maven or JDK is required. Existing releases are not uninstalled. - Add a Foundry ADBC CRUD/restart smoke test, partial-failure cleanup, offline chart/runner tests, and CI coverage. Validation: - 30 offline tests passed, including startup paths, configuration rendering, Java properties round-trips, and cleanup after partial preparation. - Helm lint, shell syntax checks, `git diff --check`, and `mvn -N -o apache-rat:check -DskipTests -Dstyle.color=never` passed. - Earlier local validation built the pinned image on Linux ARM64 and deployed it to an isolated kind cluster with HMS/MinIO. Foundry Spark ADBC 0.2.1 passed CREATE TABLE, SELECT, INSERT, UPDATE, DELETE, and readback after an endpoint restart. Failure injection also verified cleanup after a failed write. - The full minikube startup has not been run locally; its orchestration is covered by mocked command tests. Other CPU architectures and production load/security scenarios remain unverified. ### Impact Adds an opt-in endpoint running one shared Spark driver in local mode; it does not use Spark Operator or distributed executor pods. Catalog metadata and table data live outside the endpoint pod. A restart drops active sessions, so clients must reconnect. `--spark-connect` selects the independent stack rather than adding Connect to the full stack. Default Trino/writer behavior is unchanged; its existing stale build chain is not repaired here. The gateway backend and write guardrails are unchanged. ADBC drivers are installed on clients, not the server. ### Risk Level Medium. The endpoint is shared and has no client authentication or TLS. Anyone with network access can use its storage permissions, including writes. It defaults to ClusterIP and is intended for trusted development networks, not public exposure. Offline checks and local CRUD/restart tests cover the functional path; authentication, isolation, concurrency, cancellation, and large-result handling need further validation. ### Documentation Update Update the lakehouse/local-dev READMEs and add a chart README covering pinned versions, standalone startup, catalog/storage/Secret configuration, client installation, CRUD/restart checks, teardown, and security/runtime limits. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
