jiangxt2 opened a new pull request, #13493:
URL: https://github.com/apache/gravitino/pull/13493
# [#13489] feat(spark-connector): Add the governed Doris batch-read baseline
### What changes were proposed in this pull request?
This pull request adds the first governed Apache Doris batch-read baseline
for the Gravitino Spark connector.
- Add the explicit `spark.sql.gravitino.enableDorisSupport` opt-in for Spark
3.5.
- Preserve the existing generic JDBC path when the opt-in is disabled or
when another JDBC provider is used.
- Add the specialized read-only Doris catalog facade with vended JDBC
credential enforcement.
- Reject protected connection and credential overrides while preserving
ordinary Spark JDBC read options.
- Validate ordinary Doris tables against both FE `SHOW COLUMNS` metadata and
JDBC metadata, including column order, names, ordinary type signatures, and
nullability.
- Fail closed for unsupported special types and metadata mismatches.
- Reject Spark writes and catalog DDL through the specialized Doris path.
- Add documentation and focused unit/Doris integration coverage.
This PR does not add Doris special-type normalization, the native tablet
reader, Doris-specific semantic routing, partition planning, or write support.
### Why are the changes needed?
The parent issue is being delivered as independently reviewable read
contributions. This PR establishes the governed JDBC read path that later
Doris-specific capabilities can extend while keeping generic JDBC behavior
unchanged by default.
The specialized path uses MySQL Connector/J to access the Doris FE MySQL
protocol. The driver and other required runtime dependencies remain external to
the connector artifact.
Fix: #13489
Part of: #12846
### Does this PR introduce _any_ user-facing change?
Yes.
- Adds the `spark.sql.gravitino.enableDorisSupport` Spark configuration key.
Its default is `false`.
- When enabled for a `jdbc-doris` catalog, the connector selects the
governed read-only Doris path.
- The specialized path requires a vended JDBC credential and catalog-managed
JDBC URL and driver properties.
- Protected connection and credential overrides supplied through Spark
catalog options are rejected.
- Writes and Spark catalog DDL are rejected for the specialized Doris path.
- Generic JDBC behavior for other providers and for `jdbc-doris` without the
opt-in remains unchanged.
### How was this patch tested?
- `git diff --check` passed.
- Earlier targeted Spark 3.5.3 / Scala 2.12 / Doris 3.0.6.2 validation
passed four Doris integration tests covering catalog registration, scalar read,
write-aware load rejection, and unsupported JSON rejection.
- The current worktree includes subsequent ordinary type-signature,
nullability, and nullable-fixture changes; the targeted unit and Doris
integration suites must be rerun against this final draft before the PR is
published.
Commands for the final validation:
```bash
./gradlew :spark-connector:spark-3.5:test \
--tests
org.apache.gravitino.spark.connector.plugin.TestGravitinoDriverPlugin \
--tests
org.apache.gravitino.spark.connector.jdbc.doris.TestDorisPropertiesConverter35 \
--tests
org.apache.gravitino.spark.connector.jdbc.doris.TestDorisPhysicalSchemaValidator35
\
--tests
org.apache.gravitino.spark.connector.jdbc.doris.TestGravitinoDorisCatalogSpark35
\
-PskipITs
./gradlew :spark-connector:spark-3.5:test \
--tests
org.apache.gravitino.spark.connector.integration.test.jdbc.SparkJdbcDorisReadBaselineIT35
\
-PskipDockerTests=false -PtestMode=embedded
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
