jerryshao opened a new pull request, #13468:
URL: https://github.com/apache/gravitino/pull/13468
### What changes were proposed in this pull request?
- Placeholders take a default value, written as `{{name:-default}}`, or
`{{name:-}}` for an empty one. A value is resolved from the job configuration
first, where an explicit `""` counts as a value and `null` counts as absent,
and from the default otherwise. A default declared on one occurrence applies to
every occurrence of that parameter.
- A placeholder with neither a value nor a default now fails the run with an
error listing every missing parameter. This happens before the staging
directory is created or any file is fetched, so a rejected run leaves nothing
behind.
- `\{{` escapes a literal `{{`, for templates that carry another tool's
syntax.
- A default value is used as is, may span lines, and may contain braces as
long as they are balanced, so a JSON object works as a default. Malformed
placeholders are rejected when a template is registered or updated.
- Job configuration keys the template does not use are logged as a warning.
- Placeholder parsing moves out of `JobManager` into
`JobTemplatePlaceholderUtils`, and the runtime job template is built by
`JobTemplateResolver`, which parses and validates a template once per job run.
`JobManager` depends only on the resolver.
- Built-in templates get default values and move to `v2`. The
expire-snapshots `--stream-results` argument now takes a `true`/`false` value,
and Spark resource defaults such as `spark_master` move from the optimizer's
code into the templates. The catalog connection keys and the rewrite
`where_clause` stay required.
### Why are the changes needed?
Every placeholder of a job template had to be supplied in `jobConf`. A
missing key was passed through as the literal `{{name}}`, which then failed
inside the job: `--strategy {{strategy}}` failed validation and
`spark.master={{spark_master}}` broke spark-submit. Callers such as the
optimizer therefore had to send every key, using `""` for the optional ones,
and keep the defaults in their own code.
Fix: #13385
### Does this PR introduce _any_ user-facing change?
- A run whose job configuration misses a required parameter is rejected with
HTTP 400 listing the missing parameters, instead of failing inside the job.
- **Behavior change**: a template that relied on an unresolved placeholder
being passed through as literal text, which is how templates carried another
tool's syntax such as `{{ds}}` or `{{.Values.image}}`, now fails every run.
Such text has to be escaped as `\{{ds}}`, or the parameter has to be given a
default value. This is called out in the docs.
- Built-in Iceberg and SparkPi templates move to `v2` and are refreshed on
startup. `jobConf` now only needs the required keys: `catalog_name`,
`table_identifier`, the catalog connection keys, and `where_clause` for
rewrite-data-files.
- `stream_results` is now passed as `true`/`false`. A bare
`--stream-results` on the command line still means true.
- No new configuration keys, no REST or client API change.
### How was this patch tested?
- New `TestJobTemplatePlaceholderUtils` (20 cases) covers the syntax:
default values, JSON and multi-line defaults, escapes, malformed placeholders,
text that only looks like a placeholder, and the compatibility break above.
- New `TestJobTemplateResolver` (renamed from `TestJobTemplate`) covers
building the runtime template, `checkJobConf`, and that nothing is fetched when
a parameter is missing.
- `TestJobManager` covers `runJob` with defaults, rejection before the
staging directory is created, staging cleanup when resolution fails, and
rejection of malformed templates at register and alter.
- Updated the built-in job tests for the `v2` templates, and added one for
`--stream-results` parsing.
- `./gradlew :core:test :maintenance:jobs:test
:maintenance:optimizer-api:test :maintenance:optimizer:test -PskipITs` — all
pass (2315 / 144 / 16 / 233). `spotlessCheck` and `:core:javadoc` pass.
Integration tests were not run; the job ITs all supply every placeholder they
declare.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
