LiJie20190102 opened a new issue, #13113:
URL: https://github.com/apache/gravitino/issues/13113
### Version
main branch
### Describe what's wrong
The `printUsage()` output of `IcebergExpireSnapshotsJob`,
`IcebergRewriteDataFilesJob`, and `IcebergUpdateStatsAndMetricsJob` all state:
> Note: Cannot override catalog, extensions, or app name configs
However, the actual code never enforces this restriction. Custom configs
passed via `--spark-conf` are applied directly to `SparkSession.Builder`
without any filtering:
```java
// IcebergExpireSnapshotsJob.java:136-148
// IcebergRewriteDataFilesJob.java:148-160
// IcebergUpdateStatsAndMetricsJob.java:105-110
Map<String, String> customConfigs =
IcebergJobUtils.parseCustomSparkConfigs(sparkConfJson);
for (Map.Entry<String, String> entry : customConfigs.entrySet()) {
sparkBuilder.config(entry.getKey(), entry.getValue());
}
### Error message and/or stacktrace
none
### How to reproduce
# This should fail but currently succeeds silently
bin/run-iceberg-expire-snapshots.sh \
--catalog iceberg_prod --table db.sample \
--spark-conf
'{"spark.sql.extensions":"org.apache.spark.sql.SparkSessionExtensions"}'
### Additional context
- The reserved keys are already defined in
`IcebergSparkConfigUtils.buildTemplateSparkConfigs()`: `spark.master`,
`spark.sql.extensions`, `spark.sql.catalog.*`
- `IcebergJobUtils.parseCustomSparkConfigs` is the natural place to add the
filter
- Additionally, `--older-than` timestamps and `--table` identifiers are not
validated before being interpolated into SQL, which is inconsistent with the
existing `validateRetainLast()` / `validateStrategy()` validators
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
