jerryshao opened a new pull request, #13563: URL: https://github.com/apache/gravitino/pull/13563
### What changes were proposed in this pull request? > This PR is stacked on #13468 and stays a draft until that PR merges. To see only the changes of this PR, use this compare view: https://github.com/jerryshao/gravitino/compare/13385-job-template-placeholder-defaults...13555-job-resource-utils - `JobTemplateResolver.resolve(jobConf)` now only resolves the placeholders of a job template. The resources of the runtime job template (executable, scripts, jars, files, archives) are kept as their resolved URIs. It no longer takes a staging directory. - New `JobResourceUtils` in `org.apache.gravitino.connector.job` (`@DeveloperApi`) fetches the resources of a runtime job template into a local directory: - `localizeJobTemplate(jobTemplate, dir)` returns a copy of the job template whose resources point to the fetched local files. - `fetchFiles(uris, dir, timeoutInMs)` and `fetchFile(uri, dir, timeoutInMs)` fetch individual files. - The fetch logic is moved as is from `JobTemplateResolver`. - `JobManager` resolves the template and then fetches its resources into the job's staging directory: - Missing parameters are still rejected before the staging directory is created. - A failed fetch still cleans up the staging directory. ### Why are the changes needed? This is the first step of #13554, which adds a production job executor on Kubernetes. An executor that runs jobs outside the Gravitino server needs the original resource URIs rather than paths on the server, so fetching must become a step that each job executor can choose to use. This PR only separates the two steps. Behavior is unchanged: `JobManager` still fetches every resource before submitting the job. Fix: #13555 ### Does this PR introduce _any_ user-facing change? No user-facing change. `JobTemplateResolver.resolve` changes its signature, but the class is only used inside `core`. `JobResourceUtils` is a new developer API for job executor implementations. ### How was this patch tested? - New `TestJobResourceUtils`: - `testLocalizeShellJobTemplate` and `testLocalizeSparkJobTemplate` check that every resource is fetched and the other fields are kept. - The existing fetch tests move here from `TestJobTemplateResolver` as `testFetchFile*`: local files, a missing file, SSRF blocking, and loopback when blocking is disabled. - New `TestJobTemplateResolver#testResolveKeepsResourceUris` checks that resolving keeps resources as URIs. - `TestJobManager` and `TestLocalJobExecutor` now resolve and localize explicitly. - `./gradlew :core:test --tests 'org.apache.gravitino.job.*' --tests 'org.apache.gravitino.connector.job.*' -PskipITs`: 212 tests pass. - `:core:spotlessCheck` and `:core:javadoc` pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
