DanielLeens commented on issue #12766: URL: https://github.com/apache/seatunnel/issues/12766#issuecomment-6595079693
Thanks @CryoThrust. The per-instance UGI approach is the right direction. I found no competing Hudi Kerberos PR, so please go ahead with one focused PR linked here. I checked the proposal against `dev`, and three things should change before you start. **1. Option names.** `use.kerberos` / `kerberos.principal` / `kerberos.principal.file` come from the JDBC connector (`JdbcCommonOptions`). Hudi's own options use snake_case (`conf_files_path`, `table_dfs_path`). The Hadoop-family connectors all use `kerberos_principal`, `kerberos_keytab_path`, and `krb5_path`: see `FileBaseOptions`, `HiveBaseOptions`, and `IcebergCommonOptions`. The reporter's working Iceberg job uses exactly those keys. Hudi has never declared the dotted keys, so there is no compatibility reason to adopt them. Please use the Hadoop-family names, enable login only when both principal and keytab are set, and add both to the sink and `HudiCatalogFactory` option rules. If you want the reporter's existing config to keep working, register the dotted keys with `withFallbackKeys`. Either way, the docs should name the canonical keys. **2. Security mode (your question 4).** I would not add a worker-startup prerequisite that the sibling connectors do not have. The existing precedent is `HadoopLoginFactory.loginWithKerberos` in connector-file-base, used by `HadoopFileSystemProxy`. It requires `hadoop.security.authentication=kerberos` in the supplied configuration and fails clearly otherwise. It then runs `UserGroupInformation.setConfiguration` and `loginUserFromKeytabAndReturnUGI` under `synchronized (UserGroupInformation.class)` and returns the UGI for `doAs`. That is how HdfsFile coexists with other jobs in the same worker today. Please follow the same contract using the configuration loaded from `conf_files_path`. Use a small Hudi-local helper rather than a new dependency on connector-file-base. **3. Coverage.** Hadoop's `FileSystem` cache is keyed by UGI, so each client and its filesystem have to be created and used under the same UGI. These paths are in scope: - `HudiCatalog.open()` (`HadoopFSUtils.getFs`), which also runs where save mode is handled, so the keytab must be readable there too; - `HudiUtil.createHoodieJavaWriteClient`, reached from both `HudiClientManager.getClient` and `HudiWriteClientProvider`; - write, commit, and close in `HudiRecordWriter`. Please leave the unused `HudiUtil.initKerberosAuthentication` uncalled, because it uses the JVM-global `loginUserFromKeytab`. Also note that open PR #12440 (Hudi exactly-once) touches `HudiSinkOptions`, `HudiSinkConfig`, `HudiUtil`, and `HudiRecordWriter`, and adds an aggregated committer. Whichever PR merges second must also cover that commit path. Your test plan in point 5 looks right. Please also add a negative test where the configuration does not enable Kerberos while the options are set. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
