surawut-jirasaktavee opened a new pull request, #7680:
URL: https://github.com/apache/datafusion-comet/pull/7680
## Which issue does this PR close?
Closes #7667.
## Rationale for this change
The Iceberg docs' assume-role setup for Spark on EKS (`io-impl=S3FileIO`,
`client.factory=org.apache.iceberg.aws.AssumeRoleAwsClientFactory`,
`client.assume-role.arn`, `client.assume-role.region`) makes every native
Iceberg write fall back:
```
IcebergWrite [COMET: unsupported S3 FileIO settings:
client.assume-role.region, client.factory]
```
The native writer already supports `client.assume-role.arn`, `external-id`
and `session-name`. The fallback comes from the two keys iceberg-java uses to
turn that path on: `client.factory`, without which iceberg-java ignores the
arn, and `client.assume-role.region`, which the pinned iceberg-rust does not
read.
## What changes are included in this PR?
- `client.factory` and `client.assume-role.region` are admitted only when
all of these hold:
- the factory is exactly
`org.apache.iceberg.aws.AssumeRoleAwsClientFactory`;
- `client.assume-role.arn` and `client.assume-role.region` are set and not
blank;
- the data location is written through `S3FileIO`, including a
`ResolvingFileIO` that resolves to it. `HadoopFileIO` ignores the factory and
writes through S3A, so it still falls back.
- The assume-role region is passed to the native writer as `client.region`
when neither `client.region` nor `s3.region` is set.
- With the factory, the write still falls back on settings the native writer
would read but the factory's S3 client ignores, and the fallback reason names
the key:
- a `client.region` or `s3.region` different from the assume-role region;
- `s3.access-key-id`, `s3.secret-access-key` or `s3.session-token`
(opendal would use them as the source credentials for AssumeRole);
- a Comet credential provider class;
- `s3.allow-anonymous`, `s3.disable-ec2-metadata` or
`s3.disable-config-load` set to `true`.
- Other assume-role keys the native writer cannot honour, such as
`client.assume-role.tags.*` and `client.assume-role.timeout-sec`, still fall
back.
- The Iceberg user guide describes the rule.
## How are these changes tested?
- New unit tests in `CometIcebergWriteDetectionSuite` cover the admitted
configuration, the `HadoopFileIO` fallback, each conflicting key, other factory
classes, missing or blank arn or region, and the region mapping. The suite
passes (93 tests).
- End to end on EKS (Spark 4.1, Iceberg 1.11, Glue catalog, Pod Identity
holding only `sts:AssumeRole`, the configuration above):
- Comet 1.2.0 falls back with the message above.
- With this branch the plan uses `CometIcebergWrite` with no fallback, on
a single pod and on 3 executors in client mode (12 write tasks, 0 failures).
Reading back 200,000 rows gives the same count, sum and distinct count as
written.
- CloudTrail shows the native AssumeRole calls (session name `reqsign`)
all succeeding against the configured role.
Two follow-ups found during the end-to-end test, not changed here:
- The native Iceberg scan does not get the assume-role region. Without
`AWS_REGION` in the environment, reading the table goes to `us-east-1` and
fails with status 400. Pods that set `AWS_REGION` are not affected.
- The native path makes one STS AssumeRole call per data file (280 calls for
140 files written and read back), where iceberg-java reuses one session per
client. Large writes could hit STS throttling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]