FANNG1 commented on PR #66805: URL: https://github.com/apache/doris/pull/66805#issuecomment-5370182956
Sorry for the delayed reply. There are still a few small things I am fixing; I expect this to be ready to review by the weekend. One design point worth recording here, since it is the thing this PR went back and forth on. The options a namespace vends cannot simply be passed through untouched. Doris normalizes its own S3 configuration onto the canonical AWS names (`aws_access_key_id`, `aws_endpoint`, ...), while neither Lance nor Gravitino's `lance-rest` service constrains the names it returns — Gravitino vends the unprefixed spelling (`access_key_id`, `endpoint`, ...). So a catalog that configures `s3.access_key` and also uses a namespace that vends credentials ends up holding both spellings of one option: ``` aws_access_key_id = <from the catalog> access_key_id = <from the namespace> ``` `object_store` parses both to `AmazonS3ConfigKey::AccessKeyId` and collects them into a `HashMap`, so which one survives is decided by iteration order — separately for the access key and the secret, and separately in the FE and in the BE. Two of the four combinations pair one side's key with the other side's secret, which surfaces as `SignatureDoesNotMatch` rather than as an obvious configuration error. There are two ways to fix it: 1. Doris stops normalizing the AWS-related parameters at all, and it is left to the user to keep the two sides consistent. 2. Doris normalizes what the namespace returns as well, merges it with its own, and hands Lance a single map. This PR takes the second. On any option both sides name, the namespace wins, since it just described the table. On other storage backends: supporting them is worthwhile, but this PR is trying to stay focused on S3. What had to be right now is that non-S3 datasets are *not* rewritten — the resolution is gated on the dataset URL scheme, the same way Lance itself routes a dataset to a provider, so anything that is not `s3`/`s3+ddb` is passed through exactly as the namespace wrote it. Actually translating Doris's own configuration for Azure/GCS/OSS is left for a follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
