kgyrtkirk opened a new pull request, #19249: URL: https://github.com/apache/druid/pull/19249
I've investigated some S3 related issues which appeared agains MiniIO in a forked version of Druid. After reproducing the issue consistently it was clear that disabling `druid.storage.transfer.useTransferManager` fixes the issue - but beyond disabling it I also checked that utilizing the [crt](https://docs.aws.amazon.com/sdkref/latest/guide/common-runtime.html) backed client also works without issues. Also with netty even with S3 sometime a similar 400 happened. Because my experience was that `crt` just worked more reliably all the time - I think it should be the default. * adds config option `druid.storage.transfer.asyncHttpClientType` * `crt` (default) uses aws crt lib * `netty` (old) netty-nio backed client * some stability relate scripts/wfs to validate test stability in the future exceptions: * for minio were like this ([full logs are here](https://gist.github.com/kgyrtkirk/53f1740a43ce99c504f116c3fd0cb395)) ``` java.lang.RuntimeException: java.util.concurrent.CompletionException: software.amazon.awssdk.services.s3.model.S3Exception: (Service: S3, Status Code: 400, Request ID: null) (SDK Attempt Count: 1) ``` * for aws ([full logs are here](https://gist.github.com/kgyrtkirk/b3980d438430f5c65865e93e0dd91ab5)) ``` java.lang.RuntimeException: java.util.concurrent.CompletionException: software.amazon.awssdk.services.s3.model.S3Exception: (Service: S3, Status Code: 400, Request ID: C0BFF04967DEC3EB) (SDK Attempt Count: 1) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
