snmvaughan commented on PR #6509: URL: https://github.com/apache/datafusion-comet/pull/6509#issuecomment-5937062286
Thanks for the numbers, @andygrove. Your review crossed with d9e5ff4f2, which takes a different route from an uncacheable state. Each bridge still makes one provider call at a time, but the requests that waited for it now share its outcome, a credential or an error, even when the credential can't be kept. A bridge is one policy location on a location-scoped store, and one bucket or table otherwise, so a burst on one location costs one call while different locations never wait on each other. With a provider that reports `0`, eight overlapping requests now share one 5 ms call instead of making eight in turn, so your S3Proxy scan should be back near main's time. Could you re-run it on 9ecdbcd38? On the async lock: a waiter blocks its worker for at most one provider call, about what each request blocked for on main with its own JNI call, so I left it as is. I'm happy to move to an async wait like `SharedEntry` here or in a follow-up if you'd prefer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
