Hi,
Le 02/12/2024 à 08:12, Jerry Adair a écrit :
Hi Weston,
Thank you for the reply.
IIRC, this is a limitation given to use by the AWS C++ SDK. See [1]. The AWS
C++ SDK has static state and they do not manage it with static local variables.
As
result, the initialization and finalization order is
(IIRC) undefined (or at least not very well defined).
OK, a few questions. When did this limitation begin in the AWS C++ SDK?
AFAIU, this limitation always existed in the AWS C++ SDK, it's just that
Arrow C++ didn't guard against it. It's not obvious to me that the AWS
C++ SDK supports the "multiple init/finalization cycles" use case
(nothing in their docs seems to mention that, and if they don't test for
it, it's likely to fail in some ways), but it might have happened to
work by chance.
We (Arrow developers) would rather not have to worry about AWS SDK
init/finalization *at all*. But we got multiple reports of issues and
crashes (some from users of PyArrow) that had to do with AWS SDK
init/finalization, and so were compelled to exert control over it.
Understood. Yes, a complete unloading of the Arrow library seems sufficient to
resolve the issue. The trouble is that we (ergo I) must support multi-user,
and the handles are not unique, instead a counter is used to keep track of
handle loading, similar to a shared smart pointer. Thus the scenario in which
a given user loads the handle, then unloads the handle, but the underlying
resources aren't unloaded is a very distinct possibility.
That sounds like a problem indeed. You cannot arrange for a hook to be
invoked when resource unloading actually happens?
Also, by the way, this would mean multiple users would share potentially
the same AWS configuration?
Regards
Antoine.
