there's even a reg key to tweak that flush delay but it seems I can't put the finger on it anymore.
Of course I just find it minutes after posting... it's in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer add a DWORD of name AlwaysUnloadDll and give it a 1. Though it's advertised everywhere as managing all dlls, I dunno the exact effect, and the place of the reg key makes me thinks it only affects explorer, but you'll never know.