Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-12-09 Thread Thomas Munro
On Thu, Dec 9, 2021 at 9:16 PM Thomas Munro wrote: > Slightly improvement: now I include only from > src/port/open.c and src/port/win32ntdll.c, so I avoid the extra > include for the other ~1500 translation units. That requires a small > extra step to work, see comment in win32ntdll.h. I checke

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-12-09 Thread Thomas Munro
On Mon, Dec 6, 2021 at 9:17 PM Thomas Munro wrote: > On Sat, Dec 4, 2021 at 6:18 PM Thomas Munro wrote: > > I think this was broken by WIN32_LEAN_AND_MEAN (and since gained a > > merge conflict, but that's easy to fix). I'll try to figure out the > > right system header hacks to unbreak it... >

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-12-06 Thread Thomas Munro
On Sat, Dec 4, 2021 at 6:18 PM Thomas Munro wrote: > On Thu, Dec 2, 2021 at 3:11 AM Daniel Gustafsson wrote: > > > This patch doesn't compile on Windows according to Appveyor, seemingly > > > because > > > of a syntax error in the new win32ntdll.h file, but the MSVC logs are > > > hard on > > >

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-12-03 Thread Thomas Munro
On Thu, Dec 2, 2021 at 3:11 AM Daniel Gustafsson wrote: > > On 3 Nov 2021, at 12:02, Daniel Gustafsson wrote: > > > > This patch doesn't compile on Windows according to Appveyor, seemingly > > because > > of a syntax error in the new win32ntdll.h file, but the MSVC logs are hard > > on > > the

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-12-01 Thread Daniel Gustafsson
> On 3 Nov 2021, at 12:02, Daniel Gustafsson wrote: > > This patch doesn't compile on Windows according to Appveyor, seemingly because > of a syntax error in the new win32ntdll.h file, but the MSVC logs are hard on > the eye so it might be unrelated. As the thread has stalled with a patch that d

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-11-03 Thread Daniel Gustafsson
This patch doesn't compile on Windows according to Appveyor, seemingly because of a syntax error in the new win32ntdll.h file, but the MSVC logs are hard on the eye so it might be unrelated. -- Daniel Gustafsson https://vmware.com/

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-09-29 Thread Juan José Santamaría Flecha
On Tue, Sep 28, 2021 at 2:50 AM Thomas Munro wrote: > On Thu, Sep 23, 2021 at 9:13 PM Juan José Santamaría Flecha > wrote: > > When GetTempFileName() finds a duplicated file name and the file is > pending for deletion, it fails with an "ERROR_ACCESS_DENIED" error code. > That may describe the si

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-09-27 Thread Thomas Munro
On Thu, Sep 23, 2021 at 9:13 PM Juan José Santamaría Flecha wrote: > On Thu, Sep 23, 2021 at 4:58 AM Thomas Munro wrote: >> One small detail I'd like to draw attention to is this bit, which >> differs slightly from the [non-working] previous attempts by mapping >> to two different errors: >> >> +

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-09-23 Thread Juan José Santamaría Flecha
On Thu, Sep 23, 2021 at 4:58 AM Thomas Munro wrote: > > One small detail I'd like to draw attention to is this bit, which > differs slightly from the [non-working] previous attempts by mapping > to two different errors: > > + * If there's no O_CREAT flag, then we'll pretend the file is >

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-09-22 Thread Thomas Munro
On Fri, Sep 10, 2021 at 5:04 PM Thomas Munro wrote: > On Tue, Sep 7, 2021 at 7:00 PM Alexander Lakhin wrote: > > 07.09.2021 09:05, Michael Paquier wrote: > > > On Tue, Sep 07, 2021 at 09:00:01AM +0300, Alexander Lakhin wrote: > > >> The new approach looks very promising. Knowing that the file is

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-09-09 Thread Thomas Munro
On Tue, Sep 7, 2021 at 7:00 PM Alexander Lakhin wrote: > 07.09.2021 09:05, Michael Paquier wrote: > > On Tue, Sep 07, 2021 at 09:00:01AM +0300, Alexander Lakhin wrote: > >> The new approach looks very promising. Knowing that the file is really > >> in the DELETE_PENDING state simplifies a lot. > >

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-09-07 Thread Alexander Lakhin
Hello Michael, 07.09.2021 09:05, Michael Paquier wrote: > On Tue, Sep 07, 2021 at 09:00:01AM +0300, Alexander Lakhin wrote: >> The new approach looks very promising. Knowing that the file is really >> in the DELETE_PENDING state simplifies a lot. >> I've tested the patch v2_0001_Check... with my de

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-09-06 Thread Michael Paquier
On Tue, Sep 07, 2021 at 09:00:01AM +0300, Alexander Lakhin wrote: > The new approach looks very promising. Knowing that the file is really > in the DELETE_PENDING state simplifies a lot. > I've tested the patch v2_0001_Check... with my demo tests [1] and [2], > and it definitely works. Oho, nice.

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-09-06 Thread Alexander Lakhin
06.09.2021 16:04, Thomas Munro wrote: > On Mon, Sep 6, 2021 at 6:36 PM Michael Paquier wrote: >> The last time I poked at the bear (54fb8c7d), there was a test posted >> by Alexander Lakhin that was really useful in making sure that >> concurrency is correctly handled when a file is unlinked: >> h

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-09-06 Thread Thomas Munro
On Mon, Sep 6, 2021 at 6:36 PM Michael Paquier wrote: > Having a non-invasive fix for this long-standing issue would be really > great, even if that means reducing the scope of systems where this can > be fixed. I hope those patches fix at least the basebackup problem on all relevant OS versions,

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-09-06 Thread Thomas Munro
On Mon, Sep 6, 2021 at 9:44 AM Andres Freund wrote: > Is it guaranteed, or at least reliable, that the status we fetch with > RtlGetLastNtStatus is actually from the underlying filesystem operation, > rather than some other work that happens during the win32->nt translation? > E.g. a memory alloca

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-09-05 Thread Michael Paquier
> If DeleteFile() is automatically using > FILE_DISPOSITION_POSIX_SEMANTICS by default when possible on recent > releases as per the SO link that Andres posted above ("18363.657 > definitely has the new behavior"), then that's great news and maybe we > shouldn't even bother to try to request that m

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-09-05 Thread Thomas Munro
On Mon, Sep 6, 2021 at 9:55 AM Tom Lane wrote: > Andres Freund writes: > > I'd be ok to only fix these bugs on e.g. Win10, Win Server 2019, Win Server > > 2016 or such. > > Yeah, particularly if the fix is trivial on the newer systems and > incredibly complicated otherwise. Between the effort ne

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-09-05 Thread Tom Lane
Andres Freund writes: > On 2021-09-06 01:32:55 +1200, Thomas Munro wrote: >> The best idea is probably to set FILE_DISPOSITION_DELETE | >> FILE_DISPOSITION_POSIX_SEMANTICS before unlinking. This appears to be >> a silver bullet, but isn't available on ancient Windows releases that >> we support,

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-09-05 Thread Andres Freund
Hi, On 2021-09-06 01:32:55 +1200, Thomas Munro wrote: > It's a good idea. I tested it and it certainly does fix the > basebackup problem I've seen (experimental patch attached). But, > yeah, I'm also a bit worried that that path could be fragile and need > special handling in lots of places. It

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-09-05 Thread Thomas Munro
On Fri, Sep 3, 2021 at 2:01 PM Kyotaro Horiguchi wrote: > Might be stupid, if a delete-pending'ed file can obstruct something, > couldn't we change unlink on Windows to rename to a temporary random > name then remove it? We do something like it explicitly while WAL > file removal. (It may cause d

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-09-02 Thread Kyotaro Horiguchi
At Fri, 3 Sep 2021 01:01:43 +1200, Thomas Munro wrote in > On Fri, Sep 3, 2021 at 12:44 AM Thomas Munro wrote: > > NtFileCreate() > > Erm, that's spelled NtCreateFile. I see Michael mentioned this > before[1]; I don't think it's only available in kernel mode though, > the docs[2] say "This fu

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-09-02 Thread Thomas Munro
On Fri, Sep 3, 2021 at 12:44 AM Thomas Munro wrote: > NtFileCreate() Erm, that's spelled NtCreateFile. I see Michael mentioned this before[1]; I don't think it's only available in kernel mode though, the docs[2] say "This function is the user-mode equivalent to the ZwCreateFile function", and ot

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-09-02 Thread Thomas Munro
On Thu, Sep 2, 2021 at 11:12 PM Tom Lane wrote: > Thomas Munro writes: > > I'm no expert, but not AFAICS. We managed to delete the file while > > some other backend had it open, which FILE_SHARE_DELETE allowed. We > > just can't open it or create a new file with the same name until it's > > rea

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-09-02 Thread Tom Lane
Thomas Munro writes: > On Thu, Sep 2, 2021 at 10:31 PM Tom Lane wrote: >> That seems quite horrid :-(. But if it works, doesn't that mean that >> somewhere we are opening a problematic file without the correct >> sharing flags? > I'm no expert, but not AFAICS. We managed to delete the file whi

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-09-02 Thread Thomas Munro
On Thu, Sep 2, 2021 at 10:31 PM Tom Lane wrote: > Thomas Munro writes: > > A disruptive solution that works in my tests: we could reuse the > > global barrier proposed in CF #2962. If you see EACCES, ask every > > backend to close all vfds at their next CFI() and wait for them all to > > finish,

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-09-02 Thread Tom Lane
Thomas Munro writes: > A disruptive solution that works in my tests: we could reuse the > global barrier proposed in CF #2962. If you see EACCES, ask every > backend to close all vfds at their next CFI() and wait for them all to > finish, and then retry. If you get EACCES again it really means >

Re: stat() vs ERROR_DELETE_PENDING, round N + 1

2021-09-02 Thread Thomas Munro
On Thu, Sep 2, 2021 at 10:10 AM Thomas Munro wrote: > Perhaps we need some combination of the old way (that apparently knew > how to detect pending deletes) and the new way (that knows about large > files)? I tried that, but as far as I can tell, the old approach didn't really work either :-( A