Re: Loaded footgun open_datasync on Windows

2020-07-26 Thread Michael Paquier
On Thu, Jul 23, 2020 at 01:05:04PM -0400, Jeff Janes wrote: > I have noticed this before, but since it wasn't a production machine I just > shrugged it off as being a hazard of using consumer-grade stuff; it didn't > seem to be worth investigating further. The most direct and non-invasive way to a

Re: Loaded footgun open_datasync on Windows

2020-07-23 Thread Jeff Janes
On Fri, Sep 14, 2018 at 3:32 AM Michael Paquier wrote: > On Fri, Sep 14, 2018 at 08:43:18AM +0200, Laurenz Albe wrote: > > > If it turns out not to break anything, would you consider backpatching? > > On the one hand it fixes a bug, on the other hand it affects all > > frontend executables... > >

Re: Loaded footgun open_datasync on Windows

2018-09-14 Thread Michael Paquier
On Fri, Sep 14, 2018 at 08:43:18AM +0200, Laurenz Albe wrote: > Thanks for being interested and doing the work. No problem. I have a sort of Windows-label stuck on me for ages, and those random buildfarm failures are annoying with TAP tests on Windows. > If it turns out not to break anything, wo

Re: Loaded footgun open_datasync on Windows

2018-09-13 Thread Laurenz Albe
Michael Paquier wrote: > On Thu, Sep 13, 2018 at 03:55:20PM +0200, Laurenz Albe wrote: > > Attached is the new, improved version of the patch. > > Thanks, finally pushed. I have made sure that recovery TAP tests, > upgrade tests and bin/ tests are working properly. Thanks for being interested an

Re: Loaded footgun open_datasync on Windows

2018-09-13 Thread Michael Paquier
On Thu, Sep 13, 2018 at 03:55:20PM +0200, Laurenz Albe wrote: > Attached is the new, improved version of the patch. Thanks, finally pushed. I have made sure that recovery TAP tests, upgrade tests and bin/ tests are working properly. -- Michael signature.asc Description: PGP signature

Re: Loaded footgun open_datasync on Windows

2018-09-13 Thread Laurenz Albe
Michael Paquier wrote: > On Mon, Sep 10, 2018 at 04:46:40PM +0200, Laurenz Albe wrote: > > I didn't get pg_upgrade to work without the log file hacks; I suspect > > that there is more than just log file locking going on, but my Windows > > skills are limited. > > > > How shall I proceed? > > I do

Re: Loaded footgun open_datasync on Windows

2018-09-12 Thread Laurenz Albe
On Wed, 2018-09-12 at 14:43 +0900, Michael Paquier wrote: > On Mon, Sep 10, 2018 at 04:46:40PM +0200, Laurenz Albe wrote: > > I didn't get pg_upgrade to work without the log file hacks; I suspect > > that there is more than just log file locking going on, but my Windows > > skills are limited. > >

Re: Loaded footgun open_datasync on Windows

2018-09-11 Thread Michael Paquier
On Mon, Sep 10, 2018 at 04:46:40PM +0200, Laurenz Albe wrote: > I didn't get pg_upgrade to work without the log file hacks; I suspect > that there is more than just log file locking going on, but my Windows > skills are limited. > > How shall I proceed? I do like this patch, and we have an occasi

Re: Loaded footgun open_datasync on Windows

2018-09-10 Thread Laurenz Albe
Noah Misch wrote: > On Wed, Jun 27, 2018 at 12:09:24PM +0200, Laurenz Albe wrote: > > Michael Paquier wrote: > > > > I have added it to the July commitfest. > > > > > > Have you looked at the possibility of removing the log file constraints > > > in pg_upgrade with the change you are doing here so

Re: Loaded footgun open_datasync on Windows

2018-09-03 Thread Michael Paquier
On Sat, Sep 01, 2018 at 05:07:21PM -0700, Noah Misch wrote: > If you grep src/bin/pg_upgrade for WIN32, roughly a third of the hits are > workarounds for this problem. I agree with Michael that removing those > workarounds would be a good test of frontend pgwin32_open() and worth > including in th

Re: Loaded footgun open_datasync on Windows

2018-09-01 Thread Noah Misch
On Wed, Jun 27, 2018 at 12:09:24PM +0200, Laurenz Albe wrote: > Michael Paquier wrote: > > > I have added it to the July commitfest. > > > > Have you looked at the possibility of removing the log file constraints > > in pg_upgrade with the change you are doing here so as things would be > > more c

Re: Loaded footgun open_datasync on Windows

2018-08-06 Thread Laurenz Albe
Thomas Munro wrote: > It looks like initdb is failing with this patch: > > https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.6732 > > Unfortunately cfbot is not clever enough to print out the contents of > the initdb log so I don't have any more information... Sorry for the d

Re: Loaded footgun open_datasync on Windows

2018-07-25 Thread Thomas Munro
On Thu, Jun 21, 2018 at 2:06 AM, Laurenz Albe wrote: > How about checking what the buildfarm thinks about the attached? Hi Laurenz, It looks like initdb is failing with this patch: https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.6732 Unfortunately cfbot is not clever enou

Re: Loaded footgun open_datasync on Windows

2018-06-27 Thread Laurenz Albe
Michael Paquier wrote: > > I have added it to the July commitfest. > > Have you looked at the possibility of removing the log file constraints > in pg_upgrade with the change you are doing here so as things would be > more consistent with non-Windows platforms, simplifying some code on the > way?

Re: Loaded footgun open_datasync on Windows

2018-06-25 Thread Michael Paquier
On Mon, Jun 25, 2018 at 06:10:21PM +0200, Laurenz Albe wrote: > I have added it to the July commitfest. Have you looked at the possibility of removing the log file constraints in pg_upgrade with the change you are doing here so as things would be more consistent with non-Windows platforms, simplif

Re: Loaded footgun open_datasync on Windows

2018-06-25 Thread Laurenz Albe
Kuntal Ghosh wrote: > > If I use the three-argument version throughout, which should be no problem, > > PostgreSQL builds just fine with MSVC. > > > > I don't have enough experience on MSVC/Windows to comment on the same. > > > How about checking what the buildfarm thinks about the attached? >

Re: Loaded footgun open_datasync on Windows

2018-06-25 Thread Kuntal Ghosh
On Wed, Jun 20, 2018 at 7:36 PM, Laurenz Albe wrote: > Kuntal Ghosh wrote: > [pg_test_fsync doesn't use pgwin32_open and hence doesn't respect O_DSYNC] >> On Wed, Jun 6, 2018 at 2:39 PM, Amit Kapila wrote: >> > On Wed, Jun 6, 2018 at 10:18 AM, Michael Paquier >> > wrote: >> > > On Wed, Jun 06,

Re: Loaded footgun open_datasync on Windows

2018-06-20 Thread Laurenz Albe
Kuntal Ghosh wrote: [pg_test_fsync doesn't use pgwin32_open and hence doesn't respect O_DSYNC] > On Wed, Jun 6, 2018 at 2:39 PM, Amit Kapila wrote: > > On Wed, Jun 6, 2018 at 10:18 AM, Michael Paquier > > wrote: > > > On Wed, Jun 06, 2018 at 09:58:34AM +0530, Amit Kapila wrote: > > > > One anoth

Re: Loaded footgun open_datasync on Windows

2018-06-10 Thread Michael Paquier
On Sun, Jun 10, 2018 at 10:09:26AM +0530, Amit Kapila wrote: > As per discussion till now, we have two options to proceed: > (a) Remove the define "#ifndef FRONTEND" that prevents pgwin32_open > usage in frontend modules. We have done some research and found that > it was added in the past to allo

Re: Loaded footgun open_datasync on Windows

2018-06-09 Thread Amit Kapila
On Fri, Jun 8, 2018 at 11:00 PM, Magnus Hagander wrote: > > On Fri, Jun 8, 2018 at 4:55 AM, Amit Kapila wrote: >> >> >> -#include "postgres_fe.h" >> +#include "postgres.h" >> >> I don't think that server application can use backend environment unless >> it is adapted to do so. I think the applic

Re: Loaded footgun open_datasync on Windows

2018-06-08 Thread Magnus Hagander
On Fri, Jun 8, 2018 at 4:55 AM, Amit Kapila wrote: > On Fri, Jun 8, 2018 at 7:48 AM, Laurenz Albe > wrote: > >> Amit Kapila wrote: >> > On Wed, Jun 6, 2018 at 3:06 PM, Kuntal Ghosh < >> kuntalghosh.2...@gmail.com> wrote: >> > > It seems the "#ifndef FRONTEND" restriction was added around >> > >

Re: Loaded footgun open_datasync on Windows

2018-06-07 Thread Amit Kapila
On Fri, Jun 8, 2018 at 7:48 AM, Laurenz Albe wrote: > Amit Kapila wrote: > > On Wed, Jun 6, 2018 at 3:06 PM, Kuntal Ghosh > wrote: > > > It seems the "#ifndef FRONTEND" restriction was added around > > > pgwin32_open() for building libpq with Visual C++ or Borland C++. The > > > restriction was

Re: Loaded footgun open_datasync on Windows

2018-06-07 Thread Laurenz Albe
Amit Kapila wrote: > On Wed, Jun 6, 2018 at 3:06 PM, Kuntal Ghosh > wrote: > > It seems the "#ifndef FRONTEND" restriction was added around > > pgwin32_open() for building libpq with Visual C++ or Borland C++. The > > restriction was added in commit 422d4819 to build libpq with VC++[1]. > > Late

Re: Loaded footgun open_datasync on Windows

2018-06-06 Thread Amit Kapila
On Wed, Jun 6, 2018 at 3:06 PM, Kuntal Ghosh wrote: > On Wed, Jun 6, 2018 at 2:39 PM, Amit Kapila > wrote: > > On Wed, Jun 6, 2018 at 10:18 AM, Michael Paquier > > wrote: > >> > >> On Wed, Jun 06, 2018 at 09:58:34AM +0530, Amit Kapila wrote: > >> > >> >> It could be > >> >> risky for existing c

Re: Loaded footgun open_datasync on Windows

2018-06-06 Thread Kuntal Ghosh
On Wed, Jun 6, 2018 at 2:39 PM, Amit Kapila wrote: > On Wed, Jun 6, 2018 at 10:18 AM, Michael Paquier > wrote: >> >> On Wed, Jun 06, 2018 at 09:58:34AM +0530, Amit Kapila wrote: >> >> >> It could be >> >> risky for existing callers of open() for tool maintainers, or on the >> >> contrary people c

Re: Loaded footgun open_datasync on Windows

2018-06-06 Thread Amit Kapila
On Wed, Jun 6, 2018 at 10:18 AM, Michael Paquier wrote: > On Wed, Jun 06, 2018 at 09:58:34AM +0530, Amit Kapila wrote: > > >> It could be > >> risky for existing callers of open() for tool maintainers, or on the > >> contrary people could welcome a wrapper of open() which is > >> concurrent-safe

Re: Loaded footgun open_datasync on Windows

2018-06-05 Thread Thomas Munro
On Wed, Jun 6, 2018 at 4:48 PM, Michael Paquier wrote: > On Wed, Jun 06, 2018 at 09:58:34AM +0530, Amit Kapila wrote: >> I think we need to explore a bit, if we want to remove that, for example >> some of the frontend modules (like pg_receivewal, pg_recvlogical) call two >> argument open which wou

Re: Loaded footgun open_datasync on Windows

2018-06-05 Thread Michael Paquier
On Wed, Jun 06, 2018 at 09:58:34AM +0530, Amit Kapila wrote: > On Wed, Jun 6, 2018 at 8:28 AM, Michael Paquier wrote: >> Ouch. Including directly c.h as you do here is against project policy >> code. See recent commit a72f0365 for example. pgwin32_open() is >> visibly able to handle frontend co

Re: Loaded footgun open_datasync on Windows

2018-06-05 Thread Amit Kapila
On Wed, Jun 6, 2018 at 8:28 AM, Michael Paquier wrote: > On Tue, Jun 05, 2018 at 04:15:00PM +0200, Laurenz Albe wrote: > > > --- a/src/bin/pg_test_fsync/pg_test_fsync.c > > +++ b/src/bin/pg_test_fsync/pg_test_fsync.c > > @@ -3,6 +3,8 @@ > > * tests all supported fsync() methods > >

Re: Loaded footgun open_datasync on Windows

2018-06-05 Thread Michael Paquier
On Tue, Jun 05, 2018 at 04:15:00PM +0200, Laurenz Albe wrote: > The attached patch makes pg_test_fsync use pgwin32_open on Windows, which is > what > we use elsewhere. Well, pg_upgrade may benefit from that as one example, as any other tools. > That should fix the problem. > Ist there a better w

Re: Loaded footgun open_datasync on Windows

2018-06-05 Thread Laurenz Albe
Amit Kapila wrote: > On Fri, Jun 1, 2018 at 8:18 PM, Laurenz Albe wrote: > > Amit Kapila wrote: > > > On Fri, Jun 1, 2018 at 3:13 PM, Laurenz Albe > > > wrote: > > > > I recently read our documentation about reliability on Windows: > > > > > > > > > On Windows, if wal_sync_method is open_datasy

Re: Loaded footgun open_datasync on Windows

2018-06-04 Thread Robert Haas
On Fri, Jun 1, 2018 at 2:56 PM, Michael Paquier wrote: > When things come to VMs or containers, developers and users tend to be > sloppy regarding the hardware being used, and they are not usually aware > that the database running within it is sensitive to such details. Many > folks use Postgres,

Re: Loaded footgun open_datasync on Windows

2018-06-01 Thread Michael Paquier
On Fri, Jun 01, 2018 at 07:32:26PM +0200, Magnus Hagander wrote: > Basically, this method *is* safe as long as you have proper storage. For > example, if yo have a RAID controller with cache, it is perfectly safe. If > you have a consumer level device with unsafe caching, then it's not safe. > This

Re: Loaded footgun open_datasync on Windows

2018-06-01 Thread Amit Kapila
On Fri, Jun 1, 2018 at 8:18 PM, Laurenz Albe wrote: > Amit Kapila wrote: > > On Fri, Jun 1, 2018 at 3:13 PM, Laurenz Albe > wrote: > > > I recently read our documentation about reliability on Windows: > > > > > > > On Windows, if wal_sync_method is open_datasync (the default), write > caching ca

Re: Loaded footgun open_datasync on Windows

2018-06-01 Thread Magnus Hagander
On Fri, Jun 1, 2018 at 3:26 PM, Amit Kapila wrote: > On Fri, Jun 1, 2018 at 3:13 PM, Laurenz Albe > wrote: > >> I recently read our documentation about reliability on Windows: >> >> > On Windows, if wal_sync_method is open_datasync (the default), write >> caching can >> > be disabled by unchecki

Re: Loaded footgun open_datasync on Windows

2018-06-01 Thread Laurenz Albe
Amit Kapila wrote: > On Fri, Jun 1, 2018 at 3:13 PM, Laurenz Albe wrote: > > I recently read our documentation about reliability on Windows: > > > > > On Windows, if wal_sync_method is open_datasync (the default), write > > > caching can > > > be disabled by unchecking > > > My Computer\Open\dis

Re: Loaded footgun open_datasync on Windows

2018-06-01 Thread Amit Kapila
On Fri, Jun 1, 2018 at 3:13 PM, Laurenz Albe wrote: > I recently read our documentation about reliability on Windows: > > > On Windows, if wal_sync_method is open_datasync (the default), write > caching can > > be disabled by unchecking > > My Computer\Open\disk drive\Properties\Hardware\Properti

Re: Loaded footgun open_datasync on Windows

2018-06-01 Thread Michael Paquier
On Fri, Jun 01, 2018 at 11:43:33AM +0200, Laurenz Albe wrote: > I am worried that there might be loads of Windows installations out there > happily > running productive databases with an unsafe setup, so I'd even suggest > backpatching > such a change. This is not only your imagination, there ar

Loaded footgun open_datasync on Windows

2018-06-01 Thread Laurenz Albe
I recently read our documentation about reliability on Windows: > On Windows, if wal_sync_method is open_datasync (the default), write caching > can > be disabled by unchecking > My Computer\Open\disk drive\Properties\Hardware\Properties\Policies\Enable > write caching > on the disk. Alternative