Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-02-26 Thread Michael Paquier
On Wed, Feb 26, 2020 at 06:02:22PM +0100, Bernd Helmle wrote: > My feeling is that in the case we cannot successfully resolve a > tablespace location from pg_tblspc, we should error out, but i could > imagine that people would like to have just a warning instead. Thanks, this patch is much cleaner

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-02-26 Thread Bernd Helmle
Am Dienstag, den 25.02.2020, 11:33 +0900 schrieb Michael Paquier: > I really think that > we should avoid duplicating the same logic around, and that we should > remain consistent with non-directory entries in those paths, > complaining with a proper failure if extra, unwanted files are > present.

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-02-24 Thread Michael Paquier
On Mon, Feb 24, 2020 at 01:11:10PM +0100, Bernd Helmle wrote: > The other makes scan_directories() complicated to read and special > cases just a single directory in an otherwise more or less generic > function. E.g. it makes me uncomfortable if we get a pg_tblspc > somewhere else than PGDATA (if

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-02-24 Thread David Steele
Hi Michael, On 2/24/20 7:26 AM, Michael Paquier wrote: On Sun, Feb 23, 2020 at 04:08:58PM +0900, Michael Paquier wrote: Good idea. Let's do things as you suggest. Applied and back-patched this one down to 11. FWIW, we took a slightly narrower approach to this issue in the pgBackRest patch

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-02-24 Thread Michael Paquier
On Sun, Feb 23, 2020 at 04:08:58PM +0900, Michael Paquier wrote: > Good idea. Let's do things as you suggest. Applied and back-patched this one down to 11. -- Michael signature.asc Description: PGP signature

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-02-24 Thread Bernd Helmle
On Fri, 2020-02-21 at 15:36 +0900, Michael Paquier wrote: > We don't do that for the normal directory scan path, so it does not > strike me as a good idea on consistency ground. As a whole, I don't > see much point in having a separate routine which is just roughly a > duplicate of scan_directory(

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-02-22 Thread Michael Paquier
On Fri, Feb 21, 2020 at 08:13:34AM -0500, David Steele wrote: > Do you have the thread? I'd like to see what was proposed and what the > objections were. Here you go: https://www.postgresql.org/message-id/20180205071022.ga17...@paquier.xyz -- Michael signature.asc Description: PGP signature

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-02-22 Thread Michael Paquier
On Fri, Feb 21, 2020 at 05:37:15PM +0900, Kyotaro Horiguchi wrote: > The two str[n]cmps are different only in matching length. I don't > think we don't need to differentiate the two message there, so we > could reduce the code as: > > | cmplen = strlen(excludeFiles[].name); > | if (!prefix_patch)

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-02-21 Thread David Steele
On 2/21/20 1:36 AM, Michael Paquier wrote: > On Thu, Feb 20, 2020 at 05:38:15PM +0100, Bernd Helmle wrote: >> So i propose a different approach like the attached patch tries to >> implement: instead of just blindly iterating over directory contents >> and filter them out, reference the tablespace

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-02-21 Thread David Steele
Hi Michael, On 2/20/20 11:07 PM, Michael Paquier wrote: > On Thu, Feb 20, 2020 at 07:37:15AM -0600, David Steele wrote: >> But since the name includes the backend's pid you would need to get lucky >> and have a new backend with the same pid create the file after a restart. I >> tried it and

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-02-21 Thread Kyotaro Horiguchi
Thank you David for decrypting my previous mail.., and your translation was correct. At Fri, 21 Feb 2020 15:07:12 +0900, Michael Paquier wrote in > On Thu, Feb 20, 2020 at 07:37:15AM -0600, David Steele wrote: > > But since the name includes the backend's pid you would need to get lucky > > an

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-02-20 Thread Michael Paquier
On Thu, Feb 20, 2020 at 05:38:15PM +0100, Bernd Helmle wrote: > So i propose a different approach like the attached patch tries to > implement: instead of just blindly iterating over directory contents > and filter them out, reference the tablespace location and > TABLESPACE_VERSION_DIRECTORY direc

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-02-20 Thread Michael Paquier
On Thu, Feb 20, 2020 at 07:37:15AM -0600, David Steele wrote: > But since the name includes the backend's pid you would need to get lucky > and have a new backend with the same pid create the file after a restart. I > tried it and the old temp file was left behind after restart and first > connec

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-02-20 Thread Bernd Helmle
Am Dienstag, den 18.02.2020, 15:15 +0900 schrieb Michael Paquier: > Fair point. Now, while the proposed patch is right to use > TABLESPACE_VERSION_DIRECTORY, shouldn't we use strncmp based on the > length of TABLESPACE_VERSION_DIRECTORY instead of de->d_name? It > seems also to me that the code a

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-02-20 Thread David Steele
On 2/20/20 12:55 AM, Michael Paquier wrote: On Wed, Feb 19, 2020 at 12:37:00PM -0600, David Steele wrote: As far as I can see, the pg_internal.init.XX will not be cleaned up by PostgreSQL on startup. I've only tested this in 9.6 so far, but I don't see any differences in the code since then th

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-02-19 Thread Michael Paquier
On Wed, Feb 19, 2020 at 12:37:00PM -0600, David Steele wrote: > On 2/19/20 2:13 AM, Michael Paquier wrote: >> Please note that pg_internal.init is listed within noChecksumFiles in >> basebackup.c, so we would miss any temporary pg_internal.init.PID if >> we don't check after the file prefix and the

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-02-19 Thread David Steele
On 1/31/20 3:59 AM, Michael Banck wrote: Hi, Am Freitag, den 31.01.2020, 13:53 +0900 schrieb Michael Paquier: On Thu, Jan 30, 2020 at 06:11:22PM +0100, Michael Banck wrote: Having a past tablespace version left around after an upgrade is a pilot error in my opinion because pg_upgrade generates

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-02-19 Thread David Steele
On 2/19/20 2:13 AM, Michael Paquier wrote: On Fri, Jan 31, 2020 at 05:39:36PM +0900, Kyotaro Horiguchi wrote: At Fri, 31 Jan 2020 17:30:43 +0900 (JST), Kyotaro Horiguchi wrote in I don't think that is a problem right away, of course. It looks good to me except for the possible excessive exclu

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-02-19 Thread Michael Paquier
On Fri, Jan 31, 2020 at 05:39:36PM +0900, Kyotaro Horiguchi wrote: > At Fri, 31 Jan 2020 17:30:43 +0900 (JST), Kyotaro Horiguchi > wrote in >> I don't think that is a problem right away, of course. It looks good >> to me except for the possible excessive exclusion. So, I don't object >> it if w

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-02-17 Thread Michael Paquier
On Fri, Jan 31, 2020 at 11:33:34AM +0100, Bernd Helmle wrote: > And to be honest, even PostgreSQL itself allows you to reuse tablespace > locations for multiple instances as well, so the described problem > should exist not in upgraded clusters only. Fair point. Now, while the proposed patch is r

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-01-31 Thread Bernd Helmle
Am Freitag, den 31.01.2020, 13:53 +0900 schrieb Michael Paquier: > Indeed, with a bad timing and a crash in the middle of > write_relcache_init_file(), it could be possible to have such a file > left around in the data folder. Having a past tablespace version > left > around after an upgrade is a

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-01-31 Thread Michael Banck
Hi, Am Freitag, den 31.01.2020, 13:53 +0900 schrieb Michael Paquier: > On Thu, Jan 30, 2020 at 06:11:22PM +0100, Michael Banck wrote: > Having a past tablespace version left > around after an upgrade is a pilot error in my opinion because > pg_upgrade generates a script to cleanup past tablespaces

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-01-31 Thread Kyotaro Horiguchi
At Fri, 31 Jan 2020 17:30:43 +0900 (JST), Kyotaro Horiguchi wrote in > I don't think that is a problem right away, of course. It looks good > to me except for the possible excessive exclusion. So, I don't object > it if we don't mind that. That's a bit wrong. All the discussion is only on exc

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-01-31 Thread Kyotaro Horiguchi
At Fri, 31 Jan 2020 13:53:52 +0900, Michael Paquier wrote in > On Thu, Jan 30, 2020 at 06:11:22PM +0100, Michael Banck wrote: > > The other question is whether it is possible to end up with a > > pg_internal.init.$PID file in a running cluster. E.g. if an instance > > crashes and gets started up

Re: [Patch] Make pg_checksums skip foreign tablespace directories

2020-01-30 Thread Michael Paquier
On Thu, Jan 30, 2020 at 06:11:22PM +0100, Michael Banck wrote: > The other question is whether it is possible to end up with a > pg_internal.init.$PID file in a running cluster. E.g. if an instance > crashes and gets started up again - are those cleaned up during crash > recovery, or should pg_chec

[Patch] Make pg_checksums skip foreign tablespace directories

2020-01-30 Thread Michael Banck
folgenden Bestimmungen: https://www.credativ.de/datenschutz From 5a0a1e7ff543749815bd4d87476550b01009dbba Mon Sep 17 00:00:00 2001 From: Michael Banck Date: Thu, 30 Jan 2020 17:48:37 +0100 Subject: [PATCH] Make pg_checksums skip foreign tablespace directories. Until now, pg_checksums blindly desc