Re: pgsql: Avoid duplicate XIDs at recovery when building initial snapshot

2018-12-02 Thread Michael Paquier
On Sat, Dec 01, 2018 at 10:51:10AM +, Simon Riggs wrote: > On Fri, 30 Nov 2018 at 23:08, Michael Paquier wrote: >> It is not complicated to >> handle duplicates for xacts and subxacts however holding ProcArrayLock >> for a longer time stresses me as it is already a bottleneck. > > I hadn't re

Re: pgsql: Avoid duplicate XIDs at recovery when building initial snapshot

2018-12-01 Thread Simon Riggs
On Fri, 30 Nov 2018 at 23:08, Michael Paquier wrote: > On Fri, Nov 30, 2018 at 02:55:47PM +, Simon Riggs wrote: > > 1df21ddb looks OK to me and was simple enough to backpatch safely. > > Thanks for the feedback! > > > Seems excessive to say that the WAL record is corrupt, it just contains > >

Re: pgsql: Avoid duplicate XIDs at recovery when building initial snapshot

2018-11-30 Thread Michael Paquier
On Fri, Nov 30, 2018 at 02:55:47PM +, Simon Riggs wrote: > 1df21ddb looks OK to me and was simple enough to backpatch safely. Thanks for the feedback! > Seems excessive to say that the WAL record is corrupt, it just contains > duplicates, just as exported snapshots do. There's a few other imp

Re: pgsql: Avoid duplicate XIDs at recovery when building initial snapshot

2018-11-30 Thread Simon Riggs
On Thu, 1 Nov 2018 at 06:09, Michael Paquier wrote: > On Tue, Oct 23, 2018 at 10:43:38AM +0900, Michael Paquier wrote: > > Well, following the same kind of thoughts, txid_current_snapshot() uses > > sort_snapshot() to remove all the duplicates after fetching its data > > from GetSnapshotData(), s

Re: pgsql: Avoid duplicate XIDs at recovery when building initial snapshot

2018-11-30 Thread Dmitry Dolgov
> On Thu, Nov 1, 2018 at 7:09 AM Michael Paquier wrote: > > On Tue, Oct 23, 2018 at 10:43:38AM +0900, Michael Paquier wrote: > > Well, following the same kind of thoughts, txid_current_snapshot() uses > > sort_snapshot() to remove all the duplicates after fetching its data > > from GetSnapshotData

Re: pgsql: Avoid duplicate XIDs at recovery when building initial snapshot

2018-10-31 Thread Michael Paquier
On Tue, Oct 23, 2018 at 10:43:38AM +0900, Michael Paquier wrote: > Well, following the same kind of thoughts, txid_current_snapshot() uses > sort_snapshot() to remove all the duplicates after fetching its data > from GetSnapshotData(), so wouldn't we want to do something about > removal of duplicat

Re: pgsql: Avoid duplicate XIDs at recovery when building initial snapshot

2018-10-22 Thread Michael Paquier
On Mon, Oct 22, 2018 at 07:15:38PM -0300, Alvaro Herrera wrote: > On 2018-Oct-22, Andres Freund wrote: >> Hm? My point is that this fix just puts a band-aid onto *one* of the >> places that read a XLOG_RUNNING_XACTS. Which still leaves the contents >> of WAL record corrupted. There's not even a not

Re: pgsql: Avoid duplicate XIDs at recovery when building initial snapshot

2018-10-22 Thread Alvaro Herrera
On 2018-Oct-22, Andres Freund wrote: > Hi, > > On 2018-10-22 12:36:25 -0300, Alvaro Herrera wrote: > > On 2018-Oct-14, Andres Freund wrote: > > > > > On 2018-10-14 13:26:24 +, Michael Paquier wrote: > > > > Avoid duplicate XIDs at recovery when building initial snapshot > > > > > I'm unhapp

Re: pgsql: Avoid duplicate XIDs at recovery when building initial snapshot

2018-10-22 Thread Andres Freund
Hi, On 2018-10-22 12:03:26 +0900, Michael Paquier wrote: > (moving to -hackers) > > On Sun, Oct 14, 2018 at 10:42:40AM -0700, Andres Freund wrote: > > I'm unhappy this approach was taken over objections. Without a real > > warning. > > Oops, that was not clear to me. Sorry about that! I did no

Re: pgsql: Avoid duplicate XIDs at recovery when building initial snapshot

2018-10-22 Thread Andres Freund
Hi, On 2018-10-22 12:36:25 -0300, Alvaro Herrera wrote: > On 2018-Oct-14, Andres Freund wrote: > > > On 2018-10-14 13:26:24 +, Michael Paquier wrote: > > > Avoid duplicate XIDs at recovery when building initial snapshot > > > I'm unhappy this approach was taken over objections. Without a rea

Re: pgsql: Avoid duplicate XIDs at recovery when building initial snapshot

2018-10-22 Thread Alvaro Herrera
On 2018-Oct-14, Andres Freund wrote: > On 2018-10-14 13:26:24 +, Michael Paquier wrote: > > Avoid duplicate XIDs at recovery when building initial snapshot > I'm unhappy this approach was taken over objections. Without a real > warning. Even leaving the crummyness aside, did you check other

Re: pgsql: Avoid duplicate XIDs at recovery when building initial snapshot

2018-10-21 Thread Michael Paquier
(moving to -hackers) On Sun, Oct 14, 2018 at 10:42:40AM -0700, Andres Freund wrote: > I'm unhappy this approach was taken over objections. Without a real > warning. Oops, that was not clear to me. Sorry about that! I did not see you objecting again after the last arguments I raised. The end of