Re: Parallel copy

On Tue, Apr 7, 2020 at 9:38 AM Ants Aasma wrote: > I think the element based approach and requirement that all tuples fit > into the queue makes things unnecessarily complex. The approach I > detailed earlier allows for tuples to be bigger than the buffer. In > that case a worker will claim the lo

Re: [HACKERS] make async slave to wait for lsn to be replayed

On 2020-04-08 04:09, Kyotaro Horiguchi wrote: How about something like the follows. BEGIN AFTER ColId Sconst BEGIN FOLOWING ColId Sconst UNTIL ; LIMIT BY ; WITHIN Iconst; regards. I like your suggested keywords! I think that "AFTER" + "WITHIN" sound the most natural. We could completely giv

Re: where should I stick that backup?

On Wed, Apr 8, 2020 at 2:06 PM Stephen Frost wrote: > * Robert Haas (robertmh...@gmail.com) wrote: > > On Wed, Apr 8, 2020 at 1:05 PM Stephen Frost wrote: > > > What if %f.bz2 already exists? > > > > That cannot occur in the scenario I described. > > Of course it can. Not really. The steps I des

Re: backup manifests and contemporaneous buildfarm failures

On Wed, Apr 8, 2020 at 1:59 PM Tom Lane wrote: > I guess we could commit it and find out. I'm all for the simpler > coding if it works. I don't understand what the local $ENV{MSYS2_ARG_CONV_EXCL} = $source_ts_prefix does, but the remove/unlink condition was suggested by Amit Kapila on the basis

Re: where should I stick that backup?

Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Wed, Apr 8, 2020 at 2:06 PM Stephen Frost wrote: > > * Robert Haas (robertmh...@gmail.com) wrote: > > > On Wed, Apr 8, 2020 at 1:05 PM Stephen Frost wrote: > > > > What if %f.bz2 already exists? > > > > > > That cannot occur in the sce

Re: backup manifests and contemporaneous buildfarm failures

On 4/8/20 3:41 PM, Robert Haas wrote: > On Wed, Apr 8, 2020 at 1:59 PM Tom Lane wrote: >> I guess we could commit it and find out. I'm all for the simpler >> coding if it works. > I don't understand what the local $ENV{MSYS2_ARG_CONV_EXCL} = > $source_ts_prefix does, You don't want to know .

Re: backup manifests and contemporaneous buildfarm failures

Andrew Dunstan writes: > On 4/8/20 3:41 PM, Robert Haas wrote: >> I don't understand what the local $ENV{MSYS2_ARG_CONV_EXCL} = >> $source_ts_prefix does, > You don't want to know > See for the > gory details. I don't want to know

Re: [HACKERS] make async slave to wait for lsn to be replayed

Anna Akenteva writes: > I'd like to hear others' opinions on the syntax as well. Pardon me for coming very late to the party, but it seems like there are other questions that ought to be answered before we worry about any of this. Why is this getting grafted onto BEGIN/START TRANSACTION in the f

Re: explain HashAggregate to report bucket and memory stats

On Fri, Mar 20, 2020 at 03:44:42AM -0500, Justin Pryzby wrote: > On Fri, Mar 13, 2020 at 10:57:43AM -0700, Andres Freund wrote: > > On 2020-03-13 10:53:17 -0700, Jeff Davis wrote: > > > On Fri, 2020-03-13 at 10:27 -0700, Andres Freund wrote: > > > > On 2020-03-13 10:15:46 -0700, Jeff Davis wrote: >

Re: Improving connection scalability: GetSnapshotData()

Hi, On 2020-04-08 09:24:13 -0400, Robert Haas wrote: > On Tue, Apr 7, 2020 at 4:27 PM Andres Freund wrote: > > The main reason is that we want to be able to cheaply check the current > > state of the variables (mostly when checking a backend's own state). We > > can't access the "dense" ones with

Re: WIP: WAL prefetch (another approach)

On Thu, Apr 9, 2020 at 12:27 AM David Steele wrote: > On 4/8/20 8:12 AM, Thomas Munro wrote: > > Judging by the comments made in this thread and > > elsewhere, I think the feature is in demand so I hope there is a way > > we could get it into 13 in the next couple of days, but I totally > > accept

Re: Improving connection scalability: GetSnapshotData()

On Wed, Apr 8, 2020 at 09:44:16AM -0400, Robert Haas wrote: > I don't know what the right thing to do is. I agree with everyone who > says this is a very important problem, and I have the highest respect > for Andres's technical ability. On the other hand, I have been around > here long enough to

Re: Improving connection scalability: GetSnapshotData()

Hi, On 2020-04-08 09:26:42 -0400, Jonathan S. Katz wrote: > On 4/8/20 8:59 AM, Alexander Korotkov wrote: > > On Wed, Apr 8, 2020 at 3:43 PM Andres Freund wrote: > >> Realistically it still 2-3 hours of proof-reading. > >> > >> This makes me sad :( > > > > Can we ask RMT to extend feature freeze f

Re: Improving connection scalability: GetSnapshotData()

Hi, On 2020-04-08 09:44:16 -0400, Robert Haas wrote: > Moreover, shakedown time will be minimal because we're so late in the > release cycle My impression increasingly is that there's very little actual shakedown before beta :(. As e.g. evidenced by the fact that 2PC did basically not work for se

Re: explain HashAggregate to report bucket and memory stats

On Wed, 2020-04-08 at 16:00 -0500, Justin Pryzby wrote: > 90% of the initial goal of this patch was handled by instrumentation > added by > "hash spill to disk" (1f39bce02), but this *also* adds: > > - separate accounting for tuples vs hashtable; > - number of hash buckets; > - handles other ag

Re: Parallel copy

On Wed, 8 Apr 2020 at 22:30, Robert Haas wrote: > - If we're unable to supply data to the COPY process as fast as the > workers could load it, then speed will be limited at that point. We > know reading the file from disk is pretty fast compared to what a > single process can do. I'm not sure we'v

Re: Improving connection scalability: GetSnapshotData()

Hi, On 2020-04-08 18:06:23 -0400, Bruce Momjian wrote: > If we don't commit this, where does this leave us with the > old_snapshot_threshold feature? We remove it in back branches and have > no working version in PG 13? That seems kind of bad. I don't think this patch changes the situation for

Perl modules for testing/viewing/corrupting/repairing your heap files

Hackers, Recently, as part of testing something else, I had need of a tool to create surgically precise corruption within heap pages. I wanted to make the corruption from within TAP tests, so I wrote the tool as a set of perl modules. The modules allow you to "tie" a perl array to a heap file, i

Report error position in partition bound check

Hi, I'm playing with partitioned tables and found a minor thing with the error reporting of bounds checking when create partitions. In function check_new_partition_bound(), there are three places where we call ereport() with a parser_errposition(pstate, spec->location) argument. However, that ps

Re: Improving connection scalability: GetSnapshotData()

On Wed, Apr 8, 2020 at 03:25:34PM -0700, Andres Freund wrote: > Hi, > > On 2020-04-08 18:06:23 -0400, Bruce Momjian wrote: > > If we don't commit this, where does this leave us with the > > old_snapshot_threshold feature? We remove it in back branches and have > > no working version in PG 13? T

Re: Report error position in partition bound check

Forgot to run make installcheck. Here's the new version of the patch that updated the test answer file. From 9071918648412383e41976a01106257bd6a2539e Mon Sep 17 00:00:00 2001 From: Alexandra Wang Date: Wed, 8 Apr 2020 16:07:28 -0700 Subject: [PATCH v2] Report error position in partition bound chec

Re: Report error position in partition bound check

On Wed, Apr 08, 2020 at 05:15:57PM -0700, Alexandra Wang wrote: > I have attached a patch to pass in a ParseState to > check_new_partition_bound() to enable the reporting of the error > position. Below is what the error message looks like before and after > applying the patch. > > Another option i

Re: Thoughts on "killed tuples" index hint bits support on standby

On Wed, Apr 8, 2020 at 5:23 AM Michail Nikolaev wrote: > Yes, it is a brilliant idea to use uniqueness to avoid bloating the index. I > am > not able to understand all the code now, but I’ll check the patch in more > detail later. The design is probably simpler than you imagine. The algorithm tr

Re: Improving connection scalability: GetSnapshotData()

On Wed, Apr 08, 2020 at 03:17:41PM -0700, Andres Freund wrote: > On 2020-04-08 09:26:42 -0400, Jonathan S. Katz wrote: >> Lastly, with the ongoing world events, perhaps time that could have been >> dedicated to this and other patches likely affected their completion. I >> know most things in my lif

Re: Commitfest 2020-03 Now in Progress

On Wed, Apr 08, 2020 at 12:39:53PM -0400, Tom Lane wrote: > David Steele writes: >> The 2020-03 Commitfest is officially closed. >> Good job everyone! > > Thanks for running it! I know it's a tedious responsibility. Nice, David. Thanks a lot! -- Michael signature.asc Description: PGP signatu

Re: Vacuum o/p with (full 1, parallel 0) option throwing an error

On Wed, Apr 08, 2020 at 01:38:54PM -0500, Justin Pryzby wrote: > I think the behavior is correct, but the error message could be improved, > like: > |ERROR: cannot specify FULL with PARALLEL jobs > or similar. Perhaps "cannot use FULL and PARALLEL options together"? I think that this patch need

Re: Commitfest 2020-03 Now in Progress

On Wed, Apr 08, 2020 at 12:36:37PM -0400, David Steele wrote: > On 4/1/20 10:09 AM, David Steele wrote: > > On 3/17/20 8:10 AM, David Steele wrote: > > > On 3/1/20 4:10 PM, David Steele wrote: > > > > The last Commitfest for v13 is now in progress! > > > > > > > > Current stats for the Commitfest

Re: Commitfest 2020-03 Now in Progress

On Wed, Apr 8, 2020 at 6:45 PM David Fetter wrote: > Thanks so much for your hard work managing this one! +1 -- Peter Geoghegan

Re: Commitfest 2020-03 Now in Progress

On 2020-Apr-08, David Steele wrote: > The 2020-03 Commitfest is officially closed. > > Final stats are (for entire CF, not just from March 1 this time): Thanks for managing! -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training

Re: Report error position in partition bound check

On Wed, Apr 8, 2020 at 6:11 PM Michael Paquier wrote: > Please note that you forgot to update the regression test output. Yep thanks! Please see my previous email for the updated patch.

Re: adding partitioned tables to publications

On Wed, Apr 8, 2020 at 11:07 PM Amit Langote wrote: > On Wed, Apr 8, 2020 at 9:21 PM Peter Eisentraut > wrote: > > On 2020-04-08 13:16, Amit Langote wrote: > > > prion seems to have failed: > > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=prion&dt=2020-04-08%2009%3A53%3A13 > > > > Th

Re: Report error position in partition bound check

On Wed, Apr 08, 2020 at 08:17:55PM -0700, Alexandra Wang wrote: > On Wed, Apr 8, 2020 at 6:11 PM Michael Paquier wrote: > > Please note that you forgot to update the regression test output. > > Yep thanks! Please see my previous email for the updated patch. Thanks, I saw the update. It looks li

Re: [PATCH] Support built control file in PGXS VPATH builds

On Mon, 30 Mar 2020 at 11:50, Craig Ringer wrote: > > > On Mon, 9 Mar 2020, 17:27 Peter Eisentraut, < > peter.eisentr...@2ndquadrant.com> wrote: > >> On 2020-02-07 04:14, Craig Ringer wrote: >> > The attached patch fixes this by having PGXS resolve >> > $(EXTENSION).control along the VPATH. >> >>

Re: Planning counters in pg_stat_statements (using pgss_store)

On 2020/04/08 18:31, Julien Rouhaud wrote: On Wed, Apr 08, 2020 at 05:37:27PM +0900, Fujii Masao wrote: On 2020/04/03 16:26, Julien Rouhaud wrote: On Thu, Apr 02, 2020 at 01:04:28PM -0700, legrand legrand wrote: Fujii Masao-4 wrote On 2020/04/01 18:19, Fujii Masao wrote: Finally I pushe

Re: Planning counters in pg_stat_statements (using pgss_store)

On 2020/04/08 21:32, legrand legrand wrote: Fujii Masao-4 wrote On 2020/04/03 16:26 [...] "Note that planning and execution statistics are updated only at their respective end phase, and only for successful operations. For example the execution counters of a long running query will only be

Re: A problem about partitionwise join

On Thu, Apr 9, 2020 at 1:07 AM Tom Lane wrote: > Richard Guo writes: > > On Sun, Apr 5, 2020 at 4:38 AM Tom Lane wrote: > >> There is already something in equivclass.c that would almost do what > >> we want here: exprs_known_equal() would tell us whether the partkeys > >> can be found in the sa

Re: [HACKERS] advanced partition matching algorithm for partition-wise join

Hi, On Thu, Apr 9, 2020 at 12:06 AM Kuntal Ghosh wrote: > Both of your patches fix the problem. I don't have much exposure in > this area to comment on whether we should keep/remove the assertion > from the code. But, here is my opinion: > > The code structure looks like following: > Assert(condi

Re: A problem about partitionwise join

Richard Guo writes: > On Thu, Apr 9, 2020 at 1:07 AM Tom Lane wrote: >> I have hopes of being able to incorporate outer >> joins into the EC logic in a less squishy way in the future, by making >> the representation of Vars distinguish explicitly between >> value-before-outer-join and value-after

Re: Vacuum o/p with (full 1, parallel 0) option throwing an error

On Thu, Apr 9, 2020 at 7:07 AM Michael Paquier wrote: > > On Wed, Apr 08, 2020 at 01:38:54PM -0500, Justin Pryzby wrote: > > I think the behavior is correct, but the error message could be improved, > > like: > > |ERROR: cannot specify FULL with PARALLEL jobs > > or similar. > > Perhaps "cannot

Re: [HACKERS] advanced partition matching algorithm for partition-wise join

Etsuro Fujita writes: > Yeah, partition_bounds_merge() is currently called only from > try_partitionwise_join(), which guarantees that the strategies are the > same. The assertion cost would be cheap, but not zero, so I still > think it would be better to remove the assertion from > partition_bou

Fast DSM segments

Hello PostgreSQL 14 hackers, FreeBSD is much faster than Linux (and probably Windows) at parallel hash joins on the same hardware, primarily because its DSM segments run in huge pages out of the box. There are various ways to convince recent-ish Linux to put our DSMs on huge pages (see below for

Re: Vacuum o/p with (full 1, parallel 0) option throwing an error

On Thu, Apr 9, 2020 at 12:09 AM Justin Pryzby wrote: > > On Thu, Apr 09, 2020 at 12:06:04AM +0530, Mahendra Singh Thalor wrote: > > On Wed, 8 Apr 2020 at 22:11, Justin Pryzby wrote: > > > > > > > Thanks Justin for the patch. > > > > Patch looks fine to me and it is fixing the issue. After applyin

Multiple FPI_FOR_HINT for the same block during killing btree index items

Hi all, During investigating the issue our customer had, I realized that _bt_killitems() can record the same FPI pages multiple times simultaneously. This can happen when several concurrent index scans are processing pages that contain killable tuples. Because killing index items could be performe

Re: Commitfest 2020-03 Now in Progress

Le jeu. 9 avr. 2020 à 04:12, Alvaro Herrera a écrit : > On 2020-Apr-08, David Steele wrote: > > > The 2020-03 Commitfest is officially closed. > > > > Final stats are (for entire CF, not just from March 1 this time): > > Thanks for managing! > Thanks a lot for the hard work! >

Re: segmentation fault using currtid and partitioned tables

On Wed, Apr 08, 2020 at 04:13:31PM +0900, Michael Paquier wrote: > I have been looking at the tree and the use of the table AM APIs, and > those TID lookups are really a particular case compared to the other > callers of the table AM callbacks. Anyway, I have not spotted similar > problems, so I f

Re: Vacuum o/p with (full 1, parallel 0) option throwing an error

On Thu, 9 Apr 2020 at 14:52, Amit Kapila wrote: > > On Thu, Apr 9, 2020 at 12:09 AM Justin Pryzby wrote: > > > > On Thu, Apr 09, 2020 at 12:06:04AM +0530, Mahendra Singh Thalor wrote: > > > On Wed, 8 Apr 2020 at 22:11, Justin Pryzby wrote: > > > > > > > > > > Thanks Justin for the patch. > > > >

Re: [HACKERS] advanced partition matching algorithm for partition-wise join

On Thu, Apr 9, 2020 at 2:36 PM Tom Lane wrote: > Etsuro Fujita writes: > > Yeah, partition_bounds_merge() is currently called only from > > try_partitionwise_join(), which guarantees that the strategies are the > > same. > If there's only one caller and there's not likely to ever be more, > then

Re: Vacuum o/p with (full 1, parallel 0) option throwing an error

On Thu, Apr 09, 2020 at 11:05:50AM +0530, Amit Kapila wrote: > On Thu, Apr 9, 2020 at 7:07 AM Michael Paquier wrote: >> I think that >> this patch needs tests in sql/vacuum.sql. > > We already have one test that is testing an invalid combination of > PARALLEL and FULL option, not sure of adding

<    1   2