Re: print_path is missing GatherMerge and CustomScan support

2018-07-18 Thread Ashutosh Bapat
On Thu, Jul 19, 2018 at 11:56 AM, Michael Paquier wrote: > On Thu, Jul 19, 2018 at 10:41:10AM +0530, Ashutosh Bapat wrote: >> I had extensive testcases in my original patch-set to exercise that >> code but 1. that testset was too extensive; even today >> partition_join.sql is a separate testcase a

Re: Have an encrypted pgpass file

2018-07-18 Thread Marco van Eck
On Thu, Jul 19, 2018 at 5:19 AM Tom Lane wrote: > "Joshua D. Drake" writes: > > On 07/18/2018 04:25 PM, Tom Lane wrote: > >> This is exactly the kind of area in which I'm concerned for the > >> possibility of sloppily-written scripts being a net negative for > >> security. > > > Although I appre

Re: print_path is missing GatherMerge and CustomScan support

2018-07-18 Thread Michael Paquier
On Thu, Jul 19, 2018 at 10:41:10AM +0530, Ashutosh Bapat wrote: > I had extensive testcases in my original patch-set to exercise that > code but 1. that testset was too extensive; even today > partition_join.sql is a separate testcase and it's quite large. 2. > that function returns NULL rather tha

Re: Make foo=null a warning by default.

2018-07-18 Thread David Fetter
On Wed, Jul 18, 2018 at 04:15:30PM -0400, Fabien COELHO wrote: > > >[...] Done. > > All looks well, but I just noticed a warning: > > gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement > -Wendif-labels -Wmissing-format-attribute -Wformat-security > -fno-strict-aliasi

Re: More consistency for some file-related error message

2018-07-18 Thread Michael Paquier
On Thu, Jul 19, 2018 at 02:05:51PM +0900, Kyotaro HORIGUCHI wrote: > Agreed to it's not necessary and a developer ought to know about > the errno behavior. However, I can sympathize with Michael. I am fine to remove them if folks here push for that. > CopyGetData has a variant of it. > > |

project updates

2018-07-18 Thread Charles Cui
Hi mentors and hackers, Here is my current working status. 1. Complete the thrift_binary_in and thrift_binary_out functions, so that users can express their thrift struct using json. These two functions support both simple data struct and complex data structure like struct and map. 2. added te

Re: print_path is missing GatherMerge and CustomScan support

2018-07-18 Thread Ashutosh Bapat
On Thu, Jul 19, 2018 at 5:37 AM, Michael Paquier wrote: > On Wed, Jul 18, 2018 at 12:15:25PM +0530, Ashutosh Bapat wrote: >> Yes that's right. Thanks for taking care of it. > > Okay, I have pushed a fix for this one as that's wrong and > back-patched to v11. The coverage of reparameterize_path_by

Re: More consistency for some file-related error message

2018-07-18 Thread Kyotaro HORIGUCHI
Hello. At Thu, 19 Jul 2018 12:33:30 +0900, Michael Paquier wrote in <2018071900.gh3...@paquier.xyz> > On Wed, Jul 18, 2018 at 11:24:05PM -0400, Tom Lane wrote: > > read() is required by spec to set errno when returning a negative result. > > I think the previous coding paid attention to errn

Re: Possible bug in logical replication.

2018-07-18 Thread Erik Rijkers
On 2018-07-19 03:42, Michael Paquier wrote: On Wed, Jul 18, 2018 at 02:30:53PM -0400, Alvaro Herrera wrote: In the immortal words of Julian Bream: "yeah, I didn't like any of that". One wikipedia lookup later, I still don't know where this quote comes from, but at least I understand who the ma

Re: [HACKERS] PATCH: Keep one postmaster monitoring pipe per process

2018-07-18 Thread Thomas Munro
On Wed, Jul 18, 2018 at 8:30 PM, Kyotaro HORIGUCHI wrote: > At Wed, 18 Jul 2018 14:02:47 +1200, Thomas Munro > wrote in > >> Here are some of the places I had to add WL_EXIT_ON_PM_DEATH: >> gather_readnext(), shm_mq_send_bytes(), shm_mq_receive_bytes(), >> shm_mq_wait_internal(), ProcSleep(),

Re: Possible bug in logical replication.

2018-07-18 Thread Michael Paquier
On Thu, Jul 19, 2018 at 12:38:53AM -0400, Alvaro Herrera wrote: > On 2018-Jul-19, Michael Paquier wrote: >> I am fine either way if you want to have the last call. So please feel >> free to choose what you prefer here. That's no big deal. > > Okay. You want to push it, or shall I? It seems to

Re: Background worker/idle sessions and caching

2018-07-18 Thread Craig Ringer
On 19 July 2018 at 04:30, Jeremy Finzel wrote: My use case is similar to the example of worker_spi. A plpgsql function > runs every 1 minute and processes records in audit tables in order to > update fact tables with records that have changed. I noticed for example > renaming a column in the fu

Re: Possible bug in logical replication.

2018-07-18 Thread Alvaro Herrera
On 2018-Jul-19, Michael Paquier wrote: > I am fine either way if you want to have the last call. So please feel > free to choose what you prefer here. That's no big deal. Okay. You want to push it, or shall I? -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Developme

Re: untrusted PLs should be GRANTable

2018-07-18 Thread Craig Ringer
On 19 July 2018 at 08:23, Stephen Frost wrote: > Greetings, > > * Craig Ringer (cr...@2ndquadrant.com) wrote: > > Untrusted PLs should be GRANTable with a NOTICE or WARNING telling the > > admin that GRANTing an untrusted PL effectively gives the user the > ability > > to escape to superuser. > >

Re: Expression errors with "FOR UPDATE" and postgres_fdw with partition wise join enabled.

2018-07-18 Thread Etsuro Fujita
(2018/07/13 23:05), Ashutosh Bapat wrote: On Thu, Jul 12, 2018 at 4:32 PM, Etsuro Fujita wrote: In this example, the value of the whole-row reference to the child table ptp1 for that record is ('foo',1), and that of the index expression for that record is (1,'foo'). Those have different colum

Re: documentation about explicit locking

2018-07-18 Thread Amit Langote
On 2018/07/18 18:30, Peter Eisentraut wrote: > On 06.07.18 04:00, Amit Langote wrote: >> On 2018/07/05 23:02, Robert Haas wrote: >>> On Wed, Jul 4, 2018 at 3:09 AM, Amit Langote >>> wrote: I wonder why we mention on the following page that CREATE COLLATION requires SHARE ROW EXCLUSIVE lo

Re: patch to allow disable of WAL recycling

2018-07-18 Thread Kyotaro HORIGUCHI
At Thu, 19 Jul 2018 12:59:26 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20180719.125926.257896670.horiguchi.kyot...@lab.ntt.co.jp> > At Thu, 19 Jul 2018 12:37:26 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI > wrote in > <20180719.123726.00899102.horiguchi.kyot...@lab.ntt.co.jp

Re: patch to allow disable of WAL recycling

2018-07-18 Thread Kyotaro HORIGUCHI
At Thu, 19 Jul 2018 12:37:26 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20180719.123726.00899102.horiguchi.kyot...@lab.ntt.co.jp> > At Tue, 17 Jul 2018 21:01:03 -0400, Robert Haas wrote > in > > On Tue, Jul 17, 2018 at 3:12 PM, Peter Eisentraut > > wrote: > > > The actual imple

Re: patch to allow disable of WAL recycling

2018-07-18 Thread Kyotaro HORIGUCHI
At Thu, 19 Jul 2018 12:37:26 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20180719.123726.00899102.horiguchi.kyot...@lab.ntt.co.jp> > While considering this, I found a bug in 4b0d28de06, which > removed prior checkpoint from control file. It actually trims the > segments before the l

Re: patch to allow disable of WAL recycling

2018-07-18 Thread Kyotaro HORIGUCHI
At Tue, 17 Jul 2018 21:01:03 -0400, Robert Haas wrote in > On Tue, Jul 17, 2018 at 3:12 PM, Peter Eisentraut > wrote: > > The actual implementation could use another round of consideration. I > > wonder how this should interact with min_wal_size. Wouldn't > > min_wal_size = 0 already do what

Re: More consistency for some file-related error message

2018-07-18 Thread Michael Paquier
On Wed, Jul 18, 2018 at 11:24:05PM -0400, Tom Lane wrote: > read() is required by spec to set errno when returning a negative result. > I think the previous coding paid attention to errno regardless of the sign > of the result, which would justify pre-zeroing it ... but the new coding > definitely

Re: More consistency for some file-related error message

2018-07-18 Thread Tom Lane
Michael Paquier writes: > On Wed, Jul 18, 2018 at 10:57:16AM -0400, Tom Lane wrote: >> BTW, isn't the initial "errno = 0" dead code now? > Hm. I have not bothered touching those as it could be possible that > read() may not initialize errno to 0, so errno would remain set to any > previous value

Re: Have an encrypted pgpass file

2018-07-18 Thread Tom Lane
"Joshua D. Drake" writes: > On 07/18/2018 04:25 PM, Tom Lane wrote: >> This is exactly the kind of area in which I'm concerned for the >> possibility of sloppily-written scripts being a net negative for >> security. > Although I appreciate the concern, can we not worried about this? Your > argum

Re: Possible bug in logical replication.

2018-07-18 Thread Michael Paquier
On Wed, Jul 18, 2018 at 10:45:27PM -0400, Alvaro Herrera wrote: > On 2018-Jul-19, Michael Paquier wrote: > >> On Wed, Jul 18, 2018 at 02:30:53PM -0400, Alvaro Herrera wrote: >> > In the immortal words of Julian Bream: "yeah, I didn't like any of >> > that". >> >> One wikipedia lookup later, I sti

Re: More consistency for some file-related error message

2018-07-18 Thread Michael Paquier
On Wed, Jul 18, 2018 at 10:57:16AM -0400, Tom Lane wrote: > +1. Okay, thanks. I can always get that pushed first if there are no objections. More can be always done, but that's already a nice cut. > The short-read ereport has no errcode() call, meaning it will report > XX000, which seems like i

Re: Possible bug in logical replication.

2018-07-18 Thread Alvaro Herrera
On 2018-Jul-19, Michael Paquier wrote: > On Wed, Jul 18, 2018 at 02:30:53PM -0400, Alvaro Herrera wrote: > > In the immortal words of Julian Bream: "yeah, I didn't like any of > > that". > > One wikipedia lookup later, I still don't know where this quote comes > from, but at least I understand wh

Re: Add SKIP LOCKED to VACUUM and ANALYZE

2018-07-18 Thread Michael Paquier
On Wed, Jul 18, 2018 at 04:58:48PM +, Bossart, Nathan wrote: > On 7/17/18, 1:22 AM, "Michael Paquier" wrote: > Perhaps we could extend RangeVarGetRelidExtended() to only lock if > has_subclass() is true. However, I also understand Robert's position > on calling RangeVarGetRelidExtended() with

Re: Possible bug in logical replication.

2018-07-18 Thread Michael Paquier
On Wed, Jul 18, 2018 at 02:30:53PM -0400, Alvaro Herrera wrote: > In the immortal words of Julian Bream: "yeah, I didn't like any of > that". One wikipedia lookup later, I still don't know where this quote comes from, but at least I understand who the man is. I may be missing something, but I can

Re: [PATCH] pg_hba.conf : new auth option : clientcert=verify-full

2018-07-18 Thread Thomas Munro
On Sun, Jul 15, 2018 at 12:47 AM, Julian Markwort wrote: > Also, while writing this part of the docs, I tried to stay below 80 > characters, but I've exceeded it in some places. > There are several other places (several in the .sgml files touched by this > patch), where 80 characters are exceede

Re: YA race condition in 001_stream_rep.pl (was Re: pgsql: Allow using the updated tuple while moving it to a different par)

2018-07-18 Thread Michael Paquier
On Wed, Jul 18, 2018 at 05:42:40PM -0400, Tom Lane wrote: > Hearing nobody speaking in favor of the other alternatives, > I've removed the test. I had this problem running in the background for a couple of days, but I could not come up with a solution cleaner than just removing the test. Thanks fo

Re: print_path is missing GatherMerge and CustomScan support

2018-07-18 Thread Michael Paquier
On Wed, Jul 18, 2018 at 03:22:02PM +0900, Michael Paquier wrote: > Good catch. Those should be backpatched. While I am looking at this > stuff, I have noticed that pathnode.c/reparameterize_path_by_child uses > T_MergeAppend and not T_MergeAppendPath. Okay, I have checked the full list of path n

RE: Recovery performance of standby for multiple concurrent truncates on large tables

2018-07-18 Thread Jamison, Kirk
Hi, Thank you for your replies. On Tue, July 10, 2018 4:15 PM, Andres Freund wrote: >I think you'd run into a lot of very hairy details with this approach. >Consider what happens if client processes need fresh buffers and need to write >out a victim buffer. You'll need to know that the relevant

Re: [HACKERS] possible self-deadlock window after bad ProcessStartupPacket

2018-07-18 Thread Asim R P
On Thu, Jun 22, 2017 at 10:50 AM, Andres Freund wrote: > > Or, probably more robust: Simply _exit(2) without further ado, and rely > on postmaster to output an appropriate error message. Arguably it's not > actually useful to see hundreds of "WARNING: terminating connection because of > crash of a

Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is (fwd)

2018-07-18 Thread Fabien COELHO
Hello Heikki, [...] So threadRun() would not have the opportunity to stop the scheduled transaction, even if beyond the end of run, because it would not have got out of doCustom, in the case I outlined above. I see. Instead of moving to FINISHED state, then, we could stay in THROTTLE state,

Re: untrusted PLs should be GRANTable

2018-07-18 Thread Stephen Frost
Greetings, * Craig Ringer (cr...@2ndquadrant.com) wrote: > Untrusted PLs should be GRANTable with a NOTICE or WARNING telling the > admin that GRANTing an untrusted PL effectively gives the user the ability > to escape to superuser. I don't know that we really want to get into the business of iss

Re: Have an encrypted pgpass file

2018-07-18 Thread Joshua D. Drake
On 07/18/2018 04:25 PM, Tom Lane wrote: Alvaro Herrera writes: Seems to me that passing %-specifiers to the command would make it more useful (%u for "user", "host" etc) -- your command could refuse to give you a password for the superuser account for instance but grant one for a read-only user

Re: untrusted PLs should be GRANTable

2018-07-18 Thread Stephen Frost
Greetings, * Peter Eisentraut (peter.eisentr...@2ndquadrant.com) wrote: > On 17.07.18 07:20, Craig Ringer wrote: > > A user has raised the point that our refusal to GRANT rights to > > untrusted PLs is counterproductive and inconsistent with how we behave > > elsewhere. > > Previous discussion: >

Re: print_path is missing GatherMerge and CustomScan support

2018-07-18 Thread Michael Paquier
On Wed, Jul 18, 2018 at 12:15:25PM +0530, Ashutosh Bapat wrote: > Yes that's right. Thanks for taking care of it. Okay, I have pushed a fix for this one as that's wrong and back-patched to v11. The coverage of reparameterize_path_by_child is actually quite poor if you look at the reports: https:/

Re: Have an encrypted pgpass file

2018-07-18 Thread Tom Lane
Alvaro Herrera writes: > Seems to me that passing %-specifiers to the command would make it more > useful (%u for "user", "host" etc) -- your command could refuse to give > you a password for the superuser account for instance but grant one for > a read-only user. It would also provide a *very* f

Re: Postgres, fsync, and OSs (specifically linux)

2018-07-18 Thread Thomas Munro
On Thu, Jul 19, 2018 at 7:23 AM, Robert Haas wrote: > 2. I don't like promote_ioerr_to_panic() very much, partly because the > same pattern gets repeated over and over, and partly because it would > be awkwardly-named if we discovered that another 2 or 3 errors needed > similar handling (or some o

Re: Have an encrypted pgpass file

2018-07-18 Thread Alvaro Herrera
On 2018-Jul-18, Marco van Eck wrote: > Since .pgpass files contain plain-text passwords, I searched for an > alternative. > In the attached patch I've added the possibility to run a command to > produce the content of the pgpass file, in exactly the same format. In this > way I could use gpg or an

Re: psql's \d versus included-index-column feature

2018-07-18 Thread Alexander Korotkov
On Wed, Jul 18, 2018 at 11:14 PM David G. Johnston < david.g.johns...@gmail.com> wrote: > On Wed, Jul 18, 2018 at 12:55 PM, Tom Lane wrote: > >> >> regression=# \d tbl_include_reg_idx >> Index "public.tbl_include_reg_idx" >> Column | Type | Key | Definition >> +-+-

Re: Have an encrypted pgpass file

2018-07-18 Thread Thomas Munro
On Thu, Jul 19, 2018 at 9:52 AM, Tom Lane wrote: > Thomas Munro writes: >> On Thu, Jul 19, 2018 at 5:46 AM, Marco van Eck >> wrote: >>> Since .pgpass files contain plain-text passwords, I searched for an >>> alternative. >>> In the attached patch I've added the possibility to run a command to p

Re: GSOC 2018 Project - A New Sorting Routine

2018-07-18 Thread Alvaro Herrera
On 2018-Jul-18, Tomas Vondra wrote: > I don't have any script for that - load the files into a spreadsheet, > create pivot tables and you're done. What!? You don't use psql's \crosstabview !? ... walks away disappointed ... -- Álvaro Herrerahttps://www.2ndQuadrant.com/ Postgre

Re: Have an encrypted pgpass file

2018-07-18 Thread Tom Lane
Thomas Munro writes: > On Thu, Jul 19, 2018 at 5:46 AM, Marco van Eck wrote: >> Since .pgpass files contain plain-text passwords, I searched for an >> alternative. >> In the attached patch I've added the possibility to run a command to produce >> the content of the pgpass file, in exactly the sam

Re: YA race condition in 001_stream_rep.pl (was Re: pgsql: Allow using the updated tuple while moving it to a different par)

2018-07-18 Thread Tom Lane
Amit Kapila writes: > On Sat, Jul 14, 2018 at 11:46 PM, Tom Lane wrote: >> So the issue boils down to this: the test script is, effectively, >> assuming that it's guaranteed that the walreceiver will send a feedback >> message before it shuts down; but there is no such guarantee. Is this >> a bu

Re: Have an encrypted pgpass file

2018-07-18 Thread Christophe Pettus
> On Jul 18, 2018, at 14:33, Thomas Munro wrote: > Here you side step those questions completely and make that the end > user's problem. I like it. +1. This is a clever solution, since any kind of key vault or other system could be dropped in there. -- -- Christophe Pettus x...@thebuild

Re: Have an encrypted pgpass file

2018-07-18 Thread Thomas Munro
On Thu, Jul 19, 2018 at 5:46 AM, Marco van Eck wrote: > Since .pgpass files contain plain-text passwords, I searched for an > alternative. > In the attached patch I've added the possibility to run a command to produce > the content of the pgpass file, in exactly the same format. In this way I > co

Re: GSOC 2018 Project - A New Sorting Routine

2018-07-18 Thread Tomas Vondra
I don't have any script for that - load the files into a spreadsheet, create pivot tables and you're done. regards On 07/18/2018 11:13 PM, Kefan Yang wrote: > Hey Tomas! > >   > > I am trying to reproduce the results on my machine. Could you please > share the script to generate .ods files? >

RE: GSOC 2018 Project - A New Sorting Routine

2018-07-18 Thread Kefan Yang
Hey Tomas! I am trying to reproduce the results on my machine. Could you please share the script to generate .ods files? Regards, Kefan From: Tomas Vondra Sent: July 18, 2018 2:05 AM To: Andrey Borodin Cc: Peter Geoghegan; Kefan Yang; PostgreSQL Hackers Subject: Re: GSOC 2018 Project - A New So

Re: GiST VACUUM

2018-07-18 Thread Heikki Linnakangas
On 18/07/18 21:27, Andrey Borodin wrote: Hi! 18 июля 2018 г., в 16:02, Heikki Linnakangas написал(а): , but I think it would be better to split this into two patches as follows: 1st patch: Scan the index in physical rather than logical order. No attempt at deleting empty pages yet. 2nd patc

Re: psql's \d versus included-index-column feature

2018-07-18 Thread Alvaro Herrera
On 2018-Jul-18, Tom Lane wrote: > I can sympathize with the eyestrain argument against t/f, but the > above doesn't seem like an improvement --- in particular, "Data" > as the column header seems quite content-free. My counterproposal > is to keep "Key" as the header and use "Yes"/"No" as the val

Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is (fwd)

2018-07-18 Thread Heikki Linnakangas
On 18/07/18 23:29, Fabien COELHO wrote: Hmm. How about we just remove this special case from doCustom(): case CSTATE_START_THROTTLE: // ... if (duration > 0 && st->txn_scheduled > end_time) { st->state = CSTATE_FINISHED; break; } That way, we let the cli

Re: psql's \d versus included-index-column feature

2018-07-18 Thread Tom Lane
Alvaro Herrera writes: > On 2018-Jul-18, David G. Johnston wrote: >> -1 for printing a boolean t/f; would rather spell it out: >> >> CASE WHEN "Key" THEN 'Key' ELSE 'Included' END AS "Data" > +1 I can sympathize with the eyestrain argument against t/f, but the above doesn't seem like an improve

Re: Background worker/idle sessions and caching

2018-07-18 Thread Jeremy Finzel
On Wed, Jul 18, 2018 at 3:05 PM, Tom Lane wrote: > Jeremy Finzel writes: > > I have a background worker running SQL functions, and I believe I have > > noticed that when I do things like change function definitions, or even > add > > tables, the background worker does not pick up the schema chan

Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is (fwd)

2018-07-18 Thread Fabien COELHO
Hmm. How about we just remove this special case from doCustom(): case CSTATE_START_THROTTLE: // ... if (duration > 0 && st->txn_scheduled > end_time) { st->state = CSTATE_FINISHED; break; } That way, we let the client go into CSTATE_THROTTLE state, even tho

Re: psql's \d versus included-index-column feature

2018-07-18 Thread Alvaro Herrera
On 2018-Jul-18, David G. Johnston wrote: > ​-1 for printing a boolean t/f; would rather spell it out: > > CASE WHEN "Key" THEN 'Key' ELSE 'Included' END AS "Data" +1 -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Servi

Re: Make foo=null a warning by default.

2018-07-18 Thread Fabien COELHO
[...] Done. All looks well, but I just noticed a warning: gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -O2 -I. -I. -I

Re: psql's \d versus included-index-column feature

2018-07-18 Thread David G. Johnston
On Wed, Jul 18, 2018 at 12:55 PM, Tom Lane wrote: > > regression=# \d tbl_include_reg_idx > Index "public.tbl_include_reg_idx" > Column | Type | Key | Definition > +-+-- > c1 | integer | t | c1 > c2 | integer | t | c2 > c3 | integer | f | c

Re: Background worker/idle sessions and caching

2018-07-18 Thread Tom Lane
Jeremy Finzel writes: > I have a background worker running SQL functions, and I believe I have > noticed that when I do things like change function definitions, or even add > tables, the background worker does not pick up the schema changes until I > restart the worker. Maybe you need some Accept

Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is (fwd)

2018-07-18 Thread Heikki Linnakangas
On 18/07/18 22:56, Fabien COELHO wrote: Hello Heikki, Yep. The attached version does only the tailing stuff under -R and not all threads were stopped on errors, with comments to tell about the why. Hmm. How about we just remove this special case from doCustom(): case CSTATE_START_THROTTL

psql's \d versus included-index-column feature

2018-07-18 Thread Tom Lane
I noticed that psql's \d command doesn't do very well with included index columns. Given the regression db's test case, CREATE INDEX tbl_include_reg_idx ON tbl_include_reg (c1, c2) INCLUDE (c3, c4); we get regression=# \d tbl_include_reg_idx Index "public.tbl_include_reg_idx" Column | Type

Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is (fwd)

2018-07-18 Thread Fabien COELHO
Hello Heikki, Yep. The attached version does only the tailing stuff under -R and not all threads were stopped on errors, with comments to tell about the why. Hmm. How about we just remove this special case from doCustom(): case CSTATE_START_THROTTLE: // ... if (duration > 0 && st->t

Re: patch to allow disable of WAL recycling

2018-07-18 Thread Robert Haas
On Tue, Jul 17, 2018 at 4:47 PM, Tomas Vondra wrote: > Makes sense, I guess. But I think many claims made in this thread are > mostly just assumptions at this point, based on our beliefs how CoW or > non-CoW filesystems work. The results from ZFS (showing positive impact) > are an exception, but t

Re: patch to allow disable of WAL recycling

2018-07-18 Thread Robert Haas
On Wed, Jul 18, 2018 at 3:22 PM, Jerry Jelinek wrote: > I've gotten a wide variety of feedback on the proposed patch. The comments > range from rough approval through various discussion about alternative > solutions. At this point I am unsure if this patch is rejected or if it > would be accepted

Re: Postgres, fsync, and OSs (specifically linux)

2018-07-18 Thread Robert Haas
On Tue, May 29, 2018 at 4:53 AM, Craig Ringer wrote: > I've revised the fsync patch with the cleanups discussed and gone through > the close() calls. > > AFAICS either socket closes, temp file closes, or (for WAL) already PANIC on > close. It's mainly fd.c that needs amendment. Which I've done pe

Re: patch to allow disable of WAL recycling

2018-07-18 Thread Jerry Jelinek
I've gotten a wide variety of feedback on the proposed patch. The comments range from rough approval through various discussion about alternative solutions. At this point I am unsure if this patch is rejected or if it would be accepted once I had the updated man page changes that were discussed las

Re: [PG-11] Potential bug related to INCLUDE clause of CREATE INDEX

2018-07-18 Thread Tom Lane
Yugo Nagata writes: > To fix this, we agree with Tom about getting rid of "must not intersect" > restriction. > A patch is attached for this Pushed, after fixing documentation and regression tests to match. regards, tom lane

Background worker/idle sessions and caching

2018-07-18 Thread Jeremy Finzel
I have a background worker running SQL functions, and I believe I have noticed that when I do things like change function definitions, or even add tables, the background worker does not pick up the schema changes until I restart the worker. Is this expected behavior? If I use background workers i

Re: Faster str to int conversion (was Table with large number of int columns, very slow COPY FROM)

2018-07-18 Thread Robert Haas
On Sat, Jul 7, 2018 at 4:01 PM, Andres Freund wrote: > FWIW, here's a rebased version of this patch. Could probably be polished > further. One might argue that we should do a bit more wide ranging > changes, to convert scanint8 and pg_atoi to be also unified. But it > might also just be worthwhile

Re: Possible bug in logical replication.

2018-07-18 Thread Alvaro Herrera
On 2018-Jul-12, Michael Paquier wrote: > On Wed, Jul 04, 2018 at 10:50:28AM +0900, Michael Paquier wrote: > > On Tue, Jul 03, 2018 at 01:17:48AM -0400, Alvaro Herrera wrote: > > > Let me review tomorrow. > > > > Of course, please feel free. > > Alvaro, are you planning to look at that to close t

Re: GiST VACUUM

2018-07-18 Thread Andrey Borodin
Hi! > 18 июля 2018 г., в 16:02, Heikki Linnakangas написал(а): > > In the corresponding B-tree code, we use don't do actual recursion, but a > hand-optimized "tail recursion", to avoid stack overflow if there are a lot > of splits. I think we need to do something like tha there, too. I don't t

Re: Subplan result caching

2018-07-18 Thread Robert Haas
On Mon, Jul 9, 2018 at 5:08 AM, David Rowley wrote: > From looking at the code I see that the caching is being done inside > nodeSubplan.c. I don't think this is the right approach to the > problem. The problem exists for any parameterized path, so I think a > more general approach would be much b

Re: ENOSPC FailedAssertion("!(RefCountErrors == 0)"

2018-07-18 Thread Tom Lane
Robert Haas writes: > On Wed, Jul 18, 2018 at 10:33 AM, Tom Lane wrote: >>> So couldn't we use TopTransactionResourceOwner instead of >>> AuxProcessResrouceOwner? I feel a bit uneasy that bootstrap and >>> standalone-backend have *AuxProcess*ResourceOwner. >> Since the aux processes aren't runni

Have an encrypted pgpass file

2018-07-18 Thread Marco van Eck
Hi, Since .pgpass files contain plain-text passwords, I searched for an alternative. In the attached patch I've added the possibility to run a command to produce the content of the pgpass file, in exactly the same format. In this way I could use gpg or any other command to decrypt a pgpass file. I

Re: ENOSPC FailedAssertion("!(RefCountErrors == 0)"

2018-07-18 Thread Robert Haas
On Wed, Jul 18, 2018 at 10:33 AM, Tom Lane wrote: >> So couldn't we use TopTransactionResourceOwner instead of >> AuxProcessResrouceOwner? I feel a bit uneasy that bootstrap and >> standalone-backend have *AuxProcess*ResourceOwner. > > Since the aux processes aren't running transactions, I didn't

Re: Add SKIP LOCKED to VACUUM and ANALYZE

2018-07-18 Thread Bossart, Nathan
Hi Michael, Thanks for taking a look. On 7/17/18, 1:22 AM, "Michael Paquier" wrote: > The first thing which is striking me is that we may actually *not* want > to check for lock skipping within expand_vacuum_rel() as that's mainly a > function aimed at building the relations which are going to b

Re: [HACKERS] logical decoding of two-phase transactions

2018-07-18 Thread Robert Haas
On Wed, Jul 18, 2018 at 11:27 AM, Tomas Vondra wrote: >> One idea is that maybe the running transaction could communicate with >> the decoding process through shared memory. For example, suppose that >> before you begin decoding an ongoing transaction, you have to send >> some kind of notificatio

Re: One transaction and several processes

2018-07-18 Thread David G. Johnston
On Wed, Jul 18, 2018 at 8:31 AM, Valery Kuzmin wrote: > Hi, > > Can multiple processes participate in a single transaction's execution? > I need to do the following actions sequence: > 1. Calling service begins transaction. At this step transaction > manager generate new XID. > 2. Calling service

Re: One transaction and several processes

2018-07-18 Thread Tomas Vondra
On 07/18/2018 05:31 PM, Valery Kuzmin wrote: Hi, Can multiple processes participate in a single transaction's execution? I need to do the following actions sequence: 1. Calling service begins transaction. At this step transaction manager generate new XID. 2. Calling service updates some data. 3.

One transaction and several processes

2018-07-18 Thread Valery Kuzmin
Hi, Can multiple processes participate in a single transaction's execution? I need to do the following actions sequence: 1. Calling service begins transaction. At this step transaction manager generate new XID. 2. Calling service updates some data. 3. Need to detach a transaction from a service co

Re: [HACKERS] logical decoding of two-phase transactions

2018-07-18 Thread Tomas Vondra
On 07/18/2018 04:56 PM, Robert Haas wrote: On Wed, Jul 18, 2018 at 10:08 AM, Tomas Vondra wrote: The problem is you don't know if a transaction does DDL sometime later, in the part that you might not have decoded yet (or perhaps concurrently with the decoding). So I don't see how you could easi

Re: Make foo=null a warning by default.

2018-07-18 Thread David Fetter
On Wed, Jul 18, 2018 at 08:25:46AM -0400, Fabien COELHO wrote: > > Hello David, > > >I assure you that you expression yourself in English a good deal > >better than I do in Portuguese. > > Alas, despite a Portuguese "rabbit" name, I cannot speak the language which > got lost between generations.

Re: Segfault logical replication PG 10.4

2018-07-18 Thread Tom Lane
Mai Peng writes: > Here the backtrace Hmm .. so this can be summarized as "logical replication workers should provide an ActiveSnapshot in case the user functions they call want one". Makes me wonder how much other transactional infrastructure is needed but not present. r

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-18 Thread Heikki Linnakangas
On 18/07/18 16:29, Robert Haas wrote: On Wed, Jul 18, 2018 at 9:06 AM, Michael Paquier wrote: What's wrong with the approach proposed in http://postgr.es/m/55afc302.1060...@iki.fi ? For back-branches that's very invasive so that seems risky to me particularly seeing the low number of complain

Re: More consistency for some file-related error message

2018-07-18 Thread Tom Lane
Michael Paquier writes: > While looking at the source code for more consistency work with error > messages, I have bumped into a couple of messages which could be > simplified, as those include in the name of the file manipulated > basically the same information as the context added. > I have fin

Re: [HACKERS] logical decoding of two-phase transactions

2018-07-18 Thread Robert Haas
On Wed, Jul 18, 2018 at 10:08 AM, Tomas Vondra wrote: > The problem is you don't know if a transaction does DDL sometime later, in > the part that you might not have decoded yet (or perhaps concurrently with > the decoding). So I don't see how you could easily exclude such transactions > from the

Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is (fwd)

2018-07-18 Thread Heikki Linnakangas
On 18/07/18 16:01, Fabien COELHO wrote: I don't think you want to wait in that situation. I think we should wait at the end only if there some threads still alive, with nothing to do only because of --rate. Yep. The attached version does only the tailing stuff under -R and not all threads were

Re: partition tree inspection functions

2018-07-18 Thread Jesper Pedersen
Hi Amit, On 06/28/2018 01:49 AM, Amit Langote wrote: OK, I've added an example below the table of functions added by the patch. Attached updated patch. You forgot to remove the test output in create_table.out, so check-world is failing. In pg_partition_parent + else + /* Not

Re: Segfault logical replication PG 10.4

2018-07-18 Thread Mai Peng
Hello, Here the backtrace ```Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `postgres: bgworker: logical replication worker for subscription'. Program terminated with signal SIGSEGV, Segmentation fault. #0 GetActiveSnapshot () at ./build/../src/b

Re: ENOSPC FailedAssertion("!(RefCountErrors == 0)"

2018-07-18 Thread Tom Lane
Kyotaro HORIGUCHI writes: > Hello. I confirmed that this patch fixes the crash. Thanks for double-checking. > At Tue, 17 Jul 2018 20:01:05 -0400, Tom Lane wrote in > <14892.1531872...@sss.pgh.pa.us> >> Further investigation showed that the part of that code that was >> actually needed was not

Re: [HACKERS] logical decoding of two-phase transactions

2018-07-18 Thread Tomas Vondra
On 07/17/2018 08:10 PM, Robert Haas wrote: On Mon, Jul 16, 2018 at 3:25 PM, Tomas Vondra wrote: Oh, right, I forgot the patch also adds the leader into the group, for some reason (I agree it's unclear why that would be necessary, as you pointed out later). But all this is happening while hol

Re: [HACKERS] PoC: full merge join on comparison clause

2018-07-18 Thread Ashutosh Bapat
On Tue, Jul 10, 2018 at 10:06 PM, Alexander Kuzmenkov wrote: > I tried to fix the things you mentioned and improve the comments. Among > other changes, there is now a description of how merge join works with > inequalities at the top of nodeMergejoin.c. It also explains why we only > support one i

Re: why partition pruning doesn't work?

2018-07-18 Thread Tom Lane
Amit Langote writes: > On 2018/07/16 2:02, Tom Lane wrote: >> BTW, there'd be a lot to be said for having InitPlan just open all >> the rels and build an array of Relation pointers that parallels the >> RTE list, rather than doing heap_opens in random places elsewhere. > +1 to this. Actually I h

Re: Alter index rename concurrently to

2018-07-18 Thread Peter Eisentraut
On 18.07.18 11:44, Andrey Klychkov wrote: > If lower locking is safe and possible to be used by default in renaming > it will be great. > What stage is solving of this issue? Does anybody do it? We wait to see if anyone has any concerns about this change. -- Peter Eisentraut http://

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-18 Thread Robert Haas
On Wed, Jul 18, 2018 at 9:06 AM, Michael Paquier wrote: >> What's wrong with the approach proposed in >> http://postgr.es/m/55afc302.1060...@iki.fi ? > > For back-branches that's very invasive so that seems risky to me > particularly seeing the low number of complaints on the matter. Hmm. I think

Re: PG 10: could not generate random cancel key

2018-07-18 Thread Dean Rasheed
On 18 July 2018 at 14:01, Michael Paquier wrote: > Thanks for the updated version. This looks safer to me. It is possible > to simplify the code by removing the external RAND_status() call and > check for RAND_status() first in the loop as per the attached. OK, thanks. Barring any further comm

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-18 Thread Michael Paquier
On Wed, Jul 18, 2018 at 06:42:10AM -0400, Robert Haas wrote: > On Tue, Jul 17, 2018 at 8:28 AM, Andrew Dunstan > wrote: >> Well, we'd be getting rid of it because of a danger of data loss which we >> can't otherwise mitigate. Maybe it does need to be backpatched, even if we >> haven't had complain

Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is (fwd)

2018-07-18 Thread Fabien COELHO
Hello Heikki, I did that in the attached version: no more environment variable hack, and no execution shortcut even if there is nothing to do. I also had to reproduce the progress logic to keep on printing report of (no) progress in this tailing phase. On second thoughts, there's one problem

Re: PG 10: could not generate random cancel key

2018-07-18 Thread Michael Paquier
On Wed, Jul 18, 2018 at 10:14:56AM +0100, Dean Rasheed wrote: > OK, I guess that it is possible that an older version of OpenSSL > requires RAND_poll() to be called multiple times. Here's an updated > patch doing that (with up to 8 retries, based on the old OpenSSL > code). Thanks for the updated

  1   2   >