Re: logical copy_replication_slot issues

On Mon, 10 Feb 2020 at 23:01, Arseny Sher wrote: > > > Masahiko Sawada writes: > > > I've attached the draft patch fixing this issue but I'll continue > > investigating it more deeply. > > There also should be a check that source slot itself has consistent > snapshot (valid confirmed_flush) -- ot

Re: [Patch] Make pg_checksums skip foreign tablespace directories

On Fri, Jan 31, 2020 at 05:39:36PM +0900, Kyotaro Horiguchi wrote: > At Fri, 31 Jan 2020 17:30:43 +0900 (JST), Kyotaro Horiguchi > wrote in >> I don't think that is a problem right away, of course. It looks good >> to me except for the possible excessive exclusion. So, I don't object >> it if w

Re: [HACKERS] WAL logging problem in 9.4.3?

At Tue, 18 Feb 2020 23:44:52 -0800, Noah Misch wrote in > I think attached v35nm is ready for commit to master. Would anyone like to > talk me out of back-patching this? I would not enjoy back-patching it, but > it's hard to justify lack of back-patch for a data-loss bug. > > Notable changes s

Re: [HACKERS] WAL logging problem in 9.4.3?

Sorry, just one fix. (omitting some typos, though..) At Wed, 19 Feb 2020 17:29:08 +0900 (JST), Kyotaro Horiguchi wrote in > At Tue, 18 Feb 2020 23:44:52 -0800, Noah Misch wrote in > > I think attached v35nm is ready for commit to master. Would anyone like to > > talk me out of back-patching

Re: Parallel copy

On Wed, 19 Feb 2020 at 06:22, Amit Kapila wrote: > > On Tue, Feb 18, 2020 at 8:08 PM Ants Aasma wrote: > > > > On Tue, 18 Feb 2020 at 15:21, Amit Kapila wrote: > > > > > > On Tue, Feb 18, 2020 at 5:59 PM Ants Aasma wrote: > > > > > > > > On Tue, 18 Feb 2020 at 12:20, Amit Kapila > > > > wrote

Re: Parallel copy

On Wed, Feb 19, 2020 at 11:02:15AM +0200, Ants Aasma wrote: On Wed, 19 Feb 2020 at 06:22, Amit Kapila wrote: On Tue, Feb 18, 2020 at 8:08 PM Ants Aasma wrote: > > On Tue, 18 Feb 2020 at 15:21, Amit Kapila wrote: > > > > On Tue, Feb 18, 2020 at 5:59 PM Ants Aasma wrote: > > > > > > On Tue, 1

Re: Updating row and width estimates in postgres_fdw

Hi Ashutosh, Long time no see! On Thu, Feb 13, 2020 at 1:17 AM Ashutosh Bapat wrote: > In postgresGetForeignJoinPaths(), I see > >/* Estimate costs for bare join relation */ > estimate_path_cost_size(root, joinrel, NIL, NIL, NULL, > &rows, &width, &startup_cos

Re: Portal->commandTag as an enum

Hi Mark, On Wed, Feb 19, 2020 at 10:40 AM Mark Dilger wrote: > This would only make sense to me if the string held in $_ had already been > checked for safety, but Catalog.pm does very little to verify that the string > is safe to eval. The assumption here, so far as I can infer, is that we >

Re: small improvement of the elapsed time for truncating heap in vacuum

On 2020/02/17 17:52, Fujii Masao wrote: On 2020/02/17 14:28, Kasahara Tatsuhito wrote: Hi, On Mon, Feb 17, 2020 at 1:07 PM Masahiko Sawada wrote: For the patch, we can put also the declaration of ru0 into the loop. Thanks for your reply. Hmm, certainly that it may be better. Fix the v2

Re: RecoveryWalAll and RecoveryWalStream wait events

On 2020/02/18 14:20, Kyotaro Horiguchi wrote: Hello. At Tue, 18 Feb 2020 12:25:51 +0900, Fujii Masao wrote in Hi, RecoveryWalAll and RecoveryWalStream wait events are documented as follows. RecoveryWalAll Waiting for WAL from any kind of source (local, archive or stream) at

Re: pg_stat_progress_basebackup - progress reporting for pg_basebackup, in the server side

On 2020/02/19 11:22, Amit Langote wrote: On Tue, Feb 18, 2020 at 9:32 PM Fujii Masao wrote: OK, I changed the doc that way. Attached the updated version of the patch. Thank you. Looks good to me. Thanks for the review! You think that the patch can be marked as "ready for committer"? Re

Re: pg_stat_progress_basebackup - progress reporting for pg_basebackup, in the server side

On Wed, Feb 19, 2020 at 9:49 PM Fujii Masao wrote: > On 2020/02/19 11:22, Amit Langote wrote: > > On Tue, Feb 18, 2020 at 9:32 PM Fujii Masao > > wrote: > >> OK, I changed the doc that way. Attached the updated version of the patch. > > > > Thank you. Looks good to me. > > Thanks for the review

Re: Parallel copy

On Wed, Feb 19, 2020 at 4:08 PM Tomas Vondra wrote: > > The one piece of information I'm missing here is at least a very rough > quantification of the individual steps of CSV processing - for example > if parsing takes only 10% of the time, it's pretty pointless to start by > parallelising this pa

Re: Increase psql's password buffer size

On 2020/01/22 11:01, Fujii Masao wrote: On 2020/01/22 0:12, Bruce Momjian wrote: On Tue, Jan 21, 2020 at 02:42:07PM +0900, Fujii Masao wrote: I have no strong opinion about the maximum length of password, for now. But IMO it's worth committing that 0001 patch as the first step for this prob

Constraint's NO INHERIT option is ignored in CREATE TABLE LIKE statement

Hi hackers, My colleague Chris Travers discovered something that looks like a bug. Let's say we have a table with a constraint that is declared as NO INHERIT. CREATE TABLE test ( x INT CHECK (x > 0) NO INHERIT ); \d test Table "public.test" Column | Type | Collation | Null

Re: DROP OWNED CASCADE vs Temp tables

The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:not tested I have tested the patch with REL_12_STABLE for the given error sc

Re: Do we need to handle orphaned prepared transactions in the server?

All, Attached is version 1 of POC patch for notifying of orphaned prepared transactions via warnings emitted to a client application and/or log file. It applies to PostgreSQL branch "master" on top of "e2e02191" commit. I've tried to keep the patch as less invasive as I could with minimal impact

Re: Internal key management system

On Sat, 15 Feb 2020 at 01:00, Robert Haas wrote: > > On Thu, Feb 6, 2020 at 9:19 PM Masahiko Sawada > wrote: > > This feature protects data from disk thefts but cannot protect data > > from attackers who are able to access PostgreSQL server. In this > > design application side still is responsibl

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

On 14.02.2020 05:57, Peter Geoghegan wrote: Attached is v33, which adds the last piece we need: opclass infrastructure that tells nbtree whether or not deduplication can be applied safely. This is based on work by Anastasia that was shared with me privately. Thank you for this work. I've looked

Re: BUG #16108: Colorization to the output of command-line has unproperly behaviors at Windows platform

On Tue, Feb 18, 2020 at 11:39 PM Michail Nikolaev < michail.nikol...@gmail.com> wrote: > > I have checked the patch source code and it seems to be working. But a > few moments I want to mention: > Thanks for looking into this. > I think it is not good idea to mix the logic of detecting the fact

Re: Make ringbuffer threshold and ringbuffer sizes configurable?

On Wed, Feb 05, 2020 at 08:00:26PM -0800, Andres Freund wrote: > I think it would make sense to have seqscan_ringbuffer_threshold, > {bulkread,bulkwrite,vacuum}_ringbuffer_size. I suggest the possibility of somehow forcing a ringbuffer for nonbulk writes for the current session. In our use-case,

Re: Clean up some old cruft related to Windows

On Wed, Feb 19, 2020 at 4:49 AM Michael Paquier wrote: > On Tue, Feb 18, 2020 at 12:26:06PM +0100, Juan José Santamaría Flecha > wrote: > > I cannot point when SetEnv.bat was exactly dropped, probably Windows 7 > SDK > > was the place where it was *last* included [1], so that needs to be > update

Re: [Patch] Make pg_checksums skip foreign tablespace directories

On 2/19/20 2:13 AM, Michael Paquier wrote: On Fri, Jan 31, 2020 at 05:39:36PM +0900, Kyotaro Horiguchi wrote: At Fri, 31 Jan 2020 17:30:43 +0900 (JST), Kyotaro Horiguchi wrote in I don't think that is a problem right away, of course. It looks good to me except for the possible excessive exclu

Re: [Patch] Make pg_checksums skip foreign tablespace directories

On 1/31/20 3:59 AM, Michael Banck wrote: Hi, Am Freitag, den 31.01.2020, 13:53 +0900 schrieb Michael Paquier: On Thu, Jan 30, 2020 at 06:11:22PM +0100, Michael Banck wrote: Having a past tablespace version left around after an upgrade is a pilot error in my opinion because pg_upgrade generates

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

On Wed, Feb 19, 2020 at 8:14 AM Anastasia Lubennikova wrote: > Thank you for this work. I've looked through the patches and they seem > to be ready for commit. > I haven't yet read recent documentation and readme changes, so maybe > I'll send some more feedback tomorrow. Great. > Is there any sp

Re: Memory-Bounded Hash Aggregation

Hi, I've started reviewing the 20200218 version of the patch. In general it seems fine, but I have a couple minor comments and two crashes. 1) explain.c currently does this: I wonder if we could show something for plain explain (without analyze). At least the initial estimate of partitions, et

Re: Resolving the python 2 -> python 3 mess

On 2020-02-19 05:39, Tom Lane wrote: After thinking about this awhile longer, I'm starting to believe we should do some of each. That is, the stub replacement for plpython2.so should redirect "plpythonu" functions to plpython3.so, but throw errors for "plpython2u" functions. I'm not sure these

Re: Resolving the python 2 -> python 3 mess

Peter Eisentraut writes: > Your scheme appears to center around the assumption that people will > want to port their functions at the same time as not building plpython2u > anymore. Not really; use of the proposed porting infrastructure is the same whether plpython2u still works or not. You en

Re: Unicode normalization SQL functions

On 2020-02-17 20:14, Daniel Verite wrote: The comment in full says: /* * unicode_normalize - Normalize a Unicode string to the specified form. * * The

Re: explain HashAggregate to report bucket and memory stats

On Sun, Feb 16, 2020 at 11:53:07AM -0600, Justin Pryzby wrote: > Updated: > > . remove from explain analyze those tests which would display sort >Memory/Disk. Oops. . Rebased on top of 5b618e1f48aecc66e3a9f60289491da520faae19 . Updated to avoid sort's Disk output, for real this time. . A

Re: Unicode normalization SQL functions

On 2020-02-17 20:08, Daniel Verite wrote: Concerning execution speed, there's an excessive CPU usage when normalizing into NFC or NFKC. Looking at pre-existing code, it looks like recompose_code() in unicode_norm.c looping over the UnicodeDecompMain array might be very costly. Yes, this is a kn

Re: error context for vacuum to include block number

Rebased on top of 007491979461ff10d487e1da9bcc87f2fd834f26 Also, I was thinking that lazy_scan_heap doesn't needs to do this: + /* Pop the error context stack while calling vacuum */ + error_context_stack = errcallback.previous; ... + /* Set the error context while c

Re: Increase psql's password buffer size

Fujii Masao writes: > Attached is the patch that Nathan proposed at [1] and I think that > it's worth applying. I'd like to add this to next CommitFest. > Thought? I can't get excited about this in the least. For any "normal" use of passwords, 100 bytes is surely far more than sufficient. Furth

Re: Delaying/avoiding BTreeTupleGetNAtts() call within _bt_compare()

Peter Geoghegan writes: > It also inlines (in the second patch) by marking the _bt_compare > definition as inline, while not changing anything in nbtree.h. I > believe that this is portable C99 -- let's see what CF Tester thinks > of it. Boy, I'd be pretty darn hesitant to go there, even with our

Re: pg_regress cleans up tablespace twice.

Kyotaro Horiguchi writes: > But in the first place it comes from [1] and the comment says: >> * XXX it would be better if pg_regress.c had nothing at all to do with >> * testtablespace, and this were handled by a .BAT file or similar on >> * Windows. See pgsql-hackers discussion of 2008-01-18.

Re: Yet another fast GiST build

On Wed, Feb 19, 2020 at 8:00 PM Thomas Munro wrote: > Could this function please have a comment that explains why it works? > I mean, just a breadcrumb... the name of the technique or something... > so that uninitiated hackers can google their way to a clue (is it > "Morton encoding"?) Ok I think

Re: Delaying/avoiding BTreeTupleGetNAtts() call within _bt_compare()

On Wed, Feb 19, 2020 at 12:55 PM Tom Lane wrote: > Boy, I'd be pretty darn hesitant to go there, even with our new > expectation of C99 compilers. What we found out when we last experimented > with non-static inlines was that the semantics were not very portable nor > desirable. I've forgotten t

Re: Delaying/avoiding BTreeTupleGetNAtts() call within _bt_compare()

Peter Geoghegan writes: > On Wed, Feb 19, 2020 at 12:55 PM Tom Lane wrote: >> Boy, I'd be pretty darn hesitant to go there, even with our new >> expectation of C99 compilers. What we found out when we last experimented >> with non-static inlines was that the semantics were not very portable nor

bad wal on replica / incorrect resource manager data checksum in record / zfs

Hello Postgres Hackers - We are having a reoccurring issue on 2 of our replicas where replication stops due to this message: "incorrect resource manager data checksum in record at ..." This has been occurring on average once every 1 to 2 weeks during large data imports (100s of GBs being written)

Re: Delaying/avoiding BTreeTupleGetNAtts() call within _bt_compare()

Hi, On 2020-02-19 15:55:38 -0500, Tom Lane wrote: > Peter Geoghegan writes: > > It also inlines (in the second patch) by marking the _bt_compare > > definition as inline, while not changing anything in nbtree.h. I > > believe that this is portable C99 -- let's see what CF Tester thinks > > of it.

Re: Delaying/avoiding BTreeTupleGetNAtts() call within _bt_compare()

Andres Freund writes: > On 2020-02-19 15:55:38 -0500, Tom Lane wrote: >> Boy, I'd be pretty darn hesitant to go there, even with our new >> expectation of C99 compilers. What we found out when we last experimented >> with non-static inlines was that the semantics were not very portable nor >> des

Re: Delaying/avoiding BTreeTupleGetNAtts() call within _bt_compare()

On Wed, Feb 19, 2020 at 2:45 PM Tom Lane wrote: > The buildfarm would only tell you if it compiles, not whether the > performance results are what you hoped for. Right. Plus I think that more granular control might make more sense in this particular instance anyway. -- Peter Geoghegan

Re: Constraint's NO INHERIT option is ignored in CREATE TABLE LIKE statement

Ildar Musin writes: > My colleague Chris Travers discovered something that looks like a bug. > Let's say we have a table with a constraint that is declared as NO INHERIT. > ... > Now when we want to make a copy of the table structure into a new table > the `NO INHERIT` option is ignored. Hm, I ag

Re: Constraint's NO INHERIT option is ignored in CREATE TABLE LIKE statement

On Wed, Feb 19, 2020 at 4:02 PM Tom Lane wrote: > Ildar Musin writes: > > My colleague Chris Travers discovered something that looks like a bug. > > Let's say we have a table with a constraint that is declared as NO > INHERIT. > > ... > > Now when we want to make a copy of the table structure in

Re: Parallel copy

On Fri, Feb 14, 2020 at 01:41:54PM +0530, Amit Kapila wrote: > This work is to parallelize the copy command and in particular "Copy > from 'filename' Where ;" command. Apropos of the initial parsing issue generally, there's an interesting approach taken here: https://github.com/robertdavidgraham/

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

On Wed, Feb 19, 2020 at 11:16 AM Peter Geoghegan wrote: > On Wed, Feb 19, 2020 at 8:14 AM Anastasia Lubennikova > wrote: > > Thank you for this work. I've looked through the patches and they seem > > to be ready for commit. > > I haven't yet read recent documentation and readme changes, so maybe

Re: WIP: expression evaluation improvements

Hey Andres, > Awesome! +1. Attached 2nd version of patch rebased on master. > (v2-0001-Resolve-PL-handler-names-for-JITed-code-instead-o.patch) > > > > > Did you check whether there's any cases this fails in the tree with your > > patch applied? The way I usually do that is by running the regressi

Re: Constraint's NO INHERIT option is ignored in CREATE TABLE LIKE statement

On Thu, Feb 20, 2020 at 8:02 AM Tom Lane wrote: > Ildar Musin writes: > > My colleague Chris Travers discovered something that looks like a bug. > > Let's say we have a table with a constraint that is declared as NO INHERIT. > > ... > > Now when we want to make a copy of the table structure into

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

Hi, On 2020-02-19 11:12:18 +0530, Amit Kapila wrote: > I think till we know the real need for changing group locking, going > in the direction of what Tom suggested to use an array of LWLocks [1] > to address the problems in hand is a good idea. -many I think that building yet another locking su

Re: Constraint's NO INHERIT option is ignored in CREATE TABLE LIKE statement

On Thu, Feb 20, 2020 at 8:20 AM David G. Johnston wrote: > On Wed, Feb 19, 2020 at 4:02 PM Tom Lane wrote: >> Ildar Musin writes: >> > My colleague Chris Travers discovered something that looks like a bug. >> > Let's say we have a table with a constraint that is declared as NO INHERIT. >> > ...

Re: Clean up some old cruft related to Windows

On Wed, Feb 19, 2020 at 07:02:55PM +0100, Juan José Santamaría Flecha wrote: > Please find a patched for so. I have tried to make it more version > neutral. + You can change certain build options, such as the targeted CPU + architecture, build type, and the selection of the SDK by using either +

Re: Print physical file path when checksum check fails

Hi, On 2020-02-19 16:48:45 +0900, Michael Paquier wrote: > On Wed, Feb 19, 2020 at 03:00:54PM +0900, Kyotaro Horiguchi wrote: > > I have had support requests related to broken block several times, and > > (I think) most of *them* had hard time to locate the broken block or > > even broken file. I

Re: backend type in log_line_prefix?

Hi, On 2020-02-13 09:56:38 +0100, Peter Eisentraut wrote: > Attached is a demo patch that adds a placeholder %b for log_line_prefix (not > in the default setting) that contains the backend type, the same that you > see in pg_stat_activity and in the ps status. I would have found this > occasional

Experimenting with transactional memory for SERIALIZABLE

Hello hackers, Here's a *highly* experimental patch set that tries to skip the LWLock protocol in predicate.c and use HTM[1] instead. HTM is itself a sort of hardware-level implementation of SSI for shared memory. My thinking was that if your workload already suits the optimistic nature of SSI,

Re: Memory-Bounded Hash Aggregation

On Wed, Feb 19, 2020 at 08:16:36PM +0100, Tomas Vondra wrote: > 4) lookup_hash_entries says > > /* check to see if we need to spill the tuple for this grouping set */ > > But that seems bogus, because AFAIK we can't spill tuples for grouping > sets. So maybe this should say just "grouping"? As

Re: Improve heavyweight locks instead of building new lock managers?

Hi, Some of the discussions about improving the locking code, in particular the group locking / deadlock detector discussion with Robert, again made me look at lock.c. While looking at how much work it'd be to a) remove the PROCLOCK hashtable b) move more of the group locking logic into lock.c, r

Re: [PoC] Non-volatile WAL buffer

Hi, On 2020-02-17 13:12:37 +0900, Takashi Menjo wrote: > I applied my patchset that mmap()-s WAL segments as WAL buffers to > refs/tags/REL_12_0, and measured and analyzed its performance with > pgbench. Roughly speaking, When I used *SSD and ext4* to store WAL, > it was "obviously worse" than th

Re: pg_regress cleans up tablespace twice.

On Wed, Feb 19, 2020 at 04:06:33PM -0500, Tom Lane wrote: > I think the existing coding dates from before we had a Perl driver for > this, or else we had it but there were other less-functional ways to > replace "make check" on Windows. +1 for taking the code out of > pg_regress.c --- but I'm not

Re: Memory-Bounded Hash Aggregation

On Wed, Feb 19, 2020 at 08:16:36PM +0100, Tomas Vondra wrote: > 5) Assert(nbuckets > 0); > ... > This however quickly fails on this assert in BuildTupleHashTableExt (see > backtrace1.txt): > > Assert(nbuckets > 0); > > The value is computed in hash_choose_num_buckets, and there seem to be > no

Re: Improve heavyweight locks instead of building new lock managers?

Hi Andres, On Thu, Feb 20, 2020 at 1:14 PM Andres Freund wrote: > Here's a *draft* patch series for removing all use of SHM_QUEUE, and > subsequently removing SHM_QUEUE. There's a fair bit of polish needed, > but I do think it makes the code considerably easier to read > (particularly for predica

Re: Autovacuum on partitioned table

Hello, I'm sorry for the delay. Attach the latest patch based on discussion in this thread. > > Yeah that is what I meant. In addition, adding partition's > > changes_since_analyze to its parent needs to be done recursively as > > the parent table could also be a partitioned table. > > That's a g

Re: Print physical file path when checksum check fails

Thanks, On Thu, Feb 20, 2020 at 11:36 AM Andres Freund wrote: > Hi, > > On 2020-02-19 16:48:45 +0900, Michael Paquier wrote: > > On Wed, Feb 19, 2020 at 03:00:54PM +0900, Kyotaro Horiguchi wrote: > > > I have had support requests related to broken block several times, and > > > (I think) most of

Re: Print physical file path when checksum check fails

Thanks Kyotaro, On Wed, Feb 19, 2020 at 2:02 PM Kyotaro Horiguchi wrote: > At Wed, 19 Feb 2020 13:28:04 +0900, Michael Paquier > wrote in > > On Wed, Feb 19, 2020 at 01:07:36PM +0900, Kyotaro Horiguchi wrote: > > > If we also verify checksum in md layer, callback is overkill since the > > > imm

Re: [Patch] Make pg_checksums skip foreign tablespace directories

On Wed, Feb 19, 2020 at 12:37:00PM -0600, David Steele wrote: > On 2/19/20 2:13 AM, Michael Paquier wrote: >> Please note that pg_internal.init is listed within noChecksumFiles in >> basebackup.c, so we would miss any temporary pg_internal.init.PID if >> we don't check after the file prefix and the

Re: Error on failed COMMIT

Am 17.02.2020 um 23:12 schrieb Dave Cramer: On Mon, 17 Feb 2020 at 13:02, Haumacher, Bernhard > wrote: ... would be an appropriate solution. PostgreSQL reports the "unsuccessful" commit through the "ROLLBACK" status code and the driver translates this in

Re: Internal key management system

On Wed, 19 Feb 2020 at 09:29, Cary Huang wrote: > > Hi > > I have tried the attached kms_v3 patch and have some comments: > > 1) In the comments, I think you meant hmac + iv + encrypted data instead of > iv + hmac + encrypted data? > > ---> in kmgr_wrap_key( ): > + /* > +* Assemble

Re: Autovacuum on partitioned table

Hosoya-san, On Thu, Feb 20, 2020 at 3:34 PM yuzuko wrote: > Attach the latest patch based on discussion in this thread. > > > > Yeah that is what I meant. In addition, adding partition's > > > changes_since_analyze to its parent needs to be done recursively as > > > the parent table could also be

Re: Bug in pg_restore with EventTrigger in parallel mode

On Wed, Feb 12, 2020 at 01:59:05PM -0300, Fabrízio de Royes Mello wrote: > In parallel mode it's firing the EventTrigger and it can't be happen. > Poking around it I did some test with attached just to leave EventTriggers > in pending_list to process it in restore_toc_entries_postfork and > everyth