Re: Protect syscache from bloating with negative cache entries

2019-02-13 Thread Kyotaro HORIGUCHI
At Wed, 13 Feb 2019 02:15:42 +, "Tsunakawa, Takayuki" wrote in <0A3221C70F24FB45833433255569204D1FB97CF1@G01JPEXMBYT05> > From: Tomas Vondra [mailto:tomas.von...@2ndquadrant.com] > > > I didin't consider planning that happen within a function. If > > > 5min is the default for catalog_cache_p

Re: libpq compression

2019-02-13 Thread Konstantin Knizhnik
On 12.02.2019 17:05, Andreas Karlsson wrote: On 2/11/19 5:28 PM, Peter Eisentraut wrote: One mitigation is to not write code like that, that is, don't put secret parameters and user-supplied content into the same to-be-compressed chunk, or at least don't let the end user run that code at will

Re: Protect syscache from bloating with negative cache entries

2019-02-13 Thread Kyotaro HORIGUCHI
At Tue, 12 Feb 2019 18:33:46 +0100, Tomas Vondra wrote in > > "catalog_cache_prune_min_age", "catalog_cache_memory_target", (if > > exists) "catalog_cache_entry_limit" and > > "catalog_cache_prune_ratio" make sense? > > > > I think "catalog_cache" sounds about right, although my point was simp

Re: 2019-02-14 Press Release Draft

2019-02-13 Thread Daniel Gustafsson
> On 13 Feb 2019, at 04:00, Jonathan S. Katz wrote: > Attached is a draft of the press release for the upcoming 2019-02-14 > release. Feedback & suggestions are welcome. I think it should be “versions” in the below sentence? "all currently supported version of PostgreSQL will only contain"

RE: [PROPOSAL]a new data type 'bytea' for ECPG

2019-02-13 Thread Matsumura, Ryo
Meskes-san At first, I find my mistake that the following member is not used in my patch. Sorry... [ecpglib_extern.h] 120 struct descriptor_item 130 int data_len; > Why is handling a bytea so different from handling a varchar? Current architecture: Internal expression of varch

Re: Prevent extension creation in temporary schemas

2019-02-13 Thread Chris Travers
On Sat, Jan 12, 2019 at 12:48 AM Michael Paquier wrote: > On Sat, Jan 12, 2019 at 08:34:37AM +0900, Michael Paquier wrote: > > Then the extension is showing up as beginning to be present for other > > users. I am mainly wondering if this case has actually been thought > > about in the past or di

Re: Challenges preventing us moving to 64 bit transaction id (XID)?

2019-02-13 Thread Chris Travers
On Fri, Mar 2, 2018 at 12:07 AM Andres Freund wrote: > On 2018-03-02 01:56:00 +0300, Alexander Korotkov wrote: > > On Fri, Mar 2, 2018 at 1:51 AM, Andres Freund > wrote: > > > > > On 2018-03-02 01:48:03 +0300, Alexander Korotkov wrote: > > > > Also, the last commitfest is already too late for su

Re: [PATCH] xlogreader: do not read a file block twice

2019-02-13 Thread Arthur Zakirov
On 12.02.2019 20:47, Andrey Lepikhov wrote: I looked at the history of the code changes: --- 7fcbf6a405f (Alvaro Herrera 2013-01-16 16:12:53 -0300 539) reqLen < state->readLen) 1bb2558046c (Heikki Linnakangas 2010-01-27 15:27:51 +000

Re: Challenges preventing us moving to 64 bit transaction id (XID)?

2019-02-13 Thread Chris Travers
On Thu, Mar 1, 2018 at 11:48 PM Alexander Korotkov < a.korot...@postgrespro.ru> wrote: > Hi! > > On Fri, Mar 2, 2018 at 1:41 AM, Andres Freund wrote: > >> On 2018-01-11 01:02:52 +0300, Alexander Korotkov wrote: >> > As I get from cputube, patchset doesn't compiles again. Please find >> > revised

Re: more unconstify use

2019-02-13 Thread Peter Eisentraut
On 07/02/2019 09:14, Peter Eisentraut wrote: > Here is a patch that makes more use of unconstify() by replacing casts > whose only purpose is to cast away const. Also a preliminary patch to > remove casts that were useless to begin with. committed -- Peter Eisentraut http://www.2nd

Re: subscriptionCheck failures on nightjar

2019-02-13 Thread Thomas Munro
On Mon, Feb 11, 2019 at 7:31 PM Tom Lane wrote: > 2019-02-10 23:55:58.798 EST [40728] sub1 PANIC: could not open file > "pg_logical/snapshots/0-160B578.snap": No such file or directory They get atomically renamed into place, which seems kosher even if snapshots for the same LSN are created co

Re: [PROPOSAL]a new data type 'bytea' for ECPG

2019-02-13 Thread Michael Meskes
Matsumura-san, > Current architecture: > Internal expression of varchar is C-string that includes length > information implicitly > because the length can be computed by strlen(). > ECPGdo(ecpg_build_params) assumes that the data in descriptor is C- > string encoded. > > In other hand, bytea data

WAL insert delay settings

2019-02-13 Thread Peter Eisentraut
Bulk operations like CREATE INDEX, ALTER TABLE, or bulk loads can create a lot of WAL. A lot of WAL at once can cause delays in replication. For synchronous replication, this can make seemingly unrelated sessions hang. But also for asynchronous replication, it will increase latency. One idea to

Re: WAL insert delay settings

2019-02-13 Thread Andres Freund
Hi, On February 13, 2019 1:16:07 PM GMT+01:00, Peter Eisentraut wrote: >Bulk operations like CREATE INDEX, ALTER TABLE, or bulk loads can >create >a lot of WAL. A lot of WAL at once can cause delays in replication. >For synchronous replication, this can make seemingly unrelated sessions >hang.

Re: [HACKERS] Block level parallel vacuum

2019-02-13 Thread Haribabu Kommi
On Sat, Feb 9, 2019 at 11:47 PM Masahiko Sawada wrote: > On Tue, Feb 5, 2019 at 12:14 PM Haribabu Kommi > wrote: > > > > > > On Fri, Feb 1, 2019 at 8:19 AM Masahiko Sawada > wrote: > >> > >> > >> The passing stats = NULL to amvacuumcleanup and ambulkdelete means the > >> first time execution. F

Re: 2019-02-14 Press Release Draft,2019-02-14 Press Release Draft

2019-02-13 Thread Tatsuo Ishii
> 2019-02-14 Cumulative Update Release > > > The PostgreSQL Global Development Group has released an update to all > supported versions of our database system, including 11.2, 10.7, 9.6.12, > 9.5.16, and 9.4.21. This release changes the behavior in how Postgr

Re: 2019-02-14 Press Release Draft

2019-02-13 Thread Jonathan S. Katz
On 2/13/19 1:35 PM, Tatsuo Ishii wrote: >> 2019-02-14 Cumulative Update Release >> >> >> The PostgreSQL Global Development Group has released an update to all >> supported versions of our database system, including 11.2, 10.7, 9.6.12, >> 9.5.16, and 9.4.21. Th

Re: Libpq support to connect to standby server as priority

2019-02-13 Thread Haribabu Kommi
On Fri, Feb 8, 2019 at 8:16 PM Tsunakawa, Takayuki < tsunakawa.ta...@jp.fujitsu.com> wrote: > From: Haribabu Kommi [mailto:kommi.harib...@gmail.com] > > target_session_attrs checks for the default_transaction_readonly or not? > > PG 11 uses transaction_read_only, not default_transaction_readonly.

Re: 2019-02-14 Press Release Draft

2019-02-13 Thread Michael Banck
Hi, On Tue, Feb 12, 2019 at 10:14:55PM -0500, Jonathan S. Katz wrote: > On 2/13/19 4:13 AM, Chapman Flack wrote: > > On 02/12/19 22:00, Jonathan S. Katz wrote: > >> Attached is a draft of the press release for the upcoming 2019-02-14 > >> release. Feedback & suggestions are welcome. > > > >

Re: 2019-02-14 Press Release Draft

2019-02-13 Thread Jonathan S. Katz
On 2/13/19 8:15 AM, Michael Banck wrote: > Hi, > > On Tue, Feb 12, 2019 at 10:14:55PM -0500, Jonathan S. Katz wrote: >> On 2/13/19 4:13 AM, Chapman Flack wrote: >>> On 02/12/19 22:00, Jonathan S. Katz wrote: Attached is a draft of the press release for the upcoming 2019-02-14 release. Fe

Re: WAL insert delay settings

2019-02-13 Thread dataegret
Hi 13.02.2019 17:16, Peter Eisentraut пишет: Bulk operations like CREATE INDEX, ALTER TABLE, or bulk loads can create a lot of WAL. A lot of WAL at once can cause delays in replication. For synchronous replication, this can make seemingly unrelated sessions hang. But also for asynchronous repl

Re: Challenges preventing us moving to 64 bit transaction id (XID)?

2019-02-13 Thread Andres Freund
Hi, On 2019-02-13 12:16:33 +0100, Chris Travers wrote: > As a note here, I have worked on projects where there could be 2-week-long > idle-in-transaction states (no joke, we tuned things to only use virtual > xids for most of that time), and having an ability to set > idle-in-transaction timeouts

Re: libpq compression

2019-02-13 Thread Dmitry Dolgov
On Mon, Feb 11, 2019 at 4:52 PM Andres Freund wrote: > > On 2019-02-11 12:46:07 -0300, Alvaro Herrera wrote: > > On 2019-Feb-11, Andreas Karlsson wrote: > > > > > Imagine the following query which uses the session ID from the cookie to > > > check if the logged in user has access to a file. > > >

Re: reducing isolation tests runtime

2019-02-13 Thread Tom Lane
Andres Freund writes: > I'm working an updated version of this. Adding the new tests is a bit > painful because of conflicting names making it harder than necessary to > schedule tests. While it's possible to work out a schedule that doesn't > conflict, it's pretty annoying to do and more importan

[Patch][WiP] Tweaked LRU for shared buffers

2019-02-13 Thread Andrey Borodin
Hi, hackers! We have held education project at Sirius edu center (Sochi, Russia) with mentors from Yandex. The group of 5 students was working on improving the shared buffers eviction algorithm: Andrey Chausov, Yuriy Skakovskiy, Ivan Edigaryev, Arslan Gumerov, Daria Filipetskaya. I’ve been a me

Re: libpq compression

2019-02-13 Thread Konstantin Knizhnik
Hi Dmitry, On 13.02.2019 16:56, Dmitry Dolgov wrote: On Mon, Feb 11, 2019 at 4:52 PM Andres Freund wrote: On 2019-02-11 12:46:07 -0300, Alvaro Herrera wrote: On 2019-Feb-11, Andreas Karlsson wrote: Imagine the following query which uses the session ID from the cookie to check if the logged

Re: libpq compression

2019-02-13 Thread Dmitry Dolgov
> On Wed, Feb 13, 2019 at 3:46 PM Konstantin Knizhnik > wrote: > > Moreover, please notice that your implementation is still passing functions > tx/rx functions to stream constructor and so zpq_read is still able to read > data itself. So I do not understand which problem you have solved by > repl

Re: libpq compression

2019-02-13 Thread Dmitry Dolgov
> On Wed, Feb 13, 2019 at 3:52 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > On Wed, Feb 13, 2019 at 3:46 PM Konstantin Knizhnik > > wrote: > > > > Moreover, please notice that your implementation is still passing functions > > tx/rx functions to stream constructor and so zpq_read is still

Re: WAL insert delay settings

2019-02-13 Thread Tom Lane
Andres Freund writes: > On February 13, 2019 1:16:07 PM GMT+01:00, Peter Eisentraut > wrote: >> One idea to address this is to slow down WAL-generating maintenance >> operations. This is similar to the vacuum delay. Where the vacuum >> delay counts notional I/O cost before sleeping, here we wo

Re: libpq compression

2019-02-13 Thread Konstantin Knizhnik
On 13.02.2019 17:54, Dmitry Dolgov wrote: On Wed, Feb 13, 2019 at 3:52 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: On Wed, Feb 13, 2019 at 3:46 PM Konstantin Knizhnik wrote: Moreover, please notice that your implementation is still passing functions tx/rx functions to stream constructo

Re: WAL insert delay settings

2019-02-13 Thread Stephen Frost
Greetings, * Peter Eisentraut (peter.eisentr...@2ndquadrant.com) wrote: > Bulk operations like CREATE INDEX, ALTER TABLE, or bulk loads can create > a lot of WAL. A lot of WAL at once can cause delays in replication. Agreed, though I think VACUUM should certainly be included in this. I'm all fo

Re: reducing isolation tests runtime

2019-02-13 Thread Alvaro Herrera
On 2019-Feb-13, Tom Lane wrote: > Andres Freund writes: > > I'm working an updated version of this. Adding the new tests is a bit > > painful because of conflicting names making it harder than necessary to > > schedule tests. While it's possible to work out a schedule that doesn't > > conflict, i

replace_text optimization (StringInfo to varlena)

2019-02-13 Thread Daniel Verite
Hi, replace_text() in varlena.c builds the result in a StringInfo buffer, and finishes by copying it into a freshly allocated varlena structure with cstring_to_text_with_len(), in the same memory context. It looks like that copy step could be avoided by preprending the varlena header to the Str

Re: WAL insert delay settings

2019-02-13 Thread Peter Eisentraut
On 13/02/2019 13:18, Andres Freund wrote: > But I don't think the way you did it is acceptable - we can't just delay > while holding buffer locks, in critical sections, while not interruptible. The code I added to XLogInsertRecord() is not inside the critical section. -- Peter Eisentraut

Re: WAL insert delay settings

2019-02-13 Thread Andres Freund
On February 13, 2019 4:39:21 PM GMT+01:00, Peter Eisentraut wrote: >On 13/02/2019 13:18, Andres Freund wrote: >> But I don't think the way you did it is acceptable - we can't just >delay while holding buffer locks, in critical sections, while not >interruptible. > >The code I added to XLogInse

Re: reducing isolation tests runtime

2019-02-13 Thread Tom Lane
Alvaro Herrera writes: > On 2019-Feb-13, Tom Lane wrote: >> Some of the slower buildfarm critters use MAX_CONNECTIONS to limit >> the load on their hosts. As long as the isolation tests honor that, >> I don't see a real need for a separate serial schedule. > MAX_CONNECTIONS was the only reason I

Re: Reaping Temp tables to avoid XID wraparound

2019-02-13 Thread Magnus Hagander
On Wed, Feb 13, 2019 at 2:26 AM Michael Paquier wrote: > On Wed, Feb 13, 2019 at 12:38:51AM +, Andrew Gierth wrote: > > Doesn't work - that function's idea of "backend id" doesn't match the > > real one, since it's looking at a local copy of the stats from which > > unused slots have been rem

Re: subscriptionCheck failures on nightjar

2019-02-13 Thread Tom Lane
Thomas Munro writes: > On Mon, Feb 11, 2019 at 7:31 PM Tom Lane wrote: >> 2019-02-10 23:55:58.798 EST [40728] sub1 PANIC: could not open file >> "pg_logical/snapshots/0-160B578.snap": No such file or directory > > They get atomically renamed into place, which seems kosher even if > snapshots

Re: reducing isolation tests runtime

2019-02-13 Thread Andres Freund
On 2019-02-13 10:58:50 -0500, Tom Lane wrote: > Alvaro Herrera writes: > > On 2019-Feb-13, Tom Lane wrote: > >> Some of the slower buildfarm critters use MAX_CONNECTIONS to limit > >> the load on their hosts. As long as the isolation tests honor that, > >> I don't see a real need for a separate s

Re: Reaping Temp tables to avoid XID wraparound

2019-02-13 Thread Tom Lane
Magnus Hagander writes: > And while at it, what would in this particular case have been even more > useful to the OP would be to actually identify that there is a temp table > *and which xid it's blocking at*. For regular transactions we can look at > backend_xid, but IIRC that doesn't work for te

Re: Ryu floating point output patch

2019-02-13 Thread Andrew Gierth
> "Andrew" == Andrew Gierth writes: Andrew> 2. How far do we need to go to support existing uses of Andrew>extra_float_digits? For example, do we need a way for clients to Andrew>request that they get the exact same output as previously for Andrew>extra_float_digits=2 or 3, ra

Re: Reaping Temp tables to avoid XID wraparound

2019-02-13 Thread Magnus Hagander
On Wed, Feb 13, 2019 at 6:05 PM Tom Lane wrote: > Magnus Hagander writes: > > And while at it, what would in this particular case have been even more > > useful to the OP would be to actually identify that there is a temp table > > *and which xid it's blocking at*. For regular transactions we ca

Re: subscriptionCheck failures on nightjar

2019-02-13 Thread Andres Freund
Hi, On 2019-02-13 11:57:32 -0500, Tom Lane wrote: > I've managed to reproduce this locally, and obtained this PANIC: Cool. How exactly? Nice catch. > Anyway, I think we might be able to fix this along the lines of > > CloseTransientFile(fd); > > + /* ensure snapshot file is down to stab

Re: subscriptionCheck failures on nightjar

2019-02-13 Thread Tom Lane
Andres Freund writes: > On 2019-02-13 11:57:32 -0500, Tom Lane wrote: >> I've managed to reproduce this locally, and obtained this PANIC: > Cool. How exactly? Andrew told me that nightjar is actually running in a qemu VM, so I set up freebsd 9.0 in a qemu VM, and boom. It took a bit of fiddling

Re: subscriptionCheck failures on nightjar

2019-02-13 Thread Andres Freund
Hi, On 2019-02-13 12:37:35 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2019-02-13 11:57:32 -0500, Tom Lane wrote: > >> I've managed to reproduce this locally, and obtained this PANIC: > > > Cool. How exactly? > > Andrew told me that nightjar is actually running in a qemu VM, > so I se

Re: reducing isolation tests runtime

2019-02-13 Thread Tom Lane
Andres Freund writes: > Do you have an idea why we have both max_concurrent_tests *and* > max_connections in pg_regress? ISTM the former isn't really useful given > the latter? No, the former is a static restriction on what the schedule file is allowed to contain, the latter is a dynamic restrict

Re: reducing isolation tests runtime

2019-02-13 Thread Andres Freund
Hi, On 2019-02-13 12:41:41 -0500, Tom Lane wrote: > Andres Freund writes: > > Do you have an idea why we have both max_concurrent_tests *and* > > max_connections in pg_regress? ISTM the former isn't really useful given > > the latter? > > No, the former is a static restriction on what the schedu

Re: subscriptionCheck failures on nightjar

2019-02-13 Thread Tom Lane
Andres Freund writes: > On 2019-02-13 12:37:35 -0500, Tom Lane wrote: >> Bleah. But in any case, the rename should not create a situation >> in which we need to fsync the file data again. > Well, it's not super well defined which of either you need to make the > rename durable, and it appears to

Re: reducing isolation tests runtime

2019-02-13 Thread Tom Lane
Andres Freund writes: > Right, but why don't we allow for more tests in a group, and then use a > default max_connections to limit concurrency? Having larger groups is > advantageous wrt test runtime - it reduces the number of artificial > serialization point where the slowest test slows things do

Re: subscriptionCheck failures on nightjar

2019-02-13 Thread Andres Freund
Hi, On 2019-02-13 12:59:19 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2019-02-13 12:37:35 -0500, Tom Lane wrote: > >> Bleah. But in any case, the rename should not create a situation > >> in which we need to fsync the file data again. > > > Well, it's not super well defined which of

Re: subscriptionCheck failures on nightjar

2019-02-13 Thread Tom Lane
Andres Freund writes: > On 2019-02-13 12:59:19 -0500, Tom Lane wrote: >> Perhaps more to the point, the way this was coded, the PANIC applies >> to open() failures in fsync_fname_ext() not just fsync() failures; >> that's painting with too broad a brush isn't it? > That indeed seems wrong. Thomas

Ought to use heap_multi_insert() for pg_attribute/depend insertions?

2019-02-13 Thread Andres Freund
Hi, Turns out in portions of the regression tests a good chunk of the runtime is inside AddNewAttributeTuples() and recordMultipleDependencies()'s heap insertions. Looking at a few profiles I had lying around I found that in some production cases too. ISTM we should use heap_multi_insert() for bot

Re: subscriptionCheck failures on nightjar

2019-02-13 Thread Andres Freund
Hi, On 2019-02-13 13:24:03 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2019-02-13 12:59:19 -0500, Tom Lane wrote: > >> Perhaps more to the point, the way this was coded, the PANIC applies > >> to open() failures in fsync_fname_ext() not just fsync() failures; > >> that's painting with t

Re: more unconstify use

2019-02-13 Thread Mark Dilger
> On Feb 7, 2019, at 12:14 AM, Peter Eisentraut > wrote: > > Here is a patch that makes more use of unconstify() by replacing casts > whose only purpose is to cast away const. Also a preliminary patch to > remove casts that were useless to begin with. > > -- > Peter Eisentraut

Re: subscriptionCheck failures on nightjar

2019-02-13 Thread Tom Lane
Andres Freund writes: > I was kinda pondering just open coding it. I am not yet convinced that > my idea of just using an open FD isn't the least bad approach for the > issue at hand. What precisely is the NFS issue you're concerned about? I'm not sure that fsync-on-FD after the rename will wor

Re: Using POPCNT and other advanced bit manipulation instructions

2019-02-13 Thread Alvaro Herrera
On 2019-Feb-04, David Rowley wrote: > On Fri, 1 Feb 2019 at 11:45, Alvaro Herrera wrote: > > > > I only have cosmetic suggestions for this patch. For one thing, I think > > the .c file should be in src/port and its header should be in > > src/include/port/, right beside the likes of pg_bswap.h a

Re: libpq compression

2019-02-13 Thread Robbie Harwood
Konstantin Knizhnik writes: > First of all thank you for attempting to push this patch, because > there is really seems to be some disagreement which blocks progress of > this patch. Unfortunately first reviewer (Robbie Harwood) think that > my approach cause some layering violation and should be

Re: Using POPCNT and other advanced bit manipulation instructions

2019-02-13 Thread Tom Lane
Alvaro Herrera writes: > I've pushed this now. Let's see what the buildfarm has to say about it. It's likely to be hard to tell, given the amount of pink from the Ryu patch. If Andrew is not planning to clean that up PDQ, I'd suggest reverting that patch pending having some repairs for it.

Re: Using POPCNT and other advanced bit manipulation instructions

2019-02-13 Thread Andres Freund
Hi, On February 13, 2019 8:40:14 PM GMT+01:00, Tom Lane wrote: >Alvaro Herrera writes: >> I've pushed this now. Let's see what the buildfarm has to say about >it. > >It's likely to be hard to tell, given the amount of pink from the Ryu >patch. If Andrew is not planning to clean that up PDQ, I'

Re: Using POPCNT and other advanced bit manipulation instructions

2019-02-13 Thread Tom Lane
Andres Freund writes: > I'd assume that breaking bit counting would cause distinct enough damage > (compile time or crashes). That's not to say that reverting ryu shouldn't be > considered (although I'm not that bothered by cross version, ia64 and cygwin > failures, especially because the latt

Re: Using POPCNT and other advanced bit manipulation instructions

2019-02-13 Thread Andrew Gierth
> "Andres" == Andres Freund writes: >> It's likely to be hard to tell, given the amount of pink from the >> Ryu patch. If Andrew is not planning to clean that up PDQ, Besides crake (x-version), fulmar (icc) and lorikeet (cygwin), I hope the rest of the known failures should pass on the nex

Re: Commit Fest 2019-01 is now closed

2019-02-13 Thread David Steele
On 2/5/19 7:08 PM, Andres Freund wrote: Hi, On 2019-02-05 11:55:37 -0500, Tom Lane wrote: Bruce Momjian writes: On Mon, Feb 4, 2019 at 06:34:50AM +, Tsunakawa, Takayuki wrote: Wow, thank you so much for your hard work. The last CF for PG 12 should be tough... Agreed. I am somewhat

Re: Using POPCNT and other advanced bit manipulation instructions

2019-02-13 Thread Alvaro Herrera
On 2019-Feb-13, Andres Freund wrote: > Hi, > > On February 13, 2019 8:40:14 PM GMT+01:00, Tom Lane > wrote: > >Alvaro Herrera writes: > >> I've pushed this now. Let's see what the buildfarm has to say about > >it. > > > >It's likely to be hard to tell, given the amount of pink from the Ryu >

Re: Protect syscache from bloating with negative cache entries

2019-02-13 Thread Bruce Momjian
On Tue, Feb 12, 2019 at 02:53:40AM +0100, Tomas Vondra wrote: > Right. But the logic behind time-based approach is that evicting such > entries should not cause any issues exactly because they are accessed > infrequently. It might incur some latency when we need them for the > first time after the

Re: Using POPCNT and other advanced bit manipulation instructions

2019-02-13 Thread Andrew Gierth
> "Andrew" == Andrew Gierth writes: Andrew> IA64 is working fine as far as I can see (specifically, anole Andrew> is passing); it's ICC on x86_64 that broke (fulmar). And I know what's wrong on fulmar now, so that'll be fixed shortly. -- Andrew (irc:RhodiumToad)

Re: Using POPCNT and other advanced bit manipulation instructions

2019-02-13 Thread Andrew Gierth
> "Alvaro" == Alvaro Herrera writes: Alvaro> I've pushed this now. Let's see what the buildfarm has to say Alvaro> about it. Lapwing's latest failure looks like yours rather than mine now? (the previous two were mine) -- Andrew (irc:RhodiumToad)

Re: Commit Fest 2019-01 is now closed

2019-02-13 Thread Tom Lane
David Steele writes: > I'm planning to do CFM duties again this year if there are no objections... Excellent! Particularly for the last CF of a cycle, we badly need an active CFM, so it's great that you're willing to take it on. regards, tom lane

Re: subscriptionCheck failures on nightjar

2019-02-13 Thread Thomas Munro
On Thu, Feb 14, 2019 at 8:11 AM Tom Lane wrote: > Andres Freund writes: > > I was kinda pondering just open coding it. I am not yet convinced that > > my idea of just using an open FD isn't the least bad approach for the > > issue at hand. What precisely is the NFS issue you're concerned about?

Re: Using POPCNT and other advanced bit manipulation instructions

2019-02-13 Thread Alvaro Herrera
On 2019-Feb-13, Andrew Gierth wrote: > > "Alvaro" == Alvaro Herrera writes: > > Alvaro> I've pushed this now. Let's see what the buildfarm has to say > Alvaro> about it. > > Lapwing's latest failure looks like yours rather than mine now? (the > previous two were mine) It definitely is ..

Re: Using POPCNT and other advanced bit manipulation instructions

2019-02-13 Thread Andrew Gierth
> "Alvaro" == Alvaro Herrera writes: >> Lapwing's latest failure looks like yours rather than mine now? (the >> previous two were mine) Alvaro> It definitely is ... plans have changed from using IndexOnly Alvaro> scans to Seqscans, which is likely fallout from the Alvaro> visibilitymap_

Re: Proposed refactoring of planner header files

2019-02-13 Thread Tom Lane
Pursuant to this discussion about breaking up selfuncs.c, here's a proposed patch to shove all the planner logic associated with LIKE and regex operators into the recently-created like_support.c file. This takes a bit under 1400 lines out of selfuncs.c. I was fairly pleased at how neatly this brok

Re: subscriptionCheck failures on nightjar

2019-02-13 Thread Tom Lane
Thomas Munro writes: > I found 3 examples of this failing with an ERROR (though not turning > the BF red, so nobody noticed) before the PANIC patch went in: Yeah, I suspected that had happened before with less-obvious consequences. Now that we know where the problem is, you could probably make it

Re: subscriptionCheck failures on nightjar

2019-02-13 Thread Andres Freund
Hi, On 2019-02-14 09:52:33 +1300, Thomas Munro wrote: > On Thu, Feb 14, 2019 at 8:11 AM Tom Lane wrote: > > Andres Freund writes: > > > I was kinda pondering just open coding it. I am not yet convinced that > > > my idea of just using an open FD isn't the least bad approach for the > > > issue

Re: Using POPCNT and other advanced bit manipulation instructions

2019-02-13 Thread Alvaro Herrera
On 2019-Feb-13, Alvaro Herrera wrote: > It definitely is ... plans have changed from using IndexOnly scans to > Seqscans, which is likely fallout from the visibilitymap_count() change. I think the problem here is that "unsigned long" is 32 bits in this machine: checking whether long int is 64 b

Re: Using POPCNT and other advanced bit manipulation instructions

2019-02-13 Thread Tom Lane
Alvaro Herrera writes: > and we have defined pg_popcount64() like this: > static int > pg_popcount64_sse42(uint64 word) > { > return __builtin_popcountl(word); > } That is clearly completely broken. > If that's correct, then I think we need something like this patch. But > it makes me w

Re: Using POPCNT and other advanced bit manipulation instructions

2019-02-13 Thread Tom Lane
... btw, why is pg_popcount casting away the const from its pointer argument? regards, tom lane

proposal: pg_restore --convert-to-text

2019-02-13 Thread Andrew Gierth
One of the remarkably common user errors with pg_restore is users leaving off the -d option. (We get cases of it regularly on the IRC channel, including one just now which prompted me to finally propose this) I propose we add a new option: --convert-to-text or some such name, and then make pg_rest

Re: Reaping Temp tables to avoid XID wraparound

2019-02-13 Thread James Sewell
It's easy to identify the temp tables which are causing the problem, yes. The issue here is just getting rid of them. In an ideal world I wouldn't actually have to care about the session and I could just drop the table (or vacuum the table?). Dropping the session was just the best way I could fin

Re: Using POPCNT and other advanced bit manipulation instructions

2019-02-13 Thread Alvaro Herrera
On 2019-Feb-13, Tom Lane wrote: > Alvaro Herrera writes: > > and we have defined pg_popcount64() like this: > > > static int > > pg_popcount64_sse42(uint64 word) > > { > > return __builtin_popcountl(word); > > } > > That is clearly completely broken. Pushed my proposed fix, which includes

Re: proposal: pg_restore --convert-to-text

2019-02-13 Thread Euler Taveira
Em qua, 13 de fev de 2019 às 19:56, Andrew Gierth escreveu: > One of the remarkably common user errors with pg_restore is users > leaving off the -d option. (We get cases of it regularly on the IRC > channel, including one just now which prompted me to finally propose > this) > I'm not sure it is

Re: Reaping Temp tables to avoid XID wraparound

2019-02-13 Thread Michael Paquier
On Wed, Feb 13, 2019 at 05:48:39PM +0100, Magnus Hagander wrote: > On Wed, Feb 13, 2019 at 2:26 AM Michael Paquier wrote: >> The temporary namespace OID is added to PGPROC since v11, so it could >> be easy enough to add a system function which maps a temp schema to a >> PID. Now, it could actuall

Re: Proposed refactoring of planner header files

2019-02-13 Thread Tom Lane
[ moving this discussion to a more appropriate thread; let's keep the original thread for discussing whether bloom actually needs fixed ] Over in https://www.postgresql.org/message-id/CAMkU=1yHfC+Gu84UFsz4hWn=c7tgqfmlieqcto1y-8wde96...@mail.gmail.com Jeff Janes writes: > On Tue, Feb 12, 2019 a

Re: proposal: pg_restore --convert-to-text

2019-02-13 Thread Tom Lane
Euler Taveira writes: > Em qua, 13 de fev de 2019 às 19:56, Andrew Gierth > escreveu: >> I propose we add a new option: --convert-to-text or some such name, and >> then make pg_restore throw a usage error if neither -d nor the new >> option is given. > However, I agree that pg_restore to stdout

grouping_planner refactoring

2019-02-13 Thread Amit Langote
Hi, It would help the project to "speed up partition planning" [1] a bit if grouping_planner didn't call query_planner directly. grouping_planner's main role seems to be adding Path(s) for the "top-level" operations of the query such as grouping, aggregation, etc. on top of Path(s) for scan/join

RE: Protect syscache from bloating with negative cache entries

2019-02-13 Thread Tsunakawa, Takayuki
From: Bruce Momjian [mailto:br...@momjian.us] > > That being said, having a "minimal size" threshold before starting with > > the time-based eviction may be a good idea. > > Agreed. I see the minimal size as a way to keep the systems tables in > cache, which we know we will need for the next quer

RE: Libpq support to connect to standby server as priority

2019-02-13 Thread Tsunakawa, Takayuki
From: Haribabu Kommi [mailto:kommi.harib...@gmail.com] > No. It's not good if the user has to be bothered by > default_transaction_read_only when he simply wants to a standby. > > > > OK. Understood. > so if we are going to differentiate between readonly and standby types, > then I still

Re: Using POPCNT and other advanced bit manipulation instructions

2019-02-13 Thread Tom Lane
And, while I'm complaining: why the devil is use of the compiler builtins gated by HAVE__GET_CPUID? This is unbelievably Intel-centric, because it prevents use of the builtins on other architectures. If the builtin exists, we should use it, full stop. There's no reason to expect that it would be

Re: pg_basebackup ignores the existing data directory permissions

2019-02-13 Thread Michael Paquier
On Wed, Feb 13, 2019 at 06:42:36PM +1100, Haribabu Kommi wrote: > This should back-patch till 11 where the group access is introduced. > Because of lack of complaints, I agree with you that there is no need of > further back-patch. I am confused by the link with group access. The patch you are se

Re: Using POPCNT and other advanced bit manipulation instructions

2019-02-13 Thread Thomas Munro
On Thu, Feb 14, 2019 at 4:38 PM Tom Lane wrote: > And, while I'm complaining: why the devil is use of the compiler builtins > gated by HAVE__GET_CPUID? This is unbelievably Intel-centric, because > it prevents use of the builtins on other architectures. If the builtin > exists, we should use it,

Re: Using POPCNT and other advanced bit manipulation instructions

2019-02-13 Thread Tom Lane
Thomas Munro writes: > On Thu, Feb 14, 2019 at 4:38 PM Tom Lane wrote: >> I'd be inclined to rip out all of the run-time-detection logic here; >> I doubt any of it is buying anything that's worth the price of an >> indirect call. > No view on that but apparently there were Intel Atom and AMD C c

Re: Using POPCNT and other advanced bit manipulation instructions

2019-02-13 Thread Tom Lane
Some further thoughts here ... Does the "lzcnt" runtime probe actually do anything useful? On the x86_64 compilers I tried (gcc 8.2.1 and 4.4.7), __builtin_clz and __builtin_ctz compile to sequences involving bsrq and bsfq regardless of -mpopcnt. It's fairly hard to see how lzcnt would buy anythi

Re: Using POPCNT and other advanced bit manipulation instructions

2019-02-13 Thread Gavin Flower
On 14/02/2019 11:17, Alvaro Herrera wrote: On 2019-Feb-13, Alvaro Herrera wrote: It definitely is ... plans have changed from using IndexOnly scans to Seqscans, which is likely fallout from the visibilitymap_count() change. I think the problem here is that "unsigned long" is 32 bits in this ma

Re: Using POPCNT and other advanced bit manipulation instructions

2019-02-13 Thread Tom Lane
Gavin Flower writes: > From my memory of reading of K&R many moons ago, it said that C only > guarantees that the lengths are such that > byte <= half word <= word <= long Indeed. > But I don't recall ever seeing a long less than 32 bits! I'm not sure offhand what C89 said, but C99 requires "s

Re: WAL insert delay settings

2019-02-13 Thread Andrey Lepikhov
On 13.02.2019 19:57, Tom Lane wrote: Andres Freund writes: On February 13, 2019 1:16:07 PM GMT+01:00, Peter Eisentraut wrote: One idea to address this is to slow down WAL-generating maintenance operations. This is similar to the vacuum delay. Where the vacuum delay counts notional I/O c

Re: Making all nbtree entries unique by having heap TIDs participate in comparisons

2019-02-13 Thread Peter Geoghegan
On Mon, Feb 11, 2019 at 12:54 PM Peter Geoghegan wrote: > Notable improvements in v12: I've been benchmarking v12, once again using a slightly modified BenchmarkSQL that doesn't do up-front CREATE INDEX builds [1], since the problems with index bloat don't take so long to manifest themselves when

Re: [PATCH] xlogreader: do not read a file block twice

2019-02-13 Thread Michael Paquier
On Tue, Feb 12, 2019 at 11:44:14AM +0300, Arthur Zakirov wrote: > Of course. Agree, it may be a non trivial case. Added as a bug fix: > https://commitfest.postgresql.org/22/1994/ I have been looking at the patch, and I agree that the current coding is a bit crazy. If the wanted data has already b

Re: Incorrect visibility test function assigned to snapshot

2019-02-13 Thread Michael Paquier
On Fri, Feb 08, 2019 at 11:59:05AM +0100, Antonin Houska wrote: > Sorry, I forgot. Patch is below and I'm going to add an entry to the > next CF. > @@ -615,6 +615,8 @@ SnapBuildInitialSnapshot(SnapBuild *builder) > > TransactionIdAdvance(xid); > } > + /* And of course, ad

Re: WAL insert delay settings

2019-02-13 Thread Peter Geoghegan
On Wed, Feb 13, 2019 at 4:16 AM Peter Eisentraut wrote: > Bulk operations like CREATE INDEX, ALTER TABLE, or bulk loads can create > a lot of WAL. A lot of WAL at once can cause delays in replication. > For synchronous replication, this can make seemingly unrelated sessions > hang. But also for

RE: Protect syscache from bloating with negative cache entries

2019-02-13 Thread Tsunakawa, Takayuki
From: Kyotaro HORIGUCHI [mailto:horiguchi.kyot...@lab.ntt.co.jp] > It is too complex as I was afraid. The indirect calls causes siginicant > degradation. (Anyway the previous code was bogus in that it passes > CACHELINEALIGN'ed pointer to get_chunk_size..) > > Instead, I added an accounting(?) int

Re: Spurious "apparent wraparound" via SimpleLruTruncate() rounding

2019-02-13 Thread Noah Misch
On Sat, Feb 02, 2019 at 03:38:22AM -0500, Noah Misch wrote: > The main consequence is the false alarm. That conclusion was incorrect. On further study, I was able to reproduce data loss via either of two weaknesses in the "apparent wraparound" test: 1. The result of the test is valid only until

  1   2   >