Re: Race conditions with checkpointer and shutdown

2019-04-18 Thread Thomas Munro
04-16 08:23:24.178 CEST [8393] FATAL: terminating walreceiver process due to administrator command 2019-04-16 08:28:23.166 CEST [8337] LOG: restartpoint starting: time LogCheckpointStart() is the thing that writes "starting: ...", and it prefers to report "shutdown" over &q

Re: Race conditions with checkpointer and shutdown

2019-04-18 Thread Thomas Munro
re-existing race condition more likely to go wrong. -- Thomas Munro https://enterprisedb.com

Re: Race conditions with checkpointer and shutdown

2019-04-18 Thread Thomas Munro
On Fri, Apr 19, 2019 at 10:22 AM Tom Lane wrote: > Thomas Munro writes: > > 2019-04-16 08:23:24.178 CEST [8393] FATAL: terminating walreceiver > > process due to administrator command > Maybe what we should be looking for is "why doesn't the walreceiver > shut d

Re: Do PostgreSQL have map and set structure(like STL in C++)?

2019-04-25 Thread Thomas Munro
y dealing in void pointers all over the place. I suppose it would be theoretically possible to make some of these design choices into policies you could select when instantiating, rather than having separate library components. Another thing that is on the radar for future development is concurrent lock-free extensible hash tables as seen in some other projects. -- Thomas Munro https://enterprisedb.com

Re: Race conditions with checkpointer and shutdown

2019-04-28 Thread Thomas Munro
n it. +1 That reminds me, we should probably also clean up at least the ereport-from-signal-handler hazard identified over in this thread: https://www.postgresql.org/message-id/CAEepm%3D10MtmKeDc1WxBM0PQM9OgtNy%2BRCeWqz40pZRRS3PNo5Q%40mail.gmail.com -- Thomas Munro https://enterprisedb.com

Re: Adding a test for speculative insert abort case

2019-04-30 Thread Thomas Munro
lation tests could add/remove names and signal a condition variable to release waiters. I contemplated that while working on SKIP LOCKED, which had a bunch of weird edge cases that I tested by inserting throw-away wait-point code like this: https://www.postgresql.org/message-id/CADLWmXXss83oiYD0pn_SfQfg%2ByNEpPbPvgDb8w6Fh--jScSybA%40mail.gmail.com -- Thomas Munro https://enterprisedb.com

Re: How to estimate the shared memory size required for parallel scan?

2019-05-02 Thread Thomas Munro
On Mon, Aug 20, 2018 at 11:09 AM Thomas Munro wrote: > 2. Teach your GetForeignPath function to do something like this: > [blah blah blah] I was pinged off-list by someone who is working on a parallel-aware FDW, who asked if I still had the test code I mentioned up-thread. While diggin

Fixing order of resowner cleanup in 12, for Windows

2019-05-02 Thread Thomas Munro
the beta period. I'll need to find someone to test this with a fault injection scenario on Windows before committing it, but wanted to sound out the list for any objections to this late change? [1] https://www.postgresql.org/message-id/CAEepm%3D2ikUtjmiJ18bTnwaeUBoiYN%3DwMDSdhU1jy%3D8Wz

Re: How to estimate the shared memory size required for parallel scan?

2019-05-02 Thread Thomas Munro
On Thu, May 2, 2019 at 8:06 PM Thomas Munro wrote: > I was pinged off-list by someone who is working on a parallel-aware > FDW, who asked if I still had the test code I mentioned up-thread. > While digging that out, I couldn't resist hacking it a bit more until > it gave the ri

Re: First-draft release notes for back branches are up

2019-05-03 Thread Thomas Munro
ync calls (Thomas Munro, + James Sewell) Nit-picking: ENOSYS is for sync_file_range, EINVAL is for fsync. -- Thomas Munro https://enterprisedb.com

Re: First-draft release notes for back branches are up

2019-05-03 Thread Thomas Munro
On Sat, May 4, 2019 at 1:29 PM Tom Lane wrote: > Thomas Munro writes: > > On Sat, May 4, 2019 at 10:29 AM Tom Lane wrote: > > + Tolerate EINVAL and ENOSYS > > + error results, where appropriate, for fsync calls (Thomas Munro, > > + James Sewell) >

Re: First-draft release notes for back branches are up

2019-05-04 Thread Thomas Munro
ime` like other identifier/file/whatever names? > * Several fixes for `conritb/postgres_fdw`, including one for remote > partitions where an UPDATE could lead to incorrect results or a crash Typo: contrib -- Thomas Munro https://enterprisedb.com

Re: Fixing order of resowner cleanup in 12, for Windows

2019-05-05 Thread Thomas Munro
On Fri, May 3, 2019 at 2:15 AM Tom Lane wrote: > Thomas Munro writes: > > A while back I posted a patch[1] to change the order of resowner > > cleanup so that DSM handles are released last. That's useful for the > > error cleanup path on Windows, when a Shar

Re: Fixing order of resowner cleanup in 12, for Windows

2019-05-05 Thread Thomas Munro
else being still held when they run, because that's fragile, so calling them last is the best and most conservative choice. I think if someone does come with legitimate reasons to want that, we should discuss it then, and perhaps consider something a bit like the ResourceRelease_callbacks li

Re: [HACKERS] Weaker shmem interlock w/o postmaster.pid

2019-05-05 Thread Thomas Munro
On Wed, May 1, 2019 at 12:57 PM Noah Misch wrote: > On Tue, Apr 30, 2019 at 10:47:37PM +1200, Thomas Munro wrote: > > --- a/src/test/perl/PostgresNode.pm > > +++ b/src/test/perl/PostgresNode.pm > > @@ -1098,17 +1098,12 @@ sub get_new_node > >

Re: Fixing order of resowner cleanup in 12, for Windows

2019-05-05 Thread Thomas Munro
On Mon, May 6, 2019 at 3:44 PM Tom Lane wrote: > Thomas Munro writes: > > On Mon, May 6, 2019 at 11:07 AM Tom Lane wrote: > >> ... You're not doing future > >> hackers any service by failing to include a comment that explains that > >> DSM detach M

Re: Fixing order of resowner cleanup in 12, for Windows

2019-05-06 Thread Thomas Munro
On Mon, May 6, 2019 at 9:26 PM Amit Kapila wrote: > On Mon, May 6, 2019 at 3:43 AM Thomas Munro wrote: > > Here's a way to produce an error which might produce the log message > > on Windows. Does anyone want to try it? > > I can give it a try. Thanks! > > If

Re: accounting for memory used for BufFile during hash joins

2019-05-06 Thread Thomas Munro
, which involves sorting the inner and outer batch, plus the entirely new concept of switching to another node at execution time. I also wondered about reducing the buffer size of the BufFiles, but that doesn't seem to be fixing the real problem. -- Thomas Munro https://enterprisedb.com

Re: [HACKERS] Broken O(n^2) avoidance in wal segment recycling.

2019-05-06 Thread Thomas Munro
On Tue, May 7, 2019 at 2:45 PM Andres Freund wrote: > I just hit this bad a couple times during some testing. Under load, with > 2500 segments to recycle, it took well over a minute. I wonder if this played a part in the wal_recycle=off-for-ZFS thing. -- Thomas Munro https://enterprisedb.com

Re: accounting for memory used for BufFile during hash joins

2019-05-06 Thread Thomas Munro
On Tue, May 7, 2019 at 3:15 PM Tomas Vondra wrote: > On Tue, May 07, 2019 at 01:48:40PM +1200, Thomas Munro wrote: > >Seems expensive for large numbers of slices -- you need to join the > >outer batch against each inner slice. > > Nope, that's not how it works. It's

Re: We're leaking predicate locks in HEAD

2019-05-07 Thread Thomas Munro
.. Investigating. -- Thomas Munro https://enterprisedb.com

Re: We're leaking predicate locks in HEAD

2019-05-07 Thread Thomas Munro
On Wed, May 8, 2019 at 6:56 AM Thomas Munro wrote: > On Wed, May 8, 2019 at 5:46 AM Tom Lane wrote: > > I'd have to say that my first suspicion falls on bb16aba50 ... > > Investigating. Reproduced here. Once the system reaches a state where it's leaking (which happens

Re: We're leaking predicate locks in HEAD

2019-05-07 Thread Thomas Munro
On Wed, May 8, 2019 at 3:53 PM Tom Lane wrote: > Thomas Munro writes: > > Reproduced here. Once the system reaches a state where it's leaking > > (which happens only occasionally for me during installcheck-parallel), > > it keeps leaking for future SSI t

Re: Copy data to DSA area

2019-05-07 Thread Thomas Munro
ny backend crashes, the postmaster will terminate all backends and reinitialise everything anyway, because shared memory might be corrupted. -- Thomas Munro https://enterprisedb.com

Adding SMGR discriminator to buffer tags

2019-05-07 Thread Thomas Munro
? [1] https://www.postgresql.org/message-id/flat/CA%2BhUKG%2BDE0mmiBZMtZyvwWtgv1sZCniSVhXYsXkvJ_Wo%2B83vvw%40mail.gmail.com -- Thomas Munro https://enterprisedb.com 0001-Add-SmgrId-to-smgropen-and-BufferTag.patch Description: Binary data 0002-Move-tablespace-dir-creation-from-smgr.c-to-md.c.patch Description: Binary data

Re: [HACKERS] Weaker shmem interlock w/o postmaster.pid

2019-05-08 Thread Thomas Munro
On Mon, May 6, 2019 at 3:30 PM Thomas Munro wrote: > I put the second sentence back and tweaked it thus: s/fails/might > fail/. Maybe I'm being too pedantic here, but binding to 127.0.0.2 > works on other OSes too, as long as you've configured an interface or > alia

Re: We're leaking predicate locks in HEAD

2019-05-09 Thread Thomas Munro
On Wed, May 8, 2019 at 4:50 PM Thomas Munro wrote: > On Wed, May 8, 2019 at 3:53 PM Tom Lane wrote: > > Thomas Munro writes: > > > Reproduced here. Once the system reaches a state where it's leaking > > > (which happens only occasionally for me during instal

Re: Fixing order of resowner cleanup in 12, for Windows

2019-05-09 Thread Thomas Munro
trace through this and identify the cause for-sure before we go > > designing an invasive fix. > > Huh, OK. The reason the patch didn't solve the problem is that AtEOXact_Parallel() calls DestroyParallelContext(). So DSM segments that happen to belong to ParallelContext objects are already gone by the time resowner.c gets involved. -- Thomas Munro https://enterprisedb.com

Re: pg12 release notes

2019-05-09 Thread Thomas Munro
parallel query when in SERIALIZABLE ISOLATION MODE (Thomas Munro)" Only SERIALIZABLE should be in all-caps IMHO. -- Thomas Munro https://enterprisedb.com

ldapbindpasswdfile

2019-05-13 Thread Thomas Munro
e-id/flat/20140617175511.2589.45249%40wrigleys.postgresql.org -- Thomas Munro https://enterprisedb.com 0001-Add-ldapbindpasswdfile-option-for-pg_hba.conf.patch Description: Binary data

Re: ldapbindpasswdfile

2019-05-13 Thread Thomas Munro
On Tue, May 14, 2019 at 1:49 PM Thomas Munro wrote: > ... or a named pipe that performs arbitrary magic. (Erm, that bit might not make much sense...) -- Thomas Munro https://enterprisedb.com

Tab completion for CREATE TYPE

2019-05-13 Thread Thomas Munro
Hi, Since I keep forgetting the syntax and options, here is $SUBJECT. -- Thomas Munro https://enterprisedb.com 0001-Tab-completion-for-CREATE-TYPE.patch Description: Binary data

Re: Tab completion for CREATE TYPE

2019-05-13 Thread Thomas Munro
ves the same way among > existing completions. Hmm. True. Here's one way to fix that. -- Thomas Munro https://enterprisedb.com 0001-Tab-completion-for-CREATE-TYPE-v2.patch Description: Binary data 0002-Fix-tab-completion-of-UPDATE-v2.patch Description: Binary data

Re: Tab completion for CREATE TYPE

2019-05-14 Thread Thomas Munro
On Tue, May 14, 2019 at 8:32 PM Edgy Hacker wrote: > Hmm... just got here. Welcome. > What happens around here? Please see https://wiki.postgresql.org/wiki/So,_you_want_to_be_a_developer%3F . -- Thomas Munro https://enterprisedb.com

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-05-14 Thread Thomas Munro
have no clue how to set it, but at least users have a fighting chance of experimenting and figuring it out that way. I volunteer to write the patch if we get a consensus. -- Thomas Munro https://enterprisedb.com

Re: PSA: New intel MDS vulnerability mitigations cause measurable slowdown

2019-05-14 Thread Thomas Munro
f not, there would also be a couple of lseek(SEEK_END) calls in between for planning... I wonder how many more syscall-taxing mitigations we need before relation size caching pays off. -- Thomas Munro https://enterprisedb.com

Re: Emacs vs pg_indent's weird indentation for function declarations

2019-05-15 Thread Thomas Munro
ith that case eventually if so. > Attached is working draft of pg_bsd_indent changes (still sans comments) > as well as a patch showing the difference between current pgindent > results on HEAD and the results of this version. I think there's no > question that this is an improvement. +1 Thanks for looking into it! -- Thomas Munro https://enterprisedb.com

Re: pgsql: Compute XID horizon for page level index vacuum on primary.

2019-05-15 Thread Thomas Munro
On Thu, May 16, 2019 at 3:53 AM Tom Lane wrote: > Andres Freund writes: > > On 2019-05-15 12:01:07 +1200, Thomas Munro wrote: > >> This is listed as an open item to resolve for 12. IIUC the options on > >> the table are: > >> > >> 1. Do nothing

Avoiding hash join batch explosions with extreme skew and weird stats

2019-05-15 Thread Thomas Munro
mail.com [2] https://www.postgresql.org/message-id/flat/20190504003414.bulcbnge3rhwhcsh%40development -- Thomas Munro https://enterprisedb.com fix.patch Description: Binary data

Re: Parallel Foreign Scans - need advice

2019-05-15 Thread Thomas Munro
[1] https://www.postgresql.org/message-id/CA%2BhUKG%2BqK3E2RF75PKfsV0sn2s018%2Bft--hUuCmd2R_yQ9tmPQ%40mail.gmail.com -- Thomas Munro https://enterprisedb.com

Re: PSA: New intel MDS vulnerability mitigations cause measurable slowdown

2019-05-16 Thread Thomas Munro
off 16.6716.8617.33 onon 18.5818.8318.99 -- Thomas Munro https://enterprisedb.com

Re: PSA: New intel MDS vulnerability mitigations cause measurable slowdown

2019-05-16 Thread Thomas Munro
On Fri, May 17, 2019 at 5:26 AM Chapman Flack wrote: > On 5/16/19 12:24 PM, Albert Cervera i Areny wrote: > > Missatge de Thomas Munro del dia dj., 16 de > > maig 2019 a les 13:09: > >> With all three mitigations activated, my little dev machine has gone > >> fr

Re: PSA: New intel MDS vulnerability mitigations cause measurable slowdown

2019-05-16 Thread Thomas Munro
On Fri, May 17, 2019 at 9:42 AM Thomas Munro wrote: > pgbench -M prepared read-only transactions per second, 16 client threads: (That second "16 client threads" line should read "32 client threads".) -- Thomas Munro https://enterprisedb.com

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-05-16 Thread Thomas Munro
oins, I haven't figured out how to do it for parallel-aware hash join, because then each loop over the outer batch could see different tuples in each participant. You could use the match bit in HashJoinTuple header, but then you'd have to write all the tuples out again, which is more IO than I want to do. I'll probably start another thread about that. -- Thomas Munro https://enterprisedb.com

Re: Parallel Foreign Scans - need advice

2019-05-16 Thread Thomas Munro
if it works in simple examples. Your node isn't necessarily the only node in the plan, and higher up nodes get to decide when, if at all, you run, in each worker. -- Thomas Munro https://enterprisedb.com

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-05-16 Thread Thomas Munro
On Fri, May 17, 2019 at 11:46 AM Tomas Vondra wrote: > On Thu, May 16, 2019 at 06:58:43PM -0400, Tom Lane wrote: > >Thomas Munro writes: > >> On Fri, May 17, 2019 at 4:39 AM Tomas Vondra > >> wrote: > >>> I kinda like the idea with increasing the space

Re: Fixing order of resowner cleanup in 12, for Windows

2019-05-16 Thread Thomas Munro
On Thu, May 9, 2019 at 10:23 PM Thomas Munro wrote: > The reason the patch didn't solve the problem is that > AtEOXact_Parallel() calls DestroyParallelContext(). So DSM segments > that happen to belong to ParallelContext objects are already gone by > the time resowner.c gets in

Re: sample scans and predicate locking

2019-05-18 Thread Thomas Munro
theoretical argument about TABLESAMPLE(100) behaving differently if done per page rather than if done at relation level, wrt new pages added to the end later and therefore missed. And then by logical extension, TABLESAMPLE of any percentage. I'm not sure. I made a little list of small SERIALIZABLE projects for v13 and added this, over here: https://wiki.postgresql.org/wiki/SerializableToDo -- Thomas Munro https://enterprisedb.com

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-05-19 Thread Thomas Munro
On Sat, May 18, 2019 at 12:15 PM Melanie Plageman wrote: > On Thu, May 16, 2019 at 3:22 PM Thomas Munro wrote: >> Admittedly I don't have a patch, just a bunch of handwaving. One >> reason I haven't attempted to write it is because although I know how >> to do t

Re: sample scans and predicate locking

2019-05-19 Thread Thomas Munro
On Mon, May 20, 2019 at 10:23 AM Andres Freund wrote: > On 2019-05-19 13:57:42 +1200, Thomas Munro wrote: > > Yeah, we could probably predicate-lock pages in > > heapam_scan_sample_next_block() and tuples in > > heapam_scan_sample_next_tuple(), instead of doing this. Seems

Re: Avoiding hash join batch explosions with extreme skew and weird stats

2019-05-19 Thread Thomas Munro
On Mon, May 20, 2019 at 12:22 PM Tomas Vondra wrote: > On Mon, May 20, 2019 at 11:07:03AM +1200, Thomas Munro wrote: > >First let me restate the PostgreSQL terminology for this stuff so I > >don't get confused while talking about it: > > > >* The inner side of the

Parallel Append subplan order instability on aye-aye

2019-05-19 Thread Thomas Munro
5-19%2018%3A30%3A10 -- Thomas Munro https://enterprisedb.com

Re: Do we expect tests to work with default_transaction_isolation=serializable

2019-05-19 Thread Thomas Munro
bb16aba5. -- Thomas Munro https://enterprisedb.com remove-bogus-iso-change.patch Description: Binary data

Re: Parallel Append subplan order instability on aye-aye

2019-05-20 Thread Thomas Munro
On Mon, May 20, 2019 at 4:46 PM Tom Lane wrote: > Thomas Munro writes: > > Here's a one-off regression test failure of a sort that commit > > 624e440a intended to fix. > > Note that in the discussion that led up to 624e440a, we never did > think that we'd

Re: Parallel Append subplan order instability on aye-aye

2019-05-21 Thread Thomas Munro
gle-page tables, the relpages and reltuples counts > should be machine-independent. In theory anyway. Huh, idiacanthus failed showing vacuum_count 0, in select_parallel. So ... the VACUUM command somehow skipped those tables? -- Thomas Munro https://enterprisedb.com

Re: Teach pg_upgrade test to honor NO_TEMP_INSTALL

2019-05-22 Thread Thomas Munro
l -Werror" > src/Makefile.custom && make -j4 all contrib docs && make check-world I added --prefix=$HOME/something and added "make install" before "make check-world", and now it's happy again. Was that expected? -- Thomas Munro https://enterprisedb.com

Re: Typo: llvm*.cpp files identified as llvm*.c

2019-05-22 Thread Thomas Munro
On Wed, Jan 23, 2019 at 7:24 PM Andrew Gierth wrote: > >>>>> "Andres" == Andres Freund writes: > > [IDENTIFICATION] > > Andres> I think we should just rip them out. It's useless noise. > > +1 Here's a patch like that. 9

Minor typos and copyright year slippage

2019-05-22 Thread Thomas Munro
Hi, Here are some tiny things I noticed in passing. -- Thomas Munro https://enterprisedb.com 0001-Fix-typos.patch Description: Binary data 0002-Update-copyright-year.patch Description: Binary data

Re: Question about BarrierAttach spinlock

2019-05-23 Thread Thomas Munro
h I think that'd be slightly harder to follow. I suppose I could introduce a variable phase. -- Thomas Munro https://enterprisedb.com

Re: Minor typos and copyright year slippage

2019-05-23 Thread Thomas Munro
On Thu, May 23, 2019 at 1:55 PM Michael Paquier wrote: > On Thu, May 23, 2019 at 01:28:45PM +1200, Thomas Munro wrote: > > Here are some tiny things I noticed in passing. > > Good catches. And you have spotted all the blank spots for the > copyright notices. Thanks, pushed.

Re: Collation versioning

2019-05-28 Thread Thomas Munro
ning can be cleared for the indexes on that one table like so: postgres=# reindex table t; REINDEX You can see that it's captured the new version: postgres=# select refobjversion from pg_depend where refobjversion != ''; refobjversion --- 153.72 (1 row) -- Thomas Munro https:/

Re: Time range

2019-05-31 Thread Thomas Kellerer
Donald Shtjefni schrieb am 31.05.2019 um 13:35: > I was wondering why there is not a type Range of time without time zone, I > think it may be useful for someone, Is good if i do PR. you can easily create one: create type timerange as range (subtype = time); Thomas

Re: Odd 9.4, 9.3 buildfarm failure on s390x

2018-10-04 Thread Thomas Munro
ld be an integral variable. + */ +#define INVERT_SIGN(var) \ +((var) = ((var) < 0) ? 1 : -(var)) I suppose someone might mistake this for a function that converts -42 to 42... would something like INVERT_COMPARE_RESULT() be better? -- Thomas Munro http://www.enterprisedb.com

Re: WIP: Avoid creation of the free space map for small tables

2018-10-05 Thread Thomas Munro
l table so make check-world fails. -- Thomas Munro http://www.enterprisedb.com

Re: Relax transactional restrictions on ALTER ENUM ... ADD TYPE (redux)

2018-10-05 Thread Thomas Munro
On Sat, Oct 6, 2018 at 2:31 AM Andrew Dunstan wrote: > >> On Wed, Oct 3, 2018 at 2:24 PM Thomas Munro > >> wrote: > >>> Over the thread for bug #14825 I posted some draft code to show one > >>> way to save/restore the enum blacklist for parallel workers.

Re: Relax transactional restrictions on ALTER ENUM ... ADD TYPE (redux)

2018-10-06 Thread Thomas Munro
On Sun, Oct 7, 2018 at 5:53 AM Tom Lane wrote: > Thomas Munro writes: > > Thanks. Here is a version squashed into one commit, with a decent > > commit message and a small improvement: the code to create the hash > > table is moved into a static function, to avoid repetit

DSM segment handle generation in background workers

2018-10-07 Thread Thomas Munro
hould be moved into InitPostmasterChild(). If we put it in there right after MyStartTime is assigned a new value, we could use the same incantation that PostmasterMain() uses. I noticed that the comment in PostmasterMain() refers to PostmasterRandom(), which is gone. -- Thomas Munro http://www.enterprisedb.com

Re: PostgreSQL 12, JIT defaults

2018-10-08 Thread Thomas Munro
On Mon, Oct 8, 2018 at 10:52 PM Pavel Stehule wrote: > > Hi > > I configured PostgreSQL without JIT support, but JIT is active by default I think that happens when llvmjit.so is present (ie from last time you built with JIT support and ran make install). You need to remove it. --

Re: Relax transactional restrictions on ALTER ENUM ... ADD TYPE (redux)

2018-10-08 Thread Thomas Munro
On Sun, Oct 7, 2018 at 3:30 PM Thomas Munro wrote: > On Sun, Oct 7, 2018 at 5:53 AM Tom Lane wrote: > > Thomas Munro writes: > > > Thanks. Here is a version squashed into one commit, with a decent > > > commit message and a small improvement: the code to create th

Re: DSM segment handle generation in background workers

2018-10-08 Thread Thomas Munro
On Mon, Oct 8, 2018 at 1:17 AM Thomas Munro wrote: > That's because the bgworker startup path doesn't contain a call to > srandom(...distinguishing stuff...), unlike BackendRun(). I suppose > do_start_bgworker() could gain a similar call... or perhaps that call >

Re: pread() and pwrite()

2018-10-08 Thread Thomas Munro
On Fri, Sep 28, 2018 at 2:03 AM Jesper Pedersen wrote: > Thanks for v5 too. Rebased again. Patches that touch AC_CHECK_FUNCS are fun like that! -- Thomas Munro http://www.enterprisedb.com 0001-Use-pread-pwrite-instead-of-lseek-read-write-v6.patch Description: Binary data

Re: DSM segment handle generation in background workers

2018-10-08 Thread Thomas Munro
On Tue, Oct 9, 2018 at 1:53 PM Tom Lane wrote: > Thomas Munro writes: > > On Mon, Oct 8, 2018 at 1:17 AM Thomas Munro > > wrote: > >> That's because the bgworker startup path doesn't contain a call to > >> srandom(...distinguishing

Re: pread() and pwrite()

2018-10-08 Thread Thomas Munro
On Tue, Oct 9, 2018 at 2:55 PM Tom Lane wrote: > Thomas Munro writes: > > Rebased again. Patches that touch AC_CHECK_FUNCS are fun like that! > > Yeah, I've been burnt by that too recently. It occurs to me we could make > that at least a little less painful if we forma

Re: [HACKERS] SERIALIZABLE with parallel query

2018-10-08 Thread Thomas Munro
On Tue, Oct 2, 2018 at 4:53 PM Thomas Munro wrote: > Thanks for the review! And sorry for my delayed response. Here is a > rebased patch, with changes as requested. Rebased. -- Thomas Munro http://www.enterprisedb.com 0001-Enable-parallel-query-with-SERIALIZABLE-isolatio-v16

Re: [HACKERS] Why does logical replication launcher exit with exit code 1?

2018-10-09 Thread Thomas Munro
On Fri, Oct 5, 2018 at 12:36 PM Thomas Munro wrote: > * We really should get rid of that "exited with exit code 1". Robert and I just discussed this subproblem (the original complaint of this thread) off-list. Our questions are: does anyone actually want that message from the post

Re: pread() and pwrite()

2018-10-09 Thread Thomas Munro
On Tue, Oct 9, 2018 at 5:08 PM Tom Lane wrote: > I wrote: > > Thomas Munro writes: > >> On Tue, Oct 9, 2018 at 2:55 PM Tom Lane wrote: > >>> Yeah, I've been burnt by that too recently. It occurs to me we could make > >>> that at least a little

Re: [HACKERS] kqueue

2018-10-09 Thread Thomas Munro
On Tue, Oct 2, 2018 at 6:28 AM Andres Freund wrote: > On 2018-10-01 19:25:45 +0200, Matteo Beccati wrote: > > On 01/10/2018 01:09, Thomas Munro wrote: > > > I don't know why the existence of the kqueue should make recvfrom() > > > slower on the pgbench side. T

Re: background worker shudown (was Re: [HACKERS] Why does logical replication launcher exit with exit code 1?)

2018-10-09 Thread Thomas Munro
On Wed, Oct 10, 2018 at 7:29 AM Robert Haas wrote: > On Thu, Oct 4, 2018 at 7:37 PM Thomas Munro > wrote: > > 0. The default SIGTERM handler for bgworkers is bgworker_die(), which > > generates a FATAL ereport "terminating background worker \"%s\" due to > &

Re: DSM segment handle generation in background workers

2018-10-10 Thread Thomas Munro
On Tue, Oct 9, 2018 at 3:21 PM Thomas Munro wrote: > On Tue, Oct 9, 2018 at 1:53 PM Tom Lane wrote: > > Thomas Munro writes: > > > On Mon, Oct 8, 2018 at 1:17 AM Thomas Munro > > > wrote: > > >> That's because the bgworker startup path doesn't c

Re: DSM segment handle generation in background workers

2018-10-13 Thread Thomas Munro
for the function that does those things. I went with InitProcessGlobals(). Better suggestions welcome. -- Thomas Munro http://www.enterprisedb.com 0001-Refactor-random-seed-and-start-time-initialization.patch Description: Binary data

Experimenting with hash join prefetch

2018-10-13 Thread Thomas Munro
ch mode. [1] https://wiki.postgresql.org/wiki/Parallel_Hash [2] https://15721.courses.cs.cmu.edu/spring2016/papers/balkesen-icde2013.pdf (table VI) [3] http://www.cs.cmu.edu/~chensm/papers/hashjoin_icde04.pdf -- Thomas Munro http://www.enterprisedb.com

Re: Experimenting with hash join prefetch

2018-10-14 Thread Thomas Munro
On Mon, Oct 15, 2018 at 12:16 AM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > On Sun, 14 Oct 2018 at 06:19, Thomas Munro > > wrote: > > Cache-oblivious hash joins cause a lot of TLB and cache misses. > > ... > > (There is another class of cache-aware

Re: B-tree cache prefetches

2018-10-14 Thread Thomas Munro
On Mon, Oct 15, 2018 at 12:03 AM Andrey Borodin wrote: > 31 авг. 2018 г., в 2:40, Thomas Munro > написал(а): > A related topic is the cache-unfriendliness of traditional binary > searches of sorted data. Check out "Array Layouts for > Comparison-Based Searching"[1] if

Re: "SELECT ... FROM DUAL" is not quite as silly as it appears

2018-10-14 Thread Thomas Munro
that there's > a bit of method in that madness after all. We can still make fun of that table. Apparently it had two rows, so you could double rows by cross joining against it, but at some point one of them went missing, leaving a strange name behind. Source: https://en.wikipedia.org

Re: [RFC] Removing "magic" oids

2018-10-14 Thread Thomas Munro
from SELECT * for other reasons too: https://www.postgresql.org/message-id/flat/CAEepm%3D3ZHh%3Dp0nEEnVbs1Dig_UShPzHUcMNAqvDQUgYgcDo-pA%40mail.gmail.com -- Thomas Munro http://www.enterprisedb.com

Re: Large writable variables

2018-10-15 Thread Thomas Munro
often, nor in all processes... Thomas? Yeah, here's a patch to move it in to the heap. -- Thomas Munro http://www.enterprisedb.com 0001-Move-the-replication-lag-tracker-into-heap-memory.patch Description: Binary data

Re: Large writable variables

2018-10-15 Thread Thomas Munro
e it in heap > > memory there. > > Cool, let's do that. Pushed. -- Thomas Munro http://www.enterprisedb.com

Re: DSM robustness failure (was Re: Peripatus/failures)

2018-10-17 Thread Thomas Munro
> garbage old state prevent it from establishing a valid new segment. Looking. > BTW, the header comment on dsm_postmaster_startup is a lie, which > is probably not unrelated to its failure to consider this situation. Agreed. -- Thomas Munro http://www.enterprisedb.com

Re: DSM robustness failure (was Re: Peripatus/failures)

2018-10-17 Thread Thomas Munro
On Thu, Oct 18, 2018 at 9:43 AM Thomas Munro wrote: > On Thu, Oct 18, 2018 at 9:00 AM Tom Lane wrote: > > I would argue that both dsm_postmaster_shutdown and dsm_postmaster_startup > > are broken here; the former because it makes no attempt to unmap > > the old control seg

Re: DSM robustness failure (was Re: Peripatus/failures)

2018-10-17 Thread Thomas Munro
On Thu, Oct 18, 2018 at 11:08 AM Thomas Munro wrote: > On Thu, Oct 18, 2018 at 9:43 AM Thomas Munro > wrote: > > On Thu, Oct 18, 2018 at 9:00 AM Tom Lane wrote: > > > I would argue that both dsm_postmaster_shutdown and dsm_postmaster_startup > > > are broken here;

Re: DSM robustness failure (was Re: Peripatus/failures)

2018-10-17 Thread Thomas Munro
ritten? Gah. So basically that's hiding in shm_dictionary in the kernel and I don't know a way to look at it from userspace (other than trying to open all 2^32 random paths we're capable of generating). -- Thomas Munro http://www.enterprisedb.com

Re: DSM robustness failure (was Re: Peripatus/failures)

2018-10-17 Thread Thomas Munro
HEAD > aren't whinging about lost semaphores is not hard to understand --- we > stopped using SysV semas. But why don't the older branches look like v10 > here? I think v10 is where we switched to POSIX unnamed (= sem_destroy()), so it's 10, 11 and master that should be the same in this respect, no? -- Thomas Munro http://www.enterprisedb.com

Re: DSM robustness failure (was Re: Peripatus/failures)

2018-10-18 Thread Thomas Munro
I'll see if there is some way to improve that situation for next time something like this happens. -- Thomas Munro http://www.enterprisedb.com

Re: DSM robustness failure (was Re: Peripatus/failures)

2018-10-18 Thread Thomas Munro
, so ... hmm, we'd better forget about it if we plan to continue running, by clearing those variables explicitly. That's actually another pre-existing way for the assertion to fail in current master. (Realistically, that unmmap() could only fail if our global variables are trashed so we try t

Re: Postgres, fsync, and OSs (specifically linux)

2018-10-18 Thread Thomas Munro
Hello hackers, Let's try to get this issue resolved. Here is my position on the course of action we should take in back-branches: 1. I am -1 on back-patching the fd-transfer code. It's a significant change, and even when sufficiently debugged (I don't think it's there yet), we have no idea wha

Re: lowering pg_regress privileges on Windows

2018-10-18 Thread Thomas Munro
do with other binaries. This is useful in setups > > like Appveyor where running under any other account is ... difficult. For > > the cfbot Thomas has had to make the script hack the schedule file to omit > > the tablespace test. This would make that redundant. > > > > I p

Re: DSM segment handle generation in background workers

2018-10-18 Thread Thomas Munro
On Sat, Oct 13, 2018 at 11:45 PM Thomas Munro wrote: > On Thu, Oct 11, 2018 at 5:51 PM Tom Lane wrote: > > The comment adjacent to the change in InitStandaloneProcess bothers me. > > In particular, it points out that what BackendRun() is currently doing > > creates more ent

Re: [HACKERS] PATCH: Keep one postmaster monitoring pipe per process

2018-10-22 Thread Thomas Munro
tuff that changes directory on the way out the door for gprof users. It looks like per-pid gmon files could have been achieved by setting environment variables[1][2], but I guess there is some value in doing it the same way on every platform. [1] https://sourceware.org/git/?p=glibc.git;a=blob;f=gmon/gm

Re: Synchronous replay take III

2018-10-22 Thread Thomas Munro
On Mon, Sep 24, 2018 at 10:39 AM Thomas Munro wrote: > On Mon, Jul 2, 2018 at 12:39 PM Thomas Munro > wrote: > > On Sat, Mar 3, 2018 at 2:11 AM, Adam Brusselback > > wrote: > > > Thanks Thomas, appreciate the rebase and the work you've done on this. > >

Re: Log timestamps at higher resolution

2018-10-23 Thread Thomas Munro
be helpful in this context. Wouldn't you want to choose a new letter or some other way to make existing format control strings do what they always did? -- Thomas Munro http://www.enterprisedb.com

<    8   9   10   11   12   13   14   15   16   17   >