Re: Reduce timing overhead of EXPLAIN ANALYZE using rdtsc?

2021-03-12 Thread Thomas Munro
On Sat, Jun 13, 2020 at 11:28 AM Andres Freund wrote: > [PATCH v1 1/2] WIP: Change instr_time to just store nanoseconds, that's > cheaper. Makes a lot of sense. If we do this, I'll need to update pgbench, which just did something similar locally. If I'd been paying attention to this thread I m

Re: A qsort template

2021-03-12 Thread Thomas Munro
On Fri, Mar 12, 2021 at 7:58 AM Andres Freund wrote: > I wish we had the same for bsearch... :) Glibc already has the definition of the traditional void-based function in /usr/include/bits/stdlib-bsearch.h, so the generated code when the compiler can see the comparator definition is already good

Re: pgbench: option delaying queries till connections establishment?

2021-03-12 Thread Thomas Munro
On Sat, Mar 13, 2021 at 3:47 PM Tom Lane wrote: > Checking the man pages, it seems that this ancient HPUX version > is using some pre-POSIX API spec in which pthread_cond_init takes a > pthread_condattr_t rather than a pointer to pthread_condattr_t. > Similarly for pthread_mutex_init. Wow. > Whi

Re: pgbench: option delaying queries till connections establishment?

2021-03-12 Thread Thomas Munro
On Sat, Mar 13, 2021 at 4:09 PM Tom Lane wrote: > OK, cool. I don't think it's hard, just do > > if test "$enable_thread_safety" = yes; then > AC_REPLACE_FUNCS(pthread_barrier_wait) > fi > > Probably this check should be likewise conditional: > > AC_SEARCH_LIBS(pthread_barrier_wait, pthread) T

Re: pgbench: option delaying queries till connections establishment?

2021-03-12 Thread Thomas Munro
On Sat, Mar 13, 2021 at 4:59 PM Tom Lane wrote: > Thomas Munro writes: > > Thanks. This seems to work for me on a Mac. I confirmed with nm that > > we don't define or reference any pthread_XXX symbols with > > --disable-thread-safety, and we do otherwise, and the pgb

Re: pgbench: option delaying queries till connections establishment?

2021-03-13 Thread Thomas Munro
On Sat, Mar 13, 2021 at 9:08 PM Fabien COELHO wrote: > I must say that I'm not a big fan of the macro-based all-in-capitals API > for threads because it exposes some platform specific uglyness (eg > THREAD_FUNC_CC) and it does not look much like clean C code when used. I > liked the previous parti

Re: A qsort template

2021-03-13 Thread Thomas Munro
On Sat, Mar 13, 2021 at 3:49 PM Thomas Munro wrote: > On Fri, Mar 12, 2021 at 7:58 AM Andres Freund wrote: > > I wish we had the same for bsearch... :) > > Glibc already has the definition of the traditional void-based > function in /usr/include/bits/stdlib-bsearch.h, so

Re: fdatasync performance problem with large number of DB files

2021-03-14 Thread Thomas Munro
On Thu, Mar 11, 2021 at 2:32 PM Thomas Munro wrote: > On Thu, Mar 11, 2021 at 2:25 PM Tom Lane wrote: > > Trolling the net, I found a newer-looking version of the man page, > > and behold it says > > > >In mainline kernel versions prior to 5.8, syncfs() wil

Re: Boundary value check in lazy_tid_reaped()

2021-03-14 Thread Thomas Munro
27;t work... should be << 16, but > > > even then it would only work with a bsearch that uses int64_t > > > comparators, so I take that part back. > > > > Yeah, it seems to worth benchmarking the speed-up with an inlining. > > I'll do some perform

Re: fdatasync performance problem with large number of DB files

2021-03-14 Thread Thomas Munro
On Mon, Mar 15, 2021 at 11:52 AM Thomas Munro wrote: > Time being of the essence, here is the patch I posted last year, this > time with a GUC and some docs. You can set sync_after_crash to > "fsync" (default) or "syncfs" if you have it. Cfbot told me to add HAVE

Re: A qsort template

2021-03-14 Thread Thomas Munro
On Sun, Mar 14, 2021 at 5:03 PM Zhihong Yu wrote: > For 0001-Add-bsearch-and-unique-templates-to-sort_template.h.patch : > > + * Remove duplicates from an array. Return the new size. > + */ > +ST_SCOPE size_t > +ST_UNIQUE(ST_ELEMENT_TYPE *array, > > The array is supposed to be sorted, right ? > T

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-14 Thread Thomas Munro
[Dropping pgsql-general@ from the CC, because cross-posting triggers moderation; sorry I didn't notice that on my first reply] On Mon, Mar 15, 2021 at 2:05 PM Avinash Kumar wrote: > On Sun, Mar 14, 2021 at 10:01 PM Avinash Kumar > wrote: >> Also the datatype is bigint Ok. Collation changes ar

Re: Collation versioning

2021-03-14 Thread Thomas Munro
On Fri, Nov 6, 2020 at 10:56 AM Thomas Munro wrote: > On Wed, Nov 4, 2020 at 9:11 PM Michael Paquier wrote: > > On Wed, Nov 04, 2020 at 08:44:15AM +0100, Juan José Santamaría Flecha wrote: > > > We could create a static table with the conversion based on what was > >

Re: cleanup temporary files after crash

2021-03-14 Thread Thomas Munro
On Wed, Mar 10, 2021 at 1:31 AM Michael Paquier wrote: > On Tue, Mar 09, 2021 at 02:28:43AM +0100, Tomas Vondra wrote: > > Let's move this patch forward. Based on the responses, I agree the > > default behavior should be to remove the temp files, and I think we > > should have the GUC (on the off

Re: Change JOIN tutorial to focus more on explicit joins

2021-03-14 Thread Thomas Munro
On Thu, Mar 11, 2021 at 2:06 AM David Steele wrote: > On 12/1/20 3:38 AM, Jürgen Purtz wrote: > > OK. Patch attached. +Queries which access multiple tables (including repeats) at once are called I'd write "Queries that" here (that's is a transatlantic difference in usage; I try to proofread

Regression tests vs SERIALIZABLE

2021-03-14 Thread Thomas Munro
read committed + serializable (1 row) From 21115a0721aedec9c39a1d41bc38ec2b96960ff1 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Mon, 15 Mar 2021 17:08:25 +1300 Subject: [PATCH] Parallel regression tests don't need workaround for SERIALIZABLE. SERIALIZABLE no longer inhibits parallelism

Re: Consider Parallelism While Planning For REFRESH MATERIALIZED VIEW

2021-03-14 Thread Thomas Munro
On Mon, Jan 4, 2021 at 9:05 PM Luc Vlaming wrote: > The new status of this patch is: Ready for Committer I think the comments above this might as well be removed, because they aren't very convincing: +-- Allow parallel planning of the underlying query for refresh materialized +-- view. We can be

Re: Regression tests vs SERIALIZABLE

2021-03-14 Thread Thomas Munro
On Mon, Mar 15, 2021 at 6:14 PM Bharath Rupireddy wrote: > On Mon, Mar 15, 2021 at 9:54 AM Thomas Munro wrote: > > While reviewing the patch for parallel REFRESH MATERIALIZED VIEW, I > > noticed that select_parallel.sql and write_parallel.sql believe that > > (1) the tests

Re: Regression tests vs SERIALIZABLE

2021-03-15 Thread Thomas Munro
On Mon, Mar 15, 2021 at 8:00 PM Bharath Rupireddy wrote: > Thanks. v2 LGTM, both make check and make check-world passes on my dev system. Pushed. Thanks!

Re: Regression tests vs SERIALIZABLE

2021-03-15 Thread Thomas Munro
On Mon, Mar 15, 2021 at 5:24 PM Thomas Munro wrote: > However, since commit 862ef372d6b, there *is* one test that fails if > you run make installcheck against a cluster running with -c > default_transaction_isolation=serializable: transaction.sql. Is that > a mistake? Is it a goal t

subscriptionCheck failures

2021-03-15 Thread Thomas Munro
Hi, This seems to be a new low frequency failure, I didn't see it mentioned already: # Failed test 'DROP SUBSCRIPTION during error can clean up the slots on the publisher' # at t/004_sync.pl line 171. # got: '1' # expected: '0' https://buildfarm.postgresql.org/cgi-bin/show_log

Re: Postgres crashes at memcopy() after upgrade to PG 13.

2021-03-15 Thread Thomas Munro
On Mon, Mar 15, 2021 at 1:29 PM Avinash Kumar wrote: > Is this expected when replication is happening between PostgreSQL databases > hosted on different OS versions like Ubuntu 16 and Ubuntu 20 ? Or, do we > think this is some sort of corruption ? Is this index on a text datatype, and using a c

Re: Calendar support in localization

2021-03-15 Thread Thomas Munro
sensus. One key question here is whether you need a different date type or just different operations (functions, operators etc) on the existing types. > I cc Thomas Munro and Vik because they have interest on this area Last time it came up[1], I got as far as wondering if the best way would be

Re: fdatasync performance problem with large number of DB files

2021-03-15 Thread Thomas Munro
On Tue, Mar 16, 2021 at 3:30 AM Paul Guo wrote: > By the way, there is a usual case that we could skip fsync: A fsync-ed > already standby generated by pg_rewind/pg_basebackup. > The state of those standbys are surely not > DB_SHUTDOWNED/DB_SHUTDOWNED_IN_RECOVERY, so the > pgdata directory is fs

Re: Consider Parallelism While Planning For REFRESH MATERIALIZED VIEW

2021-03-15 Thread Thomas Munro
On Mon, Mar 15, 2021 at 8:25 PM Bharath Rupireddy wrote: > > > The problem with a case like REFRESH MATERIALIZED VIEW is that there's > > > nothing to prevent something that gets run in the course of the query > > > from trying to access the view (and the heavyweight lock won't prevent > > > that,

Re: Calendar support in localization

2021-03-16 Thread Thomas Munro
On Wed, Mar 17, 2021 at 6:31 AM Surafel Temesgen wrote: > Ethiopice calendar have 13 months so it can not be stored as date and > timestamp type and you approach seems more complicated and i suggest to have > this feature on the purpose of PostgreSQL popularities too not only for my > need I k

Re: Consider Parallelism While Planning For REFRESH MATERIALIZED VIEW

2021-03-16 Thread Thomas Munro
On Tue, Mar 16, 2021 at 2:41 PM Thomas Munro wrote: > On Mon, Mar 15, 2021 at 8:25 PM Bharath Rupireddy > wrote: > > > > The problem with a case like REFRESH MATERIALIZED VIEW is that there's > > > > nothing to prevent something that gets run in the course

Re: cutting down the TODO list thread

2020-10-27 Thread Thomas Munro
On Wed, Oct 28, 2020 at 8:36 AM Andres Freund wrote: > On 2020-10-27 15:24:35 -0400, John Naylor wrote: > > - Allow WAL replay of CREATE TABLESPACE to work when the directory > > structure on the recovery computer is different from the original > > Thread quote: "part of the difficult, perhaps-n

-Wformat-signedness

2020-10-29 Thread Thomas Munro
Hi hackers, There're probably mostly harmless, being mostly error and debug messages and the like, and considering that eg OID parsing tolerates negative numbers when reading them back in, but for what it's worth: GCC complains about many %d vs %u type mixups if you build with $SUBJECT.

Re: how to replicate test results in cf-bot on travis

2020-11-01 Thread Thomas Munro
On Mon, Nov 2, 2020 at 1:58 AM Dave Cramer wrote: > For my patch https://commitfest.postgresql.org/30/2522/ > > When I run > > make -j4 all contrib && make check-world > locally > > I see 2 errors. > > When cf-bot runs this it sees > 35 out of 93 failed. > > How can I see the same errors? Hi Dave

Re: Dereference before NULL check (src/backend/storage/ipc/latch.c)

2020-11-01 Thread Thomas Munro
On Mon, Nov 2, 2020 at 1:49 PM Kyotaro Horiguchi wrote: > At Sat, 31 Oct 2020 11:40:53 -0300, Ranier Vilela wrote > in > > Per Coverity. > > > > If test set->latch against NULL, is why it can be NULL. > > ResetEvent can dereference NULL. > > If the returned event is WL_LATCH_SET, set->latch cann

Re: Collation versioning

2020-11-02 Thread Thomas Munro
On Mon, Nov 2, 2020 at 10:28 PM Julien Rouhaud wrote: > + /* Remember not to complain about this collation again. */ > + context->checked_colls = lappend_oid(context->checked_colls, > +otherObject->objectId); > > It's maybe not immediately obvious that i

Re: Collation versioning

2020-11-02 Thread Thomas Munro
On Tue, Nov 3, 2020 at 2:08 AM Julien Rouhaud wrote: > On Mon, Nov 2, 2020 at 9:04 PM Thomas Munro wrote: > > Ok, moved and renamed, and finally pushed. Thanks for all the help! > > \o/ Thanks a lot Thomas! Hmm, a failure from dory (WIndows) during pg_upgrade: performing

Re: Collation versioning

2020-11-02 Thread Thomas Munro
On Tue, Nov 3, 2020 at 6:51 AM Tom Lane wrote: > Thomas Munro writes: > > Hmm, a failure from dory (WIndows) during pg_upgrade: > > > performing post-bootstrap initialization ... 2020-11-02 08:08:22.213 > > EST [5392] FATAL: could not get collation version for locale &g

Re: Collation versioning

2020-11-02 Thread Thomas Munro
On Tue, Nov 3, 2020 at 1:51 PM David Rowley wrote: > On Tue, 3 Nov 2020 at 12:29, David Rowley wrote: > > Running low on ideas for now, so thought I'd post this in case it > > someone thinks of something else. > > FWIW, the attached does fix the issue for me. It basically just calls > the functi

Re: Dereference before NULL check (src/backend/storage/ipc/latch.c)

2020-11-02 Thread Thomas Munro
On Tue, Nov 3, 2020 at 12:50 AM Kyotaro Horiguchi wrote: > With the fix patch, it changes to: > > [16632] LOG: FALSE LATCH: Nice repo. But is it OK to not reset the Win32 event in this case? Does it still work correctly if you wait on the latch after that happened, and perhaps

Re: Collation versioning

2020-11-03 Thread Thomas Munro
On Tue, Nov 3, 2020 at 4:38 PM Thomas Munro wrote: > On Tue, Nov 3, 2020 at 1:51 PM David Rowley wrote: > > On Tue, 3 Nov 2020 at 12:29, David Rowley wrote: > > > Running low on ideas for now, so thought I'd post this in case it > > > someone thinks of something

Re: Collation versioning

2020-11-03 Thread Thomas Munro
On Wed, Nov 4, 2020 at 10:52 AM Tom Lane wrote: > Thomas Munro writes: > > We want the same algorithm that Windows uses internally to resolve the > > old style name to a collation; in other words we probably want > > something more like the code path that they took away in

Re: Collation versioning

2020-11-03 Thread Thomas Munro
On Wed, Nov 4, 2020 at 10:56 AM Thomas Munro wrote: > On Wed, Nov 4, 2020 at 10:52 AM Tom Lane wrote: > > Thomas Munro writes: > > > We want the same algorithm that Windows uses internally to resolve the > > > old style name to a collation; in other words we probabl

Re: PANIC: could not fsync file "pg_multixact/..." since commit dee663f7843

2020-11-03 Thread Thomas Munro
On Wed, Nov 4, 2020 at 2:32 PM Tomas Vondra wrote: > After a while (~1h on my machine) the pg_multixact gets over 10GB, which > triggers a more aggressive cleanup (per MultiXactMemberFreezeThreshold). > My guess is that this discards some of the files, but checkpointer is > not aware of that, or s

Re: Collation versioning

2020-11-03 Thread Thomas Munro
On Wed, Nov 4, 2020 at 2:56 PM David Rowley wrote: > initdb works fine. I ran vcregress upgradecheck and it passes. > > With my default locale of English.New Zealand.1252 I get zero rows from: > > select * from pg_depend where coalesce(refobjversion,'') <> ''; > > if I initdb with --lc-collate=en-

Re: PANIC: could not fsync file "pg_multixact/..." since commit dee663f7843

2020-11-03 Thread Thomas Munro
On Wed, Nov 4, 2020 at 2:57 PM Tomas Vondra wrote: > On Wed, Nov 04, 2020 at 02:49:24PM +1300, Thomas Munro wrote: > >On Wed, Nov 4, 2020 at 2:32 PM Tomas Vondra > > wrote: > >> After a while (~1h on my machine) the pg_multixact gets over 10GB, which > >> trigge

Re: PANIC: could not fsync file "pg_multixact/..." since commit dee663f7843

2020-11-04 Thread Thomas Munro
On Thu, Nov 5, 2020 at 12:07 PM Tomas Vondra wrote: > It's been running for hours on both machines, without any crashes etc. > While that's not a definitive proof the fix is correct, it certainly > behaves differently. Thanks! Embarrassed to have missed that. Pushed.

Re: Collation versioning

2020-11-05 Thread Thomas Munro
On Wed, Nov 4, 2020 at 9:11 PM Michael Paquier wrote: > On Wed, Nov 04, 2020 at 08:44:15AM +0100, Juan José Santamaría Flecha wrote: > > We could create a static table with the conversion based on what was > > discussed for commit a169155, please find attached a spreadsheet with the > > compariso

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-11-05 Thread Thomas Munro
On Thu, Nov 5, 2020 at 10:47 PM Amit Kapila wrote: > I still feel 'cached' is a better name. Amusingly, this thread is hitting all the hardest problems in computer science according to the well known aphorism... Here's a devil's advocate position I thought about: It's OK to leave stray buffers

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-11-05 Thread Thomas Munro
On Fri, Nov 6, 2020 at 5:09 PM Amit Kapila wrote: > On Fri, Nov 6, 2020 at 5:02 AM Thomas Munro wrote: > > Here's a devil's advocate position I thought about: It's OK to leave > > stray buffers (clean or dirty) in the buffer pool if files are > > truncated u

Re: Advance xmin aggressively on Read Commit isolation level

2020-11-06 Thread Thomas Munro
On Fri, Nov 6, 2020 at 9:48 PM Andy Fan wrote: > I have 2 ideas about this. One is in the Read Committed level, we can > advance xmin > aggressively. suppose it started at t1, and complete a query at t2. the xmin > should > be t1 currently. Can we advance the xmin to t2 since it is read comm

Re: Collation versioning

2020-11-08 Thread Thomas Munro
On Fri, Oct 4, 2019 at 1:25 AM Thomas Munro wrote: > Ok. Here's one like that. Also, a WIP patch for FreeBSD. Here's an updated patch for FreeBSD, which I'll sit on for a bit longer. It needs bleeding edge 13-CURRENT (due out Q1ish). From b9cb5562457c214c48c0a6334b8ed3264f50

Re: -Wformat-signedness

2020-11-09 Thread Thomas Munro
On Tue, Nov 10, 2020 at 4:25 AM Tom Lane wrote: > Peter Eisentraut writes: > > 1. enums are unsigned by default in gcc, so all those internal error > > messages "unrecognized blah kind: %d" need to be changed to %u. > > Do we have reason to think that that is true in every C compiler? > My own pr

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-11-09 Thread Thomas Munro
On Sat, Nov 7, 2020 at 12:40 AM Amit Kapila wrote: > I think one of the problems is returning fewer rows and that too > without any warning or error, so maybe that is a bigger problem but we > seem to be okay with it as that is already a known thing though I > think that is not documented anywhere

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-11-09 Thread Thomas Munro
On Tue, Nov 10, 2020 at 6:18 PM Amit Kapila wrote: > Do we always truncate all the blocks? What if the vacuum has cleaned > last N (say 100) blocks then how do we handle it? Oh, hmm. Yeah, that idea only works for DROP, not for truncate last N.

Re: MultiXact\SLRU buffers configuration

2020-11-10 Thread Thomas Munro
On Wed, Nov 11, 2020 at 7:07 AM Tomas Vondra wrote: > Seems we haven't made much progress in reproducing the issue :-( I guess > we'll need to know more about the machine where this happens. Is there > anything special about the hardware/config? Are you monitoring size of > the pg_multixact direct

Re: Background writer and checkpointer in crash recovery

2020-11-11 Thread Thomas Munro
On Wed, Nov 11, 2020 at 9:57 PM Simon Riggs wrote: > Having said that, we did raise the checkpoint_timeout by a lot, so the > situation today might be quite different. A large checkpoint_timeout > could eventually overflow shared buffers, with the right workload. FWIW Jakuk Wartak did manage to s

Re: Optimising latch signals

2020-11-12 Thread Thomas Munro
On Sun, Aug 9, 2020 at 11:48 PM Thomas Munro wrote: > Here are some more experimental patches to reduce system calls. > > 0001 skips sending signals when the recipient definitely isn't > waiting, using a new flag-and-memory-barrier dance. This seems to > skip around 12% of

Re: POC: Cleaning up orphaned files using undo logs

2020-11-12 Thread Thomas Munro
On Thu, Nov 12, 2020 at 10:15 PM Antonin Houska wrote: > Thomas Munro wrote: > > Thanks. We decided to redesign a couple of aspects of the undo > > storage and record layers that this patch was intended to demonstrate, > > and work on that is underway. More on that

Re: "pg_ctl: the PID file ... is empty" at end of make check

2020-11-13 Thread Thomas Munro
On Fri, Oct 18, 2019 at 2:26 PM Thomas Munro wrote: > On Fri, Oct 18, 2019 at 1:21 AM Tom Lane wrote: > > Thomas Munro writes: > > > On Tue, Oct 15, 2019 at 1:55 PM Tom Lane wrote: > > >> and now prairiedog has shown it too: > > >> https://build

Re: Cache relation sizes?

2020-11-15 Thread Thomas Munro
On Tue, Aug 4, 2020 at 2:21 PM Thomas Munro wrote: > On Tue, Aug 4, 2020 at 3:54 AM Konstantin Knizhnik > wrote: > > This shared relation cache can easily store relation size as well. > > In addition it will solve a lot of other problems: > > - noticeable overhead o

Re: Cache relation sizes?

2020-11-16 Thread Thomas Munro
On Mon, Nov 16, 2020 at 11:01 PM Konstantin Knizhnik wrote: > I will look at your implementation more precisely latter. Thanks! Warning: I thought about making a thing like this for a while, but the patch itself is only a one-day prototype, so I am sure you can find many bugs... Hmm, I guess t

Re: ERROR: too many dynamic shared memory segment

2020-11-17 Thread Thomas Munro
On Wed, Nov 18, 2020 at 4:21 AM Ben Kanouse wrote: > My database is experiencing a 'ERROR: too many dynamic shared memory > segment' error from time to time. It seems to happen most when traffic > is high, and it happens with semi-simple select statements that run a > parallel query plan with a pa

Re: ERROR: too many dynamic shared memory segment

2020-11-17 Thread Thomas Munro
On Wed, Nov 18, 2020 at 10:55 AM Thomas Munro wrote: > a few times those ways, also due to an ancient DSM leak that was fixed > in 93745f1e (fix present in 12.4). I take that bit back -- I misremembered -- that was a leak of the actual memory once slots were exhausted, not a leak of the slot.

Re: Optimising latch signals

2020-11-18 Thread Thomas Munro
On Fri, Nov 13, 2020 at 12:42 PM Thomas Munro wrote: > 1. It's a bit clunky that pqinitmask() takes a new argument to say > whether SIGURG should be blocked; that's because the knowledge of > which latch implementation we're using is private to latch.c, and only >

Re: Cache relation sizes?

2020-11-18 Thread Thomas Munro
e changes AKA dirty updates are serialised, but clean updates can run concurrently, so we just have to make sure that clean updates never clobber dirty updates -- do you think that is on the right path? From ac3c61926bf947a3288724bd02cf8439ff5c14bc Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Fri

Re: ERROR: too many dynamic shared memory segment

2020-11-19 Thread Thomas Munro
On Wed, Nov 18, 2020 at 10:55 AM Thomas Munro wrote: > > Should this commit be back-ported to earlier versions of postgres to > > prevent this error in other versions? > > Yeah, that seems like a good idea anyway. I will do that tomorrow, > barring objections. Done, for 10, 11 and 12.

Re: Migration Oracle multitenant database to PostgreSQL ?

2020-11-24 Thread Thomas Kellerer
own a single database as you can do with a PDB. Database users in Postgres are like Oracle's "common users", they are global for the whole instance (aka "cluster" in Postgres' terms). There are no database specific users. Thomas

Re: WIP: WAL prefetch (another approach)

2020-11-24 Thread Thomas Munro
On Thu, Nov 19, 2020 at 10:00 AM Stephen Frost wrote: > * Thomas Munro (thomas.mu...@gmail.com) wrote: > > Hmm. Every time I try to think of a protocol change for the > > restore_command API that would be acceptable, I go around the same > > circle of thoughts about event

Re: Dereference before NULL check (src/backend/storage/ipc/latch.c)

2020-11-24 Thread Thomas Munro
On Thu, Nov 5, 2020 at 10:47 AM Ranier Vilela wrote: > Em ter., 3 de nov. de 2020 às 22:09, Kyotaro Horiguchi > escreveu: >> If we call WaitLatch(OrSocket) that waits on the latch, it immediately >> returns because the latch is set. If we called ResetLatch before the >> next call to WaitLatch(),

Re: Optimising latch signals

2020-11-25 Thread Thomas Munro
On Thu, Nov 19, 2020 at 4:49 PM Thomas Munro wrote: > I'll add this to the next commitfest. Let's see if this version fixes the Windows compile error and warning reported by cfbot. From 3eb542891a11d39047b28f6f33ae4e3d25bdd510 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Sat,

Re: BUG #16663: DROP INDEX did not free up disk space: idle connection hold file marked as deleted

2020-11-29 Thread Thomas Munro
cate("base/14208/16396.1", 0) = -1 ENOENT truncate("base/14208/16396_fsm", 0) = -1 ENOENT truncate("base/14208/16396_vm", 0) = -1 ENOENT truncate("base/14208/16396_init", 0)= -1 ENOENT From 8c2c189ce0c2aa25d0c3f414034b939abe4eb4ee Mon Sep 17 00:

Re: pg_preadv() and pg_pwritev()

2021-01-10 Thread Thomas Munro
On Wed, Dec 23, 2020 at 12:06 AM Thomas Munro wrote: > On Mon, Dec 21, 2020 at 11:40 AM Andres Freund wrote: > > Can we come up with a better name than 'uio'? I find that a not exactly > > meaningful name. > > Ok, let's try port/pg_iovec.h. I pushed it with t

Re: pg_preadv() and pg_pwritev()

2021-01-10 Thread Thomas Munro
On Mon, Jan 11, 2021 at 3:34 PM Thomas Munro wrote: > On Wed, Dec 23, 2020 at 12:06 AM Thomas Munro wrote: > > On Mon, Dec 21, 2020 at 11:40 AM Andres Freund wrote: > > > Can we come up with a better name than 'uio'? I find that a not exactly > > > meanin

O(n^2) system calls in RemoveOldXlogFiles()

2021-01-10 Thread Thomas Munro
Hi, I noticed that RemoveXlogFile() has this code: /* * Before deleting the file, see if it can be recycled as a future log * segment. Only recycle normal files, pg_standby for example can create * symbolic links pointing to a separate archive directory.

Re: O(n^2) system calls in RemoveOldXlogFiles()

2021-01-11 Thread Thomas Munro
On Tue, Jan 12, 2021 at 6:55 PM Andres Freund wrote: > I found this before as well: > https://postgr.es/m/CAB7nPqTB3VcKSSrW2Qj59tYYR2H4+n=5pzbdwou+x9iqvnm...@mail.gmail.com Hah, I even knew that, apparently, but forgot. Adding Michael who wrote a patch. It'd be nice to fix this, at least in 14

Re: O(n^2) system calls in RemoveOldXlogFiles()

2021-01-12 Thread Thomas Munro
On Tue, Jan 12, 2021 at 8:27 PM Michael Paquier wrote: > Yeah, this rings a bell. I never went back to it even if the thing > looks rather clean at quick glance (not tested), but I may be able > to spend some cycles on that. I don't think that's critical enough > for a backpatch, so doing someth

Re: pg_preadv() and pg_pwritev()

2021-01-12 Thread Thomas Munro
On Mon, Jan 11, 2021 at 3:59 PM Thomas Munro wrote: > On Mon, Jan 11, 2021 at 3:34 PM Thomas Munro wrote: > > I pushed it with that name, and a couple more cosmetic changes. I'll > > keep an eye on the build farm. > > Since only sifaka has managed to return a result

Re: pg_preadv() and pg_pwritev()

2021-01-13 Thread Thomas Munro
On Wed, Jan 13, 2021 at 10:40 PM Sergey Shinderuk wrote: > /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk/usr/include/sys/uio.h:104:9: > note: 'pwritev' has been marked as being introduced in macOS 11.0 here, > but the deployment target is macOS

Re: pg_preadv() and pg_pwritev()

2021-01-13 Thread Thomas Munro
627bb80acb243d9e56 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Thu, 14 Jan 2021 08:19:18 +1300 Subject: [PATCH] Move our p{read,write}v replacements into their own files. macOS's ranlib issued a warning about an empty pread.o file with the previous arrangement, on systems new enough to

Re: pg_preadv() and pg_pwritev()

2021-01-13 Thread Thomas Munro
On Thu, Jan 14, 2021 at 9:26 AM Tom Lane wrote: > * You need to remove pread.o and pwrite.o from the hard-wired > part of the list in src/port/Makefile, else they get built > whether needed or not. Right, done. > * I don't much like this in fd.h: > > @@ -46,6 +46,7 @@ > #include > > > +struct

Re: Cirrus CI (Windows help wanted)

2021-01-14 Thread Thomas Munro
On Wed, Jan 13, 2021 at 3:04 AM Andrew Dunstan wrote: > OK, I got this working. Thanks Andrew. This is great! > # cirrus does something odd with this command, so it's stuck in a bat file > # and copied to the docker container and then executed > COPY ci/inst-tools.bat . > RUN \

Re: O(n^2) system calls in RemoveOldXlogFiles()

2021-01-14 Thread Thomas Munro
On Fri, Jan 15, 2021 at 3:07 PM Michael Paquier wrote: > On Wed, Jan 13, 2021 at 04:27:25PM +0900, Michael Paquier wrote: > > I have been looking again at that, and the rebased version that Andres > > has provided would take care of that. Any thoughts? > > Hearing nothing, I have applied the thin

fdatasync(2) on macOS

2021-01-14 Thread Thomas Munro
Hello hackers, While following along with the nearby investigation into weird cross-version Apple toolchain issues that confuse configure, I noticed that the newer buildfarm Macs say: checking for fdatasync... (cached) yes That's a bit strange because there's no man page and no declaration: che

Re: fdatasync(2) on macOS

2021-01-15 Thread Thomas Munro
On Fri, Jan 15, 2021 at 7:53 PM Thomas Munro wrote: > That was fun, but now I'm asking myself: do we really want to use an > IO synchronisation facility that's not declared by the vendor? I should add, the default wal_sync_method is open_datasync, not fdatasync. I'm pret

Re: pgbench: option delaying queries till connections establishment?

2021-01-17 Thread Thomas Munro
On Sat, Jan 9, 2021 at 8:13 AM Thomas Munro wrote: > On Sun, Jan 3, 2021 at 9:49 AM Fabien COELHO wrote: > > > Just for fun, the attached 0002 patch is a quick prototype of a > > > replacement for that stuff that seems to work OK on a Mac here. (I'm > > >

Re: pg_collation_actual_version() ERROR: cache lookup failed for collation 123

2021-01-17 Thread Thomas Munro
On Mon, Jan 18, 2021 at 10:59 AM Justin Pryzby > |postgres=# SELECT pg_collation_actual_version(123); > |ERROR: cache lookup failed for collation 123 Yeah, not a great user experience. Will fix next week; perhaps get_collation_version_for_oid() needs missing_ok and found flags, or something like

Re: fdatasync(2) on macOS

2021-01-17 Thread Thomas Munro
On Sat, Jan 16, 2021 at 6:55 AM Tom Lane wrote: > So it's not a no-op, but on the other hand it's not succeeding in getting > bits down to the platter. I'm not inclined to dike it out, but it does > seem problematic that we're defaulting to open_datasync, which is also > not getting bits down to

Re: compression libraries and CF bot

2021-01-19 Thread Thomas Munro
On Wed, Jan 20, 2021 at 9:56 AM Justin Pryzby wrote: > Do you know if the old travis build environment had liblz4 installed ? It sounds like it. > I'm asking regarding Dilip's patch, which was getting to "check world" 2 weeks > ago but now failing to even compile, not apparently due to any chang

Re: cfbot building docs - serving results

2021-01-19 Thread Thomas Munro
On Wed, Jan 20, 2021 at 10:22 AM Erik Rijkers wrote: > I am wondering if the cfbot at the moment is building the docs > (html+pdf), for the patches that it tests. I suppose that it does? If > so, what happens with the resulting (doc)files? To /dev/null? They are > not available as far as I can

Re: [bug?] EXPLAIN outputs 0 for rows and width in cost estimate for update nodes

2021-01-20 Thread Thomas Munro
n, This was a change made deliberately. Do you see a problem? commit f0f13a3a08b2757997410f3a1c38bdc22973c525 Author: Thomas Munro Date: Mon Oct 12 20:41:16 2020 +1300 Fix estimates for ModifyTable paths without RETURNING. In the past, we always estimated that a ModifyTable node would emi

Re: [bug?] EXPLAIN outputs 0 for rows and width in cost estimate for update nodes

2021-01-20 Thread Thomas Munro
On Wed, Jan 20, 2021 at 9:42 PM Laurenz Albe wrote: > On Wed, 2021-01-20 at 08:35 +, tsunakawa.ta...@fujitsu.com wrote: > > > This was a change made deliberately. Do you see a problem? > > > > Thank you, I was surprised at your very quick response. > > I just wanted to confirm I can believe E

Re: [PATCH 1/1] Initial mach based shared memory support.

2021-01-20 Thread Thomas Munro
On Sun, Nov 22, 2020 at 9:19 PM James Hilliard wrote: > In order to avoid hitting these limits we can bypass the wrapper layer > and just use mach directly. FWIW I looked into using mach_vm_alllocate() years ago because I wanted to be able to use its VM_FLAGS_SUPERPAGE_SIZE_2MB flag to implement

Re: Jsonpath ** vs lax mode

2021-01-21 Thread Thomas Kellerer
es strict mode to work properly. It should probably be documented in chapter 9.16.2 "The SQL/JSON Path Language", maybe with a little warning in the description of jsonb_path_query** and in chapter 8.14.16 as well (or at least that's were I would expect such a warning) Regards Thomas

Re: Why does creating logical replication subscriptions require superuser?

2021-01-22 Thread Thomas Kellerer
uns the CREATE SUBSCRIPTION command. Thomas

Re: Use pg_pwrite() in pg_test_fsync

2021-01-23 Thread Thomas Munro
On Sun, Jan 10, 2021 at 9:21 AM Thomas Munro wrote: > I left the fsync-after-closing and non-sync'd tests using write(), > because they weren't using lseek(). The latter case is arguably a bit > odd because it's not overwriting pre-allocated blocks, unlike the >

Re: fdatasync(2) on macOS

2021-01-25 Thread Thomas Munro
On Mon, Jan 18, 2021 at 5:08 PM Tom Lane wrote: > (1) other platforms weren't safe-by-default either. Perhaps the > state of the art is better now, though? Generally the answer seems to be yes, but there are still some systems out there that don't send flushes when volatile write cache is enable

Re: shared tempfile was not removed on statement_timeout

2021-01-26 Thread Thomas Munro
On Wed, Jan 27, 2021 at 12:22 AM Kyotaro Horiguchi wrote: > At Tue, 26 Jan 2021 11:00:56 +0200, Heikki Linnakangas > wrote in > > Don't we potentially have the same problem with all on_dsm_detach > > callbacks? Looking at the other on_dsm_detach callbacks, I don't see > > any CHECK_FOR_INTERRUPT

Re: [PATCH] remove pg_standby

2021-01-26 Thread Thomas Munro
rigger file when recovery ends - - - - - remove no-longer-needed files from the archive directory - - - - - - - The copy command on Windows sets the final file size - before the file is completely copied, which would ordinarily confuse - pg_standby. Therefo

Re: WIP: WAL prefetch (another approach)

2021-01-26 Thread Thomas Munro
On Sat, Dec 5, 2020 at 7:27 AM Stephen Frost wrote: > * Thomas Munro (thomas.mu...@gmail.com) wrote: > > I just noticed this thread proposing to retire pg_standby on that > > basis: > > > > https://www.postgresql.org/message-id/flat/20201029024412.GP5380%40telsasoft.

Re: [PATCH] remove pg_standby

2021-01-26 Thread Thomas Munro
On Wed, Jan 27, 2021 at 6:06 PM Michael Paquier wrote: > On Wed, Jan 27, 2021 at 04:13:24PM +1300, Thomas Munro wrote: > > I would like to commit this, because "waiting restore commands" have > > confusing interactions with my proposed prefetching-during-recovery > &

Re: [PATCH] remove pg_standby

2021-01-28 Thread Thomas Munro
On Thu, Jan 28, 2021 at 8:36 PM Michael Paquier wrote: > On Wed, Jan 27, 2021 at 05:08:56PM +0900, Fujii Masao wrote: > > But one question is; shouldn't we follow "usual" way to retire the > > feature instead of dropping that immediately? That is, mark > > pg_standby as obsolete, announce that pg_

Re: [PATCH] remove pg_standby

2021-01-28 Thread Thomas Munro
On Fri, Jan 29, 2021 at 11:13 AM Thomas Munro wrote: > On Thu, Jan 28, 2021 at 8:36 PM Michael Paquier wrote: > > On Wed, Jan 27, 2021 at 05:08:56PM +0900, Fujii Masao wrote: > > > But one question is; shouldn't we follow "usual" way to retire the >

Fix DROP TABLESPACE on Windows with ProcSignalBarrier?

2021-01-30 Thread Thomas Munro
r code, I couldn't resist replacing its poll/sleep loop with condition variables. Thoughts? From bf32e7426cf780d33ae1140c443ae8964397a3c9 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Sat, 30 Jan 2021 14:02:32 +1300 Subject: [PATCH 1/2] Use a global barrier to fix DROP TABLESPACE on Windows

<    1   2   3   4   5   6   7   8   9   10   >