Re: Parallel query hangs after a smart shutdown is issued

2020-08-13 Thread Thomas Munro
On Fri, Aug 14, 2020 at 4:45 AM Tom Lane wrote: > I wrote: > > Hmmm ... maybe that should be more like > > if (smartShutState != SMART_NORMAL_USAGE && > > backend_type == BACKEND_TYPE_NORMAL) > > After some more rethinking and testing, here's a v5 that feels > fairly final to m

Re: fixing old_snapshot_threshold's time->xid mapping

2020-08-14 Thread Thomas Munro
On Fri, Aug 14, 2020 at 1:04 PM Thomas Munro wrote: > On Fri, Aug 14, 2020 at 12:52 PM Thomas Munro wrote: > > Here's a rebase. > > And another, since I was too slow and v6 is already in conflict... > sorry for the high frequency patches. And ... now that this has a commi

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-08-14 Thread Thomas Munro
On Fri, Aug 14, 2020 at 6:14 PM Amit Kapila wrote: > Yeah, that makes sense. I will take care of that later today or > tomorrow. We have not noticed that because currently none of the > extensions is using those functions. BTW, I noticed that after > failure, the next run is green, why so? Is the

One-off failure in "cluster" test

2020-08-16 Thread Thomas Munro
Hi, I wonder what caused this[1] one-off failure to see tuples in clustered order: diff -U3 /home/pgbfarm/buildroot/REL_13_STABLE/pgsql.build/src/test/regress/expected/cluster.out /home/pgbfarm/buildroot/REL_13_STABLE/pgsql.build/src/test/regress/results/cluster.out --- /home/pgbfarm/buildroot/

Re: One-off failure in "cluster" test

2020-08-16 Thread Thomas Munro
On Mon, Aug 17, 2020 at 1:20 PM Tom Lane wrote: > Thomas Munro writes: > > I wonder what caused this[1] one-off failure to see tuples in clustered > > order: > > ... > > I guess a synchronised scan could cause that, but I wouldn't expect one > > here. >

Re: One-off failure in "cluster" test

2020-08-16 Thread Thomas Munro
On Mon, Aug 17, 2020 at 1:27 PM Thomas Munro wrote: > > On Mon, Aug 17, 2020 at 1:20 PM Tom Lane wrote: > > Thomas Munro writes: > > > I wonder what caused this[1] one-off failure to see tuples in clustered > > > order: > > > ... > > > I g

Optimising compactify_tuples()

2020-08-17 Thread Thomas Munro
.postgresql.org/29/2669/ [2] https://www.postgresql.org/message-id/flat/3c6ff1d3a2ff429ee80d0031e6c69cb7%40postgrespro.ru [3] https://www.postgresql.org/message-id/flat/546B89DE.7030906%40vmware.com From 4f3e6ccab72b3241da9afebac311eb32400eaa61 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Mon,

Re: EDB builds Postgres 13 with an obsolete ICU version

2020-08-18 Thread Thomas Kellerer
om SQL Server to Postgres very often keep Windows as the operating system and they want to get SQL Server's case-insensitivity (at least partially) using ICU collations. Thomas

Re: Optimising compactify_tuples()

2020-08-19 Thread Thomas Munro
out a bit more, and replace the three other copies of qsort() while I'm at it. From 8d0b569fcf6141b622c63fc4bc102c762f01ca9e Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Mon, 17 Aug 2020 21:31:56 +1200 Subject: [PATCH 1/4] Add sort_template.h for making fast sort functions. Move our qso

Re: Optimising compactify_tuples()

2020-08-19 Thread Thomas Munro
On Wed, Aug 19, 2020 at 11:41 PM Thomas Munro wrote: > On Tue, Aug 18, 2020 at 6:53 AM Peter Geoghegan wrote: > > I definitely think that we should have something like this, though. > > It's a relatively easy win. There are plenty of workloads that spend > > lots of ti

Re: Handing off SLRU fsyncs to the checkpointer

2020-08-25 Thread Thomas Munro
On Tue, Aug 25, 2020 at 9:16 PM Jakub Wartak wrote: > I just wanted to help testing this patch (defer SLRU fsyncs during recovery) > and also faster compactify_tuples() patch [2] as both are related to the WAL > recovery performance in which I'm interested in. This is my first message to > this

Re: Handing off SLRU fsyncs to the checkpointer

2020-08-26 Thread Thomas Munro
On Thu, Aug 27, 2020 at 6:15 AM Alvaro Herrera wrote: > > --4.90%--smgropen > > |--2.86%--ReadBufferWithoutRelcache > > Looking at an earlier report of this problem I was thinking whether it'd > make sense to replace SMgrRelationHash with a simplehash tabl

Re: Help needed configuring postgreSQL with xml support

2020-08-27 Thread Thomas Munro
On Thu, Aug 27, 2020 at 8:17 PM Khanna, Sachin 000 wrote: > I am getting following error in configuration.log of installation . Please > help You didn't mention what operating system this is, but, for example, if it's Debian, Ubuntu or similar you might need to install libxml2-dev and pkg-confi

Re: Handing off SLRU fsyncs to the checkpointer

2020-08-27 Thread Thomas Munro
On Thu, Aug 27, 2020 at 8:48 PM Jakub Wartak wrote: > I've tried to get cache misses ratio via PMCs, apparently on EC2 they are > (even on bigger) reporting as not-supported or zeros. I heard some of the counters are only allowed on their dedicated instance types. > However interestingly the wo

Re: Handing off SLRU fsyncs to the checkpointer

2020-08-27 Thread Thomas Munro
On Thu, Aug 27, 2020 at 8:48 PM Jakub Wartak wrote: > >> 29.62% postgres [kernel.kallsyms] [k] > >> copy_user_enhanced_fast_string > >> ---copy_user_enhanced_fast_string > >>|--17.98%--copyin > >> [..] > >>| __pwrite_nocancel > >>

Re: Handing off SLRU fsyncs to the checkpointer

2020-08-28 Thread Thomas Munro
On Sat, Aug 29, 2020 at 12:43 AM Jakub Wartak wrote: > USERPID %CPU %MEMVSZ RSS TTY STAT START TIME COMMAND > postgres 120935 0.9 0.0 866052 3824 ?Ss 09:47 0:00 postgres: > checkpointer > postgres 120936 61.9 0.0 865796 3824 ?Rs 09:47 0:22 postgre

Re: Handing off SLRU fsyncs to the checkpointer

2020-08-28 Thread Thomas Munro
On Sat, Aug 29, 2020 at 12:43 AM Jakub Wartak wrote: > ... %CPU ... COMMAND > ... 97.4 ... postgres: startup recovering 00010089 So, what else is pushing this thing off CPU, anyway? For one thing, I guess it might be stalling while reading the WAL itself, because (1) we only read

Background writer and checkpointer in crash recovery

2020-08-29 Thread Thomas Munro
flags and possibly an end-of-recovery record. What problems do you foresee with that? Why should we have "fast" promotion but not "fast" crash recovery? [1] https://www.postgresql.org/message-id/flat/CA+hUKGLJ=84yt+nvhkeedauutvhmfq9i-n7k_o50jmq6rpj...@mail.gmail.com From 21d7d4

Re: Windows buildfarm members vs. new async-notify isolation test

2020-08-29 Thread Thomas Munro
On Thu, Dec 12, 2019 at 9:11 AM Tom Lane wrote: > Amit Kapila writes: > > I am convinced by your points. So +1 for your proposed patch. I have > > already reviewed it yesterday and it appears fine to me. > > OK, pushed. Thanks for reviewing! I made a thing to watch out for low probability BF

Rare link canary failure in dblink test

2020-08-29 Thread Thomas Munro
Hi, A couple of recent cases where an error "libpq is incorrectly linked to backend functions" broke the dblink test: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=lorikeet&dt=2020-08-22%2002:55:22 | REL9_6_STABLE | Windows https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=l

Re: Terminate the idle sessions

2020-08-30 Thread Thomas Munro
On Tue, Aug 18, 2020 at 2:13 PM Li Japin wrote: > On Aug 18, 2020, at 9:19 AM, Kyotaro Horiguchi > wrote: > The same already happens for idle_in_transaction_session_timeout and > we can use "ALTER ROLE/DATABASE SET" to dislable or loosen them, it's > a bit cumbersome, though. I don't think we sh

Re: Terminate the idle sessions

2020-08-30 Thread Thomas Munro
On Mon, Aug 31, 2020 at 2:40 PM Li Japin wrote: > Could you give the more details about the test instructions? Hi Japin, Sure. Because I wasn't trying to get reliable TPS number or anything, I just used a simple short read-only test with one connection, like this: pgbench -i -s10 postgres pgbe

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2020-08-30 Thread Thomas Munro
On Sat, Aug 29, 2020 at 3:33 AM Robert Haas wrote: > I think David's points elsewhere on the thread about ProjectSet and > Materialize nodes are interesting. Indeed, I'm now finding it very difficult to look past the similarity with: postgres=# explain select count(*) from t t1 cross join t t2;

Re: Boundary value check in lazy_tid_reaped()

2020-08-31 Thread Thomas Munro
On Sun, Aug 30, 2020 at 11:08 PM Masahiko Sawada wrote: > So my proposal is to add boundary value check in lazy_tid_reaped() > before executing bsearch(3). This will help when index vacuum happens > multiple times or when garbage tuples are concentrated to a narrow > range. Makes sense if it's of

Re: Boundary value check in lazy_tid_reaped()

2020-08-31 Thread Thomas Munro
On Tue, Sep 1, 2020 at 7:21 AM Thomas Munro wrote: > On Sun, Aug 30, 2020 at 11:08 PM Masahiko Sawada > wrote: > > So my proposal is to add boundary value check in lazy_tid_reaped() > > before executing bsearch(3). This will help when index vacuum happens > > multip

Re: WIP: WAL prefetch (another approach)

2020-09-01 Thread Thomas Munro
On Wed, Sep 2, 2020 at 1:14 AM Tomas Vondra wrote: > from the archive Ahh, so perhaps that's the key. > I've tested this applied on 6ca547cf75ef6e922476c51a3fb5e253eef5f1b6, > and the failure seems fairly similar to what I reported before, except > that now it happened right at the very beginnin

A micro-optimisation for walkdir()

2020-09-01 Thread Thomas Munro
Hello hackers, You don't need to call stat() just to find out if a dirent is a file or directory, most of the time. Please see attached. From cc2f0fd4a078728a67d862e2deec0332fb8b3555 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Wed, 2 Sep 2020 16:15:09 +1200 Subject: [PATCH]

Re: Two fsync related performance issues?

2020-09-02 Thread Thomas Munro
On Wed, May 27, 2020 at 12:31 AM Craig Ringer wrote: > On Tue, 12 May 2020, 08:42 Paul Guo, wrote: >> 1. StartupXLOG() does fsync on the whole data directory early in the crash >> recovery. I'm wondering if we could skip some directories (at least the >> pg_log/, table directories) since wal, e

Re: Two fsync related performance issues?

2020-09-02 Thread Thomas Munro
On Tue, May 12, 2020 at 12:43 PM Paul Guo wrote: > 2. CheckPointTwoPhase() > > This may be a small issue. > > See the code below, > > for (i = 0; i < TwoPhaseState->numPrepXacts; i++) > RecreateTwoPhaseFile(gxact->xid, buf, len); > > RecreateTwoPhaseFile() writes a state file for a prepared t

Re: A micro-optimisation for walkdir()

2020-09-02 Thread Thomas Munro
On Thu, Sep 3, 2020 at 3:52 AM Juan José Santamaría Flecha wrote: > On Wed, Sep 2, 2020 at 4:35 PM Tom Lane wrote: >> Thomas Munro writes: >> > You don't need to call stat() just to find out if a dirent is a file >> > or directory, most of the time. Please s

Re: A micro-optimisation for walkdir()

2020-09-02 Thread Thomas Munro
On Thu, Sep 3, 2020 at 5:36 PM Tom Lane wrote: > [request for better comments] Ack. > Both of these concerns would abate if we had get_dirent_type() > just throw an error itself when stat() fails, thereby removing the > PGFILETYPE_ERROR result code. I'm not 100% sold either way on > that, but i

Re: A micro-optimisation for walkdir()

2020-09-03 Thread Thomas Munro
On Fri, Sep 4, 2020 at 3:31 AM Tom Lane wrote: > Thomas Munro writes: > > Hmm. Well I had it like that in an earlier version, but then I > > couldn't figure out the right way to write code that would work in > > both frontend and backend code, without writing two cop

Re: Rare deadlock failure in create_am test

2020-09-04 Thread Thomas Munro
On Fri, Sep 4, 2020 at 2:13 PM Tom Lane wrote: > (There might be more such failures, but I only looked back six months, > and these two are all I found in that window.) Right: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=conchuela&dt=2020-09-03%2023:00:36 | HEAD | Dragonfl

Re: [PATCH] Comments related to " buffer descriptors“ cache line size"

2020-09-04 Thread Thomas Munro
On Sat, Sep 5, 2020 at 5:28 AM Bruce Momjian wrote: > On Fri, Sep 4, 2020 at 11:31:55PM +0800, Kelly Min wrote: > > I found a comment error in buffer descriptors comment. The cache line size > > is > > 64 bytes, not 64 bits > > Thanks, fix applied to all active branches. I find it a bit strange

Re: A micro-optimisation for walkdir()

2020-09-04 Thread Thomas Munro
On Sat, Sep 5, 2020 at 9:45 AM Andres Freund wrote: > On 2020-09-02 17:51:27 +0200, Juan José Santamaría Flecha wrote: > > + attrib = GetFileAttributes(d->ret.d_name); > > Is this really an optimization? The benefit of Thomas' patch is that > that information somet

Re: WIP: WAL prefetch (another approach)

2020-09-04 Thread Thomas Munro
On Wed, Sep 2, 2020 at 2:18 AM Tomas Vondra wrote: > On Wed, Sep 02, 2020 at 02:05:10AM +1200, Thomas Munro wrote: > >On Wed, Sep 2, 2020 at 1:14 AM Tomas Vondra > > wrote: > >> from the archive > > > >Ahh, so perhaps that's the key. > > May

Re: Implementing Incremental View Maintenance

2020-09-04 Thread Thomas Munro
Hi Nagata-san, On Mon, Aug 31, 2020 at 5:32 PM Yugo NAGATA wrote: > https://wiki.postgresql.org/wiki/Incremental_View_Maintenance Thanks for writing this! + /* +* Wait for concurrent transactions which update this materialized view at +* READ COMMITED. This is needed to see changes co

Re: A micro-optimisation for walkdir()

2020-09-06 Thread Thomas Munro
On Sun, Sep 6, 2020 at 5:23 AM Juan José Santamaría Flecha wrote: > On Sat, Sep 5, 2020 at 2:13 AM Andres Freund wrote: >> > However, it looks like we might be missing a further opportunity >> > here... Doesn't Windows already give us the flags we need in the >> > dwFileAttributes member of the

Re: A micro-optimisation for walkdir()

2020-09-06 Thread Thomas Munro
On Mon, Sep 7, 2020 at 10:36 AM Tom Lane wrote: > Thomas Munro writes: > > Excellent. I'd like to commit these soon, unless someone has a better > > idea for how to name file_utils_febe.c. > > As long as it's in src/port or src/common, isn't it implicit th

Re: A micro-optimisation for walkdir()

2020-09-07 Thread Thomas Munro
On Mon, Sep 7, 2020 at 9:42 PM Magnus Hagander wrote: > On Mon, Sep 7, 2020 at 12:27 AM Thomas Munro wrote: >> I think the following is a little mysterious, but it does seem to be >> what people do for this in other projects. It is the documented way >> to detect moun

Re: A micro-optimisation for walkdir()

2020-09-07 Thread Thomas Munro
On Tue, Sep 8, 2020 at 2:05 AM Juan José Santamaría Flecha wrote: > On Mon, Sep 7, 2020 at 1:41 PM Thomas Munro wrote: >> Thanks for confirming. I ran the Windows patch through pgindent, >> fixed a small typo, and pushed. > > Great, thanks. Should we include something fr

Re: Optimising compactify_tuples()

2020-09-07 Thread Thomas Munro
On Mon, Sep 7, 2020 at 7:48 PM David Rowley wrote: > I was wondering today if we could just get rid of the sort in > compactify_tuples() completely. It seems to me that the existing sort > is there just so that the memmove() is done in order of tuple at the > end of the page first. We seem to be j

Re: Division in dynahash.c due to HASH_FFACTOR

2020-09-07 Thread Thomas Munro
On Sat, Sep 5, 2020 at 2:34 AM Alvaro Herrera wrote: > On 2020-Sep-04, Jakub Wartak wrote: > > After removing HASH_FFACTOR PostgreSQL still compiles... Would > > removing it break some external API/extensions ? > > FWIW, HASH_FFACTOR has *never* been used in Postgres core code. > > https://postgr

Re: Improving connection scalability: GetSnapshotData()

2020-09-07 Thread Thomas Munro
On Tue, Sep 8, 2020 at 4:11 PM Andres Freund wrote: > At first I was very confused as to why none of the existing tests have > found this significant issue. But after thinking about it for a minute > that's because they all use psql, and largely separate psql invocations > for each query :(. Which

Re: Optimising compactify_tuples()

2020-09-08 Thread Thomas Munro
On Wed, Sep 9, 2020 at 3:47 AM David Rowley wrote: > On Tue, 8 Sep 2020 at 12:08, Thomas Munro wrote: > > One thought is that if we're going to copy everything out and back in > > again, we might want to consider doing it in a > > memory-prefetcher-friendly order.

Re: Implementing Incremental View Maintenance

2020-09-08 Thread Thomas Munro
On Wed, Sep 9, 2020 at 12:29 PM Yugo NAGATA wrote: > I also thought it might be resolved using tuple locks and EvalPlanQual > instead of table level lock, but there is still a unavoidable case. For > example, suppose that tuple dR is inserted into R in T1, and dS is inserted > into S in T2. Also,

Re: Two fsync related performance issues?

2020-09-08 Thread Thomas Munro
On Thu, Sep 3, 2020 at 11:30 AM Thomas Munro wrote: > On Wed, May 27, 2020 at 12:31 AM Craig Ringer wrote: > > On Tue, 12 May 2020, 08:42 Paul Guo, wrote: > >> 1. StartupXLOG() does fsync on the whole data directory early in the crash > >> recovery. I'm

Re: Optimising compactify_tuples()

2020-09-09 Thread Thomas Munro
On Thu, Sep 10, 2020 at 2:34 AM David Rowley wrote: > I think you were adequately caffeinated. You're right that this is > fairly simple to do, but it looks even more simple than looping twice > of the array. I think it's just a matter of looping over the > itemidbase backwards and putting the h

Re: Division in dynahash.c due to HASH_FFACTOR

2020-09-09 Thread Thomas Munro
man, I don't like the mash-up of int, long, uint32, Size dynahash uses in various places and that are brought into relief by that comparison... From e5d890bc1178861dc1a5b1f430289a5ee2b4a2fa Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Thu, 10 Sep 2020 12:27:25 +1200 Subject: [PATCH]

Re: Two fsync related performance issues?

2020-09-10 Thread Thomas Munro
On Thu, Sep 3, 2020 at 12:09 PM Thomas Munro wrote: > On Tue, May 12, 2020 at 12:43 PM Paul Guo wrote: > > RecreateTwoPhaseFile(gxact->xid, buf, len); > I hadn't previously focused on this second part of your email. I > think the fsync() call in RecreateTwoPhaseFile

Re: Optimising compactify_tuples()

2020-09-10 Thread Thomas Munro
On Fri, Sep 11, 2020 at 1:45 AM David Rowley wrote: > On Thu, 10 Sep 2020 at 10:40, Thomas Munro wrote: > > I wonder if we could also identify a range at the high end that is > > already correctly sorted and maximally compacted so it doesn't even > > need to be copied

Re: Optimising compactify_tuples()

2020-09-10 Thread Thomas Munro
On Fri, Sep 11, 2020 at 3:53 AM David Rowley wrote: > That gets my benchmark down to 60.8 seconds, so 2.2 seconds better than v4b. . o O ( I wonder if there are opportunities to squeeze some more out of this with __builtin_prefetch... ) > I've attached v6b and an updated chart showing the result

Re: ldap/t/001_auth.pl fails with openldap 2.5

2021-10-13 Thread Thomas Munro
not like ldap > by experience). +1 > The only reason I'm hesitating a bit is that f0e60ee4bc0, the commit adding > the ldap test suite, used an ldap:// uri for the server, but then 27cd521e6e7 > (adding the ldapsearch) didn't use that for the ldapsearch? Thomas? My mistake, I p

Re: [RFC] building postgres with meson

2021-10-13 Thread Thomas Munro
On Thu, Oct 14, 2021 at 4:51 AM John Naylor wrote: > /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/LDAP.framework/Headers/ldap.h:1:10: > error: #include nested too deeply > #include > ^ I vaguely recall that PostgreSQL should build OK against Apple's cop

Re: [RFC] building postgres with meson

2021-10-14 Thread Thomas Munro
On Fri, Oct 15, 2021 at 11:00 AM Tom Lane wrote: > Peter E. did some hacking towards another solution awhile ago, > but IIRC it involved changing the built binaries, and I think > we concluded that the benefits didn't justify that. Yeah, by now there are lots of useful blogs from various projects

Re: [RFC] building postgres with meson

2021-10-14 Thread Thomas Munro
On Fri, Oct 15, 2021 at 12:04 PM Tom Lane wrote: > [tgl@pro ~]$ cat checkenv.c > #include > #include > > int > main(int argc, char **argv) > { > char *pth = getenv("DYLD_LIBRARY_PATH"); > > if (pth) > printf("DYLD_LIBRARY_PATH = %s\n", pth); > else > printf("DYLD_LIBRARY_PATH is un

Re: [PATCH] Make ENOSPC not fatal in semaphore creation

2021-10-17 Thread Thomas Munro
On Mon, Oct 18, 2021 at 4:49 AM Mikhail wrote: > The logic works - the initial call to semget() in > InternalIpcSemaphoreCreate returns -1 and errno is set to ENOSPC - I > tested the patch on OpenBSD 7.0, it successfully recycles sem's after > previous "pkill -6 postgres". Verified it with 'ipcs -

Interrupts vs signals

2021-10-20 Thread Thomas Munro
red it was good enough to demonstrate the concept. From aa330cf1ecf8665b6fa73caf07c01da0fceeb770 Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Sat, 5 Jun 2021 17:13:11 +1200 Subject: [PATCH] WIP: Refactor procsignals and interrupts. Previously, "procsignals" were implemented with shared memory flags and

Re: Thinking about ANALYZE stats and autovacuum and large non-uniform tables

2021-10-21 Thread Thomas Munro
On Fri, Oct 22, 2021 at 10:13 AM Greg Stark wrote: > Obviously this could get complex quickly. Perhaps it should be > something users could declare. Some kind of "partitioned statistics" > where you declare a where clause and we generate statistics for the > table where that where clause is true.

Re: [PATCH] Make ENOSPC not fatal in semaphore creation

2021-10-22 Thread Thomas Munro
On Sat, Oct 23, 2021 at 8:43 AM Tom Lane wrote: > Mikhail writes: > > In your patch I've removed testing for 5.x versions, because official > > releases are supported only for one year, no need to worry about them. > > Official support or no, we have OpenBSD 5.9 in our buildfarm, so > ignoring th

Re: [PATCH] Make ENOSPC not fatal in semaphore creation

2021-10-24 Thread Thomas Munro
On Mon, Oct 18, 2021 at 10:07 AM Thomas Munro wrote: > Note: The best kind would be *unnamed* POSIX semas, where we get to > control their placement in existing memory; that's what we do on Linux > and FreeBSD. They weren't supported on OpenBSD last time we checked: > i

Re: Spelling change in LLVM 14 API

2021-10-25 Thread Thomas Munro
On Mon, Sep 27, 2021 at 10:54 AM Thomas Munro wrote: > And pushed. Probably won't be the last change and seawasp only tests > master, so no back-patch for now. According to my crystal ball, seawasp will shortly break again[1][2] and the attached patch will fix it. [1] https://gith

Re: Spelling change in LLVM 14 API

2021-10-25 Thread Thomas Munro
On Tue, Oct 26, 2021 at 1:52 PM Andres Freund wrote: > On 2021-10-26 13:39:53 +1300, Thomas Munro wrote: > > According to my crystal ball, seawasp will shortly break again[1][2] > > and the attached patch will fix it. > > That feels lot more convincing though. Based on pas

Re: [PATCH] Make ENOSPC not fatal in semaphore creation

2021-10-28 Thread Thomas Munro
On Sun, Oct 24, 2021 at 10:50 PM Thomas Munro wrote: > Sadly, although the attached proof-of-concept patch allows a > PREFERRED_SEMAPHORES=FUTEX build to pass tests on macOS (which also > lacks native unnamed semas), FreeBSD and Linux (which don't need this > but are interesting

Re: Make relfile tombstone files conditional on WAL level

2021-10-28 Thread Thomas Munro
On Tue, Oct 5, 2021 at 4:21 PM Thomas Munro wrote: > On Thu, Sep 30, 2021 at 11:32 PM Thomas Munro wrote: > > I managed to produce a case where live data is written to an unlinked > > file and lost In conclusion, there *is* something else that would break, so I'm withd

Re: Time to drop plpython2?

2021-10-31 Thread Thomas Munro
On Mon, Nov 1, 2021 at 8:05 AM Andres Freund wrote: > > Yeah, it's a bit hard to justify continuing support in HEAD. +1, it's dropping out of distros, it'd be unsupportable without unreasonable effort.

Re: Portability report: ninja

2021-11-01 Thread Thomas Munro
On Tue, Nov 2, 2021 at 8:25 AM Tom Lane wrote: > 6. While configure.py thinks it knows what to do on AIX, it fails > on AIX 7.1 and 7.2: > > Traceback (most recent call last): > File "./configure.py", line 544, in > if platform.is_aix() and not platform.is_os400_pase(): > File "./configur

Re: Portability report: ninja

2021-11-01 Thread Thomas Munro
On Tue, Nov 2, 2021 at 8:25 AM Tom Lane wrote: > 5. It built and passed self-test on Solaris 11, but failed self-test > on Solaris 10. I haven't had time to actually do anything with it yet, but I can report that meson and ninja are, as of quite recently, available in the package repository of Op

AArch64 has single-copy 64 bit atomicity

2021-11-02 Thread Thomas Munro
urn this on for __aarch64__. It goes back to the original ARMv8-A so should cover all 64 bit ARM systems. From 1721c7b15b132c2aa16b18cf0b3eea054247833d Mon Sep 17 00:00:00 2001 From: Thomas Munro Date: Wed, 3 Nov 2021 10:23:33 +1300 Subject: [PATCH] AArch64 has single-copy 64 bit atomicity. Th

Re: [PATCH] Native spinlock support on RISC-V

2021-11-02 Thread Thomas Munro
On Wed, Sep 1, 2021 at 9:22 PM Christoph Berg wrote: > Re: Tom Lane > > I did not like confusing the RISC-V case with the ARM case: duplicating > > the code block seems better, in case there's ever reason to add > > arch-specific stuff like SPIN_DELAY. So I split it off to its own > > code block

Re: [PATCH] Native spinlock support on RISC-V

2021-11-02 Thread Thomas Munro
On Wed, Nov 3, 2021 at 5:13 PM Andres Freund wrote: > Any chance you could enable jit_dump_bitcode and manually try a failing > query? That should dump. bc files in the data directory. That'd might allow > debugging this outside the emulated environment. > > I don't see where the undef is origin

Re: [PATCH] Native spinlock support on RISC-V

2021-11-03 Thread Thomas Munro
On Wed, Nov 3, 2021 at 5:41 PM Thomas Munro wrote: > 'generic' is not a recognized processor for this target (ignoring processor) I still don't know what's wrong but I spent 20 minutes searching for more clues this morning... First, 'generic' is coming from L

Re: Missing include in be-secure-openssl.c?

2021-11-03 Thread Thomas Munro
On Thu, Nov 4, 2021 at 3:16 PM Michael Paquier wrote: > On Thu, Nov 04, 2021 at 10:56:13AM +0900, 近藤雄太 wrote: > > We'll check it again, but there is no difference between HEAD and v14 > > branch. > > We simply added v14 branch to build recently, and did not change any > > settings of HEAD at tha

Re: Missing include in be-secure-openssl.c?

2021-11-03 Thread Thomas Munro
On Thu, Nov 4, 2021 at 3:39 PM Thomas Munro wrote: > On Thu, Nov 4, 2021 at 3:16 PM Michael Paquier wrote: > > Could it be possible to copy-paste on this thread some of the > > buildfarm logs that show the compilation failure? No issues from me > > even if these are in

Re: Missing include in be-secure-openssl.c?

2021-11-03 Thread Thomas Munro
On Thu, Nov 4, 2021 at 4:33 PM Tom Lane wrote: > But I don't get the point about where HEAD is different from v14? > be-secure-openssl.c isn't. I don't understand what's going on and I don't have the headers to look at, but I was thinking that WIN32_LEAN_AND_MEAN must be causing a different state

Re: Weird failure in explain.out with OpenBSD

2021-11-10 Thread Thomas Munro
On Wed, Dec 30, 2020 at 8:17 PM Michael Paquier wrote: > On Tue, Dec 29, 2020 at 04:16:06PM -0500, Tom Lane wrote: > > Hmph, no, a look at explain.c shows that the "Execution Time" is just > > based on the difference of INSTR_TIME_SET_CURRENT measurements taken > > within the current process. It'

Re: Weird failure in explain.out with OpenBSD

2021-11-10 Thread Thomas Munro
On Thu, Nov 11, 2021 at 1:15 PM Michael Paquier wrote: > morepork uses 6.9 now, but it has been recently upgraded from 5.4 or > a version close to that, right? I am wondering if 7.0 may help > regarding this issue. Postgres is not wrong here. I dunno. Clocks on virtualised systems and even met

Re: Interrupts vs signals

2021-11-10 Thread Thomas Munro
On Thu, Oct 21, 2021 at 8:27 AM Andres Freund wrote: > On 2021-10-21 07:55:54 +1300, Thomas Munro wrote: > > I wonder if we really need signals to implement interrupts. Given > > that they are non-preemptive/cooperative (work happens at the next > > CFI()), why not just us

Re: Interrupts vs signals

2021-11-11 Thread Thomas Munro
On Fri, Nov 12, 2021 at 9:24 AM Robert Haas wrote: > On Thu, Nov 11, 2021 at 2:50 PM Andres Freund wrote: > > They can count, but only when used for network sockets or pipes ("slow > > devices" or whatever the posix language is). Disk IO doesn't count as that. > > So > > I don't think it'd be a

Re: Should AT TIME ZONE be volatile?

2021-11-11 Thread Thomas Munro
On Fri, Nov 12, 2021 at 12:09 PM Robert Haas wrote: > On Thu, Nov 11, 2021 at 5:04 PM Isaac Morland wrote: > > Wouldn't an existing index only have characters that were already part of > > the collation? Attempting to use one not covered by the collation I would > > have expected to cause an er

Re: Should AT TIME ZONE be volatile?

2021-11-12 Thread Thomas Munro
On Sat, Nov 13, 2021 at 11:47 AM Ilya Anfimov wrote: > Currently for glibc the version looks like glibc version at > initdb, and that doesn't seem very reliable, but that could be a > different task (to find LC_COLLATE file and put hash of the > usuable data into version string, for e

Re: Invalid Unicode escape value at or near "\u0000"

2021-11-12 Thread Thomas Munro
On Sat, Nov 13, 2021 at 4:32 PM Japin Li wrote: > When I try to insert an Unicode "\u", there is an error $subject. > > postgres=# CREATE TABLE tbl (s varchar(10)); > CREATE TABLE > postgres=# INSERT INTO tbl VALUES (E'\u'); > ERROR: invalid Unicode escape value at or near "\u" > LINE

Re: [RFC] building postgres with meson

2021-11-15 Thread Thomas Munro
On Tue, Nov 16, 2021 at 8:23 AM Andres Freund wrote: > On 2021-11-15 14:11:25 -0500, Tom Lane wrote: > > Having said that, I don't plan to be the one trying to get meson > > to add xlc support ... > > It'd probably not be too hard. But given that it's quite hard to get access to > AIX + xlc, I'm n

Re: [RFC] building postgres with meson

2021-11-15 Thread Thomas Munro
On Tue, Nov 16, 2021 at 11:08 AM Thomas Munro wrote: > FWIW there's a free-as-in-beer edition of xlc for Linux (various > distros, POWER only) so you could use qemu, (It's also known to be possible to run AIX 7.2 on qemu, but the install media is not made available to developer

Re: Windows: Wrong error message at connection termination

2021-11-17 Thread Thomas Munro
On Thu, Nov 18, 2021 at 10:13 AM Lars Kanis wrote: > Unfortunately each connection is closed hard by a Windows PostgreSQL server > with TCP flag RST. That in turn is another Winsock API behavior, that is that > every socket, that wasn't closed by the application is closed hard with the > RST fl

Re: Should rename "startup process" to something else?

2021-11-18 Thread Thomas Munro
On Tue, Nov 16, 2021 at 3:33 AM Rushabh Lathia wrote: > I think a better name for the process may be “recovery” +1

Re: [RFC] building postgres with meson

2021-11-18 Thread Thomas Munro
On Tue, Nov 16, 2021 at 8:23 AM Andres Freund wrote: > On 2021-11-15 14:11:25 -0500, Tom Lane wrote: > > (In fact, unless somebody renews fossa/husky's > > icc license, the three xlc animals will be an outright majority of > > them, because wrasse and anole are the only other active animals with >

Re: [PATCH] Make ENOSPC not fatal in semaphore creation

2021-11-19 Thread Thomas Munro
On Fri, Oct 29, 2021 at 4:54 PM Thomas Munro wrote: > On Sun, Oct 24, 2021 at 10:50 PM Thomas Munro wrote: > > Sadly, although the attached proof-of-concept patch allows a > > PREFERRED_SEMAPHORES=FUTEX build to pass tests on macOS (which also > > lacks native unnamed sema

Re: [PATCH] Make ENOSPC not fatal in semaphore creation

2021-11-19 Thread Thomas Munro
On Sat, Nov 20, 2021 at 9:34 AM Tom Lane wrote: > Thomas Munro writes: > > This has been fixed. So now there are working basic futexes on Linux, > > macOS, {Free,Open,Net,Dragonfly}BSD (though capabilities beyond basic > > wait/wake vary, as do APIs). So the question i

Re: Adding CI to our tree

2021-11-19 Thread Thomas Munro
On Sat, Nov 20, 2021 at 11:17 AM Melanie Plageman wrote: > - do you not need to change the default core resource limits for > FreeBSD? Unfortunately the performance really sucks on that FreeBSD CI system if you crank it up, and I haven't had time to figure out why yet :-/ Possibly something to

Re: Windows: Wrong error message at connection termination

2021-11-21 Thread Thomas Munro
On Mon, Nov 22, 2021 at 8:19 AM Lars Kanis wrote: > The other way around would be to make sure on the client side, that the > last message is retrieved before the RST packet arrives, so that no data > is lost. This works mostly well through the sync API of libpq, but with > the async API the trigg

Re: Windows: Wrong error message at connection termination

2021-11-21 Thread Thomas Munro
On Mon, Nov 22, 2021 at 9:24 AM Thomas Munro wrote: > Hmm. Well, if I understand how this works (and I'm not too familiar > with this Windows code so I maybe I don't), the postmaster duplicates > the socket into the child process (see > {write,read}_inheritable_socket()) a

Re: Windows: Wrong error message at connection termination

2021-11-21 Thread Thomas Munro
On Mon, Nov 22, 2021 at 10:42 AM Tom Lane wrote: > Thomas Munro writes: > > Hmm, maybe it's still not enough. Now that I have coffee, I thought > > about the well known failure of idle_in_transaction_timeout to report > > errors on Windows[1]. > > Yeah, I think

Re: A test for replay of regression tests

2021-11-23 Thread Thomas Munro
On Wed, Oct 6, 2021 at 7:10 PM Thomas Munro wrote: > I wonder if for Windows we'd want to switch to real symlinks, since, > as far as I know from some light reading, reparse points are converted > to absolute paths on creation, so the pgdata directory would be less > portable th

Re: Worth using personality(ADDR_NO_RANDOMIZE) for EXEC_BACKEND on linux?

2021-11-23 Thread Thomas Munro
ucing a failure it's clear to see using pmap that this has the right effect. I didn't bother with a check for the existence of ADDR_NO_RANDOMIZE because it's since 2.6.12 which is definitely ancient enough. From e615dc88c89142a1b7ecf7ba6f6f01ea59c61be5 Mon Sep 17 00:00:00 2001 From: T

Warning in geqo_main.c from clang 13

2021-11-23 Thread Thomas Munro
Hi, Clang 13 on my machine and peripatus (but not Apple clang 13 on eg sifika, I'm still confused about Apple's versioning but I think that's really llvm 12-based) warns: geqo_main.c:86:8: warning: variable 'edge_failures' set but not used [-Wunused-but-set-variable] int

Re: Parallel Full Hash Join

2021-11-26 Thread Thomas Munro
On Sun, Nov 21, 2021 at 4:48 PM Justin Pryzby wrote: > On Wed, Nov 17, 2021 at 01:45:06PM -0500, Melanie Plageman wrote: > > Yes, this looks like that issue. > > > > I've attached a v8 set with the fix I suggested in [1] included. > > (I added it to 0001). > > This is still crashing :( > https://c

Re: WIP: WAL prefetch (another approach)

2021-11-26 Thread Thomas Munro
On Fri, Nov 26, 2021 at 11:32 AM Tomas Vondra wrote: > The results are pretty good / similar to previous results. Replaying the > 1h worth of work on a smaller machine takes ~5:30h without prefetching > (master or with prefetching disabled). With prefetching enabled this > drops to ~2h (default co

Re: WIP: WAL prefetch (another approach)

2021-11-26 Thread Thomas Munro
On Sat, Nov 27, 2021 at 12:34 PM Tomas Vondra wrote: > One thing that's not clear to me is what happened to the reasons why > this feature was reverted in the PG14 cycle? Reasons for reverting: 1. A bug in commit 323cbe7c, "Remove read_page callback from XLogReader.". I couldn't easily revert

Re: JIT compiling with LLVM v12.2

2018-03-24 Thread Thomas Munro
Hi Andres, I spotted a couple of typos and some very minor coding details -- see please see attached. -- Thomas Munro http://www.enterprisedb.com 0001-Correct-some-minor-typos-in-the-new-JIT-code.patch Description: Binary data 0002-Minor-code-cleanup-for-llvmjit_wrap.cpp.patch Description

<    1   2   3   4   5   6   7   8   9   10   >