Re: Fast COPY FROM based on batch insert

2022-03-23 Thread Andrey V. Lepikhov
On 3/22/22 06:54, Etsuro Fujita wrote: On Fri, Jun 4, 2021 at 5:26 PM Andrey Lepikhov wrote: We still have slow 'COPY FROM' operation for foreign tables in current master. Now we have a foreign batch insert operation And I tried to rewrite the patch [1] with this machinery. The patch has been

Re: ubsan

2022-03-23 Thread Noah Misch
On Wed, Mar 23, 2022 at 03:58:09PM -0400, Tom Lane wrote: > Andres Freund writes: > > I think we should backpatch both, based on the reasoning in > > 46ab07ffda9d6c8e63360ded2d4568aa160a7700 ? > > Yeah, I suppose. Is anyone going to step up and run a buildfarm > member with ubsan enabled? thorn

Re: PG DOCS - logical replication filtering

2022-03-23 Thread Peter Smith
On Fri, Mar 11, 2022 at 9:37 AM Euler Taveira wrote: > > On Fri, Mar 4, 2022, at 12:41 AM, Peter Smith wrote: > > PSA patch v3 to address all comments received so far. > > Peter, > > I started reviewing this documentation for row filter and I noticed that I was > changing too much lines to submit

Re: shared-memory based stats collector - v67

2022-03-23 Thread Kyotaro Horiguchi
(nice work about ubsan) At Wed, 23 Mar 2022 10:42:03 -0700, Andres Freund wrote in > Hi, > > On 2022-03-23 17:27:50 +0900, Kyotaro Horiguchi wrote: > > At Mon, 21 Mar 2022 14:30:17 -0700, Andres Freund > > wrote in > > Before there, InitWalRecovery changes the state to > > DB_IN_ARCHIVE_RECO

Re: pg_stat_reset_single_*_counters vs pg_stat_database.stats_reset

2022-03-23 Thread Andres Freund
Hi, On 2022-03-24 06:27:48 +0100, Laurenz Albe wrote: > On Wed, 2022-03-23 at 17:55 -0700, Andres Freund wrote: > > Starting with the below commit, pg_stat_reset_single_function_counters, > > pg_stat_reset_single_table_counters don't just reset the stats for the > > individual function, but also s

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-23 Thread Dilip Kumar
On Wed, Mar 23, 2022 at 10:50 PM Dilip Kumar wrote: > > On Wed, Mar 23, 2022 at 10:37 PM Andres Freund wrote: > > > > Hi, > > > > On 2022-03-23 22:29:40 +0530, Dilip Kumar wrote: > > > I could not see any reason for it to fail, and I could not reproduce > > > it either. Is it possible to access

Re: pg_stat_reset_single_*_counters vs pg_stat_database.stats_reset

2022-03-23 Thread Laurenz Albe
On Wed, 2022-03-23 at 17:55 -0700, Andres Freund wrote: > Starting with the below commit, pg_stat_reset_single_function_counters, > pg_stat_reset_single_table_counters don't just reset the stats for the > individual function, but also set pg_stat_database.stats_reset. I see the point in the fine-g

Re: Removing unneeded self joins

2022-03-23 Thread Andrey V. Lepikhov
On 3/22/22 05:58, Andres Freund wrote: Hi, On 2022-03-04 15:47:47 +0500, Andrey Lepikhov wrote: Also, in new version of the patch I fixed one stupid bug: checking a self-join candidate expression operator - we can remove only expressions like F(arg1) = G(arg2). This CF entry currently fails t

Re: Remove an unnecessary errmsg_plural in dependency.c

2022-03-23 Thread Kyotaro Horiguchi
At Wed, 23 Mar 2022 17:39:52 +0100, Peter Eisentraut wrote in > On 23.03.22 17:33, Bharath Rupireddy wrote: > > It looks like the following errmsg_plural in dependency.c is > > unnecessary as numReportedClient > 1 always and numNotReportedClient > > can never be < 0. Therefore plural version of

Re: [PoC] Let libpq reject unexpected authentication requests

2022-03-23 Thread Laurenz Albe
On Wed, 2022-03-23 at 21:31 +, Jacob Champion wrote: > On Mon, 2022-03-07 at 11:44 +0100, Laurenz Albe wrote: > > I am all for the idea, but you implemented the reverse of proposal 2. > > > > Wouldn't it be better to list the *rejected* authentication methods? > > Then we could have "password"

Re: pg_walinspect - a new extension to get raw WAL data and WAL stats

2022-03-23 Thread Kyotaro Horiguchi
At Wed, 23 Mar 2022 21:36:09 +0530, Bharath Rupireddy wrote in > Here's a refactoring patch that basically moves the pg_waldump's > functions and stats structures to xlogreader.h/.c so that the > pg_walinspect can reuse them. If it looks okay, I will send the > pg_walinspect patches based on it.

Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors

2022-03-23 Thread Tatsuo Ishii
>> My hoary animal prairiedog doesn't like this [1]: >> >> # Failed test 'concurrent update with retrying stderr /(?s-xim:client >> (0|1) got an error in command 3 \\(SQL\\) of script 0; ERROR: could not >> serialize access due to concurrent update\\b.*\\g1)/' >> # at t/001_pgbench_with_ser

Re: Extensible Rmgr for Table AMs

2022-03-23 Thread Jeff Davis
On Fri, 2022-02-04 at 22:56 +0800, Julien Rouhaud wrote: > > Right, which I guess raises another question: if the maximum is > > UINT8_MAX, which BTW I find perfectly reasonable, why are we not > > just > > defining this as an array of size 256? There's no point in adding > > code > > complexity to

Re: postgres_fdw: commit remote (sub)transactions in parallel during pre-commit

2022-03-23 Thread Etsuro Fujita
On Sat, Mar 5, 2022 at 7:32 PM Etsuro Fujita wrote: > Attached is an updated version. In the 0002 patch I introduced a macro for building an abort command in preparation for the parallel abort patch (0003), but I moved it to 0003. Attached is a new patch set. The new version of 0002 is just a c

Re: pg_tablespace_location() failure with allow_in_place_tablespaces

2022-03-23 Thread Thomas Munro
On Tue, Mar 8, 2022 at 12:58 AM Michael Paquier wrote: > Junction points are directories, no? Are you sure that this works > correctly on WIN32? It seems to me that we'd better use readlink() > only for entries in pg_tlbspc/ that are PGFILETYPE_LNK on non-WIN32 > and pgwin32_is_junction() on WIN

RE: logical replication empty transactions

2022-03-23 Thread shiy.f...@fujitsu.com
On Thursday, March 24, 2022 11:19 AM Hou, Zhijie/侯 志杰 wrote: > > Attach the new version patch which include the following changes: > > - Fix a typo > - Change the requestreply flag of the newly added WalSndKeepalive to false, > because the subscriber can judge whether it's necessary to post a

Checking pgwin32_is_junction() errors

2022-03-23 Thread Thomas Munro
Hi, The comment for pgwin32_is_junction() says "Assumes the file exists, so will return false if it doesn't (since a nonexistent file is not a junction)". In fact that's the behaviour for any kind of error, and although we set errno in that case, no caller ever checks it. I think it'd be better

Re: Failed transaction statistics to measure the logical replication progress

2022-03-23 Thread Amit Kapila
On Thu, Mar 3, 2022 at 8:58 AM osumi.takami...@fujitsu.com wrote: > This patch introduces two new subscription statistics columns (apply_commit_count and apply_rollback_count) to the pg_stat_subscription_stats view for counting cumulative transactions commits/rollbacks for a particular subscripti

Re: add checkpoint stats of snapshot and mapping files of pg_logical dir

2022-03-23 Thread Bharath Rupireddy
On Wed, Mar 23, 2022 at 10:16 AM Bharath Rupireddy wrote: > > On Tue, Mar 22, 2022 at 8:12 PM Andres Freund wrote: > > > Do you mean like this? > > > ereport(LOG, > > > /* translator: the placeholders show checkpoint options */ > > > (errmsg("%s starting:%s%s%s%s%s%s%s%s", > >

Re: pg_stat_reset_single_*_counters vs pg_stat_database.stats_reset

2022-03-23 Thread Andres Freund
Hi, On 2022-03-23 18:47:58 -0700, David G. Johnston wrote: > It also seems that each tracked object type needs to have its own > last_reset field (we could choose to name it stats_reset too, leaving > pg_stat_database.last_reset as the only anomaly) added as an implied > behavior needed for such i

RE: logical replication empty transactions

2022-03-23 Thread houzj.f...@fujitsu.com
On Tuesday, March 22, 2022 7:50 PM Amit Kapila wrote: > On Tue, Mar 22, 2022 at 7:25 AM houzj.f...@fujitsu.com > wrote: > > > > > On Monday, March 21, 2022 6:01 PM Amit Kapila > > > > > > wrote: > > > > Oh, sorry, I posted the wrong patch, here is the correct one. > > > > The test change looks

Re: Column Filtering in Logical Replication

2022-03-23 Thread Amit Kapila
On Thu, Mar 24, 2022 at 4:11 AM Tomas Vondra wrote: > > On 3/21/22 12:28, Amit Kapila wrote: > > On Fri, Mar 18, 2022 at 8:13 PM Tomas Vondra > > wrote: > >> > >> Ah, thanks for reminding me - it's hard to keep track of all the issues > >> in threads as long as this one. > >> > >> BTW do you have

Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors

2022-03-23 Thread Yugo NAGATA
On Wed, 23 Mar 2022 14:26:54 -0400 Tom Lane wrote: > Tatsuo Ishii writes: > > The patch Pushed. Thank you! > > My hoary animal prairiedog doesn't like this [1]: > > # Failed test 'concurrent update with retrying stderr /(?s-xim:client (0|1) > got an error in command 3 \\(SQL\\) of script 0;

Re: Support isEmptyStringInfo

2022-03-23 Thread Kyotaro Horiguchi
At Wed, 23 Mar 2022 10:13:43 -0400, Tom Lane wrote in > Robert Haas writes: > > I think that the code is perfectly readable as it is and that this > > change makes it less so. > > Yeah, after a quick look through this patch I'm unimpressed too. > The new code is strictly longer, and it requires

Re: freeing bms explicitly

2022-03-23 Thread Amit Kapila
On Wed, Mar 23, 2022 at 9:30 AM Zhihong Yu wrote: > > On Tue, Mar 22, 2022 at 8:45 PM Amit Kapila wrote: >> >> On Tue, Mar 22, 2022 at 3:39 AM Zhihong Yu wrote: >> >> Your patch looks good to me. I have found one more similar instance in >> the same file and changed that as well accordingly. Let

Re: [PoC] Reducing planning time when tables have many partitions

2022-03-23 Thread David Rowley
On Fri, 18 Mar 2022 at 23:32, Yuya Watari wrote: > I found a problem that planning takes too much time when the tables > have many child partitions. According to my observation, the planning > time increases in the order of O(n^2). Here, n is the number of child > partitions. I attached the patch

Re: pg_stat_reset_single_*_counters vs pg_stat_database.stats_reset

2022-03-23 Thread David G. Johnston
On Wed, Mar 23, 2022 at 5:55 PM Andres Freund wrote: > > Starting with the below commit, pg_stat_reset_single_function_counters, > pg_stat_reset_single_table_counters don't just reset the stats for the > individual function, but also set pg_stat_database.stats_reset. > > commit 4c468b37a281941afd

Re: Fix overflow in DecodeInterval

2022-03-23 Thread Tom Lane
Joseph Koshakow writes: > Sorry about that, I didn't have my IDE set up quite right and > noticed a little too late that I had some auto-formatting turned > on. Thanks for doing the rebase, did it end up fixing > the whitespace issues? If not I'll go through the patch and try > and fix them all.

Re: [PATCH] Add native windows on arm64 support

2022-03-23 Thread Andres Freund
Hi, On 2022-03-23 16:30:58 +1300, Thomas Munro wrote: > On Tue, Mar 22, 2022 at 11:30 PM Julien Rouhaud wrote: > > On Tue, Mar 22, 2022 at 09:37:46AM +, Niyas Sait wrote: > > > Yes, we could look into providing a build machine. Do you have any > > > reference to what the CI system looks like

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-03-23 Thread Andres Freund
On 2022-03-24 11:54:15 +1300, Thomas Munro wrote: > Erm, is that really OK? C says "Each enumerated type shall be > compatible with char, a signed integer type, or an > unsigned integer type. The choice of type is implementation-defined, > but shall be capable of representing the values of all the

Re: pg_stat_reset_single_*_counters vs pg_stat_database.stats_reset

2022-03-23 Thread Andres Freund
Hi, On 2022-03-23 17:55:16 -0700, Andres Freund wrote: > Maybe I just don't understand what these reset functions are intended for? > Their introduction [3] didn't explain much either. To me the behaviour of > resetting pg_stat_database.stats_reset but nothing else in pg_stat_database > makes them

pg_stat_reset_single_*_counters vs pg_stat_database.stats_reset

2022-03-23 Thread Andres Freund
Hi, Starting with the below commit, pg_stat_reset_single_function_counters, pg_stat_reset_single_table_counters don't just reset the stats for the individual function, but also set pg_stat_database.stats_reset. commit 4c468b37a281941afd3bf61c782b20def8c17047 Author: Magnus Hagander Date: 2011-

Re: Add pg_freespacemap extension sql test

2022-03-23 Thread Michael Paquier
On Wed, Mar 23, 2022 at 10:45:19AM -0300, Fabrízio de Royes Mello wrote: > On Wed, Mar 23, 2022 at 3:05 AM Michael Paquier wrote: >> Another thing that itched me is that we >> could also test more with indexes, particularly with btree, BRIN and >> hash (the latter should not have a FSM with 10 pa

Re: Fix overflow in DecodeInterval

2022-03-23 Thread Joseph Koshakow
On Mon, Mar 21, 2022 at 8:31 PM Tom Lane wrote: > This isn't applying per the cfbot; looks like it got sideswiped > by 9e9858389. Here's a quick rebase. I've not reviewed it, but > I did notice (because git was in my face about this) that it's > got whitespace issues. Please try to avoid unnece

Re: Avoid erroring out when unable to remove or parse logical rewrite files to save checkpoint work

2022-03-23 Thread Thomas Munro
On Wed, Feb 16, 2022 at 12:11 PM Nathan Bossart wrote: > On Tue, Feb 15, 2022 at 10:37:58AM -0800, Nathan Bossart wrote: > > On Tue, Feb 15, 2022 at 10:10:34AM -0800, Andres Freund wrote: > >> I generally think it'd be a good exercise to go through an use > >> get_dirent_type() in nearly all ReadD

Re: [PATCH] Expose port->authn_id to extensions and triggers

2022-03-23 Thread Andres Freund
Hi, On 2022-03-23 23:06:14 +, Jacob Champion wrote: > On Wed, 2022-03-23 at 19:00 -0400, Tom Lane wrote: > > Hm. I was more envisioning getting the "sharable" info out of Port > > entirely, although I'm not quite sure where it should go instead. > > If it helps, I can move the substruct out

Re: [PATCH] Accept IP addresses in server certificate SANs

2022-03-23 Thread Jacob Champion
On Wed, 2022-03-23 at 14:20 +0900, Kyotaro Horiguchi wrote: > I tried to write out the doc part. What do you think about it? I like it, thanks! I've applied that in v10, with a tweak to two iPAddress spellings and a short expansion of the condition in the Note, and I've added you as a co-author t

Re: multithreaded zstd backup compression for client and server

2022-03-23 Thread Andres Freund
On 2022-03-23 18:07:01 -0500, Justin Pryzby wrote: > Did you try this on windows at all ? Really should get zstd installed in the windows cf environment... > It's probably no surprise that zstd implements threading differently there. Worth noting that we have a few of our own threads running on

Re: multithreaded zstd backup compression for client and server

2022-03-23 Thread Andres Freund
Hi, On 2022-03-23 18:31:12 -0400, Robert Haas wrote: > On Wed, Mar 23, 2022 at 5:14 PM Andres Freund wrote: > > The most likely source of problem would errors thrown while zstd threads are > > alive. Should make sure that that can't happen. > > > > What is the lifetime of the threads zstd spawns?

Re: multithreaded zstd backup compression for client and server

2022-03-23 Thread Justin Pryzby
On Wed, Mar 23, 2022 at 04:34:04PM -0400, Robert Haas wrote: > be, spawning threads inside the PostgreSQL backend. Short of cats and > dogs living together, it's hard to think of anything more terrifying, > because the PostgreSQL backend is very much not thread-safe. However, > a lot of the things

Re: [PATCH] Expose port->authn_id to extensions and triggers

2022-03-23 Thread Jacob Champion
On Wed, 2022-03-23 at 19:00 -0400, Tom Lane wrote: > Hm. I was more envisioning getting the "sharable" info out of Port > entirely, although I'm not quite sure where it should go instead. If it helps, I can move the substruct out and up to a new global struct (MyProcShared?). At this point I thin

Re: ubsan

2022-03-23 Thread Tom Lane
Andres Freund writes: > Running with asan found an existing use-after-free bug in pg_waldump (*), a > bug in > dshash_seq_next() next that probably can't be hit in HEAD and a bug in my > shared memory stats patch. I count that as a success. Nice! regards, tom lane

Re: [PATCH] Expose port->authn_id to extensions and triggers

2022-03-23 Thread Tom Lane
Jacob Champion writes: > On Thu, 2022-03-17 at 18:33 -0400, Tom Lane wrote: >> I think what we ought to do here is separate out the data that we think >> parallel workers need access to. It does not seem wise to say "workers >> can access fields A,B,C of MyPort but not fields X,Y,Z". I do not ha

Re: multithreaded zstd backup compression for client and server

2022-03-23 Thread Robert Haas
On Wed, Mar 23, 2022 at 5:52 PM Justin Pryzby wrote: > Also because the library may not be compiled with threading. A few days ago, > I > tried to rebase the original "parallel workers" patch over the COMPRESS DETAIL > patch but then couldn't test it, even after trying various versions of the >

Re: ubsan

2022-03-23 Thread Andres Freund
Hi, On 2022-03-23 13:12:34 -0700, Andres Freund wrote: > I'm planning to enable it on two of mine. Looks like gcc and clang find > slightly different things, so I was intending to enable it on one of each. Originally I'd planned to mix them into existing members, but I think it'd be better to hav

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-03-23 Thread Thomas Munro
On Thu, Mar 24, 2022 at 9:53 AM Peter Eisentraut wrote: > On 21.03.22 05:55, Thomas Munro wrote: > > [04:30:50.630] pg_waldump.c:963:26: error: format ‘%u’ expects > > argument of type ‘unsigned int *’, but argument 3 has type ‘ForkNumber > > *’ [-Werror=format=] > > [04:30:50.630] 963 | if (sscan

Re: [PATCH] Expose port->authn_id to extensions and triggers

2022-03-23 Thread Jacob Champion
On Thu, 2022-03-17 at 18:33 -0400, Tom Lane wrote: > Yeah. It seems to me that putting the auth info into struct Port was > a fairly random thing to do in the first place, and we are now dealing > with the fallout of that. > > I think what we ought to do here is separate out the data that we thin

Re: Column Filtering in Logical Replication

2022-03-23 Thread Tomas Vondra
On 3/21/22 12:28, Amit Kapila wrote: > On Fri, Mar 18, 2022 at 8:13 PM Tomas Vondra > wrote: >> >> Ah, thanks for reminding me - it's hard to keep track of all the issues >> in threads as long as this one. >> >> BTW do you have any opinion on the SET COLUMNS syntax? Peter Smith >> proposed to g

Re: Column Filtering in Logical Replication

2022-03-23 Thread Tomas Vondra
On 3/21/22 12:55, Amit Kapila wrote: > On Sat, Mar 19, 2022 at 3:56 AM Tomas Vondra > wrote: >> >> ... >> >> However, while looking at how pgoutput, I realized one thing - for row >> filters we track them "per operation", depending on which operations are >> defined for a given publication. Should

Re: Proposal: allow database-specific role memberships

2022-03-23 Thread Kenaniah Cerny
Hi all, cfbot is once again green as of the v7 patch: https://cirrus-ci.com/github/postgresql-cfbot/postgresql/commitfest/37/3374 - Kenaniah

Re: XID formatting and SLRU refactorings (was: Add 64-bit XIDs into PostgreSQL 15)

2022-03-23 Thread Peter Eisentraut
On 23.03.22 10:51, Maxim Orlov wrote: Thanks for updating the patchs. I have a little comment on 0002. +                                errmsg_internal("found xmax %llu" " (infomask 0x%04x) not frozen, not multi, not normal", +

Re: multithreaded zstd backup compression for client and server

2022-03-23 Thread Robert Haas
On Wed, Mar 23, 2022 at 5:14 PM Andres Freund wrote: > The most likely source of problem would errors thrown while zstd threads are > alive. Should make sure that that can't happen. > > What is the lifetime of the threads zstd spawns? Are they tied to a single > compression call? A single ZSTD_cre

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-03-23 Thread Peter Geoghegan
On Wed, Mar 23, 2022 at 2:03 PM Thomas Munro wrote: > Yeah, I found it confusing that DISABLE_PAGE_SKIPPING doesn't disable > all page skipping, so 3414099c turned out to be not enough. The proposed change to DISABLE_PAGE_SKIPPING is partly driven by that, and partly driven by a similar concern a

Re: MDAM techniques and Index Skip Scan patch

2022-03-23 Thread Dmitry Dolgov
> On Wed, Mar 23, 2022 at 05:32:46PM -0400, Tom Lane wrote: > Dmitry Dolgov <9erthali...@gmail.com> writes: > > On Tue, Mar 22, 2022 at 04:55:49PM -0400, Tom Lane wrote: > >> In short: I would throw out just about all the planner infrastructure > >> that's been proposed so far. It looks bulky, exp

Re: multithreaded zstd backup compression for client and server

2022-03-23 Thread Justin Pryzby
+* We check for failure here because (1) older versions of the library +* do not support ZSTD_c_nbWorkers and (2) the library might want to +* reject an unreasonable values (though in practice it does not seem to do +* so). +*/ + ret = ZSTD_CCtx_setPar

Re: MDAM techniques and Index Skip Scan patch

2022-03-23 Thread Tom Lane
Dmitry Dolgov <9erthali...@gmail.com> writes: > On Tue, Mar 22, 2022 at 04:55:49PM -0400, Tom Lane wrote: >> In short: I would throw out just about all the planner infrastructure >> that's been proposed so far. It looks bulky, expensive, and >> drastically undercommented, and I don't think it's bu

Re: [PoC] Let libpq reject unexpected authentication requests

2022-03-23 Thread Jacob Champion
On Mon, 2022-03-07 at 11:44 +0100, Laurenz Albe wrote: > I am all for the idea, but you implemented the reverse of proposal 2. (This email was caught in my spam filter; sorry for the delay.) > Wouldn't it be better to list the *rejected* authentication methods? > Then we could have "password" on

Re: Proposal: allow database-specific role memberships

2022-03-23 Thread Kenaniah Cerny
Thanks Andres, An updated patch is attached. - Kenaniah On Mon, Mar 21, 2022 at 5:40 PM Andres Freund wrote: > Hi, > > On 2022-01-22 22:56:44 +0800, Julien Rouhaud wrote: > > On Sat, Jan 22, 2022 at 05:28:05AM -0800, Kenaniah Cerny wrote: > > > Thank you so much for the backtrace! > > > > > >

Re: logical replication restrictions

2022-03-23 Thread Euler Taveira
On Mon, Mar 21, 2022, at 10:09 PM, Euler Taveira wrote: > On Mon, Mar 21, 2022, at 10:04 PM, Andres Freund wrote: >> On 2022-03-20 21:40:40 -0300, Euler Taveira wrote: >> > On Mon, Feb 28, 2022, at 9:18 PM, Euler Taveira wrote: >> > > Long time, no patch. Here it is. I will provide documentation in

Re: multithreaded zstd backup compression for client and server

2022-03-23 Thread Andres Freund
Hi, On 2022-03-23 16:34:04 -0400, Robert Haas wrote: > Therefore, I think it might be safe to just ... turn this on. One reason I > think that is that this whole approach was recommended to me by Andres ... I didn't do a super careful analysis of the issues... But I do think it's pretty much the

Re: [PATCH] Remove workarounds to format [u]int64's

2022-03-23 Thread Pavel Borisov
> > On 21.03.22 15:37, Aleksander Alekseev wrote: > >> It would not simplify things for them at all, just mess it up. > >> The master copies of the .po files are kept in a different repo. > >> Also, I believe that extraction of new message strings is automated > >> already. > > > > Got it, thanks.

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-03-23 Thread Thomas Munro
On Thu, Mar 24, 2022 at 9:59 AM Peter Geoghegan wrote: > On Wed, Mar 23, 2022 at 1:53 PM Robert Haas wrote: > > And therefore I favor defining it to mean that we don't > > skip any work at all. > > But even today DISABLE_PAGE_SKIPPING won't do pruning when we cannot > acquire a cleanup lock on a

Re: Adding CI to our tree

2022-03-23 Thread Justin Pryzby
On Thu, Mar 24, 2022 at 09:52:39AM +1300, Thomas Munro wrote: > On Thu, Mar 10, 2022 at 9:37 AM Justin Pryzby wrote: > > -Og > > Adding this to CXXFLAGS caused a torrent of warnings from g++ about > LLVM headers, which I also see on my local system for LLVM 11 and LLVM > 14: Yes, I mentioned see

Re: Schema variables - new implementation for Postgres 15

2022-03-23 Thread Pavel Stehule
Hi A bit more work seems to be needed for deparsing session variables: > > # create variable myvar text; > CREATE VARIABLE > > # create view myview as select myvar; > CREATE VIEW > > # \d+ myview > View "public.myview" > Column | Type | Collation | Nullable | Default |

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-03-23 Thread Peter Geoghegan
On Wed, Mar 23, 2022 at 1:53 PM Robert Haas wrote: > I see what you mean about it depending on how you define "skipping". > But I think that DISABLE_PAGE_SKIPPING is intended as a sort of > emergency safeguard when you really, really don't want to leave > anything out. I agree. > And therefore I

Re: ExecRTCheckPerms() and many prunable partitions

2022-03-23 Thread Zhihong Yu
On Wed, Mar 23, 2022 at 12:03 AM Amit Langote wrote: > On Mon, Mar 14, 2022 at 4:36 PM Amit Langote > wrote: > > Also needed fixes when rebasing. > > Needed another rebase. > > As the changes being made with the patch are non-trivial and the patch > hasn't been reviewed very significantly since

Re: SQL/JSON: functions

2022-03-23 Thread Justin Pryzby
On Wed, Mar 23, 2022 at 03:49:17PM -0400, Andrew Dunstan wrote: > > On 3/23/22 08:24, Justin Pryzby wrote: > > At least 0002-SQL-JSON-constructors-v64.patch has an issue with nodes, > > per COPY_PARSE_PLAN_TREES. > > > > +ERROR: unrecognized node type: 157 > > I just tried to reproduce this and

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-03-23 Thread Peter Eisentraut
On 21.03.22 05:55, Thomas Munro wrote: [04:30:50.630] pg_waldump.c:963:26: error: format ‘%u’ expects argument of type ‘unsigned int *’, but argument 3 has type ‘ForkNumber *’ [-Werror=format=] [04:30:50.630] 963 | if (sscanf(optarg, "%u", &config.filter_by_relation_forknum) != 1 || [04:30:50.630

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-03-23 Thread Robert Haas
On Wed, Mar 23, 2022 at 4:49 PM Peter Geoghegan wrote: > On Wed, Mar 23, 2022 at 1:41 PM Robert Haas wrote: > > It seems to me that if DISABLE_PAGE_SKIPPING doesn't completely > > disable skipping pages, we have a problem. > > It depends on how you define skipping. DISABLE_PAGE_SKIPPING was > cre

Re: Adding CI to our tree

2022-03-23 Thread Thomas Munro
On Thu, Mar 10, 2022 at 9:37 AM Justin Pryzby wrote: > -Og Adding this to CXXFLAGS caused a torrent of warnings from g++ about LLVM headers, which I also see on my local system for LLVM 11 and LLVM 14: [19:47:11.047] /usr/lib/llvm-11/include/llvm/ADT/Twine.h: In member function ‘llvm::CallInst*

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-03-23 Thread Peter Geoghegan
On Wed, Mar 23, 2022 at 1:41 PM Robert Haas wrote: > It seems to me that if DISABLE_PAGE_SKIPPING doesn't completely > disable skipping pages, we have a problem. It depends on how you define skipping. DISABLE_PAGE_SKIPPING was created at a time when a broader definition of skipping made a lot mor

Re: [PATCH] Remove workarounds to format [u]int64's

2022-03-23 Thread Peter Eisentraut
On 21.03.22 15:37, Aleksander Alekseev wrote: It would not simplify things for them at all, just mess it up. The master copies of the .po files are kept in a different repo. Also, I believe that extraction of new message strings is automated already. Got it, thanks. Here is the corrected patch.

Re: MDAM techniques and Index Skip Scan patch

2022-03-23 Thread Dmitry Dolgov
> On Tue, Mar 22, 2022 at 04:55:49PM -0400, Tom Lane wrote: > Peter Geoghegan writes: > > Like many difficult patches, the skip scan patch is not so much > > troubled by problems with the implementation as it is troubled by > > *ambiguity* about the design. Particularly concerning how skip scan >

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-03-23 Thread Robert Haas
On Wed, Mar 23, 2022 at 3:59 PM Peter Geoghegan wrote: > In other words, since DISABLE_PAGE_SKIPPING doesn't *consistently* > force lazy_scan_noprune to refuse to process a page on HEAD (it all > depends on FreezeLimit/vacuum_freeze_min_age), it is logical for > DISABLE_PAGE_SKIPPING to totally ge

multithreaded zstd backup compression for client and server

2022-03-23 Thread Robert Haas
[ Changing subject line in the hopes of attracting more eyeballs. ] On Mon, Mar 14, 2022 at 12:11 PM Dipesh Pandit wrote: > I tried to implement support for parallel ZSTD compression. Here's a new patch for this. It's more of a rewrite than an update, honestly; commit ffd53659c46a54a6978bcb8c442

Re: Parameter for planner estimate of recursive queries

2022-03-23 Thread Tom Lane
Simon Riggs writes: >> [New patch version pending] Do you have any objection if I rename the GUC to recursive_worktable_factor? That seems a bit clearer as to what it does, and it leaves more room for other knobs in the same area if we decide we need any. regards, tom la

Re: SQL/JSON: functions

2022-03-23 Thread Andrew Dunstan
On 3/23/22 15:49, Andrew Dunstan wrote: > On 3/23/22 08:24, Justin Pryzby wrote: >> At least 0002-SQL-JSON-constructors-v64.patch has an issue with nodes, >> per COPY_PARSE_PLAN_TREES. >> >> +ERROR: unrecognized node type: 157 > > > I just tried to reproduce this and was unable to.  Ubuntu 20.04

Re: ubsan

2022-03-23 Thread Andres Freund
Hi, On 2022-03-23 15:58:09 -0400, Tom Lane wrote: > Andres Freund writes: > > I think we should backpatch both, based on the reasoning in > > 46ab07ffda9d6c8e63360ded2d4568aa160a7700 ? > > Yeah, I suppose. Is anyone going to step up and run a buildfarm > member with ubsan enabled? (I'm already

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-03-23 Thread Peter Geoghegan
On Sun, Mar 13, 2022 at 9:05 PM Peter Geoghegan wrote: > Attached is v10. While this does still include the freezing patch, > it's not in scope for Postgres 15. As I've said, I still think that it > makes sense to maintain the patch series with the freezing stuff, > since it's structurally related

Re: Proposal: Support custom authentication methods using hooks

2022-03-23 Thread Peter Eisentraut
On 15.03.22 20:27, samay sharma wrote: This patch-set adds the following: * Allow multiple custom auth providers to be registered (Addressing feedback from Aleksander and Andrew) * Modify the test extension to use SCRAM to exchange secrets (Based on Andres's suggestion) * Add support for custo

Re: ubsan

2022-03-23 Thread Tom Lane
Andres Freund writes: > I think we should backpatch both, based on the reasoning in > 46ab07ffda9d6c8e63360ded2d4568aa160a7700 ? Yeah, I suppose. Is anyone going to step up and run a buildfarm member with ubsan enabled? (I'm already checking -fsanitize=alignment on longfin, but it seems advisab

Re: Documenting when to retry on serialization failure

2022-03-23 Thread Tom Lane
Simon Riggs writes: > I've tried to sum up the various points from everybody into this doc > patch. Thanks all for replies. This seemed rather badly in need of copy-editing. How do you like the attached text? regards, tom lane diff --git a/doc/src/sgml/mvcc.sgml b/doc/s

Re: SQL/JSON: functions

2022-03-23 Thread Andrew Dunstan
On 3/23/22 08:24, Justin Pryzby wrote: > At least 0002-SQL-JSON-constructors-v64.patch has an issue with nodes, > per COPY_PARSE_PLAN_TREES. > > +ERROR: unrecognized node type: 157 I just tried to reproduce this and was unable to.  Ubuntu 20.04, gcc 9.4.0. the build was done with CPPFLAGS=-

Re: Parameter for planner estimate of recursive queries

2022-03-23 Thread Simon Riggs
On Wed, 23 Mar 2022 at 18:20, Simon Riggs wrote: > [New patch version pending] -- Simon Riggshttp://www.EnterpriseDB.com/ recursive_worktable_estimate.v3.patch Description: Binary data

Re: ubsan

2022-03-23 Thread Andres Freund
Hi, On 2022-03-23 11:21:37 -0700, Andres Freund wrote: > On 2022-03-23 13:54:50 -0400, Tom Lane wrote: > > Andres Freund writes: > > > I tried to run postgres with ubsan to debug something. > > > > For 0001, could we just replace configure's dlopen check with the > > dlsym check? Or are you afr

Re: warn if GUC set to an invalid shared library

2022-03-23 Thread Tom Lane
Maciek Sakrejda writes: > In v4, the message looks fine to me for shared_preload_libraries > (except there is a doubled "is"). However, I also get the message for > a simple SET with local_preload_libraries: > postgres=# set local_preload_libraries=xyz; > WARNING: could not access file "xyz" > H

Re: speed up a logical replica setup

2022-03-23 Thread Peter Eisentraut
On 18.03.22 23:34, Andrew Dunstan wrote: On 3/15/22 09:51, Peter Eisentraut wrote: On 21.02.22 13:09, Euler Taveira wrote: A new tool called pg_subscriber does this conversion and is tightly integrated with Postgres. Are we comfortable with the name pg_subscriber?  It seems too general. Are w

Re: prevent immature WAL streaming

2022-03-23 Thread Alvaro Herrera
On 2021-Sep-03, Alvaro Herrera wrote: > The last commit is something I noticed in pg_rewind ... I had missed this one; it's pushed now. -- Álvaro HerreraBreisgau, Deutschland — https://www.EnterpriseDB.com/ "I can see support will not be a problem. 10 out of 10."(Simon Wittber)

Re: [HACKERS] WIP aPatch: Pgbench Serialization and deadlock errors

2022-03-23 Thread Tom Lane
Tatsuo Ishii writes: > The patch Pushed. Thank you! My hoary animal prairiedog doesn't like this [1]: # Failed test 'concurrent update with retrying stderr /(?s-xim:client (0|1) got an error in command 3 \\(SQL\\) of script 0; ERROR: could not serialize access due to concurrent update\\b.*\

Re: ubsan

2022-03-23 Thread Andres Freund
Hi, On 2022-03-23 13:54:50 -0400, Tom Lane wrote: > Andres Freund writes: > > I tried to run postgres with ubsan to debug something. > > For 0001, could we just replace configure's dlopen check with the > dlsym check? Or are you afraid of reverse-case failures? Yea, I was worried about that. B

Re: Reducing power consumption on idle servers

2022-03-23 Thread Simon Riggs
On Tue, 22 Mar 2022 at 00:54, Andres Freund wrote: > > On 2022-02-21 21:04:19 +, Simon Riggs wrote: > > On Mon, 21 Feb 2022 at 16:49, Chapman Flack wrote: > > > > > Shouldn't the comment be "with work_done=false" ? > > > > Good catch, thanks. > > > > I've also added docs to say that "promote_

Re: Parameter for planner estimate of recursive queries

2022-03-23 Thread Simon Riggs
On Wed, 23 Mar 2022 at 17:36, Tom Lane wrote: > > Robert Haas writes: > > On Tue, Jan 25, 2022 at 4:44 AM Peter Eisentraut > > wrote: > >> On the one hand, this smells like a planner hint. But on the other > >> hand, it doesn't look like we will come up with proper graph-aware > >> selectivity

Re: ubsan

2022-03-23 Thread Tom Lane
Andres Freund writes: > I tried to run postgres with ubsan to debug something. For 0001, could we just replace configure's dlopen check with the dlsym check? Or are you afraid of reverse-case failures? 0002: ugh, but my only real complaint is that __ubsan_default_options needs more than zero co

Re: shared-memory based stats collector - v67

2022-03-23 Thread Andres Freund
Hi, On 2022-03-23 17:27:50 +0900, Kyotaro Horiguchi wrote: > At Mon, 21 Mar 2022 14:30:17 -0700, Andres Freund wrote > in > > > Right now we reset stats for replicas, even if we start from a shutdown > > > checkpoint. That seems pretty unnecessary with this patch: > > > - 0021-pgstat-wip-only-r

Re: Parameter for planner estimate of recursive queries

2022-03-23 Thread Tom Lane
Robert Haas writes: > On Tue, Jan 25, 2022 at 4:44 AM Peter Eisentraut > wrote: >> On the one hand, this smells like a planner hint. But on the other >> hand, it doesn't look like we will come up with proper graph-aware >> selectivity estimation system any time soon, so just having all graph >>

ubsan

2022-03-23 Thread Andres Freund
Hi, I tried to run postgres with ubsan to debug something. I ran into two main issues: 1) Despite Tom's recent efforts in [1], I see two ubsan failures in HEAD. These are easy enough to fix as per the attached, although the fix for the GetConfigOptionByNum() isn't great - we should pro

Re: [BUG] Panic due to incorrect missingContrecPtr after promotion

2022-03-23 Thread alvhe...@alvh.no-ip.org
On 2022-Mar-07, Imseih (AWS), Sami wrote: > I have gone ahead and backpatched this all the way to 10 as well. Thanks! I pushed this now. I edited the test though: I don't understand why you went to the trouble of setting stuff in order to call 'pg_ctl promote' (in different ways for older branc

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-23 Thread Dilip Kumar
On Wed, Mar 23, 2022 at 10:37 PM Andres Freund wrote: > > Hi, > > On 2022-03-23 22:29:40 +0530, Dilip Kumar wrote: > > I could not see any reason for it to fail, and I could not reproduce > > it either. Is it possible to access the server log for this cfbot > > failure? > > Yes, near the top, bel

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2022-03-23 Thread Andres Freund
Hi, On 2022-03-23 22:29:40 +0530, Dilip Kumar wrote: > I could not see any reason for it to fail, and I could not reproduce > it either. Is it possible to access the server log for this cfbot > failure? Yes, near the top, below the cpu / memory graphs, there's a file navigator. Should have all f

Re: pgsql: Unbreak the build.

2022-03-23 Thread Robert Haas
On Wed, Mar 23, 2022 at 12:03 PM Robert Haas wrote: > I'm looking into this now, but that's not the same Assert(false). The > one Andres is talking about is at the end of get_bc_algorithm_name(). > This one is in tar_open_for_write(). AFAIK, the first of these is > actually unreachable or at least

  1   2   >