[PATCH] Faster, easier valgrind runs with make USE_VALGRIND=1 check

2021-05-05 Thread Craig Ringer
Hi all The attached patch adds support for running any temp-install based tests (check, isolationcheck, src/test/recovery, etc) under the control of valgrind with a simple make USE_VALGRIND=1 check It's based on a script I've been using for some time to run faster, simpler Valgrind checks o

Re: Small issues with CREATE TABLE COMPRESSION

2021-05-05 Thread Dilip Kumar
On Wed, May 5, 2021 at 11:02 AM Dilip Kumar wrote: > > On Wed, May 5, 2021 at 12:06 AM Robert Haas wrote: > > > > > > There are no tests in pg_dump to make sure that some ALTER > > > MATERIALIZED VIEW or ALTER TABLE commands are generated when the > > > compression of a matview's or table's colum

RE: Use simplehash.h instead of dynahash in SMgr

2021-05-05 Thread Jakub Wartak
Hi David, Alvaro, -hackers > Hi David, > > You're probably aware of this, but just to make it explicit: Jakub Wartak was > testing performance of recovery, and one of the bottlenecks he found in > some of his cases was dynahash as used by SMgr. It seems quite possible > that this work would ben

Re: few ideas for pgbench

2021-05-05 Thread Fabien COELHO
Hello Pavel, This is not a simple question. Personally I prefer to show this info every time, although it can be redundant. Just for check and for more simple automatic processing. When I run pgbench, I usually work with more releases together, so the server version is important info. Ok. Y

Re: few ideas for pgbench

2021-05-05 Thread Pavel Stehule
st 5. 5. 2021 v 11:55 odesílatel Fabien COELHO napsal: > > Hello Pavel, > > > This is not a simple question. Personally I prefer to show this info > every > > time, although it can be redundant. Just for check and for more simple > > automatic processing. > > > > When I run pgbench, I usually wor

Re: Small issues with CREATE TABLE COMPRESSION

2021-05-05 Thread Michael Paquier
On Wed, May 05, 2021 at 01:41:03PM +0530, Dilip Kumar wrote: > I think we already have such test cases at least through pg_upgrade. > Basically, if you see in compression.sql we are not dropping the table > so that pg_upgrade and dump them and test. So if test run --with-lz4 > then in pg_upgrade d

RFC: Detailed reorder buffer stats dumps

2021-05-05 Thread Craig Ringer
Hi all I'm thinking of piggy-backing on the approach used in the "Get memory contexts of an arbitrary backend process" patch in order to provide access to detailed reorder buffer content statistics from walsenders on request. Right now the reorder buffer is mostly a black-box. I mostly rely on gd

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-05-05 Thread Pavel Luzanov
Hello, I found this in the documentation, section '5.11.3. Partitioning Using Inheritance'[1]: "Some operations require a stronger lock when using declarative partitioning than when using table inheritance. For example, removing a partition from a partitioned table requires taking an ACCESS EX

Re: Small issues with CREATE TABLE COMPRESSION

2021-05-05 Thread Dilip Kumar
On Wed, May 5, 2021 at 4:00 PM Michael Paquier wrote: > > On Wed, May 05, 2021 at 01:41:03PM +0530, Dilip Kumar wrote: > > I think we already have such test cases at least through pg_upgrade. > > Basically, if you see in compression.sql we are not dropping the table > > so that pg_upgrade and dump

Re: Use simplehash.h instead of dynahash in SMgr

2021-05-05 Thread David Rowley
Hi Jakub, On Wed, 5 May 2021 at 20:16, Jakub Wartak wrote: > I might be a little bit out of the loop, but as Alvaro stated - Thomas did > plenty of excellent job related to recovery performance in that thread. In my > humble opinion and if I'm not mistaken (I'm speculating here) it might be >

Re: Performance degradation of REFRESH MATERIALIZED VIEW

2021-05-05 Thread Tomas Vondra
On 4/27/21 8:22 PM, Andres Freund wrote: Hi, On 2021-04-28 00:44:47 +0900, Masahiko Sawada wrote: On Wed, Apr 28, 2021 at 12:26 AM Masahiko Sawada wrote: What Andres is suggesting (I think) is to modify ExecInsert() to pass a valid bistate to table_tuple_insert, instead of just NULL, and s

Re: RFC: Detailed reorder buffer stats dumps

2021-05-05 Thread Ashutosh Bapat
On Wed, May 5, 2021 at 4:03 PM Craig Ringer wrote: > > Hi all > > I'm thinking of piggy-backing on the approach used in the "Get memory > contexts of an arbitrary backend process" patch in order to provide access to > detailed reorder buffer content statistics from walsenders on request. > > Rig

Re: Small issues with CREATE TABLE COMPRESSION

2021-05-05 Thread Robert Haas
On Wed, May 5, 2021 at 7:09 AM Dilip Kumar wrote: > So basically, if we have to write this test case in pg_dump then we > will have to use lz4 which means it will generate different output > --with-lz4 vs --without-lz4. With a simple regress test it easy to > deal with such cases by keeping multi

RE: Use simplehash.h instead of dynahash in SMgr

2021-05-05 Thread Jakub Wartak
Hey David, > I think you'd have to batch by filenode and transaction in that case. Each > batch might be pretty small on a typical OLTP workload, so it might not help > much there, or it might hinder. True, it is very workload dependent (I was chasing mainly INSERTs multiValues, INSERT-SELECT)

Re: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-05-05 Thread Robert Haas
On Tue, May 4, 2021 at 11:47 PM Greg Nancarrow wrote: > Problem is, for built-in functions, the changes are allowed, but for > some properties (like strict) the allowed changes don't actually take > effect (this is what Amit was referring to - so why allow those > changes?). > It's because some of

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Robert Haas
On Tue, May 4, 2021 at 9:24 PM Peter Geoghegan wrote: > Here is my concern: I have an obligation to make it clear that I think > that you really ought to straighten out this business with > generalizing TIDs before too long. Not because I say so, but because > it's holding up progress in general.

Re: Is txid_status() actually safe? / What is 011_crash_recovery.pl testing?

2021-05-05 Thread Craig Ringer
On Wed, 10 Feb 2021 at 04:28, Robert Haas wrote: > On Mon, Feb 8, 2021 at 4:52 PM Andres Freund wrote: > > The 011_crash_recovery.pl test test starts a transaction, creates a > > table, fetches the transaction's xid. Then shuts down the server in > > immediate mode. It then asserts that after cr

pg_receivewal makes a bad daemon

2021-05-05 Thread Robert Haas
You might want to use pg_receivewal to save all of your WAL segments somewhere instead of relying on archive_command. It has, at the least, the advantage of working on the byte level rather than the segment level. But it seems to me that it is not entirely suitable as a substitute for archiving, fo

Proposal: per expression intervalstyle

2021-05-05 Thread Michał Wadas
Problem: I have to set IntervalStyle in separate statement to convert interval type to ISO8601 string. This isn't well supported by ORMs or similar tools, requiring us to set it globally (per role or per database). Problem #2 (exotic): you can't mix two output styles in a single query. Proposed

Re: Is txid_status() actually safe? / What is 011_crash_recovery.pl testing?

2021-05-05 Thread Craig Ringer
On Tue, 9 Feb 2021 at 05:52, Andres Freund wrote: > > Craig, it kind of looks to me like you assumed it'd be guaranteed that > the xid at this point would show in-progress? > At the time I wrote that code, I don't think I understood that xid assignment wasn't necessarily durable until either (a)

Re: Is txid_status() actually safe? / What is 011_crash_recovery.pl testing?

2021-05-05 Thread Craig Ringer
On Wed, 5 May 2021 at 23:15, Craig Ringer wrote: > Which was fine as far as it went, but I failed to account for the xid > assignment not necessarily being durable when the client calls > txid_status(). Ahem, I meant "when the client calls txid_current()" -- Craig Ringer htt

COPY table_name (single_column) FROM 'unknown.txt' DELIMITER E'\n'

2021-05-05 Thread Joel Jacobson
Hi, >From time to time, I need to deal with bizarrely delimited text files, having to use tools such as awk/sed/perl to reformat the files so that they can be copied into PostgreSQL. If such files could be imported to a table with a single text column, we could then use PostgreSQL's now really fa

Re: pg_receivewal makes a bad daemon

2021-05-05 Thread Laurenz Albe
On Wed, 2021-05-05 at 11:04 -0400, Robert Haas wrote: > You might want to use pg_receivewal to save all of your WAL segments > somewhere instead of relying on archive_command. It has, at the least, > the advantage of working on the byte level rather than the segment > level. But it seems to me that

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Peter Geoghegan
On Wed, May 5, 2021 at 7:27 AM Robert Haas wrote: > It seems to me that we're doing a lot of disagreeing given that, as I > see it, there are only relatively minor differences between the > positions of the various people here. I'm being very vocal here because I'm concerned that we're going abou

Re: Printing backtrace of postgres processes

2021-05-05 Thread vignesh C
On Wed, Feb 3, 2021 at 3:24 PM Bharath Rupireddy wrote: > > On Wed, Feb 3, 2021 at 1:49 PM vignesh C wrote: > > > > On Wed, Feb 3, 2021 at 1:00 PM Tom Lane wrote: > > > > > > vignesh C writes: > > > > On Mon, Feb 1, 2021 at 11:04 AM Bharath Rupireddy > > > > wrote: > > > >> Are these superuser

Re: useless argument of ATAddForeignKeyConstraint

2021-05-05 Thread Alvaro Herrera
On 2019-Feb-14, Amit Langote wrote: > While reviewing the foreign keys referencing partitioned tables patch, I > noticed that the parentConstr argument of ATAddForeignConstraint is > rendered useless by the following commit: > Maybe remove that argument in HEAD ? Attached a patch. Indeed -- two

Re: pg_receivewal makes a bad daemon

2021-05-05 Thread Magnus Hagander
On Wed, May 5, 2021 at 5:04 PM Robert Haas wrote: > > You might want to use pg_receivewal to save all of your WAL segments > somewhere instead of relying on archive_command. It has, at the least, > the advantage of working on the byte level rather than the segment > level. But it seems to me that

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Robert Haas
On Wed, May 5, 2021 at 11:50 AM Peter Geoghegan wrote: > I'm being very vocal here because I'm concerned that we're going about > generalizing TIDs in the wrong way. To me it feels like there is a > loss of perspective about what really matters. Well, which things matter is a question of opinion,

Re: COPY table_name (single_column) FROM 'unknown.txt' DELIMITER E'\n'

2021-05-05 Thread David G. Johnston
On Wed, May 5, 2021 at 8:31 AM Joel Jacobson wrote: > Could it be an idea to exploit the fact that DELIMITER E'\n' is currently > an error? > > Why not just allow: "DELIMITER NONE" to be valid syntax meaning exactly what it says and does exactly what you desire? David J.

Re: .ready and .done files considered harmful

2021-05-05 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Tue, May 4, 2021 at 11:54 AM Dilip Kumar wrote: > > I agree that if we continue to archive one file using the archive > > command then Robert's solution of checking the existence of the next > > WAL segment (N+1) has an advantage. But,

Re: pg_receivewal makes a bad daemon

2021-05-05 Thread Robert Haas
On Wed, May 5, 2021 at 12:34 PM Magnus Hagander wrote: > Is this really a problem we should fix ourselves? Most daemon-managers > today will happily be configured to automatically restart a daemon on > failure with a single setting since a long time now. E.g. in systemd > (which most linuxen uses

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Jeff Davis
On Wed, 2021-05-05 at 10:27 -0400, Robert Haas wrote: > It's too early for the project to commit to stability in > this area; we have not managed to get a single AM apart from heapam > into core "In core" shouldn't matter. In fact, if it's in core, stability of the APIs is much less important. >

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Peter Geoghegan
On Wed, May 5, 2021 at 9:42 AM Robert Haas wrote: > On Wed, May 5, 2021 at 11:50 AM Peter Geoghegan wrote: > > I'm being very vocal here because I'm concerned that we're going about > > generalizing TIDs in the wrong way. To me it feels like there is a > > loss of perspective about what really ma

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Jeff Davis
On Wed, 2021-05-05 at 08:50 -0700, Peter Geoghegan wrote: > There just isn't that > many table AM TID designs that could ever work, and even among those > schemes that could ever work there is a pretty clear hierarchy. This > blue sky thinking about generalizing TIDs 2 years in seems *weird* to > m

Re: COPY table_name (single_column) FROM 'unknown.txt' DELIMITER E'\n'

2021-05-05 Thread Chapman Flack
On 05/05/21 13:02, David G. Johnston wrote: > Why not just allow: "DELIMITER NONE" to be valid syntax meaning exactly > what it says and does exactly what you desire? What would it mean? That you get one column, multiple rows of text corresponding to "lines" delimited by something, or that you get

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Robert Haas
On Wed, May 5, 2021 at 1:13 PM Jeff Davis wrote: > "In core" shouldn't matter. In fact, if it's in core, stability of the > APIs is much less important. I don't know what to say here. I think it's unrealistic to believe that a very new API that has only 1 in-core user is going to be fully stable,

Re: COPY table_name (single_column) FROM 'unknown.txt' DELIMITER E'\n'

2021-05-05 Thread Isaac Morland
On Wed, 5 May 2021 at 13:23, Chapman Flack wrote: > On 05/05/21 13:02, David G. Johnston wrote: > > Why not just allow: "DELIMITER NONE" to be valid syntax meaning exactly > > what it says and does exactly what you desire? > > What would it mean? That you get one column, multiple rows of text > c

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Peter Geoghegan
On Wed, May 5, 2021 at 10:33 AM Robert Haas wrote: > I don't know what to say here. I think it's unrealistic to believe > that a very new API that has only 1 in-core user is going to be fully > stable, or that we can know how it might evolve. I can understand why > you and probably other people wa

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Jeff Davis
On Wed, 2021-05-05 at 10:48 -0700, Peter Geoghegan wrote: > What we have right now has little chance of failing. It also has > little chance of succeeding (except for something like zheap, which > can presumably get by with the heapam's idea of TID). What has little chance of succeeding? Table AMs

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Robert Haas
On Wed, May 5, 2021 at 1:15 PM Peter Geoghegan wrote: > > I don't think this is true at all. If you have a clustered index - > > i.e. the table is physically arranged according to the index ordering > > - then your secondary indexes all pretty much have to be what we're > > calling indirect indexe

Re: COPY table_name (single_column) FROM 'unknown.txt' DELIMITER E'\n'

2021-05-05 Thread David G. Johnston
On Wed, May 5, 2021 at 10:34 AM Isaac Morland wrote: > On Wed, 5 May 2021 at 13:23, Chapman Flack wrote: > >> On 05/05/21 13:02, David G. Johnston wrote: >> > Why not just allow: "DELIMITER NONE" to be valid syntax meaning exactly >> > what it says and does exactly what you desire? >> >> What wo

v14 mechanical code beautification patches

2021-05-05 Thread Tom Lane
It's getting to be time to think about these steps for v14: * Renumber any manually-assigned OIDs between 8000 and to lower numbers, using renumber_oids.pl (see notes in bki.sgml) * pgindent, perltidy, reformat-dat-files * Update config.guess and config.sub (from https://savannah.gnu.or

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Andres Freund
Hi, On 2021-05-05 13:32:57 -0400, Robert Haas wrote: > I don't know what to say here. I think it's unrealistic to believe > that a very new API that has only 1 in-core user is going to be fully > stable, or that we can know how it might evolve. I can understand why > you and probably other people

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Peter Geoghegan
On Wed, May 5, 2021 at 10:57 AM Robert Haas wrote: > One advantage of indirect indexes is that you can potentially avoid a > lot of writes to the index. If a non-HOT update is performed, but the > primary key is not updated, the index does not need to be touched. I > think that's a potentially sig

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Andres Freund
Hi, On 2021-05-05 10:56:56 -0700, Jeff Davis wrote: > On Wed, 2021-05-05 at 10:48 -0700, Peter Geoghegan wrote: > > What we have right now has little chance of failing. It also has > > little chance of succeeding (except for something like zheap, which > > can presumably get by with the heapam's i

Re: RFC: Detailed reorder buffer stats dumps

2021-05-05 Thread Andres Freund
Hi, On 2021-05-05 18:33:27 +0800, Craig Ringer wrote: > I'm thinking of piggy-backing on the approach used in the "Get memory > contexts of an arbitrary backend process" patch in order to provide access > to detailed reorder buffer content statistics from walsenders on request. > > Right now the

Re: COPY table_name (single_column) FROM 'unknown.txt' DELIMITER E'\n'

2021-05-05 Thread Joel Jacobson
On Wed, May 5, 2021, at 19:58, David G. Johnston wrote: > On Wed, May 5, 2021 at 10:34 AM Isaac Morland wrote: >> On Wed, 5 May 2021 at 13:23, Chapman Flack wrote: >>> On 05/05/21 13:02, David G. Johnston wrote: >>> > Why not just allow: "DELIMITER NONE" to be valid syntax meaning exactly >>> > w

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Peter Geoghegan
On Wed, May 5, 2021 at 10:56 AM Jeff Davis wrote: > What has little chance of succeeding? Table AMs? > > And why isn't columnar an example of someting that can "get by with > heapam's idea of TID"? I mean, it's not a perfect fit, but my primary > complaint this whole thread is that it's undefined,

Re: COPY table_name (single_column) FROM 'unknown.txt' DELIMITER E'\n'

2021-05-05 Thread Joel Jacobson
On Wed, May 5, 2021, at 19:34, Isaac Morland wrote: > Would DELIMITER NULL make sense? The existing values are literal strings so > NULL fits with that. Do we already have NONE as a keyword somewhere? It's > listed in the keyword appendix to the documentation but I can't think of > where it is u

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Peter Geoghegan
On Wed, May 5, 2021 at 11:25 AM Andres Freund wrote: > Agreed. And we can increase the fit a good bit without needing invasive > all-over changes. With those changes often even helping heap. > > E.g. tidbitmap.c's harcoded use of MaxHeapTuplesPerPage is a problem > even for heap - we waste a lot o

Re: Re: COPY table_name (single_column) FROM 'unknown.txt' DELIMITER E'\n'

2021-05-05 Thread Tom Lane
"Joel Jacobson" writes: > I think you misunderstood the problem. > I don't want the entire file to be considered a single value. > I want each line to become its own row, just a row with a single column. > So I actually think COPY seems like a perfect match for the job, > since it does precisely

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Jeff Davis
On Wed, 2021-05-05 at 11:22 -0700, Andres Freund wrote: > Yea. I think it would be actively *bad* if tableam were too > stable. tableam is at best an 80% solution to the abstraction needs > (those 80% were pretty painful to achieve already, I don't think we > could have gotten much more initially).

Re: COPY table_name (single_column) FROM 'unknown.txt' DELIMITER E'\n'

2021-05-05 Thread Andrew Dunstan
On 5/5/21 2:45 PM, Tom Lane wrote: > "Joel Jacobson" writes: >> I think you misunderstood the problem. >> I don't want the entire file to be considered a single value. >> I want each line to become its own row, just a row with a single column. >> So I actually think COPY seems like a perfect mat

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Peter Geoghegan
On Wed, May 5, 2021 at 12:09 PM Jeff Davis wrote: > Like anything, we make the decision at the time we have a reason to > break something. But why are are exensions disfavored in this > calculation vs. in-core? Isn't it a lot easier to update in-core code > to new APIs? We don't really have an AP

Re: Re: COPY table_name (single_column) FROM 'unknown.txt' DELIMITER E'\n'

2021-05-05 Thread Justin Pryzby
On Wed, May 05, 2021 at 02:45:41PM -0400, Tom Lane wrote: > > I'm currently using the pg_read_file()-hack in a project, > > and even though it can read files up to 1GB, > > using e.g. regexp_split_to_table() to split on E'\n' > > seems to need 4x as much memory, so it only > > works with files less

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Matthias van de Meent
On Wed, 5 May 2021 at 19:15, Peter Geoghegan wrote: > > On Wed, May 5, 2021 at 9:42 AM Robert Haas wrote: > > On Wed, May 5, 2021 at 11:50 AM Peter Geoghegan wrote: > > > I'm being very vocal here because I'm concerned that we're going about > > > generalizing TIDs in the wrong way. To me it fee

Dubious assertion in RegisterDynamicBackgroundWorker

2021-05-05 Thread Tom Lane
I noticed this recent crash on lorikeet: https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=lorikeet&dt=2021-05-05%2009%3A19%3A29 The relevant bits of the log seem to be 2021-05-05 05:36:22.011 EDT [60926716.bb24:1] ERROR: could not map dynamic shared memory segment ... 2021-05-05 05:36:2

Re: COPY table_name (single_column) FROM 'unknown.txt' DELIMITER E'\n'

2021-05-05 Thread Andrew Dunstan
On 5/5/21 3:36 PM, Justin Pryzby wrote: > On Wed, May 05, 2021 at 02:45:41PM -0400, Tom Lane wrote: >>> I'm currently using the pg_read_file()-hack in a project, >>> and even though it can read files up to 1GB, >>> using e.g. regexp_split_to_table() to split on E'\n' >>> seems to need 4x as much

Re: COPY table_name (single_column) FROM 'unknown.txt' DELIMITER E'\n'

2021-05-05 Thread Tom Lane
Andrew Dunstan writes: > On 5/5/21 2:45 PM, Tom Lane wrote: >> Yeah, that's because of the conversion to "chr". But a regexp >> is overkill for that anyway. Don't we have something that will >> split on simple substring matches? > Not that I know of. There is split_part but I don't think that's

Re: .ready and .done files considered harmful

2021-05-05 Thread Robert Haas
On Wed, May 5, 2021 at 1:06 PM Stephen Frost wrote: > It's not just about making sure that we archive the history file for a > timeline before archiving WAL segments along that timeline but also > about making sure we get that history file into the archive as fast as > we can, and archiving a 16MB

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Peter Geoghegan
On Wed, May 5, 2021 at 12:43 PM Matthias van de Meent wrote: > I believe that it cannot be "just" an additive thing, at least not > through a normal INCLUDEd column, as you'd get duplicate TIDs in the > index, with its related problems. You also cannot add it as a key > column, as this would disab

Re: .ready and .done files considered harmful

2021-05-05 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Wed, May 5, 2021 at 1:06 PM Stephen Frost wrote: > > It's not just about making sure that we archive the history file for a > > timeline before archiving WAL segments along that timeline but also > > about making sure we get that histor

Re: .ready and .done files considered harmful

2021-05-05 Thread Robert Haas
On Wed, May 5, 2021 at 4:13 PM Stephen Frost wrote: > I would think something like that would be alright and not worse than > what we've got now. OK. > That said, in an ideal world, we'd have a way to get the new timeline to > switch to in a way that doesn't leave open race conditions, so as lon

Re: .ready and .done files considered harmful

2021-05-05 Thread Andres Freund
Hi, On 2021-05-05 16:13:08 -0400, Stephen Frost wrote: > Just as a reminder for those following along at home, as I'm sure you're > already aware, the way we figure out what timeline to switch to when a > replica is getting promoted is that we go run the restore command asking > for history files

Re: .ready and .done files considered harmful

2021-05-05 Thread Andres Freund
Hi, On 2021-05-05 16:22:21 -0400, Robert Haas wrote: > Huh, I had not thought about that problem. So, at the risk of getting > sidetracked, what exactly are you asking for here? Let the extension > pick the timeline using an algorithm of its own devising, rather than > having core do it? Or what?

Re: .ready and .done files considered harmful

2021-05-05 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Wed, May 5, 2021 at 4:13 PM Stephen Frost wrote: > > That said, in an ideal world, we'd have a way to get the new timeline to > > switch to in a way that doesn't leave open race conditions, so as long > > we're talking about big changes

Re: Dubious assertion in RegisterDynamicBackgroundWorker

2021-05-05 Thread Robert Haas
On Wed, May 5, 2021 at 3:46 PM Tom Lane wrote: > Admittedly, it seems unlikely that the difference could exceed > MAX_PARALLEL_WORKER_LIMIT = 1024 in a regression test run where > the limit on number of parallel workers is only 8. What I think is > more likely, given that these counters are unsig

Re: .ready and .done files considered harmful

2021-05-05 Thread Robert Haas
On Wed, May 5, 2021 at 4:31 PM Andres Freund wrote: > On 2021-05-05 16:22:21 -0400, Robert Haas wrote: > > Huh, I had not thought about that problem. So, at the risk of getting > > sidetracked, what exactly are you asking for here? Let the extension > > pick the timeline using an algorithm of its

Re: .ready and .done files considered harmful

2021-05-05 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Wed, May 5, 2021 at 4:31 PM Andres Freund wrote: > > On 2021-05-05 16:22:21 -0400, Robert Haas wrote: > > > Huh, I had not thought about that problem. So, at the risk of getting > > > sidetracked, what exactly are you asking for here? L

Re: Bogus collation version recording in recordMultipleDependencies

2021-05-05 Thread Andrew Dunstan
On 4/21/21 4:28 PM, Andres Freund wrote: > Hi, > > On 2021-04-20 12:05:27 +1200, Thomas Munro wrote: >> I'll hold off reverting for a few more days to see if anyone has any >> other thoughts on that, because there doesn't seem to be any advantage >> in being too hasty about it. > I'm not really c

cache lookup failed for statistics object 123

2021-05-05 Thread Justin Pryzby
Per sqlsmith. postgres=# SELECT pg_get_statisticsobjdef_expressions(123); ERROR: cache lookup failed for statistics object 123 postgres=# \errverbose ERROR: XX000: cache lookup failed for statistics object 123 LOCATION: pg_get_statisticsobjdef_expressions, ruleutils.c:1762 The expectation is

Re: Bogus collation version recording in recordMultipleDependencies

2021-05-05 Thread Thomas Munro
On Thu, May 6, 2021 at 8:58 AM Andrew Dunstan wrote: > this is an open item for release 14 . The discussion seems to have gone > silent for a couple of weeks. Are we in a position to make any > decisions? I hear what Andres says, but is anyone acting on it? I'm going to revert this and resubmit f

Re: Dubious assertion in RegisterDynamicBackgroundWorker

2021-05-05 Thread Tom Lane
Robert Haas writes: > On Wed, May 5, 2021 at 3:46 PM Tom Lane wrote: >> Admittedly, it seems unlikely that the difference could exceed >> MAX_PARALLEL_WORKER_LIMIT = 1024 in a regression test run where >> the limit on number of parallel workers is only 8. What I think is >> more likely, given th

Re: Bogus collation version recording in recordMultipleDependencies

2021-05-05 Thread Andrew Dunstan
On 5/5/21 5:12 PM, Thomas Munro wrote: > On Thu, May 6, 2021 at 8:58 AM Andrew Dunstan wrote: >> this is an open item for release 14 . The discussion seems to have gone >> silent for a couple of weeks. Are we in a position to make any >> decisions? I hear what Andres says, but is anyone acting o

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Matthias van de Meent
On Wed, 5 May 2021 at 22:09, Peter Geoghegan wrote: > > On Wed, May 5, 2021 at 12:43 PM Matthias van de Meent > wrote: > > I believe that it cannot be "just" an additive thing, at least not > > through a normal INCLUDEd column, as you'd get duplicate TIDs in the > > index, with its related proble

Make some column descriptions easier to distinguish visually

2021-05-05 Thread David Fetter
Folks, I was writing up a query on pg_constraint, and the columns whose descriptions I've changed here were pretty hard to puzzle out, as they were only distinct up to the difference between F and P, which isn't always easy to see. Please find attached a patch to disambiguate them. Best, David.

Re: v14 mechanical code beautification patches

2021-05-05 Thread Bruce Momjian
On Wed, May 5, 2021 at 02:18:04PM -0400, Tom Lane wrote: > I notice that we also list this as a pre-beta task in > src/tools/RELEASE_CHANGES: > > * Update inet/cidr data types with newest Bind patches > > However, I can't recall that anyone has ever done any such thing; > and at this point, any

Re: v14 mechanical code beautification patches

2021-05-05 Thread Tom Lane
Bruce Momjian writes: > On Wed, May 5, 2021 at 02:18:04PM -0400, Tom Lane wrote: >> I notice that we also list this as a pre-beta task in >> src/tools/RELEASE_CHANGES: >> * Update inet/cidr data types with newest Bind patches >> However, I can't recall that anyone has ever done any such thin

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Peter Geoghegan
On Wed, May 5, 2021 at 3:18 PM Matthias van de Meent wrote: > I believe that the TID is the unique identifier of that tuple, within context. > > For normal indexes, the TID as supplied directly by the TableAM is > sufficient, as the context is that table. > For global indexes, this TID must includ

Re: v14 mechanical code beautification patches

2021-05-05 Thread Bruce Momjian
On Wed, May 5, 2021 at 07:08:35PM -0400, Tom Lane wrote: > > I think it is related to these files: > > src/backend/utils/adt/inet_cidr_ntop.c > > src/backend/utils/adt/inet_net_pton.c > > which have at the top: > > * Copyright (c) 2004 by Internet Systems Consortium, Inc. ("ISC") > > * C

Re: Small issues with CREATE TABLE COMPRESSION

2021-05-05 Thread Michael Paquier
On Wed, May 05, 2021 at 09:59:41AM -0400, Robert Haas wrote: > TAP tests have a facility for conditionally skipping tests; see > perldoc Test::More. That's actually superior to what you can do with > pg_regress. We'd need to come up with some logic to determine when to > skip or not, though. Perhap

Re: .ready and .done files considered harmful

2021-05-05 Thread Robert Haas
On Wed, May 5, 2021 at 4:53 PM Stephen Frost wrote: > I do note that this comment is timeline.c is, ahem, perhaps over-stating > things a bit: > > * Note: while this is somewhat heuristic, it does positively guarantee > * that (result + 1) is not a known timeline, and therefore it should > * be

Re: Dubious assertion in RegisterDynamicBackgroundWorker

2021-05-05 Thread Robert Haas
On Wed, May 5, 2021 at 5:22 PM Tom Lane wrote: > I follow your concern, but I'm not convinced that this assertion is > a useful aid; first because the asynchrony involved makes the edge > cases rather squishy, and second because allowing 1024 bogus > increments before complaining will likely mean

Re: Replication slot stats misgivings

2021-05-05 Thread Masahiko Sawada
On Tue, May 4, 2021 at 2:34 PM vignesh C wrote: > > On Tue, May 4, 2021 at 9:48 AM Masahiko Sawada wrote: > > > > On Mon, May 3, 2021 at 10:21 PM Amit Kapila wrote: > > > > > > On Mon, May 3, 2021 at 5:48 PM Masahiko Sawada > > > wrote: > > > > > > > > On Mon, May 3, 2021 at 2:27 PM Amit Kapil

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Robert Haas
On Wed, May 5, 2021 at 3:43 PM Matthias van de Meent wrote: > I believe that it cannot be "just" an additive thing, at least not > through a normal INCLUDEd column, as you'd get duplicate TIDs in the > index, with its related problems. You also cannot add it as a key > column, as this would disabl

Re: Dubious assertion in RegisterDynamicBackgroundWorker

2021-05-05 Thread Tom Lane
I wrote: > I've not tried to trace the code, but I'm now a bit suspicious > that there is indeed a design bug here. I gather from the > comments that parallel_register_count is incremented by the > worker processes, which of course implies that a worker that > fails to reattach to shared memory wo

Re: PG in container w/ pid namespace is init, process exits cause restart

2021-05-05 Thread Robert Haas
On Tue, May 4, 2021 at 4:35 PM Tom Lane wrote: > I'm still thinking that we're best off refusing to do > that and making people install one of these shims that's meant > for the job. I have to admit that I care less about the specific issue here than about the general issue of being open to heari

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Hannu Krosing
How hard would it be to declare TID as current ItemPointerData with some values prohibited (NULL, SpecTokenOffsetNumber = 0xfffe, MovedPartitionsOffsetNumber = 0xfffd, presumably also 0x ?). And then commit to fixing usage outside access/heap/ which depend on small value for MaxHeapTuplesPerPa

Re: PG in container w/ pid namespace is init, process exits cause restart

2021-05-05 Thread Tom Lane
Robert Haas writes: > I have to admit that I care less about the specific issue here than > about the general issue of being open to hearing what the user needs > actually are. I honestly have no idea whether it's sensible to want to > run postgres as init. If people who know about container stuff

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Hannu Krosing
On Thu, May 6, 2021 at 3:07 AM Robert Haas wrote: > > On Wed, May 5, 2021 at 3:43 PM Matthias van de Meent > > Storage gains for index-oriented tables can become as large as the > > size of the primary key by not having to store all primary key values > > in both the index and the table; which can

Re: Printing backtrace of postgres processes

2021-05-05 Thread Justin Pryzby
Here's a cleaned-up copy of the doc text. Send a request to the backend with the specified process ID to log its backtrace. The backtrace will be logged at message level LOG. It will appear in the server log based on the log configuration set (See for more information), but will not be sent to t

Re: pg_receivewal makes a bad daemon

2021-05-05 Thread David Fetter
On Wed, May 05, 2021 at 01:12:03PM -0400, Robert Haas wrote: > On Wed, May 5, 2021 at 12:34 PM Magnus Hagander wrote: > > Is this really a problem we should fix ourselves? Most daemon-managers > > today will happily be configured to automatically restart a daemon on > > failure with a single setti

Re: v14 mechanical code beautification patches

2021-05-05 Thread Tom Lane
I wrote: > I dug in the archives and found the thread that prompted you to > add that bullet item: > https://www.postgresql.org/message-id/200502021700.j12H05j20872%40candle.pha.pa.us > which made the point that those were moving targets back in 2005. > I doubt they still are, so I don't see much p

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Jeff Davis
On Thu, 2021-05-06 at 03:26 +0200, Hannu Krosing wrote: > How hard would it be to declare TID as current ItemPointerData with > some values prohibited (NULL, SpecTokenOffsetNumber = 0xfffe, > MovedPartitionsOffsetNumber = 0xfffd, presumably also 0x ?). I don't think there's consensus in this t

RE: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-05-05 Thread tsunakawa.ta...@fujitsu.com
From: Robert Haas > On Tue, May 4, 2021 at 11:47 PM Greg Nancarrow > wrote: > > Problem is, for built-in functions, the changes are allowed, but for > > some properties (like strict) the allowed changes don't actually take > > effect (this is what Amit was referring to - so why allow those > > ch

Re: decoupling table and index vacuum

2021-05-05 Thread Masahiko Sawada
On Fri, Apr 23, 2021 at 5:01 AM Andres Freund wrote: > > Hi, > > On 2021-04-22 12:15:27 -0400, Robert Haas wrote: > > On Wed, Apr 21, 2021 at 5:38 PM Andres Freund wrote: > > > I'm not sure that's the only way to deal with this. While some form of > > > generic "conveyor belt" infrastructure woul

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Hannu Krosing
- Hannu Krosing On Thu, May 6, 2021 at 4:53 AM Jeff Davis wrote: > > On Thu, 2021-05-06 at 03:26 +0200, Hannu Krosing wrote: > > How hard would it be to declare TID as current ItemPointerData with > > some values prohibited (NULL, SpecTokenOffsetNumber = 0xfffe, > > MovedPartitionsOffsetNumbe

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2021-05-05 Thread Amit Kapila
On Fri, Apr 30, 2021 at 7:43 PM Dilip Kumar wrote: > > So, I > > used the other approach which led to the attached. > > The patch looks fine to me. > Thanks, pushed! -- With Regards, Amit Kapila.

Re: Identify missing publications from publisher while create/alter subscription.

2021-05-05 Thread Japin Li
On Tue, 04 May 2021 at 21:20, vignesh C wrote: > On Tue, May 4, 2021 at 2:37 PM Bharath Rupireddy > wrote: >> >> On Mon, May 3, 2021 at 7:59 PM vignesh C wrote: >> > Thanks for the comments, these comments are handle in the v7 patch >> > posted in my earlier mail. >> >> Thanks. Some comments o

Re: MaxOffsetNumber for Table AMs

2021-05-05 Thread Robert Haas
On Wed, May 5, 2021 at 10:53 PM Jeff Davis wrote: > On Thu, 2021-05-06 at 03:26 +0200, Hannu Krosing wrote: > > How hard would it be to declare TID as current ItemPointerData with > > some values prohibited (NULL, SpecTokenOffsetNumber = 0xfffe, > > MovedPartitionsOffsetNumber = 0xfffd, presumably

  1   2   >