Re: [HACKERS] Custom compression methods

2021-03-22 Thread Robert Haas
should be. It would also be nice if these names were more visually distinct from the related but very different enum values TOAST_PGLZ_COMPRESSION_ID and TOAST_LZ4_COMPRESSION_ID. Really, as the comments I added explain, we want to minimize the amount of code that knows about the 0-3

Re: [HACKERS] Custom compression methods

2021-03-22 Thread Robert Haas
On Mon, Mar 22, 2021 at 4:33 PM Robert Haas wrote: > On Mon, Mar 22, 2021 at 1:58 PM Justin Pryzby wrote: > > guc.c should not longer define this as extern: > > default_toast_compression_options > > Fixed. Fixed some more. -- Robert Haas EDB: http://www.enterprisedb.com

Re: making update/delete of inheritance trees scale better

2021-03-23 Thread Robert Haas
to contradict the first sentence and (b) that second target list isn't what you see when you run EXPLAIN. Also, there's no mention of what happens for FDWs here, but it's evidently different, as per the previous review comment. - The comments atop fix_join_expr() should be updated. Maybe you can just adjust the wording for case #2. OK, that's all I've got based on a first read-through. Thanks for your work on this. -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-03-23 Thread Robert Haas
On Thu, Mar 18, 2021 at 12:12 AM Tom Lane wrote: > Mark Dilger writes: > >> On Mar 16, 2021, at 12:52 PM, Robert Haas wrote: > >> Since we now know that shutting autovacuum off makes the problem go > >> away, I don't see a reason to commit 0001. We should

Re: pg_amcheck contrib application

2021-03-24 Thread Robert Haas
ely did not commit, then the table's current TupleDesc might differ from the one used to construct this tuple, so we must not check it." -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-03-24 Thread Robert Haas
On Wed, Mar 24, 2021 at 9:12 AM Robert Haas wrote: > On Wed, Mar 24, 2021 at 2:13 AM Mark Dilger > wrote: > > The visibility rules fix is different in v11, relying on a visibility check > > which more closely follows the implementation of > > HeapTupleSatisfiesVac

Re: default result formats setting

2021-03-24 Thread Robert Haas
ively, etc. That seems pretty wasteful and complicated for no > real gain. But ... if it's just a GUC, it can be set by code on the server side that the client knows nothing about, breaking the client. That seems pretty bad to me. -- Robert Haas EDB: http://www.enterprisedb.com

Re: pgsql: Move tablespace path re-creation from the makefiles to pg_regres

2021-03-24 Thread Robert Haas
27;@testtablespace@'; Would that work on Windows? -- Robert Haas EDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-03-24 Thread Robert Haas
rom the table column or the default, not the setting on the index column. -- Robert Haas EDB: http://www.enterprisedb.com

Re: default result formats setting

2021-03-24 Thread Robert Haas
On Wed, Mar 24, 2021 at 10:58 AM Tom Lane wrote: > Robert Haas writes: > > But ... if it's just a GUC, it can be set by code on the server side > > that the client knows nothing about, breaking the client. That seems > > pretty bad to me. > > It's impos

Re: [HACKERS] Custom compression methods

2021-03-24 Thread Robert Haas
but I'm not sure we have the luxury of worrying about that problem right now. -- Robert Haas EDB: http://www.enterprisedb.com

Re: default result formats setting

2021-03-24 Thread Robert Haas
s, for me, maybe your most interesting observation. -- Robert Haas EDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-03-24 Thread Robert Haas
so I guess this is no worse. But also ... aren't we just doing this to work around a test case that isn't especially good in the first place? Counting the number of lines in the diff between A and B is an extremely crude proxy for "they're similar enough that we probably haven't broken anything." -- Robert Haas EDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-03-24 Thread Robert Haas
On Mon, Mar 22, 2021 at 4:57 PM Robert Haas wrote: > > Fixed. > > Fixed some more. Committed. -- Robert Haas EDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-03-24 Thread Robert Haas
o where imagining that the TOAST compression property of the old database simply does not exist, rather than being pglz implicitly, is worse. But I could be wrong, and even if I'm right it's not a hill upon which I wish to die. -- Robert Haas EDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-03-24 Thread Robert Haas
it all gets dumped and restored. But, as it is, I think we should be deciding how to compress new values for an expression column based on the default_toast_compression setting at the time of compression, not the time of index creation. -- Robert Haas EDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-03-24 Thread Robert Haas
roposal, that's fine with me, as long as we're on the same page about what happens if we adopt it. -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-03-24 Thread Robert Haas
Mark, Here's a quick and very dirty sketch of what I think perhaps this logic could look like. This is pretty much untested and it might be buggy, but at least you can see whether we're thinking at all in the same direction. -- Robert Haas EDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-03-25 Thread Robert Haas
g. But I don't want to argue about it, either. It's a judgement call what's best here, and I don't pretend to have all the answers. If you're sure you've got it right ... great! -- Robert Haas EDB: http://www.enterprisedb.com

Re: shared memory stats: high level design decisions: consistency, dropping

2021-03-25 Thread Robert Haas
g all the data use a lot of memory, but having to read all the data in order to cache it is potentially a lot slower than just reading the data actually requested. I'm unwilling to dismiss that as a negligible problem. In short, I agree that there's stuff to worry about here, but

Re: [HACKERS] Custom compression methods

2021-03-25 Thread Robert Haas
On Thu, Mar 25, 2021 at 5:44 AM Dilip Kumar wrote: > Okay got it. Fixed as suggested. Committed with a bit of editing of the comments. -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-03-29 Thread Robert Haas
h a buffer lock held, and then more than that if there are other pieces that make sense to separate out? -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-03-30 Thread Robert Haas
ood idea for get_xid_status to return both a XidBoundsViolation and an XidCommitStatus. It seems to me (without checking all that carefully) that it might be better to just flatten all of that into a single enum, because right now it seems like you often end up with two consecutive switch statements where, perhaps, just one would suffice. -- Robert Haas EDB: http://www.enterprisedb.com

Re: New IndexAM API controlling index vacuum strategies

2021-03-31 Thread Robert Haas
s are obscured by moving code and changing it in the same patch. All this logic gets moved into lazy_scan_prune() and revised at the same time. Using git diff --color-moved -w sorta works, but even then there are parts of it that are pretty hard to read, because there's a bunch of other stuff that gets rejiggered at the same time. My concentration is flagging a bit so I'm going to stop reviewing here for now. I'm not deeply opposed to any of what I've seen so far. My main criticism is that I think more thought should be given to how things are named and to separating minimal code-movement patches from other changes. Thanks, -- Robert Haas EDB: http://www.enterprisedb.com

Re: making update/delete of inheritance trees scale better

2021-03-31 Thread Robert Haas
rd to understand what was happening in a lot of cases, but the recursive stuff is clearly not without its own attendant confusions. Thanks to both of you for working on this! As I said before, this seems like really important work. -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-03-31 Thread Robert Haas
ays exactly 1 remaining tuple using pointers to those TOAST values. Your assumption seems to be the opposite, that the TOASTed values would be retoasted as part of VF. If that is true, then your idea is right. Do you agree with this analysis? If so, we can check the code and find out which way it actually worked. -- Robert Haas EDB: http://www.enterprisedb.com

Re: making update/delete of inheritance trees scale better

2021-03-31 Thread Robert Haas
t; long runs of tuples from the same target relation, so as long as we > maintain a one-element cache of the last lookup result, it's only about > one comparison per tuple most of the time. OK, that's pretty fair. -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-03-31 Thread Robert Haas
her the tuples it's moving around contain TOAST pointers, so I think I had the right idea. -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-03-31 Thread Robert Haas
#x27;s not concurrent in the sense that you never have a situation where the VACUUM FULL does some of its modifications, then an overlapping transaction sees them, and then it does the rest of them. -- Robert Haas EDB: http://www.enterprisedb.com

Re: New IndexAM API controlling index vacuum strategies

2021-04-01 Thread Robert Haas
t it might help to try to write documentation for whatever behavior you choose. If it's hard to document without weasel-words, maybe it's not the best approach. -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-04-01 Thread Robert Haas
possible for it not to be running when we've *actually checked that* doesn't make nearly as much sense as it does when we haven't checked that and aren't going to do so. -- Robert Haas EDB: http://www.enterprisedb.com v15-0001-amcheck-Fix-verify_heapam-s-tuple-visibility-che.patch Description: Binary data

Re: pg_amcheck contrib application

2021-04-01 Thread Robert Haas
al complaints might complicate resolution of the issue also needs to be considered. > * If xmin_status happens to be XID_IN_PROGRESS, then in theory > > Did you mean to say XID_IS_CURRENT_XID here? Yes, I did, thanks. > /* xmax is an MXID, not an MXID. Sanity check it. */ > > Is it an MXID or isn't it? Good catch. New patch attached. -- Robert Haas EDB: http://www.enterprisedb.com v16-0001-amcheck-Fix-verify_heapam-s-tuple-visibility-che.patch Description: Binary data

Re: pg_amcheck contrib application

2021-04-01 Thread Robert Haas
On Thu, Apr 1, 2021 at 1:06 PM Mark Dilger wrote: > Seems fine other than the typo. OK, let's try that again. -- Robert Haas EDB: http://www.enterprisedb.com v17-0001-amcheck-Fix-verify_heapam-s-tuple-visibility-che.patch Description: Binary data

Re: pg_amcheck contrib application

2021-04-01 Thread Robert Haas
On Thu, Apr 1, 2021 at 1:24 PM Mark Dilger wrote: > > On Apr 1, 2021, at 10:20 AM, Robert Haas wrote: > > OK, let's try that again. > Looks good! OK, committed. We still need to deal with what you had as 0003 upthread, so I guess the next step is for me to spend some tim

Re: pg_amcheck contrib application

2021-04-01 Thread Robert Haas
On Thu, Apr 1, 2021 at 1:41 PM Robert Haas wrote: > OK, committed. We still need to deal with what you had as 0003 > upthread, so I guess the next step is for me to spend some time > reviewing that one. I did this, and it was a bit depressing. It appears that we now have duplicate c

Re: why pg_walfile_name() cannot be executed during recovery?

2021-04-02 Thread Robert Haas
rmation, but probably won't help except when WAL receiver is actually active. pg_last_wal_receive_lsn() and pg_last_wal_replay_lsn() will give the LSN at any point during recovery, but not the TLI. We might have some gaps in this area... -- Robert Haas EDB: http://www.enterprisedb.com

Re: ModifyTable overheads in generic plans

2021-04-07 Thread Robert Haas
ot there yet. So I hope that, in future releases, we can continue to find ways to whittle down the overhead. -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-04-07 Thread Robert Haas
ot sure it's a good idea to couple the concepts so closely like this. Maybe we can just #define VARLENA_SIZE_LIMIT in this file and use that, and a message that says size %u exceeds limit %u. I'm a little worried about whether the additional test cases are Endian-dependen

Re: [PATCH] PREPARE TRANSACTION unexpected behavior with TEMP TABLE

2021-04-08 Thread Robert Haas
ema-qualified object to do the same thing. Maybe that's a good idea, but we've been reluctant to go that far in the past due to performance consequences, and it's not clear whether any of those problems are related to the issue that prompted you to submit the patch. So, I'm kind of left wondering what exactly you're trying to solve here. Can you clarify? -- Robert Haas EDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-04-08 Thread Robert Haas
On Thu, Apr 8, 2021 at 11:32 AM David Steele wrote: > It looks like this CF entry should have been marked as committed so I > did that. Thanks. Here's a patch for the doc update which was mentioned as an open item upthread. -- Robert Haas EDB: http://www.enterprisedb.com upd

Re: pg_amcheck contrib application

2021-04-08 Thread Robert Haas
isn't any requirement that each possible > corruption we check actually be manifested in the regression tests. The > simplest solution is to remove this problematic test, so that's what I did. > The other two new tests corrupt c_va_toastrelid and c_va_rawsize, both of > which are read/written using unpack/pack, so perl should handle the > endianness for us (I hope). I don't immediately see why this particular thing should be an issue. The format of the varlena header itself is different on big-endian and little-endian machines, which is why postgres.h has all this stuff conditioned on WORDS_BIGENDIAN. But va_extinfo doesn't have any similar treatment, so I'm not sure what could go wrong there, as long as the 4-byte value as a whole is being packed and unpacked according to the machine's endian-ness. -- Robert Haas EDB: http://www.enterprisedb.com

Re: [HACKERS] Custom compression methods

2021-04-08 Thread Robert Haas
this patch would need to update parts about borrowing 2 spare bits, > but maybe that's the wrong header..before. We're not borrowing any more bits from the places where we were borrowing 2 bits before. We are borrowing 2 bits from places that don't seem to be discussed in detail here, w

Re: pg_amcheck contrib application

2021-04-08 Thread Robert Haas
but on the plus side, it avoids a mountain of output if the raw size has been overwritten with a gigantic bogus value. But, how is #2 different from #3? Those sound like the same thing to me. -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-04-08 Thread Robert Haas
On Thu, Apr 8, 2021 at 6:51 PM Mark Dilger wrote: > #2 is if chunk_seq goes up but skips numbers. #3 is if chunk_seq ever goes > down, meaning the index scan did something unexpected. Yeah, sure. But I think we could probably treat those the same way. -- Robert Haas EDB

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-09 Thread Robert Haas
them trace to this one bug. > > Pushed now, thanks. Does this need to worry about new partitions getting attached to a partitioned table, or old ones getting detached? (Maybe it does already, not sure.) -- Robert Haas EDB: http://www.enterprisedb.com

Re: pg_amcheck contrib application

2021-04-09 Thread Robert Haas
gt; Corruption #2: > > UPDATE $toastname SET chunk_seq = chunk_seq * 1000 Similarly here, except the extra chunk numbers are different. > Corruption #3: > > UPDATE $toastname SET chunk_id = (chunk_id::integer + 1000)::oid > WHERE chunk_seq = 3 And here we'd just get a complaint that chunk 3 is missing. -- Robert Haas EDB: http://www.enterprisedb.com

Re: More sepgsql weirdness

2021-04-13 Thread Robert Haas
the system is in permissive mode, but I think the behavior itself is deliberate. See OAT_NAMESPACE_SEARCH and commit e965e6344cfaff0708a032721b56f61eea777bc5. -- Robert Haas EDB: http://www.enterprisedb.com

Re: [PATCH] Identify LWLocks in tracepoints

2021-04-13 Thread Robert Haas
, and the block number. You can argue that we should just expose the buffer number and let the user sort out the rest with dtrace/systemtap magic, but that makes it useless in practice to an awful lot of people, including me. -- Robert Haas EDB: http://www.enterprisedb.com

Re: Default setting for enable_hashagg_disk

2020-07-22 Thread Robert Haas
ght we had evidence that this was a real problem, but if that's not the case, then I think we're fine as far as v13 goes. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: INSERT INTO SELECT, Why Parallelism is not selected?

2020-07-24 Thread Robert Haas
sday) unless > you have something more to add? Well, I think the comments could be more clear - for the insert case specifically - about which cases you think are and are not safe. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: [Patch] ALTER SYSTEM READ ONLY

2020-07-24 Thread Robert Haas
engineering, because the server would cache the state from the control file, and a bunch of blocks from the database. I guess it would work if you did a server restart afterward, but I think there are better ways of supporting online checksum enabling that don't require shutting down the serv

Re: [Patch] ALTER SYSTEM READ ONLY

2020-07-24 Thread Robert Haas
reak the feature for the originally intended use case. I'm not on board with that. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: [Patch] ALTER SYSTEM READ ONLY

2020-07-24 Thread Robert Haas
ate discussion, apart from this thread. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Issue with cancel_before_shmem_exit while searching to remove a particular registered exit callbacks

2020-07-24 Thread Robert Haas
. I guess the other question is why we're doing it at all. What resources are being allocated that wouldn't be freed up by process exit anyway? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: HOT vs freezing issue causing "cannot freeze committed xmax"

2020-07-24 Thread Robert Haas
d beyond whatever HOT-pruning already did, if it's necessary. A better solution would probably be to merge HOT-pruning with setting things all-visible and have a single function that does both, but that seems a lot more invasive, and definitely unsuitable for back-patching. -- Robert Haas

Re: Default setting for enable_hashagg_disk

2020-07-24 Thread Robert Haas
think, but this seems like the key finding. It seems like we don't really understand what's actually getting written. Whether we use hash or sort doesn't seem like it should have this kind of impact on how much data gets written, and whether we use CP_SMALL_TLIST or project when needed

Re: [HACKERS] Custom compression methods

2020-07-24 Thread Robert Haas
eund > > The "plgz performance" thread - vintage 2019 - Petr Jelinek > > > > Anyone want to point to a FOURTH implementation of this feature? > > To be clear, I don't think the 2003 patch should be considered as being > "in the running". I gue

Re: Making CASE error handling less surprising

2020-07-24 Thread Robert Haas
the previous one. How would we even document it? Sometimes things get inlined, sometimes they don't. Sometimes subqueries get pulled up, sometimes not. The current behavior isn't great, but at least it handles these cases consistently. Getting the easy cases "right" while making the behavior in more complex cases harder to understand is not necessarily a win. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Mark unconditionally-safe implicit coercions as leakproof

2020-07-24 Thread Robert Haas
s the > infrastructure for char(n) to text coercion. It looks like it > actually does qualify right now, but the code is long enough and > complex enough that I think such a marking would be a bit unsafe. > > Any objections? IMHO, this is a nice improvement. -- Robert Haas Enterprise

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-29 Thread Robert Haas
On Wed, Jul 29, 2020 at 3:23 AM Thomas Munro wrote: > On Tue, Jul 14, 2020 at 9:12 AM Robert Haas wrote: > > A number of EDB customers have had this error crop on their tables for > > reasons that we have usually not been able to determine. In many > > Do you happen to know

Re: INSERT INTO SELECT, Why Parallelism is not selected?

2020-07-29 Thread Robert Haas
don't know whether this is talking about two processes operating on the same page at the same time, or ever within a single query execution. If it's the former, perhaps we need to explain why that's a concern for parallel query but not otherwise; if it's the latter, that seem

Re: Making CASE error handling less surprising

2020-07-29 Thread Robert Haas
r this kind of thing is a complete non-starter for that reason. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Making CASE error handling less surprising

2020-07-29 Thread Robert Haas
ty of different problems depending on which release they are running. IMHO, changing the semantics of something like this is really scary and should be approached with great caution. You don't have to deny that something is a problem in order to admit that you might not have a perfect soluti

Re: [Patch] ALTER SYSTEM READ ONLY

2020-07-29 Thread Robert Haas
rite any data files or any WAL or even the control file. It would be useful for poking around on damaged clusters without making things worse. And it's somewhat related to the topic of this thread, but it's not THAT closely related. It's better to add features one at a time; you ca

Re: Online checksums patch - once again

2020-07-29 Thread Robert Haas
#x27;ve processed interrupts again, but that seems like a less-likely thing for somebody to do. If, on the other hand, there are stretches of code that fetch this value outside of a crit section and without interrupts held, then we need some other kind of mechanism here to make it safe. We have to

Re: new heapcheck contrib module

2020-07-29 Thread Robert Haas
sert/update/delete activity on the table altogether. Maybe I'm just being dense here -- exactly what problem are you worried about? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Issue with cancel_before_shmem_exit while searching to remove a particular registered exit callbacks

2020-07-30 Thread Robert Haas
On Mon, Jul 20, 2020 at 3:47 PM Robert Haas wrote: > But I also agree that what pg_start_backup() was doing before v13 was > wrong; that's why I committed > 303640199d0436c5e7acdf50b837a027b5726594. The only reason I didn't > back-patch it is because the consequences are s

Re: new heapcheck contrib module

2020-07-30 Thread Robert Haas
e extra braces. - bt_index_check | relname | relpages + bt_index_check | relname | relpages Don't include unrelated changes in the patch. I'm not really sure that the list of fields you're displaying for each reported problem really makes sense. I think the theory here should be that we want to report the information that the user needs to localize the problem but not everything that they could find out from inspecting the page, and not things that are too specific to particular classes of errors. So I would vote for keeping blkno, offnum, and attnum, but I would lose lp_flags, lp_len, and chunk. lp_off feels like it's a more arguable case: technically, it's a locator for the problem, because it gives you the byte offset within the page, but normally we reference tuples by TID, i.e. (blkno, offset), not byte offset. On balance I'd be inclined to omit it. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Threading in BGWorkers (!)

2020-07-30 Thread Robert Haas
ms likely to hurt PostgreSQL's reputation for reliability with no compensating benefit. Or alternatively they hack core, which sucks, or they switch to some non-PG database, which also sucks. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: [Proposal] Global temporary tables

2020-07-30 Thread Robert Haas
plaining also what that means in terms of features and limitations. Without that, it's really hard for anyone to jump into reviewing this code, and it will be hard for people who have to maintain it in the future to understand it, either. Or for users, for that matter. -- Robert Haas

Re: new heapcheck contrib module

2020-07-30 Thread Robert Haas
XIDs are >= relfrozenxid and <= ReadNewFullTransactionId(), then this shouldn't be a problem. It could be, if CLOG is hosed, which is possible, because if the table is corrupted, why shouldn't CLOG also be corrupted? But I'm not sure that's what your concern is here. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: new heapcheck contrib module

2020-07-30 Thread Robert Haas
vacuum bug seems strange to me. Why would the right solution not be to fix such a bug if and when we find that there is one? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: new heapcheck contrib module

2020-07-31 Thread Robert Haas
On Thu, Jul 30, 2020 at 9:38 PM Mark Dilger wrote: > > On Jul 30, 2020, at 5:53 PM, Robert Haas wrote: > > On Thu, Jul 30, 2020 at 6:10 PM Mark Dilger > Since I can't think of a plausible concrete example of corruption which would > elicit the problem I was worrying a

Re: Why is pq_begintypsend so slow?

2020-07-31 Thread Robert Haas
y that, but you have to be able to get the right amount of data into the right place in the StringInfo right from the start. I think that in some cases that will require fairly complex orchestration. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Why is pq_begintypsend so slow?

2020-07-31 Thread Robert Haas
0002: Totally mechanical, seems fine. Regarding 0003: For the same reasons as above, I suggest renaming pq_begintypsend_ex() to pq_begintypsend_extended() or pq_begintypsend_with_length() or something of that sort, rather than using _ex. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Why is pq_begintypsend so slow?

2020-07-31 Thread Robert Haas
= str->data+str->len; ep[0] = ch; ep[1] = '\0'; ++str->len; Whether or not str itself is marked restricted is another question; what I'm talking about is why we need to repeat (char *pg_restrict) (str->data + str->len). I don't have any further comment on the remainder of your reply. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: new heapcheck contrib module

2020-07-31 Thread Robert Haas
e holding a ShareLock on the buffer, this seems impossible. What's the flaw in this argument? If we do need to do something special in terms of advertising xmin, how would you do it? Normally it happens by registering a snapshot, but here all we would have is an XID; specifically, the value

Re: Why is pq_begintypsend so slow?

2020-07-31 Thread Robert Haas
few, it might be better to have a more specific explanation at the point of use, as it may be easier to understand in specific cases than in general. I imagine this only really makes sense for places that are pretty hot. > Ah, I misunderstood. Yea, there's no reason not to do that. OK, then

Re: refactoring basebackup.c

2020-07-31 Thread Robert Haas
en uploading > to S3, even when not having pre-calculated the total size of the data > directory. I don't mind a separate message type here, but if you want merging of short messages with adjacent longer messages to generate a minimal number of system calls, that might have some implica

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-31 Thread Robert Haas
that it makes it easier for translators. I see very little point in what verify_tid() is doing. Before using each block number, we should check that it's less than or equal to a cached value of RelationGetNumberOfBlocks(rel). That's necessary in any case to avoid funny errors; and then the check here against specifically InvalidBlockNumber is redundant. For the offset number, same thing: we need to check each offset against the page's PageGetMaxOffsetNumber(page); and if we do that then we don't need these checks. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: new heapcheck contrib module

2020-07-31 Thread Robert Haas
rozenxid that we observed. > > An appropriate procarray or snapmgr function would probably suffice? Not sure; I guess that'll need some investigation. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Creating a function for exposing memory usage of backend process

2020-07-31 Thread Robert Haas
e do, it might make those bugs easier to exploit. I don't think this information is useful enough to justify taking that risk. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: ModifyTable overheads in generic plans

2020-07-31 Thread Robert Haas
On Fri, Jun 26, 2020 at 8:36 AM Amit Langote wrote: > 0001 and 0002 are preparatory patches. I read through these patches a bit but it's really unclear what the point of them is. I think they need better commit messages, or better comments, or both. -- Robert Haas Enterprise

Re: COPY FREEZE and setting PD_ALL_VISIBLE/visibility map bits

2020-07-31 Thread Robert Haas
't see any reason why this can't be made to work. It needs substantially more polishing than it's had, though, I think. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-08-03 Thread Robert Haas
g_newpage_buffer() are skipped. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: public schema default ACL

2020-08-03 Thread Robert Haas
the idea of having an automatic schema creation option. I think that would be quite a cool thing to have, whether it's the default (Y) or not (Z). But I don't know how to choose between (1), (2), and (3). -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Allow ERROR from heap_prepare_freeze_tuple to be downgraded to WARNING

2020-08-03 Thread Robert Haas
an to the rest of the table is the right thing to do. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: new heapcheck contrib module

2020-08-03 Thread Robert Haas
endorse that approach, or anything like it. But neither do I accept the argument that it would be useless to report all the errors even if we could do so safely. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: new heapcheck contrib module

2020-08-03 Thread Robert Haas
sn't even matter at all, because all the code in which this is used will be new code. Even if we were churning existing code, mechanical stuff like this isn't really a huge problem most of the time, but there's no need for that here. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Reduce/eliminate the impact of FPW

2020-08-03 Thread Robert Haas
ly possible that it's just always a loser. I don't really know. It seems like a very difficult project. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Confusing behavior of create table like

2020-08-03 Thread Robert Haas
e's every possibility that there are people out there who rely on the current behavior, and whose stuff would break if it were changed. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: FailedAssertion("pd_idx == pinfo->nparts", File: "execPartition.c", Line: 1689)

2020-08-03 Thread Robert Haas
On Sun, Aug 2, 2020 at 2:11 PM Justin Pryzby wrote: > Based on commit logs, I suspect this may be an "older bug", specifically maybe > with: > > |commit 898e5e3290a72d288923260143930fb32036c00c > |Author: Robert Haas > |Date: Thu Mar 7 11:13:12 2019 -0500 > | >

Re: Can a background worker exist without shared memory access for EXEC_BACKEND cases?

2020-08-03 Thread Robert Haas
o trying to clarify the documentation and/or comments here, but "global state" is a fuzzy term that doesn't really mean anything to me. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: new heapcheck contrib module

2020-08-04 Thread Robert Haas
rom the root, by doing a pass after the fact over all the pages that you didn't otherwise reach. It would be a lot of work to build something like that and maybe not the best use of time, but if I got to wave tools into existence using my magic wand, I think that would be the gold standard. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: new heapcheck contrib module

2020-08-04 Thread Robert Haas
, I'm thinking that (2) and (4) are just bad and so we ought to either do (3) if it doesn't suck too much for performance (which I don't quite see why it should, but it might) or else fall back on (1). (1) doesn't feel clever enough but it might be better to be not clever enough than to be too clever. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: new heapcheck contrib module

2020-08-04 Thread Robert Haas
sh or worth the time it took. I'm just trying to sketch out what I see as maybe being theoretically possible, and why I think it would be useful if it did. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Can a background worker exist without shared memory access for EXEC_BACKEND cases?

2020-08-04 Thread Robert Haas
: IIUC, we start with shared memory access, then afterwards detach. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: Confusing behavior of create table like

2020-08-04 Thread Robert Haas
ly convinced that's a good idea. There's probably a lot of people (me included) who are used to the way serial and bigserial work and wouldn't necessarily be happy about a change. Plus, aren't the generated columns still depending on an underlying sequence anyway? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2020-08-04 Thread Robert Haas
tion bounds? I think the two-transaction approach is interesting and I can imagine that it possibly solves some problems, but it's not clear to me exactly which problems it solves or how it does so. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: FailedAssertion("pd_idx == pinfo->nparts", File: "execPartition.c", Line: 1689)

2020-08-04 Thread Robert Haas
the beginning and 22628862 at the end, and also in that a 0 has been removed. This code can't cope with things getting removed, so kaboom. I think Amit probably has the right idea about what's going on here and how to fix it, but I haven't yet had time to study it in detail. --

<    1   2   3   4   5   6   7   8   9   10   >