Using operators to do query hints

2022-02-22 Thread Greg Stark
I've been playing with an idea I had a while back. Basically that it would be useful to have some "noop" operators that are used purely to influence the planner. For context I've suggested in the past that there are two categories of hints: 1 Hints that override the planner's decisions with expli

Re: Design of pg_stat_subscription_workers vs pgstats

2022-02-25 Thread Greg Stark
On Tue, 25 Jan 2022 at 01:32, Andres Freund wrote: > > Hi, > > I was looking the shared memory stats patch again. Can you point me to this thread? I looked for it but couldn't find it. -- greg

Re: Commitfest manager for 2022-03

2022-02-25 Thread Greg Stark
I would like to volunteer. On Fri, 25 Feb 2022 at 05:31, Julien Rouhaud wrote: > > Hi, > > The final commitfest for pg15 will start in a few days, and I didn't see any > discussion on it or anyone volunteering to be a CFM. > > I thought it would be a good idea to send this reminder now and avoid

Re: Commitfest manager for 2022-03

2022-02-26 Thread Greg Stark
On Sat, 26 Feb 2022 at 01:33, Julien Rouhaud wrote: > > On Sat, Feb 26, 2022 at 02:42:33PM +0900, Michael Paquier wrote: > > On Fri, Feb 25, 2022 at 01:58:55PM -0600, David Steele wrote: > > > On 2/25/22 12:39, Greg Stark wrote: > > >> I would like to volu

CommitFest begins tomorrow... Get your patches in

2022-02-28 Thread Greg Stark
28 days has February ... So the March commitfest begins tomorrow. Meet your friendly neighbourhood Commitfest Manager for March. Greetings! If you have a patch you're hoping to get feedback on or you're expecting to get committed this month make sure it's in the commitfest at https://commitfest.p

Re: Removing unneeded self joins

2022-02-28 Thread Greg Stark
On Thu, 1 Jul 2021 at 02:38, Ronan Dunklau wrote: > > Well in some cases they can't, when the query is not emitting redundant > predicates by itself but they are added by something else like a view or a RLS > policy. > Maybe it would be worth it to allow spending a bit more time planning for > tho

Re: Removing unneeded self joins

2022-02-28 Thread Greg Stark
I don't think the benchmarking that's needed is to check whether pruning unnecessary joins is helpful. Obviously it's going to be hard to measure on simple queries and small tables. But the resulting plan is unambiguously superior and in more complex cases could extra i/o. The benchmarking people

Commitfest 2022-03 Patch Triage Part 1a.i

2022-03-01 Thread Greg Stark
Last November Daniel Gustafsson did a patch triage. It took him three emails to get through the patches in the commitfest back then. Since then we've had the November and the January commitfests so I was interested to see how many of these patches had advanced I'm only part way through the fi

Commitfest 2022-03 Starts Now

2022-03-01 Thread Greg Stark
The final commitfest of this release begins now. Whereas most commitfests are about getting feedback to authors so they can advance the patch -- this one is about actually committing patches to wrap up the release. Please when reviewing patches try to push yourself to make the difficult call abou

Re: Commitfest 2022-03 Patch Triage Part 1a.i

2022-03-01 Thread Greg Stark
As Justin suggested I CC the authors from these patches I'm adding them here. Some of the patches have multiple "Authors" listed in the commitfest which may just be people who posted updated patches so I may have added more people than necessary. [If you received two copies of this do not reply to

Re: Commitfest 2022-03 Patch Triage Part 1a.i

2022-03-01 Thread Greg Stark
On Tue, 1 Mar 2022 at 14:59, Greg Stark wrote: > > As Justin suggested I CC the authors from these patches I'm adding > them here. Some of the patches have multiple "Authors" listed in the > commitfest which may just be people who posted updated patches so I > m

Commitfest 2022-03 Patch Triage Part 1b

2022-03-01 Thread Greg Stark
> 2096: psql - add SHOW_ALL_RESULTS option > > Peter posted an updated version of Fabiens patch about a month ago (which at > this point no longer applies) fixing a few issues, but also point at old > review > comments still unaddressed. Since this was pus

Re: Commitfest 2022-03 Patch Triage Part 1a.i

2022-03-02 Thread Greg Stark
On Wed, 2 Mar 2022 at 07:12, Daniel Gustafsson wrote: > Thanks for picking it up and continuing with recent developments. Let me know > if you want a hand in triaging patchsets. While I have the time there may be patches I may need help coming to the right conclusions about what actions to take

Re: Commitfest 2022-03 Patch Triage Part 1b

2022-03-03 Thread Greg Stark
Just FYI. Better to follow up to the thread for the patch that's already in the CF. Otherwise your patch will missed by someone who looks at the CF entry to see the latest patch.

Re: [Proposal] Global temporary tables

2022-03-03 Thread Greg Stark
It doesn't look like this is going to get committed this release cycle. I understand more feedback could be valuable, especially on the overall design, but as this is the last commitfest of the release we should focus on other patches for now and spend that time in the next release cycle. I'm goin

Re: [PATCH] minor reloption regression tests improvement

2022-03-07 Thread Greg Stark
I don't think this is worth spending time adding tests for. I get what you're saying that this is at least semi-intentional behaviour and desirable behaviour so it should have tests ensuring that it continues to work. But it's not documented behaviour and the test is basically testing that the impl

Re: Commitfest 2022-03 One Week in. 3 Commits 213 Patches Remaining

2022-03-09 Thread Greg Stark
So it's 8 days into the commitfest. So far 3 patches have been committed: * parse/analyze API refactoring by Peter Eisentraut * FUNCAPI tuplestore helper function by Melanie Plagemen committed by Michael Paquier * Typo in pgbench messages by Kawamoto Masay committed by Tatsuo Ishii (There was als

Re: Commitfest 2022-03 One Week in. 3 Commits 213 Patches Remaining

2022-03-09 Thread Greg Stark
On Wed, 9 Mar 2022 at 15:44, David Steele wrote: > > On 3/9/22 13:38, Greg Stark wrote: > Should I do a round-robin style assignment for any of these? > > I don't think this is a good idea. Committers pick the patches they are > going to commit. > > What prefer to d

Re: Commitfest 2022-03 One Week in. 3 Commits 213 Patches Remaining

2022-03-09 Thread Greg Stark
On Wed, 9 Mar 2022 at 16:46, Greg Stark wrote: > Many of them seem to mostly have gotten > feedback from committers already and the type of feedback that leads > me to think it's ready for commit. Er. I meant *not* the type of feedback that leads me to think it's ready for c

Re: Tablesync early exit

2022-03-15 Thread Greg Stark
This patch has been through five CFs without any review. It's a very short patch and I'm assuming the only issue is that it's not clear whether it's a good idea or not and there are few developers familiar with this area of code? Amit, it looks like you were the one who asked for it to be split of

Re: [PATCH] pgbench: add multiconnect option

2022-03-15 Thread Greg Stark
Hi guys, It looks like David sent a patch and Fabien sent a followup patch. But there hasn't been a whole lot of discussion or further patches. It sounds like there are some basic questions about what the right interface should be. Are there specific questions that would be helpful for moving for

Re: Window Function "Run Conditions"

2022-03-15 Thread Greg Stark
This looks like an awesome addition. I have one technical questions... Is it possible to actually transform the row_number case into a LIMIT clause or make the planner support for this case equivalent to it (in which case we can replace the LIMIT clause planning to transform into a window functio

Re: Skip partition tuple routing with constant partition key

2022-03-15 Thread Greg Stark
There are a whole lot of different patches in this thread. However this last one https://commitfest.postgresql.org/37/3270/ created by Amit seems like a fairly straightforward optimization that can be evaluated on its own separately from the others and seems quite mature. I'm actually inclined to

Commitfest Update

2022-03-17 Thread Greg Stark
So far this commitfest these 10 patches have been marked committed. That leaves us with 175 "Needs Review" and 28 "Ready for Comitter" so quite a ways to go ... * FUNCAPI tuplestore helper function * parse/analyze API refactoring * Add wal_compression=zstd * Add id's to various elements in protoco

Re: Proposal for internal Numeric to Uint64 conversion function.

2022-03-17 Thread Greg Stark
On Fri, 11 Mar 2022 at 15:17, Tom Lane wrote: > > Amul Sul writes: > > Yeah, that's true, I am too not sure if we really need to refactor > > all those; If we want, I can give it a try. > > The patch as-presented isn't very compelling for > lack of callers of the new function Tom, are you say

Re: proposal: enhancing plpgsql debug API - returns text value of variable content

2022-03-17 Thread Greg Stark
It looks like this is -- like a lot of plpgsql patches -- having difficulty catching the attention of reviewers and committers. Aleksander asked for a test and Pavel put quite a bit of work into adding a good test case. I actually like that there's a test because it shows the API can be used effect

Re: jsonpath syntax extensions

2022-03-21 Thread Greg Stark
This patch seems to be getting ignored. Like David I'm a bit puzzled because it doesn't seem like an especially obscure or difficult patch to review. Yet it's been multiple years without even a superficial "does it meet the coding requirements" review let alone a design review. Can we get a volunt

Re: jsonpath syntax extensions

2022-03-21 Thread Greg Stark
Hm. Actually... These changes were split off from the JSON_TABLE patches? Are they still separate or have they been merged into those other patches since? I see the JSON_TABLE thread is getting more comments do those reviews include these patches? On Mon, 21 Mar 2022 at 16:09, Greg Stark wrote

Re: Temporary tables versus wraparound... again

2022-03-21 Thread Greg Stark
No problem, I can update the patch and check on the fuzz. But the actual conflict is just in the test and I'm not sure it's really worth having a test at all. It's testing a pretty low level detail. So I'm leaning toward fixing the conflict by just ripping the test out. Nathan also pointed out th

Re: Commitfest manager for 2022-03

2022-03-21 Thread Greg Stark
On Mon, 21 Mar 2022 at 21:48, Andres Freund wrote: > > Hi, > > On 2022-02-26 16:12:27 -0500, Greg Stark wrote: > > I do have the time available. What I don't necessarily have is insight > > into everything that needs to be done, especially behind the scenes. > &g

Re: Temporary tables versus wraparound... again

2022-03-22 Thread Greg Stark
is autovacuum.c). But just in case I renamed it to protect against any external modules failing from the added parameter. From eb6ec2edfcb10aafc3874262276638932a97add7 Mon Sep 17 00:00:00 2001 From: Greg Stark Date: Tue, 22 Mar 2022 15:54:59 -0400 Subject: [PATCH v3 2/2] Add test for truncat

Re: pg13.2: invalid memory alloc request size NNNN

2021-02-13 Thread Greg Stark
I think to get a size of -4 you would be trying to read a varlena pointer pointing to four nul bytes. I bet if you run dd on the corresponding block you'll find a chunk of nuls in the page. That perhaps makes sense with ZFS where if a new page was linked to the tree but never written it would be an

Re: Asynchronous and "direct" IO support for PostgreSQL.

2021-02-23 Thread Greg Stark
On Tue, 23 Feb 2021 at 05:04, Andres Freund wrote: > > ## Callbacks > > In the core AIO pieces there are two different types of callbacks at the > moment: > > Shared callbacks, which can be invoked by any backend (normally the issuing > backend / the AIO workers, but can be other backends if they

Re: Asynchronous and "direct" IO support for PostgreSQL.

2021-02-24 Thread Greg Stark
I guess what I would be looking for in stats would be a way to tell what the bandwidth, latency, and queue depth is. Ideally one day broken down by relation/index and pg_stat_statement record. I think seeing the actual in flight async requests in a connection is probably not going to be very usefu

Re: SSL SNI

2021-02-25 Thread Greg Stark
Hate to be that guy but This still doesn't seem like it is IPv6-ready. Is there any harm in having SNI with an IPv6 address there if it gets through?

Re: SSL SNI

2021-02-26 Thread Greg Stark
> Do you mean the IPv6 detection code is not correct? What is the problem? This bit, will recognize ipv4 addresses but not ipv6 addresses: + /* + * Set Server Name Indication (SNI), but not if it's a literal IP address. + * (RFC 6066) + */ + if (!(strspn(conn->pghost, "0123456789.") == strlen(co

Temporary tables versus wraparound... again

2020-11-08 Thread Greg Stark
x27;s a parallel transaction going on and even moving the test to the serial schedule might not guarantee that never happens due to autovacuum running analyze? I didn't actually add the warning to autovacuum yet. -- greg From 76eb00c43fba2a293dc4a079307e675e0eeaff06 Mon Sep 17 00:00:00 2001

Re: Temporary tables versus wraparound... again

2020-11-08 Thread Greg Stark
On Mon, 9 Nov 2020 at 00:17, Noah Misch wrote: > > > 2) adding the dependency on heapam.h to heap.c makes sense because of > > heap_inplace_update bt it may be a bit annoying because I suspect > > that's a useful sanity check that the tableam stuff hasn't been > > bypassed > > That is not terrible

Re: don't allocate HashAgg hash tables when running explain only

2020-11-18 Thread Greg Stark
On Wed, 18 Nov 2020 at 05:40, Heikki Linnakangas wrote: > > On 13/11/2020 18:10, Alexey Bashtanov wrote: >> > I would appreciate if someone could have a look at the patch attached, > > which makes executor skip initializing hash tables when doing explain only. > > Makes sense. Committed, thanks fo

Re: Is postgres ready for 2038?

2020-11-18 Thread Greg Stark
On Wed, 18 Nov 2020 at 12:22, Tom Lane wrote: > > Andrew Dunstan writes: > > On 11/18/20 9:44 AM, Tom Lane wrote: > >> Hmm. Digging around, I see that Mkvcbuild.pm intentionally absorbs > >> _USE_32BIT_TIME_T when building with a Perl that defines that. > >> I don't know what the state of play i

Re: Proposal: Save user's original authenticated identity for logging

2021-01-31 Thread Greg Stark
On Fri, 29 Jan 2021 at 18:41, Tom Lane wrote: > > Ah. So basically, this comes into play when you consider that some > outside-the-database entity is your "real" authenticated identity. > That seems reasonable when using Kerberos or the like, though it's > not real meaningful for traditional pass

Re: [PATCH] Document heuristics for parameterized paths

2021-12-15 Thread Greg Stark
On Mon, 6 Dec 2021 at 13:01, Steinar H. Gunderson wrote: > > +one that must cannot be delayed right away (because of outer join must cannot? -- greg

Re: [PATCH] Document heuristics for parameterized paths

2021-12-15 Thread Greg Stark
On Wed, 15 Dec 2021 at 23:22, Greg Stark wrote: > > On Mon, 6 Dec 2021 at 13:01, Steinar H. Gunderson > wrote: > > > > +one that must cannot be delayed right away (because of outer join > > must cannot? Actually on further reading... "delayed right away"? -- greg

Re: Transparent column encryption

2021-12-15 Thread Greg Stark
> In the server, the encrypted datums are stored in types called > encryptedr and encryptedd (for randomized and deterministic > encryption). These are essentially cousins of bytea. Does that mean someone could go in with psql and select out the data without any keys and just get a raw bytea-like

Re: Documenting when to retry on serialization failure

2021-12-15 Thread Greg Stark
Fwiw I think the real problem with automatic retries is that the SQL interface doesn't lend itself to it because the server never really knows if the command is going to be followed by a commit or more commands. I actually think if that problem were tackled it would very likely be a highly appreci

Re: Add index scan progress to pg_stat_progress_vacuum

2021-12-16 Thread Greg Stark
I had a similar question. And I'm still not clear from the response what exactly index_blks_total is and whether it addresses it. I think I agree that a user is likely to want to see the progress in a way they can understand which means for a single index at a time. I think what you're describing

Re: WIP: WAL prefetch (another approach)

2021-12-16 Thread Greg Stark
On Fri, 26 Nov 2021 at 21:47, Tom Lane wrote: > > Yeah ... on the one hand, that machine has shown signs of > hard-to-reproduce flakiness, so it's easy to write off the failures > I saw as hardware issues. On the other hand, the flakiness I've > seen has otherwise manifested as kernel crashes, wh

Re: WIP: WAL prefetch (another approach)

2021-12-16 Thread Greg Stark
The actual hardware of this machine is a Mac Mini Core 2 Duo. I'm not really clear how the emulation is done and whether it makes a reasonable test environment or not. Hardware Overview: Model Name: Mac mini Model Identifier: Macmini2,1 Processor Name: Intel Core 2 Duo

Re: Allow DELETE to use ORDER BY and LIMIT/OFFSET

2021-12-16 Thread Greg Stark
On Thu, 16 Dec 2021 at 22:18, Tom Lane wrote: > > * If the sort order is underspecified, or you omit ORDER BY > entirely, then it's not clear which rows will be operated on. > The LIMIT might stop after just some of the rows in a peer > group, and you can't predict which ones. Meh, that never see

Re: pg_dump versus ancient server versions

2021-12-17 Thread Greg Stark
On Fri, 22 Oct 2021 at 19:27, Robert Haas wrote: > > Another thing to think about in that regard: how likely is that > PostgreSQL 7.4 and PostgreSQL 15 both compile and run on the same > operating system? I suspect the answer is "not very." I seem to recall > Greg Stark

Re: WIP: WAL prefetch (another approach)

2021-12-17 Thread Greg Stark
What tools and tool versions are you using to build? Is it just GCC for PPC? There aren't any special build processes to make a fat binary involved? On Thu, 16 Dec 2021 at 23:11, Tom Lane wrote: > > Greg Stark writes: > > But if you're interested and can explain the tes

Re: WIP: WAL prefetch (another approach)

2021-12-17 Thread Greg Stark
I have IBUILD:postgresql gsstark$ ls /usr/bin/*gcc* /usr/bin/gcc /usr/bin/gcc-4.0 /usr/bin/gcc-4.2 /usr/bin/i686-apple-darwin9-gcc-4.0.1 /usr/bin/i686-apple-darwin9-gcc-4.2.1 /usr/bin/powerpc-apple-darwin9-gcc-4.0.1 /usr/bin/powerpc-apple-darwin9-gcc-4.2.1 I'm guessing I should do CC=/usr/bin/pow

Re: WIP: WAL prefetch (another approach)

2021-12-17 Thread Greg Stark
Hm. I seem to have picked a bad checkout. I took the last one before the revert (45aa88fe1d4028ea50ba7d26d390223b6ef78acc). Or there's some incompatibility with the emulation and the IPC stuff parallel workers use. 2021-12-17 17:51:51.688 EST [50955] LOG: background worker "parallel worker" (PID

Re: WIP: WAL prefetch (another approach)

2021-12-17 Thread Greg Stark
On Fri, 17 Dec 2021 at 18:40, Tom Lane wrote: > > Greg Stark writes: > > Hm. I seem to have picked a bad checkout. I took the last one before > > the revert (45aa88fe1d4028ea50ba7d26d390223b6ef78acc). > > FWIW, I think that's the first one *after* the revert. Do

Re: Getting rid of regression test input/ and output/ files

2021-12-20 Thread Greg Stark
On Sun, 19 Dec 2021 at 18:41, Corey Huinker wrote: > > Which brings up a tangential question, is there value in having something > that brings in one or more env vars as psql vars directly. I'm thinking > something like: > > \importenv pattern [prefix] Oof. That gives me the security heebie jee

Re: Unifying VACUUM VERBOSE and log_autovacuum_min_duration output

2021-12-21 Thread Greg Stark
I haven't read the patch yet. But some thoughts based on the posted output: 1) At first I was quite skeptical about losing the progress reporting. I've often found it quite useful. But looking at the examples I'm convinced. Or rather I think a better way to look at it is that the progress output

Re: new release pspg

2021-03-20 Thread Greg Stark
This is really cool. Now I just need to figure out how to integrate it with using Emacs for my terminal. I still want to use emacs enter and edit my queries but it would be cool to be able to hit a key and launch an xterm and send the query output to pspg

Re: fdatasync performance problem with large number of DB files

2021-03-21 Thread Greg Stark
On Wed, 10 Mar 2021 at 20:25, Tom Lane wrote: > > So this means that in less-than-bleeding-edge kernels, syncfs can > only be regarded as a dangerous toy. If we expose an option to use > it, there had better be large blinking warnings in the docs. Isn't that true for fsync and everything else re

Re: New IndexAM API controlling index vacuum strategies

2021-03-21 Thread Greg Stark
On Thu, 18 Mar 2021 at 14:37, Peter Geoghegan wrote: > They usually involve some *combination* of Postgres problems, > application code problems, and DBA error. Not any one thing. I've seen > problems with application code that runs DDL at scheduled intervals, > which interacts badly with vacuum

Re: shared memory stats: high level design decisions: consistency, dropping

2021-03-22 Thread Greg Stark
On Sun, 21 Mar 2021 at 18:16, Stephen Frost wrote: > > Greetings, > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > > I also believe that the snapshotting behavior has advantages in terms > > of being able to perform multiple successive queries and get consistent > > results from them. Only the most t

Re: Temporary tables versus wraparound... again

2021-10-12 Thread Greg Stark
st moving the work around. Just removing that optimization might be the easiest way to close this hole. The only alternative I see is adding a flag to PROC or somewhere where autovacuum can see if the backend has actually initialized the temporary schema yet or not. From 1315daf80e668b03cf2aab04106fe53

Re: Extending amcheck to check toast size and compression

2021-10-19 Thread Greg Stark
Right so here's a review. I think the patch is committable as is. It's an improvement and it does the job as promised. I do have some comments but I don't think they're serious issues and would actually be pretty happy committing it as is. Fwiw I didn't realize how short the patch was at first and

Re: Extending amcheck to check toast size and compression

2021-10-20 Thread Greg Stark
On Wed., Oct. 20, 2021, 12:41 Mark Dilger, wrote: > > I used a switch statement to trigger a compiler warning in such an event. > Catching better compiler diagnostics is an excellent reason to choose this structure. I guess all I could ask is that the comment saying no default branch say this is

Thinking about ANALYZE stats and autovacuum and large non-uniform tables

2021-10-21 Thread Greg Stark
One problem I've seen in multiple databases and is when a table has a mixture of data sets within it. E.g. A queue table where 99% of the entries are "done" but most queries are working with the 1% that are "new" or in other states. Often the statistics are skewed by the "done" entries and give bad

Re: How to retain lesser paths at add_path()?

2021-10-23 Thread Greg Stark
On Wed, 31 Jul 2019 at 11:45, Robert Haas wrote: > > On Wed, Jul 31, 2019 at 11:07 AM Tom Lane wrote: > > What you'd want to do for something like the above, I think, is to > > have some kind of figure of merit or other special marking for paths > > that will have some possible special advantage

hot_standby_feedback vs excludeVacuum and snapshots

2018-03-29 Thread Greg Stark
I'm poking around to see debug a vacuuming problem and wondering if I've found something more serious. As far as I can tell the snapshots on HOT standby are built using a list of running xids that the primary builds and puts in the WAL and that seems to include all xids from transactions running i

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-03 Thread Greg Stark
On 3 April 2018 at 11:35, Anthony Iliopoulos wrote: > Hi Robert, > > Fully agree, and the errseq_t fixes have dealt exactly with the issue > of making sure that the error is reported to all file descriptors that > *happen to be open at the time of error*. But I think one would have a > hard time d

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-03 Thread Greg Stark
On 3 April 2018 at 14:36, Anthony Iliopoulos wrote: > If EIO persists between invocations until explicitly cleared, a process > cannot possibly make any decision as to if it should clear the error I still don't understand what "clear the error" means here. The writes still haven't been written o

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-08 Thread Greg Stark
On 8 April 2018 at 04:27, Craig Ringer wrote: > On 8 April 2018 at 10:16, Thomas Munro > wrote: > > If the kernel does writeback in the middle, how on earth is it supposed to > know we expect to reopen the file and check back later? > > Should it just remember "this file had an error" forever, an

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-09 Thread Greg Stark
On 8 April 2018 at 22:47, Anthony Iliopoulos wrote: > On Sun, Apr 08, 2018 at 10:23:21PM +0100, Greg Stark wrote: >> On 8 April 2018 at 04:27, Craig Ringer wrote: >> > On 8 April 2018 at 10:16, Thomas Munro > > The question is, what should the kernel and application do

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-09 Thread Greg Stark
On 9 April 2018 at 15:22, Anthony Iliopoulos wrote: > On Mon, Apr 09, 2018 at 03:33:18PM +0200, Tomas Vondra wrote: >> > Sure, there could be knobs for limiting how much memory such "zombie" > pages may occupy. Not sure how helpful it would be in the long run > since this tends to be highly applic

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-10 Thread Greg Stark
On 9 April 2018 at 11:50, Anthony Iliopoulos wrote: > On Mon, Apr 09, 2018 at 09:45:40AM +0100, Greg Stark wrote: >> On 8 April 2018 at 22:47, Anthony Iliopoulos wrote: > To make things a bit simpler, let us focus on EIO for the moment. > The contract between the block layer and

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-10 Thread Greg Stark
On 10 April 2018 at 02:59, Craig Ringer wrote: > Nitpick: In most cases the kernel reserves disk space immediately, > before returning from write(). NFS seems to be the main exception > here. I'm kind of puzzled by this. Surely NFS servers store the data in the filesystem using write(2) or the i

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-11 Thread Greg Stark
On 10 April 2018 at 19:58, Joshua D. Drake wrote: > You can't unmount the file system --- that requires writing out all of the > pages > such that the dirty bit is turned off. I always wondered why Linux didn't implement umount -f. It's been in BSD since forever and it's a major annoyance that i

Partitioning versus autovacuum

2019-09-30 Thread Greg Stark
So we now support `ANALYZE partitioned_table` which will gather statistics for the main table by gathering stats from all the partitions. However as far as I can tell autovacuum will never actually trigger this analyze. Because we never generate any update records for the parent table in the stati

Re: Partitioning versus autovacuum

2019-09-30 Thread Greg Stark
Actually I did just find it in the To-do wiki: Have autoanalyze of parent tables occur when child tables are modified - http://archives.postgresql.org/pgsql-performance/2010-06/msg00137.php On Mon., Sep. 30, 2019, 1:48 p.m. Greg Stark, wrote: > So we now support `ANAL

Re: Partitioning versus autovacuum

2019-09-30 Thread Greg Stark
Actually -- I'm sorry to followup to myself (twice) -- but that's wrong. That Todo item predates the modern partitioning code. It came from when the partitioned statistics were added for inheritance trees. The resulting comment almost doesn't make sense any more since it talks about updates to the

Re: maintenance_work_mem used by Vacuum

2019-10-16 Thread Greg Stark
It's a bit unfortunate that we're doing the pending list flush while the vacuum memory is allocated at all. Is there any reason other than the way the callbacks are defined that gin doesn't do the pending list flush before vacuum does the heap scan and before it allocates any memory using maintenan

Re: Partitioning versus autovacuum

2019-10-18 Thread Greg Stark
At the risk of forking this thread... I think there's actually a planner estimation bug here too. Consider this test case of a simple partitioned table and a simple join. The cardinality estimates for each partition and the Append node are all perfectly accurate. But the estimate for the join is w

Can't we give better table bloat stats easily?

2019-08-16 Thread Greg Stark
Everywhere I've worked I've seen people struggle with table bloat. It's hard to even measure how much of it you have or where, let alone actually fix it. If you search online you'll find dozens of different queries estimating how much empty space are in your tables and indexes based on pg_stats st

LIKE foo% optimization easily defeated by OR?

2018-01-03 Thread Greg Stark
Our database has a query that looks like this -- note the OR between a simple equality qual and a LIKE qual: => explain SELECT 1 AS one FROM "redirect_routes" WHERE redirect_routes.path = 'foobar' OR redirect_routes.path LIKE 'foobar/%'; QUERY PLAN ---

Re: LIKE foo% optimization easily defeated by OR?

2018-01-03 Thread Greg Stark
On 3 January 2018 at 22:34, Alexander Korotkov wrote: > I've checked similar case on database with PostgreSQL mailing lists. It > works for me. Wow that's fascinating. I wonder why it's not kicking in for me. I have checked with enable_seqscan=off but I'll have to do some more investigations. I

Re: LIKE foo% optimization easily defeated by OR?

2018-01-03 Thread Greg Stark
I think I found the bug 18" from the monitor I'll just be over here with the paper bag over my head mumbling about running RESET ALL before running tests...

Re: [HACKERS] Restricting maximum keep segments by repslots

2018-01-11 Thread Greg Stark
On 11 January 2018 at 09:55, Sergei Kornilov wrote: > if (active_pid != 0) > status = "streaming"; > else > status = "keeping"; Perhaps "idle" by analogy to a pg_stat_activity entry for a backend that's connected but not doing anything. > s

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2018-01-11 Thread Greg Stark
On 11 January 2018 at 19:41, Peter Eisentraut wrote: > Two, what to do when the memory limit is reached. With the old > accounting, this was easy, because we'd decide for each subtransaction > independently whether to spill it to disk, when it has reached its 4096 > limit. Now, we are looking a

Re: WIP: a way forward on bootstrap data

2018-01-14 Thread Greg Stark
I'm 1000% on board with replacing oid constants with symbolic names that get substituted programmatically. However I wonder why we're bothering inventing a new syntax that doesn't actually do much more than present static tabular data. If things like magic proname->prosrc behaviour are not valuabl

Re: Bug in Physical Replication Slots (at least 9.5)?

2018-01-19 Thread Greg Stark
On 19 January 2017 at 09:37, Kyotaro HORIGUCHI wrote: > > Though I haven't look closer to how a modification is splitted > into WAL records. A tuple cannot be so long. As a simple test, I > observed rechder->xl_tot_len at the end of XLogRecordAssemble > inserting an about 400KB not-so-compressable

Re: log bind parameter values on error

2019-12-09 Thread Greg Stark
On Mon, 9 Dec 2019 at 15:17, Tom Lane wrote: > > Meh ... people will inevitably complain that they needed to see the > whole value, and we'll end up having to add another configuration > variable. Let's not go there just yet. I haven't been following this whole thread but my initial reaction is

Re: VACUUM memory management

2019-12-09 Thread Greg Stark
On Mon, 9 Dec 2019 at 14:03, Ibrar Ahmed wrote: > I'd > actually argue that the segment size should be substantially smaller > than 1 GB, like say 64MB; there are still some people running systems > which are small enough that allocating 1 GB when we may need only 6 > bytes can drive the system in

Re: verbose cost estimate

2019-12-09 Thread Greg Stark
On Mon, 9 Dec 2019 at 17:14, Tomas Vondra wrote: > > On Sat, Dec 07, 2019 at 11:34:12AM -0500, Tom Lane wrote: > >Justin Pryzby writes: > >> Jeff said: > >>> |What would I find very useful is a verbosity option to get the cost > >>> |estimates expressed as a multiplier of each *_cost parameter, r

Re: On disable_cost

2019-12-12 Thread Greg Stark
On Wed, 11 Dec 2019 at 01:24, Laurenz Albe wrote: > > On Tue, 2019-12-10 at 15:50 -0700, Jim Finnerty wrote: > > As a proof of concept, I hacked around a bit today to re-purpose one of the > > bits of the Cost structure to mean "is_disabled" so that we can distinguish > > 'disabled' from 'non-disa

Re: On disable_cost

2019-12-15 Thread Greg Stark
I think this would be ready to abstract away behind a few functions that could always be replaced by something else later... However on further thought I really think just using a 32-bit float and 32 bits of other bitmaps or counters would be a better approach. On Sun., Dec. 15, 2019, 14:54 Tom

Re: Implementing Incremental View Maintenance

2019-04-03 Thread Greg Stark
On Sun, 31 Mar 2019 at 23:22, Yugo Nagata wrote: > > Firstly, this will handle simple definition views which includes only > selection, projection, and join. Standard aggregations (count, sum, avg, > min, max) are not planned to be implemented in the first patch, but these > are commonly used in

Re: Statistical aggregate functions are not working with PARTIAL aggregation

2019-05-08 Thread Greg Stark
Don't we have a build farm animal that runs under valgrind that would have caught this?

Re: Create TOAST table only if AM needs

2019-05-19 Thread Greg Stark
Just throwing this out there Perhaps we should just disable toasting for non-heap tables entirely for now? That way at least people can use it and storage plugins just have to be able to deal with large datums in their own (or throw errors). On Fri., May 17, 2019, 5:56 p.m. Ashwin Agrawal, w

Re: New vacuum option to do only freezing

2018-11-05 Thread Greg Stark
On Mon 5 Nov 2018, 12:49 Robert Haas That seems non-orthogonal. We have an existing flag to force freezing > (FREEZE); we don't need a second option that does the same thing. > Skipping the index scans (and thus necessarily the second heap pass) > is a separate behavior which we don't currently h

Query pattern tha Postgres doesn't handle well

2018-02-24 Thread Greg Stark
At my day job I've been doing a fair amount of routine query and schema optimization and I've noticed on particular query shape that has repeatedly caused problems, and it's one we've talked about before. select * from table where simple-restriction 0 OR (complex-subquery) For example something l

Re: Online enabling of checksums

2018-02-24 Thread Greg Stark
> The change of the checksum state is WAL logged with a new xlog record. All > the buffers written by the background worker are forcibly enabled full page > writes to make sure the checksum is fully updated on the standby even if no > actual contents of the buffer changed. Hm. That doesn't soun

jsonlog logging only some messages?

2018-02-26 Thread Greg Stark
I tried loading the jsonlog module from https://github.com/michaelpq/pg_plugins into Postgres 9.6. However it seems it resulted in logs only for session log messages but not any background worker log messages. We have log_checkpoints set but there were no log messages in the json log about checkpo

Re: jsonlog logging only some messages?

2018-02-27 Thread Greg Stark
On 27 February 2018 at 02:04, Michael Paquier wrote: > On Mon, Feb 26, 2018 at 05:38:56PM +0000, Greg Stark wrote: >> I tried loading the jsonlog module from >> https://github.com/michaelpq/pg_plugins into Postgres 9.6. >> >> However it seems it resulted in logs only f

  1   2   3   4   >