Re: Statistics Import and Export

2025-01-27 Thread Corey Huinker
On Mon, Jan 27, 2025 at 9:05 AM jian he wrote: > On Tue, Jan 21, 2025 at 7:31 AM Jeff Davis wrote: > > > > On Mon, 2025-01-20 at 16:45 -0500, Corey Huinker wrote: > > > > > > What I struggle to understand is how that purpose isn't served better > > > by statistics being in SECTION_NONE like COMM

Re: Statistics Import and Export

2025-01-27 Thread jian he
On Tue, Jan 21, 2025 at 7:31 AM Jeff Davis wrote: > > On Mon, 2025-01-20 at 16:45 -0500, Corey Huinker wrote: > > > > What I struggle to understand is how that purpose isn't served better > > by statistics being in SECTION_NONE like COMMENTs are, so that they > > are imported immediately after the

Re: Statistics Import and Export

2025-01-22 Thread Michael Paquier
On Tue, Jan 21, 2025 at 10:21:51PM -0500, Corey Huinker wrote: > After some research, I think that we should treat partitioned indexes like > we were before, and just handle the existing special case for regular > indexes. Hmm, why? Sounds strange to me to not have the same locking semantics for

Re: Statistics Import and Export

2025-01-21 Thread Corey Huinker
On Tue, Jan 21, 2025 at 8:37 PM jian he wrote: > hi. > now stats_lock_check_privileges comments need to change? > > * Lock relation in ShareUpdateExclusive mode, check privileges, and close > the > * relation (but retain the lock). > > since the above comments will not be true for RELKIND_INDEX

Re: Statistics Import and Export

2025-01-21 Thread jian he
hi. now stats_lock_check_privileges comments need to change? * Lock relation in ShareUpdateExclusive mode, check privileges, and close the * relation (but retain the lock). since the above comments will not be true for RELKIND_INDEX. Yes, there are comments within the function explaining the ex

Re: Statistics Import and Export

2025-01-21 Thread Corey Huinker
> > >> >> > Sorry about that, my nvim config is auto-including stuff and it's annoying. > > Now with less includes and fewer typos: From 93e67e5304132e680b628b3e92f9fd242d998abd Mon Sep 17 00:00:00 2001 From: Corey Huinker Date: Tue, 21 Jan 2025 11:52:58 -0500 Subject: [PATCH v42] Lock table firs

Re: Statistics Import and Export

2025-01-21 Thread Corey Huinker
> > > > > /* > * For indexes, we follow what do_analyze_rel() does so as to avoid any > * deadlocks with analyze/vacuum, which is to take out a > * ShareUpdateExclusive on table/matview first and only after that take > * a a AccessShareLock on the index itself. > */ > t

Re: Statistics Import and Export

2025-01-21 Thread jian he
On Tue, Jan 21, 2025 at 2:43 PM Corey Huinker wrote: > > > > On Mon, Jan 20, 2025 at 10:59 PM Corey Huinker > wrote: >>> >>> Are you saying that there is a path for a partitioned index to have >>> stats today? If so, we can just follow that locking protocol. If not, >>> I'm concerned about tryin

Re: Statistics Import and Export

2025-01-20 Thread Corey Huinker
On Mon, Jan 20, 2025 at 10:59 PM Corey Huinker wrote: > Are you saying that there is a path for a partitioned index to have >> stats today? If so, we can just follow that locking protocol. If not, >> I'm concerned about trying to define the locking protocol for doing so >> as a side-effect of thi

Re: Statistics Import and Export

2025-01-20 Thread Corey Huinker
> > Are you saying that there is a path for a partitioned index to have > stats today? If so, we can just follow that locking protocol. If not, > I'm concerned about trying to define the locking protocol for doing so > as a side-effect of this work. > A quick test on v17 indicates that, no, there

Re: Statistics Import and Export

2025-01-20 Thread Jeff Davis
On Mon, 2025-01-20 at 22:21 -0500, Corey Huinker wrote: > > -1. > That will break pg_upgrade tests if there's any statistics out there > for any partitioned index. Are you saying that there is a path for a partitioned index to have stats today? If so, we can just follow that locking protocol. If

Re: Statistics Import and Export

2025-01-20 Thread Corey Huinker
> > > Let's just block updating stats for partitioned indexes > until there's a defined way to do so. > -1. That will break pg_upgrade tests if there's any statistics out there for any partitioned index. > Also, we should check privileges on the right object, consistent with > how ANALYZE does

Re: Statistics Import and Export

2025-01-20 Thread Jeff Davis
On Mon, 2025-01-20 at 21:25 -0500, Corey Huinker wrote: > > I tried that, adding a test for it. Treating partitioned indexes like > regular indexes causes the exact same error as was initially reported > for indexes, so I took it back out. I don't think we should declare (as a part of this featur

Re: Statistics Import and Export

2025-01-20 Thread Corey Huinker
> > > * For indexes, it looks like do_analyze_rel is opening the parent table > with ShareUpdateExclusive and the indexes with just AccessShare. Let's > follow that pattern. > Done. > > * The import code allows setting stats for partitioned indexes while > ANALYZE does not, so it's hard to say f

Re: Statistics Import and Export

2025-01-20 Thread Jeff Davis
On Mon, 2025-01-20 at 16:45 -0500, Corey Huinker wrote: > > What I struggle to understand is how that purpose isn't served better > by statistics being in SECTION_NONE like COMMENTs are, so that they > are imported immediately after the object that they reference. Tom, you expressed the strongest

Re: Statistics Import and Export

2025-01-20 Thread Jeff Davis
On Mon, 2025-01-20 at 16:26 -0500, Corey Huinker wrote: > Attached is the patch, along with the regression test output prior to > the change to stat_utils.c. Comments: * For indexes, it looks like do_analyze_rel is opening the parent table with ShareUpdateExclusive and the indexes with just Acces

Re: Statistics Import and Export

2025-01-20 Thread Corey Huinker
> > I believe you are referring to Tom's statement that "it'll be a > serious, serious error for [stats] not to be SECTION_DATA". The > statement is somewhat softened by the sentence that follows, and > slightly more by [2]. But it's pretty clear that SECTION_POST_DATA is, > at best, an implementat

Re: Statistics Import and Export

2025-01-20 Thread Corey Huinker
On Sat, Jan 18, 2025 at 7:45 PM Michael Paquier wrote: > On Sun, Jan 19, 2025 at 01:00:04AM +0800, jian he wrote: > > On Fri, Jan 17, 2025 at 10:20 PM jian he > wrote: > >> dump and execute the above query generated a warning > >> WARNING: missing lock for relation "tenk1_hundred" (OID 18431, >

Re: Statistics Import and Export

2025-01-20 Thread Jeff Davis
On Mon, 2025-01-20 at 10:58 -0500, Corey Huinker wrote: > That is where all statistics were previously. Others felt very > strongly that they should be mixed in to SECTION_DATA and > SECTION_POST_DATA. I believe you are referring to Tom's statement that "it'll be a serious, serious error for [stat

Re: Statistics Import and Export

2025-01-20 Thread Corey Huinker
On Mon, Jan 20, 2025 at 3:22 AM jian he wrote: > On Thu, Jan 16, 2025 at 4:56 AM Corey Huinker > wrote: > >> > >> > >> > >> I do like the idea of a "Statistics for ..." prefix, and I think it's > >> doable. > > > > > > And that's now implemented. > > > > > >> The caller needs some knowledge abou

Re: Statistics Import and Export

2025-01-20 Thread jian he
On Thu, Jan 16, 2025 at 4:56 AM Corey Huinker wrote: >> >> >> >> I do like the idea of a "Statistics for ..." prefix, and I think it's >> doable. > > > And that's now implemented. > > >> The caller needs some knowledge about that anyway, to correctly output >> the statistics dump when the schema i

Re: Statistics Import and Export

2025-01-18 Thread Michael Paquier
On Sun, Jan 19, 2025 at 01:00:04AM +0800, jian he wrote: > On Fri, Jan 17, 2025 at 10:20 PM jian he wrote: >> dump and execute the above query generated a warning >> WARNING: missing lock for relation "tenk1_hundred" (OID 18431, >> relkind i) @ TID (15,34) > > This seems to be an existing issue.

Re: Statistics Import and Export

2025-01-18 Thread jian he
On Fri, Jan 17, 2025 at 10:20 PM jian he wrote: > > hi. > > SELECT * FROM pg_catalog.pg_restore_relation_stats( > 'relation', 'public.tenk1_hundred'::regclass, > 'version', '18'::integer, > 'relpages', '11'::integer, > 'reltuples', '1'::real, > 'relallvisible', '0'::int

Re: Statistics Import and Export

2025-01-17 Thread jian he
hi. SELECT * FROM pg_catalog.pg_restore_relation_stats( 'relation', 'public.tenk1_hundred'::regclass, 'version', '18'::integer, 'relpages', '11'::integer, 'reltuples', '1'::real, 'relallvisible', '0'::integer ); dump and execute the above query generated a warning WARNI

Re: Statistics Import and Export

2025-01-16 Thread Corey Huinker
> > pg_dump --dbname=src2 --table=tenk1 --statistics-only --verbose > x.sql > there no pg_restore_attribute_stats, pg_restore_relation_stats there > for table tenk1. > There aren't any statistics, should there be? > pg_restore_relation_stats: yes. looking into that. pg_restore_attribute_stats: yes

Re: Statistics Import and Export

2025-01-16 Thread jian he
On Thu, Jan 16, 2025 at 4:56 AM Corey Huinker wrote: >> > Attached is just the pg_dump stuff, and only for relation/attribute stats. > The extended stats and vacuumdb work will be in their own threads going > forward. I didn't follow this thread actively, so if the following issue is already ad

Re: Statistics Import and Export

2025-01-15 Thread Corey Huinker
> > > > I do like the idea of a "Statistics for ..." prefix, and I think it's > doable. > And that's now implemented. The caller needs some knowledge about that anyway, to correctly output > the statistics dump when the schema is not requested. Tests should > cover those cases, too. > Tests for

Re: Statistics Import and Export

2025-01-07 Thread Jeff Davis
On Tue, 2025-01-07 at 01:18 -0500, Corey Huinker wrote: > Attached is the latest (and probably last) unified patchset before > parts get spun off into their own threads. In this thread I'm only looking at 0001. Please start a new thread for vacuumdb and extended stats changes. > 0001 - This is th

Re: Statistics Import and Export

2025-01-06 Thread Nathan Bossart
On Mon, Dec 30, 2024 at 03:45:03PM -0500, Bruce Momjian wrote: > On Mon, Dec 30, 2024 at 12:02:47PM -0800, Jeff Davis wrote: >> I suggest that we make a new thread about the vacuumdb changes and >> focus this thread and patch series on the pg_dump changes (and minor >> flag adjustments to pg_upgrad

Re: Statistics Import and Export

2024-12-30 Thread Bruce Momjian
On Mon, Dec 30, 2024 at 12:02:47PM -0800, Jeff Davis wrote: > On Thu, 2024-12-26 at 13:54 -0500, Bruce Momjian wrote: > > I am _again_ not happy with this part of the patch.  Please reply to > > the > > criticism in my November 19th email: > > > > > > https://www.postgresql.org/message-id

Re: Statistics Import and Export

2024-12-30 Thread Jeff Davis
On Thu, 2024-12-26 at 13:54 -0500, Bruce Momjian wrote: > I am _again_ not happy with this part of the patch.  Please reply to > the > criticism in my November 19th email: > > > https://www.postgresql.org/message-id/zz0t1benifdnx...@momjian.us > > rather than ignoring it and posting the

Re: Statistics Import and Export

2024-12-26 Thread Bruce Momjian
On Wed, Dec 11, 2024 at 10:49:53PM -0500, Corey Huinker wrote: > From cf4e731db9ffaa4e89d7c5d14b32668529c8c89a Mon Sep 17 00:00:00 2001 > From: Corey Huinker > Date: Fri, 8 Nov 2024 12:27:50 -0500 > Subject: [PATCH v34 11/11] Add --force-analyze to vacuumdb. > > The vacuumdb options of --analyze-

Re: Statistics Import and Export

2024-12-26 Thread Bruce Momjian
On Thu, Dec 19, 2024 at 09:23:20PM -0800, Jeff Davis wrote: > On Fri, 2024-12-13 at 00:22 -0500, Corey Huinker wrote: > > Per offline conversation with Jeff, adding a --no-schema to pg_dump > > option both for completeness (we already have --no-data and --no- > > statistics), but users who previous

Re: Statistics Import and Export

2024-12-21 Thread Corey Huinker
> > > I like the description "STATISTICS DATA" because it differentiates from > You have Tomas to thank for that: https://www.postgresql.org/message-id/bf724b21-914a-4497-84e3-49944f9776f6%40enterprisedb.com the extended stats definitions. It might be worth differentiating > between "RELATION ST

Re: Statistics Import and Export

2024-12-20 Thread Jeff Davis
On Thu, 2024-12-19 at 21:23 -0800, Jeff Davis wrote: > > 0001-0005 - changes to pg_dump/pg_upgrade > > Attached is a version 36j... The testing can use some work here. I noticed that if I take out the stats entirely, the tests still pass, because pg_upgrade still gets the same before/after result

Re: Statistics Import and Export

2024-12-19 Thread Corey Huinker
> > The biggest functional change is the way dependencies are handled for > matview stats. Materialized views ordinarily end up in > SECITON_PRE_DATA, but in some cases they can be postponed to > SECTION_POST_DATA. You solved that by always putting the matview stats > in SECTION_POST_DATA. > Accur

Re: Statistics Import and Export

2024-12-19 Thread Jeff Davis
On Fri, 2024-12-13 at 00:22 -0500, Corey Huinker wrote: > Per offline conversation with Jeff, adding a --no-schema to pg_dump > option both for completeness (we already have --no-data and --no- > statistics), but users who previously got the effect of --no-schema > did so by specifying --data-only,

Re: Statistics Import and Export

2024-12-12 Thread Corey Huinker
Per offline conversation with Jeff, adding a --no-schema to pg_dump option both for completeness (we already have --no-data and --no-statistics), but users who previously got the effect of --no-schema did so by specifying --data-only, which suppresses statistics as well. 0001-0005 - changes to pg_

Re: Statistics Import and Export

2024-12-11 Thread Corey Huinker
> > +1, assuming such an option is wanted at all. I suppose it should be > there for the unlikely (and hopefully impossible) case that statistics > are causing a problem during upgrade. > Here you go, rebased and re-ordered: 0001-0004 are the pg_dump/pg_upgrade related patches. 0005 is an optimiz

Re: Statistics Import and Export

2024-12-07 Thread Jeff Davis
On Sat, 2024-12-07 at 14:56 -0500, Corey Huinker wrote: > This makes sense, though perhaps instead of --schema-only perhaps we > should pass both --no-statistics and --no-data. I don't envision a > fourth option to the new data/schema/stats triumvirate, but --no- > statistics shouldn't have a beari

Re: Statistics Import and Export

2024-12-07 Thread Corey Huinker
On Sat, Dec 7, 2024 at 2:27 PM Jeff Davis wrote: > On Tue, 2024-11-26 at 17:11 -0500, Corey Huinker wrote: > > > > > * Why does binary upgrade cause statistics to be dumped? Can you > > > just > > > make pg_upgrade specify the appropriate set of flags? > > > > > > > > > That decision goes back a

Re: Statistics Import and Export

2024-12-07 Thread Jeff Davis
On Tue, 2024-11-26 at 17:11 -0500, Corey Huinker wrote: > > > * Why does binary upgrade cause statistics to be dumped? Can you > > just > > make pg_upgrade specify the appropriate set of flags? > > > > > That decision goes back a ways, I tried to dig in the archives last > night but I was getti

Re: Statistics Import and Export

2024-12-06 Thread Jeff Davis
On Wed, 2024-11-27 at 00:08 -0500, Corey Huinker wrote: > > 0003 - Re-enabling in-place updates because catalog bloat bad. Attached is my version of this patch, which I intend to commit soon. I added docs and tests, and I refactored a bit to check the arguments first. Also, I separated the mvcc

Re: Statistics Import and Export

2024-11-27 Thread Bruce Momjian
On Wed, Nov 27, 2024 at 01:15:45PM -0500, Corey Huinker wrote: > > For people that want to use the old behavior of recomputing _all_ > statistics not just the missing ones, we could add a different switch, > or an (optional) option to --analyze-in-stages. > > The current patchset prov

Re: Statistics Import and Export

2024-11-27 Thread Corey Huinker
> > > For people that want to use the old behavior of recomputing _all_ > statistics not just the missing ones, we could add a different switch, > or an (optional) option to --analyze-in-stages. > > The current patchset provides that in the form of the parameter "--force-analyze", which is a modifi

Re: Statistics Import and Export

2024-11-27 Thread Magnus Hagander
On Wed, Nov 27, 2024, 17:44 Tom Lane wrote: > Alvaro Herrera writes: > > On 2024-Nov-27, Bruce Momjian wrote: > >> Would there be a default? > > > There would be no default. Running with no option given would raise an > > error. The point is: you want to break scripts currently running > > --a

Re: Statistics Import and Export

2024-11-27 Thread Alvaro Herrera
On 2024-Nov-27, Tom Lane wrote: > I do not like the idea of breaking existing upgrade scripts, > especially not by requiring them to use a parameter that older > vacuumdb versions will reject. That makes it impossible to have a > script that is version independent. I really doubt that there is a

Re: Statistics Import and Export

2024-11-27 Thread Bruce Momjian
On Wed, Nov 27, 2024 at 11:44:25AM -0500, Tom Lane wrote: > Alvaro Herrera writes: > > On 2024-Nov-27, Bruce Momjian wrote: > >> Would there be a default? > > > There would be no default. Running with no option given would raise an > > error. The point is: you want to break scripts currently ru

Re: Statistics Import and Export

2024-11-27 Thread Bruce Momjian
On Wed, Nov 27, 2024 at 04:57:35PM +0100, Álvaro Herrera wrote: > On 2024-Nov-27, Bruce Momjian wrote: > There would be no default. Running with no option given would raise an > error. The point is: you want to break scripts currently running > --analyze-in-stages so that they can make a choice o

Re: Statistics Import and Export

2024-11-27 Thread Tom Lane
Alvaro Herrera writes: > On 2024-Nov-27, Bruce Momjian wrote: >> Would there be a default? > There would be no default. Running with no option given would raise an > error. The point is: you want to break scripts currently running > --analyze-in-stages so that they can make a choice of which of

Re: Statistics Import and Export

2024-11-27 Thread Nathan Bossart
On Wed, Nov 27, 2024 at 04:57:35PM +0100, Alvaro Herrera wrote: > On 2024-Nov-27, Bruce Momjian wrote: >> Uh, do we have parameters that require a boolean option like this? > > I'm not sure what exactly are you asking here. We do have options like initdb's --sync-method that require specifying o

Re: Statistics Import and Export

2024-11-27 Thread Alvaro Herrera
On 2024-Nov-27, Bruce Momjian wrote: > On Wed, Nov 27, 2024 at 09:18:45AM -0600, Nathan Bossart wrote: > > > Maybe not remove the option, but add a required parameter: > > > --analyze-in-stages=all / missing > > > > > > That way, if the option is missing, the user can adapt the command line > >

Re: Statistics Import and Export

2024-11-27 Thread Bruce Momjian
On Wed, Nov 27, 2024 at 09:18:45AM -0600, Nathan Bossart wrote: > On Wed, Nov 27, 2024 at 04:00:02PM +0100, Alvaro Herrera wrote: > > On 2024-Nov-27, Bruce Momjian wrote: > > > >> On Wed, Nov 27, 2024 at 02:44:01PM +0100, Magnus Hagander wrote: > >> > If you want to avoid both the surprise and con

Re: Statistics Import and Export

2024-11-27 Thread Nathan Bossart
On Wed, Nov 27, 2024 at 04:00:02PM +0100, Alvaro Herrera wrote: > On 2024-Nov-27, Bruce Momjian wrote: > >> On Wed, Nov 27, 2024 at 02:44:01PM +0100, Magnus Hagander wrote: >> > If you want to avoid both the surprise and confusion factor mentioned >> > before, >> > maybe what's needed is to *remo

Re: Statistics Import and Export

2024-11-27 Thread Alvaro Herrera
On 2024-Nov-27, Bruce Momjian wrote: > On Wed, Nov 27, 2024 at 02:44:01PM +0100, Magnus Hagander wrote: > > If you want to avoid both the surprise and confusion factor mentioned > > before, > > maybe what's needed is to *remove* --analyze-in-stages, and replace it with > > --analyze-missing-in-st

Re: Statistics Import and Export

2024-11-27 Thread Bruce Momjian
On Wed, Nov 27, 2024 at 02:44:01PM +0100, Magnus Hagander wrote: > If you want to avoid both the surprise and confusion factor mentioned before, > maybe what's needed is to *remove* --analyze-in-stages, and replace it with > --analyze-missing-in-stages and --analyze-all-in-stages (with the clear >

Re: Statistics Import and Export

2024-11-27 Thread Magnus Hagander
On Tue, Nov 19, 2024 at 1:50 PM Bruce Momjian wrote: > On Mon, Nov 18, 2024 at 08:42:35PM -0500, Bruce Momjian wrote: > > On Mon, Nov 18, 2024 at 08:29:10PM -0500, Corey Huinker wrote: > > > That's not a great surprise for group 6, but I have to believe that > group is > > > smaller than group 5,

Re: Statistics Import and Export

2024-11-26 Thread Corey Huinker
> > > > I'll be rebasing (that's done) and refactoring 0003 to get rid of the > positional column, and moving 0014 next to 0003 because they touch the same > files. > As promised, rebased (as of 8fcd80258bcf43dab93d877a5de0ce3f4d2bd471) Things have been reordered here in a mostly-priority order:

Re: Statistics Import and Export

2024-11-26 Thread Corey Huinker
> > > Comments on 0003: > > * If we commit 0003, is it a useful feature by itself or does it > require that we commit some or all of 0004-0014? Which of these need to > be in v18? > Useful by itself. 0004 seems needed to me, unless we're fine with ~50% bloat in pg_class on a new-upgraded system,

Re: Statistics Import and Export

2024-11-25 Thread Jeff Davis
On Mon, 2024-11-18 at 20:29 -0500, Corey Huinker wrote: > Attached is a re-basing of the existing patchset, plus 3 more > additions: Comments on 0003: * If we commit 0003, is it a useful feature by itself or does it require that we commit some or all of 0004-0014? Which of these need to be in v18

Re: Statistics Import and Export

2024-11-25 Thread Nathan Bossart
On Mon, Nov 25, 2024 at 01:12:52PM -0600, Nathan Bossart wrote: > On Mon, Nov 25, 2024 at 08:11:09AM -0800, Jeff Davis wrote: >> On Fri, 2024-11-22 at 15:09 -0600, Nathan Bossart wrote: >>> I took another look at v32-0001 and v32-0002, and they look >>> reasonable to >>> me.  Unless additional feed

Re: Statistics Import and Export

2024-11-25 Thread Nathan Bossart
On Mon, Nov 25, 2024 at 08:11:09AM -0800, Jeff Davis wrote: > On Fri, 2024-11-22 at 15:09 -0600, Nathan Bossart wrote: >> I took another look at v32-0001 and v32-0002, and they look >> reasonable to >> me.  Unless additional feedback materializes, I'll plan on committing >> those >> soon. > > Thos

Re: Statistics Import and Export

2024-11-25 Thread Jeff Davis
On Fri, 2024-11-22 at 15:09 -0600, Nathan Bossart wrote: > I took another look at v32-0001 and v32-0002, and they look > reasonable to > me.  Unless additional feedback materializes, I'll plan on committing > those > soon. Those refactoring patches look fine to me, the only comment I have is that

Re: Statistics Import and Export

2024-11-22 Thread Nathan Bossart
I took another look at v32-0001 and v32-0002, and they look reasonable to me. Unless additional feedback materializes, I'll plan on committing those soon. After that, it might be a good idea to take up the vacuumdb changes next, since there's been quite a bit of recent discussion about those. I

Re: Statistics Import and Export

2024-11-20 Thread Bruce Momjian
On Tue, Nov 19, 2024 at 05:40:20PM -0500, Bruce Momjian wrote: > On Tue, Nov 19, 2024 at 03:47:20PM -0500, Corey Huinker wrote: > > * create a pg_stats_health_check script that lists tables missing stats, > > with > > --fix/--fix-in-stages options, effectively replacing vacuumdb for those > > purp

Re: Statistics Import and Export

2024-11-19 Thread Bruce Momjian
On Tue, Nov 19, 2024 at 03:47:20PM -0500, Corey Huinker wrote: > I don't have a strong opinion here, but I suspect that if I was creating > vacuumdb from scratch, I'd have suggested a --missing-only flag that would > only work for --analyze-only/--analyze-in-stages.  That way, folks can

Re: Statistics Import and Export

2024-11-19 Thread Corey Huinker
> > I don't have a strong opinion here, but I suspect that if I was creating > vacuumdb from scratch, I'd have suggested a --missing-only flag that would > only work for --analyze-only/--analyze-in-stages. That way, folks can > still regenerate statistics if they want, but we also have an answer f

Re: Statistics Import and Export

2024-11-19 Thread Nathan Bossart
On Mon, Nov 18, 2024 at 08:42:35PM -0500, Bruce Momjian wrote: > We can't design an API around who is going to be surprised. We have to > look at what the options say, what people would expect it to do, and > what it does. The reason "surprise" doesn't work in the long run is > that while PG 18 u

Re: Statistics Import and Export

2024-11-19 Thread Bruce Momjian
On Mon, Nov 18, 2024 at 08:42:35PM -0500, Bruce Momjian wrote: > On Mon, Nov 18, 2024 at 08:29:10PM -0500, Corey Huinker wrote: > > That's not a great surprise for group 6, but I have to believe that group is > > smaller than group 5, and it's definitely smaller than the group of users > > that >

Re: Statistics Import and Export

2024-11-18 Thread Bruce Momjian
On Mon, Nov 18, 2024 at 08:29:10PM -0500, Corey Huinker wrote: > On Mon, Nov 18, 2024 at 2:47 PM Bruce Momjian wrote: > You seem to be optimizing for people using pg_upgrade, and for people > upgrading to PG 18, without adequately considering people using vacuumdb > in non-pg_upgrade s

Re: Statistics Import and Export

2024-11-18 Thread Corey Huinker
On Mon, Nov 18, 2024 at 3:32 PM Bruce Momjian wrote: > > How would/could this effect tables that have the vacuum and analyze > > scale_factors different from the rest of db via the ALTE RTABLE > statement? > > > > (I do this a lot) > > I don't think it would affect it since those control autovacu

Re: Statistics Import and Export

2024-11-18 Thread Bruce Momjian
On Mon, Nov 18, 2024 at 08:06:24PM +, Wetmore, Matthew (CTR) wrote: > Sorry to chime in with a dumb question: > > How would/could this effect tables that have the vacuum and analyze > scale_factors different from the rest of db via the ALTE RTABLE statement? > > (I do this a lot) > > ALTER

Re: Statistics Import and Export

2024-11-18 Thread Wetmore, Matthew (CTR)
Meent , Tom Lane , Nathan Bossart , Magnus Hagander , Stephen Frost , Ashutosh Bapat , Peter Smith , PostgreSQL Hackers , "alvhe...@alvh.no-ip.org" Subject: Re: Statistics Import and Export On Fri, Nov 8, 2024 at 01: 25: 21PM -0500, Corey Huinker wrote: > WHAT IS NOT DONE -

Re: Statistics Import and Export

2024-11-18 Thread Bruce Momjian
On Fri, Nov 8, 2024 at 01:25:21PM -0500, Corey Huinker wrote: > WHAT IS NOT DONE - EXTENDED STATISTICS > > It is a general consensus in the community that "nobody uses extended > statistics", though I've had difficulty getting actual figures to back > this > up, even from my own

Re: Statistics Import and Export

2024-11-13 Thread Michael Paquier
On Wed, Nov 13, 2024 at 01:04:32AM -0500, Corey Huinker wrote: > Makes sense, and the fix is changing a single character (unless we think it > warrants a test case). I'd suggest to add a small test case, actually. Like all the other tests of stats_import.sql, the error happens quickly meaning tha

Re: Statistics Import and Export

2024-11-12 Thread Corey Huinker
> > We should probably not allow that, because you cannot ANALYZE system > columns: > Makes sense, and the fix is changing a single character (unless we think it warrants a test case).

Re: Statistics Import and Export

2024-11-12 Thread Heikki Linnakangas
On 23/10/2024 01:27, Jeff Davis wrote: I've taken most of Jeff's work, reincorporated it into roughly the same patch structure as before, and am posting it now. I committed 0001-0004 with significant revision. This just caught my eye: postgres=# select pg_set_attribute_stats('foo', 'xmin', f

Re: Statistics Import and Export

2024-11-04 Thread Corey Huinker
> > > I'd also like feedback, though I feel very strongly that we should do what > ANALYZE does. In an upgrade situation, nearly all tables will have stats > imported, which would result in an immediate doubling of pg_class - not the > end of the world, but not great either. > > Given the recent bu

Re: Statistics Import and Export

2024-10-31 Thread Corey Huinker
> > > (c) we are considering whether to use an in-place heap update for the > relation stats, so that a large restore doesn't bloat pg_class -- I'd > like feedback on this idea > I'd also like feedback, though I feel very strongly that we should do what ANALYZE does. In an upgrade situation, nea

Re: Statistics Import and Export

2024-10-28 Thread Jeff Davis
On Sun, 2024-10-27 at 14:00 +0300, Alexander Lakhin wrote: > Please look at the following seemingly atypical behavior of the new > functions: ... > SELECT pg_restore_attribute_stats( >    'relation', 'test'::regclass, >    'attname', 'id'::name, >    'inherited', false > ) FROM generate_series(1,

Re: Statistics Import and Export

2024-10-27 Thread Alexander Lakhin
Hello Jeff and Corey, 26.10.2024 01:18, Jeff Davis wrote: On Tue, 2024-09-17 at 05:02 -0400, Corey Huinker wrote: I've taken most of Jeff's work, reincorporated it into roughly the same patch structure as before, and am posting it now. I have committed the import side of this patch series; tha

Re: Statistics Import and Export

2024-10-25 Thread Jeff Davis
On Tue, 2024-09-17 at 05:02 -0400, Corey Huinker wrote: > > I've taken most of Jeff's work, reincorporated it into roughly the > same patch structure as before, and am posting it now. I have committed the import side of this patch series; that is, the function calls that can load stats into an ex

Re: Statistics Import and Export

2024-10-23 Thread Jeff Davis
On Tue, 2024-10-22 at 23:58 +, Shinoda, Noriyoshi (SXD Japan FSIP) wrote: > Thanks for developing good features. I tried the patch that was > committed right away. > It seems that the implementation and documentation differ on the > return value of the pg_clear_attribute_stats function. > The a

RE: Statistics Import and Export

2024-10-22 Thread Shinoda, Noriyoshi (SXD Japan FSIP)
ter Smith ; PostgreSQL Hackers ; Tomas Vondra ; alvhe...@alvh.no-ip.org Subject: Re: Statistics Import and Export > I've taken most of Jeff's work, reincorporated it into roughly the > same patch structure as before, and am posting it now. I committed 0001-0004 with significant

Re: Statistics Import and Export

2024-10-22 Thread Jeff Davis
> I've taken most of Jeff's work, reincorporated it into roughly the > same patch structure as before, and am posting it now. I committed 0001-0004 with significant revision. Regards, Jeff Davis

Re: Statistics Import and Export

2024-10-22 Thread Corey Huinker
> > > If the relpages option contains -1 only for partitioned tables, shouldn't > pg_set_relation_stats restrict the values that can be > > specified by table type? The attached patch limits the value to -1 or more > if the target > > is a partition table, and 0 or more otherwise. > > Changing relp

RE: Statistics Import and Export

2024-10-21 Thread Shinoda, Noriyoshi (SXD Japan FSIP)
er 19, 2024 10:00 AM To: Jeff Davis Cc: Shinoda, Noriyoshi (SXD Japan FSIP) ; jian he ; Matthias van de Meent ; Bruce Momjian ; Tom Lane ; Nathan Bossart ; Magnus Hagander ; Stephen Frost ; Ashutosh Bapat ; Peter Smith ; PostgreSQL Hackers ; alvhe...@alvh.no-ip.org Subject: Re: S

Re: Statistics Import and Export

2024-10-18 Thread Corey Huinker
> > > > Patch that allows relation_statistics_update to continue after one failed > stat (0001) attached, along with bool->void change (0002). > Once more, with feeling: From 0cf8a90c37ec95bbbefb274bc3d48a077b4868b8 Mon Sep 17 00:00:00 2001 From: Corey Huinker Date: Wed, 16 Oct 2024 19:13:01 -040

Re: Statistics Import and Export

2024-10-18 Thread Corey Huinker
> > What is the benefit of a return value from the pg_set_*_stats variants? > As far as I can tell, there is none because they throw an ERROR if > anything goes wrong, so they should just return VOID. What am I > missing? > Probably nothing. The nuances of "this stat didn't get set but these other

Re: Statistics Import and Export

2024-10-17 Thread Jeff Davis
On Thu, 2024-10-17 at 20:54 -0400, Corey Huinker wrote: > There is some uncertainty on what, if anything, should be returned by > pg_set_relation_stats() and pg_set_attribute_stats(). ... > This re-raises the question of whether the pg_set_*_stats functions > should return anything at all. What

Re: Statistics Import and Export

2024-10-17 Thread Corey Huinker
On Wed, Oct 16, 2024 at 7:20 PM Corey Huinker wrote: > Code fix with comment on why nobody expects a relpages -1. Test case to >> demonstrate that relpages -1 can happen, and updated doc to reflect the new >> lower bound. >> > > Additional fixes, now in a patch-set: > > 1. Allow relpages to be se

Re: Statistics Import and Export

2024-10-16 Thread Corey Huinker
> > Code fix with comment on why nobody expects a relpages -1. Test case to > demonstrate that relpages -1 can happen, and updated doc to reflect the new > lower bound. > Additional fixes, now in a patch-set: 1. Allow relpages to be set to -1 (partitioned tables with partitions have this value af

Re: Statistics Import and Export

2024-10-15 Thread Corey Huinker
On Tue, Oct 15, 2024 at 2:50 PM Corey Huinker wrote: > > Oh, I see. It appears that there's a special case for partitioned >> tables that sets relpages=-1 in do_analyze_rel() around line 680. It's >> a bit inconsistent, though, because even partitioned indexes have >> relpages=0. Furthermore, the

Re: Statistics Import and Export

2024-10-15 Thread Corey Huinker
> Oh, I see. It appears that there's a special case for partitioned > tables that sets relpages=-1 in do_analyze_rel() around line 680. It's > a bit inconsistent, though, because even partitioned indexes have > relpages=0. Furthermore, the parameter is of type BlockNumber, so > InvalidBlockNumber w

Re: Statistics Import and Export

2024-10-15 Thread Jeff Davis
On Mon, 2024-10-14 at 21:46 -0400, Corey Huinker wrote: > It seems that partitioned tables have a relpages of -1 Oh, I see. It appears that there's a special case for partitioned tables that sets relpages=-1 in do_analyze_rel() around line 680. It's a bit inconsistent, though, because even partiti

Re: Statistics Import and Export

2024-10-14 Thread Corey Huinker
On Mon, Oct 14, 2024 at 3:40 PM Corey Huinker wrote: > However, this function seems to accept -1 for the relpages parameter. >> Below is an example of execution: >> --- >> postgres=> CREATE TABLE data1(c1 INT PRIMARY KEY, c2 VARCHAR(10)); >> CREATE TABLE >> postgres=> SELECT pg_set_relation_stats

Re: Statistics Import and Export

2024-10-14 Thread Corey Huinker
> > However, this function seems to accept -1 for the relpages parameter. > Below is an example of execution: > --- > postgres=> CREATE TABLE data1(c1 INT PRIMARY KEY, c2 VARCHAR(10)); > CREATE TABLE > postgres=> SELECT pg_set_relation_stats('data1', relpages=>-1); > pg_set_relation_stats > --

RE: Statistics Import and Export

2024-10-12 Thread Shinoda, Noriyoshi (SXD Japan FSIP)
ent: Saturday, October 12, 2024 8:11 AM To: jian he ; Corey Huinker Cc: Matthias van de Meent ; Bruce Momjian ; Tom Lane ; Nathan Bossart ; Magnus Hagander ; Stephen Frost ; Ashutosh Bapat ; Peter Smith ; PostgreSQL Hackers ; Tomas Vondra ; alvhe...@alvh.no-ip.org Subject: Re: Statistics Impo

Re: Statistics Import and Export

2024-10-11 Thread Jeff Davis
On Mon, 2024-09-23 at 08:57 +0800, jian he wrote: >     newtup = heap_modify_tuple_by_cols(ctup, tupdesc, ncols, > replaces, nulls); > > you just directly declared "bool nulls[3]    = {false, false, > false};" Those must be false (not NULL), because in pg_class those are non-NULL attributes. They

Re: Statistics Import and Export

2024-10-10 Thread Nathan Bossart
On Thu, Oct 10, 2024 at 03:49:16PM -0400, Corey Huinker wrote: >> One other question I had when looking at this patch is whether we could >> remove dataOnly/schemaOnly from DumpOptions and RestoreOptions. Once 0007 >> is applied, those variables become particularly hazardous, so we really >> want

  1   2   3   >