Re: Adding comments to help understand psql hidden queries

2025-01-30 Thread David Christensen
On Sat, Jan 18, 2025 at 2:37 PM Tom Lane wrote: > > David Christensen writes: > > Any further concerns/issues with this patch that I can address to help > > move it forward? > > I got around to looking at this finally --- sorry that it's been on > the back burn

Re: [PATCHES] Post-special page storage TDE support

2024-12-27 Thread David Christensen
On Fri, Dec 27, 2024 at 1:58 PM Bruce Momjian wrote: > > On Fri, Dec 27, 2024 at 12:25:11PM -0500, Greg Sabino Mullane wrote: > > On Fri, Dec 27, 2024 at 10:12 AM Bruce Momjian wrote: > > > > The value of TDE is limited from a security value perspective, but high > > on > > the list of s

Re: [PATCHES] Post-special page storage TDE support

2024-12-12 Thread David Christensen
On Tue, Dec 10, 2024 at 12:54 AM Michael Paquier wrote: > > On Wed, Mar 13, 2024 at 11:26:48AM -0500, David Christensen wrote: > > Enclosing v4 for this patch series, rebased atop the > > constant-splitting series[1]. For the purposes of having cfbot happy, > > I am inc

Re: [PATCH] GROUP BY ALL

2024-07-23 Thread David Christensen
On Tue, Jul 23, 2024 at 10:57 AM Laurenz Albe wrote: > > On Tue, 2024-07-23 at 08:37 -0500, David Christensen wrote: > > My intention here was to basically be a shorthand for "group by > > specified non-aggregate fields in the select list". Perhaps I'm not >

Re: [PATCH] GROUP BY ALL

2024-07-23 Thread David Christensen
On Mon, Jul 22, 2024 at 4:41 PM Isaac Morland wrote: > And for when this might be useful, the syntax for it already exists, although > a spurious error message is generated: > > odyssey=> select (uw_term).*, count(*) from uw_term group by uw_term; > ERROR: column "uw_term.term_id" must appear i

Re: [PATCH] GROUP BY ALL

2024-07-23 Thread David Christensen
On Tue, Jul 23, 2024 at 8:21 AM Andrei Borodin wrote: > > On 23 Jul 2024, at 00:40, Isaac Morland wrote: > > odyssey=> select (uw_term).*, count(*) from uw_term group by uw_term; > ERROR: column "uw_term.term_id" must appear in the GROUP BY clause or be > used in an aggregate function > LINE 1:

Re: [PATCH] GROUP BY ALL

2024-07-23 Thread David Christensen
On Mon, Jul 22, 2024 at 5:29 PM Tom Lane wrote: > > "David G. Johnston" writes: > > On Mon, Jul 22, 2024 at 1:55 PM David Christensen wrote: > >> I see that there'd been some chatter but not a lot of discussion about > >> a GROUP BY ALL feature/

Re: [PATCH] GROUP BY ALL

2024-07-23 Thread David Christensen
On Mon, Jul 22, 2024 at 4:34 PM David G. Johnston wrote: > > On Mon, Jul 22, 2024 at 1:55 PM David Christensen wrote: >> >> I see that there'd been some chatter but not a lot of discussion about >> a GROUP BY ALL feature/functionality. There certainly is utilit

[PATCH] GROUP BY ALL

2024-07-22 Thread David Christensen
I see that there'd been some chatter but not a lot of discussion about a GROUP BY ALL feature/functionality. There certainly is utility in such a construct IMHO. The grammar is unambiguous, so can support this construct in lieu of the traditional GROUP BY clause. Enclosed is a patch which adds t

Re: Reuse child_relids in try_partitionwise_join was Re: Assert failure on bms_equal(child_joinrel->relids, child_joinrelids)

2024-07-01 Thread David Christensen
On Jul 1, 2024, at 12:56 AM, Ashutosh Bapat wrote:On Wed, Jun 12, 2024 at 4:22 AM David Christensen <david...@pgguru.net> wrote:On Mon, Jun 10, 2024 at 8:15 AM Robert Haas <robertmh...@gmail.com> wrote: > > On Mon, Jun 10, 2024 at 3:09 AM Ashutosh Bapat > <ashutosh.ba

Re: Adding comments to help understand psql hidden queries

2024-06-11 Thread David Christensen
On Thu, Apr 4, 2024 at 11:12 AM David Christensen wrote: > > On Thu, Apr 4, 2024 at 9:32 AM Peter Eisentraut wrote: > > > > On 03.04.24 19:16, David Christensen wrote: > > > I removed _() in the output of the query/stars since there'd be no > > &g

Re: Reuse child_relids in try_partitionwise_join was Re: Assert failure on bms_equal(child_joinrel->relids, child_joinrelids)

2024-06-11 Thread David Christensen
On Mon, Jun 10, 2024 at 8:15 AM Robert Haas wrote: > > On Mon, Jun 10, 2024 at 3:09 AM Ashutosh Bapat > wrote: > > This is just one instance of measurements. If I run the experiment multiple > > times the results and the patterns will vary. Usually I have found planning > > time to vary within

Re: [PATCH] psql: \dn+ to show size of each schema (and \dA+ for AMs)

2024-06-03 Thread David Christensen
On Mon, Jun 3, 2024 at 9:10 AM Justin Pryzby wrote: > > On Thu, May 30, 2024 at 10:59:06AM -0700, David Christensen wrote: > > Hi Justin, > > > > Thanks for the patch and the work on it. In reviewing the basic > > feature, I think this is something that has utilit

Re: Proposal: Document ABI Compatibility

2024-06-03 Thread David Christensen
> > I don't think it's common for such new-fields-in-padding to cause problems > > when using an earlier minor PG version. For that the extension would need to > > actually rely on the presence of the new field, but typically that'd not be > > the case when we introduce a new field in a minor versi

Re: Add memory context type to pg_backend_memory_contexts view

2024-05-31 Thread David Christensen
> On May 30, 2024, at 5:36 PM, David Rowley wrote: > > On Fri, 31 May 2024 at 07:21, David Christensen wrote: >> Giving this a once-through, this seems straightforward and useful. I >> have a slight preference for keeping "name" the first field in the >>

Re: Reuse child_relids in try_partitionwise_join was Re: Assert failure on bms_equal(child_joinrel->relids, child_joinrelids)

2024-05-30 Thread David Christensen
Hello, I am looking through some outstanding CommitFest entries; I wonder if this particular patch is already effectively fixed by 5278d0a2, which is both attributed to the original author as well as an extremely similar approach. Can this entry (https://commitfest.postgresql.org/48/4553/) be clo

Re: Add memory context type to pg_backend_memory_contexts view

2024-05-30 Thread David Christensen
Hi David, Giving this a once-through, this seems straightforward and useful. I have a slight preference for keeping "name" the first field in the view and moving "type" to the second, but that's minor. Just confirming that the allocator types are not extensible without a recompile, since it's us

Re: [PATCH] psql: \dn+ to show size of each schema (and \dA+ for AMs)

2024-05-30 Thread David Christensen
Hi Justin, Thanks for the patch and the work on it. In reviewing the basic feature, I think this is something that has utility and is worthwhile at the high level. A few more specific notes: The pg_namespace_size() function can stand on its own, and probably has some utility for the released Po

Re: Adding comments to help understand psql hidden queries

2024-04-04 Thread David Christensen
On Thu, Apr 4, 2024 at 9:32 AM Peter Eisentraut wrote: > > On 03.04.24 19:16, David Christensen wrote: > > I removed _() in the output of the query/stars since there'd be no > > sensible existing translations for the constructed string, which > > included the query

Re: Adding comments to help understand psql hidden queries

2024-04-03 Thread David Christensen
I got Greg's blessing on squashing the commits down, and now including a v4 with additional improvements on the output formatting front. Main changes: - all generated comments are the same width - width has been bumped to 80 - removed _() functions for consumers of the new output functions This p

Re: Adding comments to help understand psql hidden queries

2024-03-22 Thread David Christensen
On Fri, Mar 22, 2024 at 9:47 AM Greg Sabino Mullane wrote: > > On Thu, Mar 21, 2024 at 6:20 PM Peter Eisentraut wrote: >> >> lines are supposed to align vertically. With your patch, the first line >> would have variable length depending on the command. > > > Yes, that is a good point. Aligning t

Re: Avoiding inadvertent debugging mode for pgbench

2024-03-21 Thread David Christensen
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:tested, passed Did a quick review of this one; CFbot is now happy, local regress

Re: Adding comments to help understand psql hidden queries

2024-03-21 Thread David Christensen
Created the CF entry in commitfest 48 but didn't see it was already in 47; closing the CFEntry in 48. (Doesn't appear to be a different status than "Withdrawn"...)

Re: Adding comments to help understand psql hidden queries

2024-03-21 Thread David Christensen
Hi Jim, Thanks for the feedback. Enclosed is a v2 of this series(?) rebased and with that warning fixed; @Greg Sabino Mullane I just created a commit on your behalf with the message to hackers. I'm also creating a commit-fest entry for this thread. Best, David v2-0002-Add-output-of-the-comma

Re: Constant Splitting/Refactoring

2024-03-13 Thread David Christensen
Here is a version 2 of this patch, rebased atop 97d85be365. As before, this is a cleanup/prerequisite patch series for the page features/reserved page size patches[1]. (Said patch series is going to be updated shortly.) This splits each of the 4 constants that care about page size into Cluster-s

Re: Constant Splitting/Refactoring

2024-02-14 Thread David Christensen
This should target PG 18, but that status is not available in the CF app yet, so just making a note here.

Constant Splitting/Refactoring

2024-02-07 Thread David Christensen
As part of the reserved page space/page features[1] work, there is a need to convert some of the existing constants into variables, since the size of those will no longer be fixed at compile time. At FOSDEM, there was some interest expressed about seeing what a cleanup patch like that would look l

Re: Adding comments to help understand psql hidden queries

2024-02-01 Thread David Christensen
On Thu, Feb 1, 2024 at 4:34 PM Greg Sabino Mullane wrote: > > The use of the --echo-hidden flag in psql is used to show people the way psql > performs its magic for its backslash commands. None of them has more magic > than "\d relation", but it suffers from needing a lot of separate queries to

Re: [PATCHES] Post-special page storage TDE support

2023-12-22 Thread David Christensen
Hi again! Per some offline discussion with Stephen, I've continued to work on some modifications here; this particular patchset is intended to facilitate review by highlighting the mechanical nature of many of these changes. As such, I have taken the following approach to this rework: 0001 - Cre

Re: [PATCHES] Post-special page storage TDE support

2023-11-29 Thread David Christensen
On Tue, Nov 7, 2023 at 6:20 PM Andres Freund wrote: > Hi, > > - IMO the patch touches many places it shouldn't need to touch, because of > essentially renaming a lot of existing macro names to *Limit, > necessitating modifying a lot of users. I think instead the few places > that > care abo

Re: [PATCHES] Post-special page storage TDE support

2023-11-13 Thread David Christensen
On Mon, Nov 13, 2023 at 2:52 PM Andres Freund wrote: > > > This scheme would open up space per page that would now be available for > > plenty of other things; the encoding in the header and the corresponding > > available space in the footer would seem to open up quite a few options > > now, no?

Re: [PATCHES] Post-special page storage TDE support

2023-11-13 Thread David Christensen
On Mon, Nov 13, 2023 at 2:27 PM Andres Freund wrote: > Hi, > > On 2023-11-08 18:47:56 -0800, Peter Geoghegan wrote: > > On Wed, Nov 8, 2023 at 6:04 AM Stephen Frost wrote: > > > In conversations with folks (my memory specifically is a discussion > with > > > Peter G, added to CC, and my apologie

Re: [PATCHES] Post-special page storage TDE support

2023-11-13 Thread David Christensen
On Tue, Nov 7, 2023 at 6:20 PM Andres Freund wrote: > Hi, > > On 2023-05-09 17:08:26 -0500, David Christensen wrote: > > From 965309ea3517fa734c4bc89c144e2031cdf6c0c3 Mon Sep 17 00:00:00 2001 > > From: David Christensen > > Date: Tue, 9 May 2023 16:56:15 -0500 >

Re: [PATCHES] Post-special page storage TDE support

2023-11-08 Thread David Christensen
On Wed, Nov 8, 2023 at 8:04 AM Stephen Frost wrote: > Greetings, > > * Andres Freund (and...@anarazel.de) wrote: > > On 2023-05-09 17:08:26 -0500, David Christensen wrote: > > > From 965309ea3517fa734c4bc89c144e2031cdf6c0c3 Mon Sep 17 00:00:00 2001 > > > From: D

Re: Moving forward with TDE [PATCH v3]

2023-11-08 Thread David Christensen
On Tue, Nov 7, 2023 at 5:49 PM Andres Freund wrote: > Hi, > > On 2023-11-06 09:56:37 -0500, Stephen Frost wrote: > > * Andres Freund (and...@anarazel.de) wrote: > > > I still am quite quite unconvinced that using the LSN as a nonce is a > good > > > design decision. > > > > This is a really impor

Re: Moving forward with TDE [PATCH v3]

2023-11-08 Thread David Christensen
On Tue, Nov 7, 2023 at 6:47 PM Andres Freund wrote: > Hi, > > On 2023-11-06 11:26:44 +0100, Matthias van de Meent wrote: > > On Sat, 4 Nov 2023 at 03:38, Andres Freund wrote: > > > On 2023-11-02 22:09:40 +0100, Matthias van de Meent wrote: > > > > I'm quite surprised at the significant number of

Re: Moving forward with TDE [PATCH v3]

2023-11-06 Thread David Christensen
On Fri, Nov 3, 2023 at 9:53 PM Andres Freund wrote: > On 2023-11-02 19:32:28 -0700, Andres Freund wrote: > > > From 327e86d52be1df8de9c3a324cb06b85ba5db9604 Mon Sep 17 00:00:00 2001 > > > From: David Christensen > > > Date: Fri, 29 Sep 2023 15:16:00 -0400 >

Re: Moving forward with TDE [PATCH v3]

2023-11-06 Thread David Christensen
Hi, thanks for the detailed feedback here. I do think it's worth addressing the question Stephen raised as far as what we use for the IV[1]; whether LSN or something else entirely, and if so what. The choice of LSN here is fairly fundamental to the existing implementation, so if we decide to do s

Re: Moving forward with TDE [PATCH v3]

2023-10-31 Thread David Christensen
On Tue, Oct 31, 2023 at 4:30 PM Bruce Momjian wrote: > On Tue, Oct 31, 2023 at 04:23:17PM -0500, David Christensen wrote: > > Greetings, > > > > I am including an updated version of this patch series; it has been > rebased > > onto 6ec62b7799 and reworked somewh

Re: Initdb-time block size specification

2023-09-05 Thread David Christensen
On Tue, Sep 5, 2023 at 2:52 PM Hannu Krosing wrote: > > Something I also asked at this years Unconference - Do we currently > have Build Farm animals testing with different page sizes ? > > I'd say that testing all sizes from 4KB up (so 4, 8, 16, 32) should be > done at least before each release i

Re: Initdb-time block size specification

2023-09-05 Thread David Christensen
On Tue, Sep 5, 2023 at 9:04 AM Robert Haas wrote: > > On Sat, Sep 2, 2023 at 3:09 PM Tomas Vondra > wrote: > > Except that no one is forcing you to actually go 130mph or 32mph, right? > > You make it seem like this patch forces people to use some other page > > size, but that's clearly not what i

Re: Initdb-time block size specification

2023-08-31 Thread David Christensen
> I was definitely hand-waving additional implementation here for > non-native 128 bit support; the modulus algorithm as presented > requires 4 times the space as the divisor, so a uint16 implementation > should work for all 64-bit machines. Certainly open to other ideas or > implementations, this

Re: Initdb-time block size specification

2023-08-31 Thread David Christensen
> + * pg_fastmod - calculates the modulus of a 32-bit number against a constant > + * divisor without using the division operator > + */ > +static inline uint32 pg_fastmod(uint32 n, uint32 divisor, uint64 fastinv) > +{ > +#ifdef HAVE_INT128 > + uint64_t lowbits = fastinv * n; > + return ((uint128)l

Re: Initdb-time block size specification

2023-08-31 Thread David Christensen
Enclosed are TPC-H results for 1GB shared_buffers, 64MB work_mem on a 64GB laptop with SSD storage; everything else is default settings. TL;DR: unpatched version: 17.30 seconds, patched version: 17.15; there are some slight variations in runtime, but seems to be within the noise level at this poin

Re: Initdb-time block size specification

2023-06-30 Thread David Christensen
On Fri, Jun 30, 2023 at 4:17 PM Andres Freund wrote: > > Hi, > > On 2023-06-30 15:05:54 -0500, David Christensen wrote: > > > I am fairly certain this is going to be causing substantial performance > > > regressions. I think we should reject this even if we

Re: Initdb-time block size specification

2023-06-30 Thread David Christensen
On Fri, Jun 30, 2023 at 4:27 PM Tomas Vondra wrote: > On 6/30/23 23:11, Andres Freund wrote: > > ... > > > > If we really wanted to do this - but I don't think we do - I'd argue for > > working on the buildsystem support to build the postgres binary multiple > > times, for 4, 8, 16 kB BLCKSZ and h

Re: Initdb-time block size specification

2023-06-30 Thread David Christensen
On Fri, Jun 30, 2023 at 4:12 PM Tomas Vondra wrote: > > > > On 6/30/23 22:05, David Christensen wrote: > > On Fri, Jun 30, 2023 at 2:39 PM Andres Freund wrote: > >> > >> ... > >> > >> Besides this, I've not really heard any convincing j

Re: Initdb-time block size specification

2023-06-30 Thread David Christensen
On Fri, Jun 30, 2023 at 3:29 PM Andres Freund wrote: >> And indeed. Comparing e.g. TPC-H, I see *massive* regressions. Some queries > are the same, sobut others regress by up to 70% (although more commonly around > 10-20%). Hmm, that is definitely not good. > That's larger than I thought, which

Re: Initdb-time block size specification

2023-06-30 Thread David Christensen
On Fri, Jun 30, 2023 at 2:39 PM Andres Freund wrote: > > Hi, > > On 2023-06-30 12:35:09 -0500, David Christensen wrote: > > As discussed somewhat at PgCon, enclosed is v1 of a patch to provide > > variable block sizes; basically instead of BLCKSZ being a compile-time >

Re: Initdb-time block size specification

2023-06-30 Thread David Christensen
On Fri, Jun 30, 2023 at 1:14 PM Tomas Vondra wrote: > Do we really want to prefix the option with CLUSTER_? That seems to just > add a lot of noise into the patch, and I don't see much value in this > rename. I'd prefer keeping BLCKSZ and tweak just the couple places that > need "default" to use

Re: Add TLI number to name of files generated by pg_waldump --save-fullpage

2023-06-27 Thread David Christensen
> Adjusted as per the v2 attached. +1

Re: Add TLI number to name of files generated by pg_waldump --save-fullpage

2023-06-27 Thread David Christensen
On Tue, Jun 27, 2023 at 1:12 AM Michael Paquier wrote: > > Hi all, > (Fujii-san and David in CC.) > > Fujii-san has reported on Twitter that we had better add the TLI > number to what pg_waldump --save-fullpage generates for the file names > of the blocks, as it could be possible that we overwrite

Re: [PATCHES] Post-special page storage TDE support

2023-05-30 Thread David Christensen
On Fri, May 12, 2023 at 7:48 PM Stephen Frost wrote: > > Greetings, > > * David Christensen (david.christen...@crunchydata.com) wrote: > > Refreshing this with HEAD as of today, v4. > > Thanks for updating this! Thanks for the patience in my response here. > &g

Re: Kerberos delegation support in libpq and postgres_fdw

2023-04-07 Thread David Christensen
Ok, based on the interdiff there, I'm happy with that last change. Marking as Ready For Committer. Best, David

Re: Kerberos delegation support in libpq and postgres_fdw

2023-04-07 Thread David Christensen
Reviewed v8; largely looking good, though I notice this hunk, which may arguably be a bug fix, but doesn't appear to be relevant to this specific patch, so could probably be debated independently (and if a bug, should probably be backpatched): diff --git a/contrib/postgres_fdw/option.c b/contrib/p

Re: Kerberos delegation support in libpq and postgres_fdw

2023-04-05 Thread David Christensen
On Wed, Apr 5, 2023 at 3:30 PM Stephen Frost wrote: > Greetings, > > * David Christensen (david...@pgguru.net) wrote: > > Did a code review pass here; here is some feedback. > > Thanks! > > > + /* If password was used to connect, make sure it was

Re: Kerberos delegation support in libpq and postgres_fdw

2023-04-03 Thread David Christensen
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:not tested Did a code review pass here; here is some feedback. + /* If p

Re: Improving btree performance through specializing by key shape, take 2

2023-01-12 Thread David Christensen
Hi Matthias, I'm going to look at this patch series if you're still interested. What was the status of your final performance testing for the 0008 patch alone vs the specialization series? Last I saw on the thread you were going to see if the specialization was required or not. Best, David

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-26 Thread David Christensen
> On Dec 26, 2022, at 1:29 AM, Michael Paquier wrote: > > On Sat, Dec 24, 2022 at 06:23:29PM +0530, Bharath Rupireddy wrote: >> Thanks for the patch. I've made the above change as well as renamed >> the test file name to be save_fpi.pl, everything else remains the same >> as v11. Here's the v12

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-23 Thread David Christensen
On Fri, Dec 23, 2022 at 12:57 PM David Christensen wrote: > > On Wed, Dec 21, 2022 at 5:47 AM Bharath Rupireddy > wrote: [snip] > > 2. +$node->init(extra => ['-k'], allows_streaming => 1); > > When enabled with allows_streaming, there are a bunch of

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-23 Thread David Christensen
On Mon, Dec 19, 2022 at 12:23 AM Michael Paquier wrote: > > On Thu, Dec 15, 2022 at 05:17:46PM -0600, David Christensen wrote: > > On Thu, Dec 15, 2022 at 12:36 AM Michael Paquier > > wrote: > > This v10 should incorporate your feedback as well as Bharath's. >

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-23 Thread David Christensen
On Wed, Dec 21, 2022 at 5:47 AM Bharath Rupireddy wrote: > > On Fri, Dec 16, 2022 at 4:47 AM David Christensen > wrote: > > > > On Thu, Dec 15, 2022 at 12:36 AM Michael Paquier > > wrote: > > > > > > On Wed, Dec 14, 2022 at 04:44:34PM -0600, David

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-15 Thread David Christensen
On Thu, Dec 15, 2022 at 12:36 AM Michael Paquier wrote: > > On Wed, Dec 14, 2022 at 04:44:34PM -0600, David Christensen wrote: > > I can get one sent in tomorrow. This v10 should incorporate your feedback as well as Bharath's. > -XLogRecordHasFPW(XLogReaderState *record)

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-12-14 Thread David Christensen
Hi Bharath, I can get one sent in tomorrow. Thanks, David

Re: Moving forward with TDE

2022-11-17 Thread David Christensen
Hi Dilip, Thanks for the feedback here. I will review the docs changes and add to my tree. Best, David

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-17 Thread David Christensen
On Wed, Nov 16, 2022 at 3:30 AM Bharath Rupireddy wrote: > 1. > -if (config.filter_by_fpw && !XLogRecordHasFPW(xlogreader_state)) > +if (config.filter_by_fpw && !XLogRecordHasFPI(xlogreader_state)) > These changes are not related to this feature, hence renaming those > variables/fu

Re: Moving forward with TDE

2022-11-17 Thread David Christensen
Hi Jacob, Thanks, I've added this patch in my tree [1]. (For now, just adding fixes and the like atop the original separate patches, but will eventually get things winnowed down into probably the same 12 parts the originals were reviewed in. Best, David [1] https://github.com/pgguru/postgres/tr

ScanSourceDatabasePgClass

2022-11-16 Thread David Christensen
Hi Robert, In 9c08aea6a you introduce the block-by-block strategy for creating a copy of the database. In the main loop, this utilizes this call: buf = ReadBufferWithoutRelcache(rlocator, MAIN_FORKNUM, blkno, RBM_NORMAL, bstrategy, false); Here, the last parameter is "false" for the permanence

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-15 Thread David Christensen
Enclosed is v9. - code style consistency (FPI instead of FPW) internally. - cleanup of no-longer needed checksum-related pieces from code and tests. - test cleanup/simplification. - other comment cleanup. Passes all CI checks. Best, David v9-0001-Teach-pg_waldump-to-extract-FPIs-from-the-WAL-

Re: Moving forward with TDE

2022-11-15 Thread David Christensen
> On Nov 15, 2022, at 1:08 PM, Jacob Champion wrote: > > On Mon, Oct 24, 2022 at 9:29 AM David Christensen > wrote: >> I would love to open a discussion about how to move forward and get >> some of these features built out. The historical threads here are >>

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-15 Thread David Christensen
On Tue, Nov 15, 2022 at 4:41 AM Bharath Rupireddy wrote: > > On Tue, Nov 15, 2022 at 1:29 AM David Christensen > wrote: > > > > Enclosed is v8, which uses the replication slot method to retain WAL > > as well as fsync'ing the output directory when everything is d

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-14 Thread David Christensen
Enclosed is v8, which uses the replication slot method to retain WAL as well as fsync'ing the output directory when everything is done. v8-0001-Teach-pg_waldump-to-extract-FPIs-from-the-WAL-str.patch Description: Binary data

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-14 Thread David Christensen
On Fri, Nov 11, 2022 at 4:57 AM Bharath Rupireddy wrote: > > Well if it's not the same output then I guess you're right and there's > > not a use for the `--fixup` mode. By the same token, I'd say > > calculating/setting the checksum also wouldn't need to be done, we > > should just include the p

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-11 Thread David Christensen
On Fri, Nov 11, 2022 at 8:15 AM Justin Pryzby wrote: > > On Wed, Nov 09, 2022 at 02:37:29PM -0600, David Christensen wrote: > > On Wed, Nov 9, 2022 at 2:08 PM David Christensen > > wrote: > > > Justin sez: > > > > I was wondering if there's a

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-10 Thread David Christensen
On Thu, Nov 10, 2022 at 2:33 AM Bharath Rupireddy wrote: > > On Thu, Nov 10, 2022 at 1:31 AM David Christensen > wrote: > > > > Thanks for providing the v7 patch, please see my comments and responses below. Hi Bharath, Thanks for the feedback. > > > 2. I'm un

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-09 Thread David Christensen
On Wed, Nov 9, 2022 at 2:08 PM David Christensen wrote: > Justin sez: > > I was wondering if there's any reason to do "CREATE DATABASE". The vast > > majority of TAP tests don't. > > > > $ git grep -ho 'safe_psql[^ ]*' '*pl&#

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-09 Thread David Christensen
> > 6. > > +if (dir_status == 0 && mkdir(config.save_fpw_path, 0700) < 0) > > Use pg_dir_create_mode instead of hard-coded 0007? > > I think I thought of that when I first looked at the patch ... but, I'm > not sure, since it says: > > src/include/common/file_perm.h-/* Modes for creating d

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-09 Thread David Christensen
On Wed, Nov 9, 2022 at 6:30 AM Bharath Rupireddy wrote: > > On Wed, Nov 9, 2022 at 5:08 AM David Christensen > wrote: > > > > Enclosed is v6, which squashes your refactor and adds the additional > > recent suggestions. > > Thanks for working on this feature.

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-08 Thread David Christensen
Enclosed is v6, which squashes your refactor and adds the additional recent suggestions. Thanks! v6-0001-Teach-pg_waldump-to-extract-FPIs-from-the-WAL-str.patch Description: Binary data

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-08 Thread David Christensen
On Tue, Nov 8, 2022 at 4:45 PM Justin Pryzby wrote: > > On Mon, Nov 07, 2022 at 05:01:01PM -0600, David Christensen wrote: > > Hi Justin et al, > > > > Enclosed is v5 of this patch which now passes the CirrusCI checks for > > all supported OSes. I went ahead and rew

Re: [PATCHES] Post-special page storage TDE support

2022-11-08 Thread David Christensen
Looking into some CF bot failures which didn't show up locally. Will send a v3 when resolved.

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-07 Thread David Christensen
Hi Justin et al, Enclosed is v5 of this patch which now passes the CirrusCI checks for all supported OSes. I went ahead and reworked the test a bit so it's a little more amenable to the OS-agnostic approach for testing. Best, David v5-0001-Teach-pg_waldump-to-extract-FPIs-from-the-WAL-str.patc

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-04 Thread David Christensen
On Fri, Nov 4, 2022 at 1:38 PM Justin Pryzby wrote: > > > As I recall, that's due to relying on "cp". And "rsync", which > > > shouldn't be assumed to exist by regression tests). Will poke around other TAP tests to see if there's a more consistent interface, what perl version we can assume and a

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-11-04 Thread David Christensen
On Nov 4, 2022, at 9:02 AM, Justin Pryzby wrote: > > On Fri, Nov 04, 2022 at 11:52:59AM +0900, Ian Lawrence Barwick wrote: >> 2022年5月3日(火) 8:45 David Christensen : >>> >>> ...and pushing a couple fixups pointed out by cfbot, so here's v4. >> >&g

Re: [PATCHES] Post-special page storage TDE support

2022-10-28 Thread David Christensen
Hi Matthias, > Did you read the related thread with related discussion from last June, "Re: > better page-level checksums" [0]? In that I argued that space at the end of a > page is already allocated for the AM, and that reserving variable space at > the end of the page for non-AM usage is wast

Re: [PATCHES] Post-special page storage TDE support

2022-10-25 Thread David Christensen
> > Explicitly > > locking (assuming you stay in your lane) should only need to guard > > against access from other > > backends of this type if using shared buffers, so will be use-case > > dependent. > > I'm not sure what you mean here? I'm mainly pointing out that the specific code that manage

[PATCH] comment fixes for delayChkptFlags

2022-10-14 Thread David Christensen
Enclosed is a trivial fix for a typo and misnamed field I noted when doing some code review. Best, David 0001-fix-comment-typos-for-delayChkptFlags.patch Description: Binary data

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-05-02 Thread David Christensen
...and pushing a couple fixups pointed out by cfbot, so here's v4. On Mon, May 2, 2022 at 8:42 AM David Christensen wrote: > > Enclosed is v3 of this patch; this adds two modes for this feature, > one with the raw page `--save-fullpage/-W` and one with the > LSN+checksum fixups

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-05-02 Thread David Christensen
Enclosed is v3 of this patch; this adds two modes for this feature, one with the raw page `--save-fullpage/-W` and one with the LSN+checksum fixups `--save-fullpage-fixup/-X`. I've added at least some basic sanity-checking of the underlying feature, as well as run the test file and the changes to

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-26 Thread David Christensen
On Mon, Apr 25, 2022 at 9:42 PM Michael Paquier wrote: > On Mon, Apr 25, 2022 at 10:24:52AM -0500, David Christensen wrote: > > On Mon, Apr 25, 2022 at 6:03 AM Bharath Rupireddy > > wrote: > >> Thanks for working on this. I'm just thinking if we can use these FPIs

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-26 Thread David Christensen
On Mon, Apr 25, 2022 at 9:54 PM Michael Paquier wrote: > On Mon, Apr 25, 2022 at 10:11:10AM -0500, David Christensen wrote: > > On Mon, Apr 25, 2022 at 1:11 AM Michael Paquier wrote: > >> I don't think that there is any need to rely on a new logic if there > >> is

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-25 Thread David Christensen
On Mon, Apr 25, 2022 at 6:03 AM Bharath Rupireddy wrote: > Thanks for working on this. I'm just thinking if we can use these FPIs > to repair the corrupted pages? I would like to understand more > detailed usages of the FPIs other than inspecting with pageinspect. My main use case was for being a

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-25 Thread David Christensen
On Mon, Apr 25, 2022 at 2:00 AM Drouvot, Bertrand wrote: > > Hi, > > On 4/25/22 8:11 AM, Michael Paquier wrote: > > On Sat, Apr 23, 2022 at 01:43:36PM -0500, David Christensen wrote: > >> Hi Matthias, great point. Enclosed is a revised version of the patch > >&

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-25 Thread David Christensen
On Mon, Apr 25, 2022 at 1:11 AM Michael Paquier wrote: > > On Sat, Apr 23, 2022 at 01:43:36PM -0500, David Christensen wrote: > > Hi Matthias, great point. Enclosed is a revised version of the patch > > that adds the fork identifier to the end if it's a non-main fork.

Re: [PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-23 Thread David Christensen
On Sat, Apr 23, 2022 at 9:49 AM Matthias van de Meent wrote: > Regardless of my (lack of) opinion on the inclusion of this patch in > PG (I did not significantly review this patch); I noticed that you do > not yet identify the 'fork' of the FPI in the file name. > > A lack of fork identifier in t

[PATCH] Teach pg_waldump to extract FPIs from the WAL

2022-04-22 Thread David Christensen
Hi -hackers, Enclosed is a patch to allow extraction/saving of FPI from the WAL stream via pg_waldump. Description from the commit: Extracts full-page images from the WAL stream into a target directory, which must be empty or not exist. These images are subject to the same filtering rules as no

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-03-24 Thread David Christensen
> On Mar 24, 2022, at 4:12 PM, Thomas Munro wrote: > > On Fri, Mar 25, 2022 at 1:43 AM David Christensen > wrote: >>>> On Mar 24, 2022, at 6:43 AM, Thomas Munro wrote: >>> On Fri, Mar 25, 2022 at 12:26 AM Thomas Munro >>> wrote: >>>&

Re: [PATCH] add relation and block-level filtering to pg_waldump

2022-03-24 Thread David Christensen
> On Mar 24, 2022, at 6:43 AM, Thomas Munro wrote: > > On Fri, Mar 25, 2022 at 12:26 AM Thomas Munro wrote: >>> On Fri, Mar 25, 2022 at 12:01 AM Peter Eisentraut >>> wrote: >>> Or even: Why are we exposing fork *numbers* in the user interface? >>> Even low-level tools such as pageinspect us

Re: [PATCH] Proof of concept for GUC improvements

2022-03-22 Thread David Christensen
> On Mar 21, 2022, at 7:53 PM, Tom Lane wrote: > > Andres Freund writes: >> My impression is that there's not a lot of enthusiasm for the concept? If >> that's true we maybe ought to mark the CF entry as rejected? > > Yeah, I'm kind of leaning that way too. I don't see how we can > incorpor

Re: [PATCH] pgbench: add multiconnect option

2022-03-22 Thread David Christensen
On Sat, Mar 19, 2022 at 11:43 AM Fabien COELHO wrote: > > Hi Sami, > > > Pgbench is a simple benchmark tool by design, and I wonder if adding > > a multiconnect feature will cause pgbench to be used incorrectly. > > Maybe, but I do not see how it would be worse that what pgbench already > allows.

  1   2   >