Re: Add LZ4 compression in pg_dump

2023-04-12 Thread Justin Pryzby
On Thu, Apr 13, 2023 at 09:37:06AM +0900, Michael Paquier wrote: > > If you don't insist on calling parse(NONE), the only change is to remove > > the empty #else, which was my original patch. > > Removing the empty else has as problem to create an empty if block, > which could be itself a cause of

v16dev: invalid memory alloc request size 8488348128

2023-04-14 Thread Justin Pryzby
I hit this elog() while testing reports under v16 and changed to PANIC to help diagnose. DETAILS: PANIC: invalid memory alloc request size 18446744072967930808 CONTEXT: PL/pgSQL function array_weight(real[],real[]) while storing call arguments into local variables I can't share the query, data

Re: v16dev: invalid memory alloc request size 8488348128

2023-04-14 Thread Justin Pryzby
On Sat, Apr 15, 2023 at 10:04:52AM +1200, David Rowley wrote: > On Sat, 15 Apr 2023 at 08:36, Justin Pryzby wrote: > > > > I hit this elog() while testing reports under v16 and changed to PANIC > > to help diagnose. > > > > DETAILS: PANIC: invalid memory alloc

Re: v16dev: invalid memory alloc request size 8488348128

2023-04-14 Thread Justin Pryzby
Maybe you'll find valgrind errors to be helpful. ==17971== Source and destination overlap in memcpy(0x1eb8c078, 0x1d88cb20, 123876054) ==17971==at 0x4C2E81D: memcpy@@GLIBC_2.14 (vg_replace_strmem.c:1035) ==17971==by 0x9C705A: memcpy (string3.h:51) ==17971==by 0x9C705A: pg_detoast_datu

Re: segfault tied to "IS JSON predicate" commit

2023-04-15 Thread Justin Pryzby
On Thu, Apr 13, 2023 at 09:14:01PM -0700, Peter Geoghegan wrote: > I find that if I run the following test against a standard debug build > on HEAD, my local installation reliably segfaults: > > $ meson test --setup running --suite test_rls_hooks-running > > Attached is a "bt full" run from gdb a

Re: Direct I/O

2023-04-15 Thread Justin Pryzby
On Sat, Apr 15, 2023 at 02:19:35PM -0400, Tom Lane wrote: > PS: I don't quite understand how it managed to get through initdb > when CREATE DATABASE doesn't work. Maybe there is a different > code path taken in standalone mode? ad43a413c4f7f5d024a5b2f51e00d280a22f1874 initdb: When running CRE

Re: v16dev: invalid memory alloc request size 8488348128

2023-04-15 Thread Justin Pryzby
On Sat, Apr 15, 2023 at 11:33:58AM +1200, David Rowley wrote: > On Sat, 15 Apr 2023 at 10:48, Justin Pryzby wrote: > > > > On Sat, Apr 15, 2023 at 10:04:52AM +1200, David Rowley wrote: > > > Which aggregate function is being called here? Is it a custom > > > aggr

Re: Can we delete the vacuumdb.sgml notes about which version each option was added in?

2023-04-16 Thread Justin Pryzby
On Sun, Apr 16, 2023 at 10:14:35PM +1200, David Rowley wrote: > I was just looking at the vacuumdb docs and noticed that I had > neglected to follow the tradition of adding a note to mention which > version we added the new option in when I committed the > --buffer-usage-limit patch. > > There are

Re: Move defaults toward ICU in 16?

2023-04-17 Thread Justin Pryzby
On Wed, Apr 05, 2023 at 09:33:25AM +0200, Peter Eisentraut wrote: > On 16.03.23 14:52, Peter Eisentraut wrote: > > On 09.03.23 20:14, Jeff Davis wrote: > > > > Let's come back to that after dealing with the other two. > > > > > > Leaving 0001 open for now. > > > > I suspect making a change like t

v16dev: TRAP: failed Assert("size > SizeOfXLogRecord"), File: "xlog.c", Line: 1055, PID: 13564

2023-04-17 Thread Justin Pryzby
I hit this assertion while pg_restoring data into a v16 instance. postgresql16-server-16-alpha_20230417_PGDG.rhel7.x86_64 wal_level=minimal and pg_dump --single-transaction both seem to be required to hit the issue. $ /usr/pgsql-16/bin/postgres -D ./pg16test -c maintenance_work_mem=1GB -c max_wa

Re: v16dev: TRAP: failed Assert("size > SizeOfXLogRecord"), File: "xlog.c", Line: 1055, PID: 13564

2023-04-17 Thread Justin Pryzby
On Mon, Apr 17, 2023 at 01:50:30PM -0400, Tom Lane wrote: > I wrote: > > Yeah, I just came to the same conclusion. One thing I don't understand > > yet: log_newpage_range is old (it looks like this back to v12), and > > that Assert is older, so why doesn't this reproduce further back? > > Maybe th

Re: Fix typos and inconsistencies for v16

2023-04-17 Thread Justin Pryzby
On Mon, Apr 17, 2023 at 09:00:00PM +0300, Alexander Lakhin wrote: > Hello hackers, > > Please consider fixing the following unique words/identifiers introduced in > v16: Well done. Note that your patches are overlapping: 3 --- a/src/backend/utils/misc/guc.c 2 --- a/src/test/perl/Po

Re: Fix typos and inconsistencies for v16

2023-04-18 Thread Justin Pryzby
On Tue, Apr 18, 2023 at 02:06:43PM +1200, David Rowley wrote: > On Tue, 18 Apr 2023 at 10:10, Justin Pryzby wrote: > > > - * USER SET values are appliciable only for PGC_USERSET > > > parameters. We > > > + * USER SET values are a

Re: Wrong results from Parallel Hash Full Join

2023-04-19 Thread Justin Pryzby
On Wed, Apr 19, 2023 at 11:17:04AM -0400, Melanie Plageman wrote: > Ultimately this is probably fine. If we wanted to modify one of the > existing tests to cover the multi-batch case, changing the select > count(*) to a select * would do the trick. I imagine we wouldn't want to > do this because of

Re: Wrong results from Parallel Hash Full Join

2023-04-19 Thread Justin Pryzby
On Wed, Apr 19, 2023 at 12:20:51PM -0700, Andres Freund wrote: > On 2023-04-19 12:16:24 -0500, Justin Pryzby wrote: > > On Wed, Apr 19, 2023 at 11:17:04AM -0400, Melanie Plageman wrote: > > > Ultimately this is probably fine. If we wanted to modify one of the > > >

Re: Wrong results from Parallel Hash Full Join

2023-04-20 Thread Justin Pryzby
On Wed, Apr 19, 2023 at 08:47:07PM -0400, Melanie Plageman wrote: > On Wed, Apr 19, 2023 at 8:41 PM Justin Pryzby wrote: > > > > On Wed, Apr 19, 2023 at 12:20:51PM -0700, Andres Freund wrote: > > > On 2023-04-19 12:16:24 -0500, Justin Pryzby wrote: > > > > On W

Re: ALTER TABLE SET ACCESS METHOD on partitioned tables

2023-04-24 Thread Justin Pryzby
On Thu, Mar 30, 2023 at 12:07:58AM -0500, Justin Pryzby wrote: > On Mon, Mar 27, 2023 at 11:34:36PM -0500, Justin Pryzby wrote: > > On Tue, Mar 28, 2023 at 09:13:10AM +0900, Michael Paquier wrote: > > > On Mon, Mar 20, 2023 at 09:30:50AM +0900, Michael Paquier wrote: > > &g

Re: Direct I/O

2023-04-30 Thread Justin Pryzby
On Sun, Apr 30, 2023 at 06:35:30PM +1200, Thomas Munro wrote: > On Sun, Apr 30, 2023 at 4:11 PM Noah Misch wrote: > > Speaking of the developer-only status, I find the io_direct name more > > enticing > > than force_parallel_mode, which PostgreSQL renamed due to overuse from > > people > > expec

Re: fix stats_fetch_consistency value in postgresql.conf.sample

2023-05-09 Thread Justin Pryzby
On Wed, Mar 29, 2023 at 11:03:59PM -0500, Justin Pryzby wrote: > On Wed, Jul 13, 2022 at 04:49:00PM -0700, Andres Freund wrote: > > On 2022-07-14 08:46:02 +0900, Michael Paquier wrote: > > > On Wed, Jul 13, 2022 at 12:30:00PM -0500, Justin Pryzby wrote: > > > > How di

Re: CREATE INDEX CONCURRENTLY on partitioned index

2022-03-25 Thread Justin Pryzby
On Fri, Mar 25, 2022 at 01:05:49AM -0400, Greg Stark wrote: > This patch is marked "waiting on author" in the CF. However the most > recent emails have patches and it's not clear to me what's left from > previous reviews that might not be addressed yet. Should this patch be > marked "Needs Review"?

Re: Add LZ4 compression in pg_dump

2022-03-25 Thread Justin Pryzby
On Fri, Mar 25, 2022 at 01:20:47AM -0400, Greg Stark wrote: > It seems development on this has stalled. If there's no further work > happening I guess I'll mark the patch returned with feedback. Feel > free to resubmit it to the next CF when there's progress. Since it's a reasonably large patch (a

Re: pg_relation_size on partitioned table

2022-03-25 Thread Justin Pryzby
On Fri, Mar 25, 2022 at 08:52:40PM +0800, Japin Li wrote: > When I try to get total size of partition tables though partitioned table > name using pg_relation_size(), it always returns zero. I can use the > following SQL to get total size of partition tables, however, it is a bit > complex. This

Re: Add psql command to list constraints

2022-03-25 Thread Justin Pryzby
On Fri, Mar 25, 2022 at 03:11:47PM -0400, Robert Haas wrote: > On Fri, Mar 25, 2022 at 12:28 AM Greg Stark wrote: > > Development of this seems to have stalled with the only review of this > > patch expressing some skepticism about whether it's needed at all. > > Now, there is some precedent for

Re: pgsql: Add 'basebackup_to_shell' contrib module.

2022-03-25 Thread Justin Pryzby
On Fri, Mar 25, 2022 at 02:27:07PM -0700, Andres Freund wrote: > On 2022-03-25 13:52:11 -0400, Robert Haas wrote: > > On Fri, Mar 25, 2022 at 12:36 PM Andres Freund wrote: > > > Create a CF entry for it, or enable CI on a github repo? > > > > I created a CF entry for it. Then I had to try to Googl

Re: Add LZ4 compression in pg_dump

2022-03-25 Thread Justin Pryzby
On Sat, Mar 26, 2022 at 02:57:50PM +0900, Michael Paquier wrote: > I have a question about 0002, actually. What has led you to the > conclusion that this code is dead and could be removed? See 0001 and the manpage. + 'pg_dump: compression is not supported by tar archive format');

Re: Add LZ4 compression in pg_dump

2022-03-26 Thread Justin Pryzby
LZ4F_HEADER_SIZE_MAX isn't defined in old LZ4. I ran into that on an ubuntu LTS, so I don't think it's so old that it shouldn't be handled more gracefully. LZ4 should either have an explicit version check, or else shouldn't depend on that feature (or should define a safe fallback version if the l

Re: Add LZ4 compression in pg_dump

2022-03-26 Thread Justin Pryzby
On Sat, Mar 26, 2022 at 11:21:56AM -0500, Justin Pryzby wrote: > You're passing both the compression method *and* level. I think there should > be a structure which includes both. In the future, that can also handle > additional options. I'm not sure if there's anything

Re: Add LZ4 compression in pg_dump

2022-03-26 Thread Justin Pryzby
On Sun, Mar 27, 2022 at 12:37:27AM +0100, Daniel Gustafsson wrote: > > On 26 Mar 2022, at 17:21, Justin Pryzby wrote: > > > I suggested off-list to add an 0099 patch to change LZ4 to the default, to > > exercise it more on CI. > > No need to change the defaults in au

Re: Add LZ4 compression in pg_dump

2022-03-27 Thread Justin Pryzby
On Sun, Mar 27, 2022 at 10:13:00AM -0400, Robert Haas wrote: > On Sat, Mar 26, 2022 at 12:22 PM Justin Pryzby wrote: > > 0002: I wonder if you're able to re-use any of the basebackup parsing stuff > > from commit ffd53659c. You're passing both the compression method *and

Re: multithreaded zstd backup compression for client and server

2022-03-27 Thread Justin Pryzby
On Wed, Mar 23, 2022 at 06:57:04PM -0400, Robert Haas wrote: > On Wed, Mar 23, 2022 at 5:52 PM Justin Pryzby wrote: > > Also because the library may not be compiled with threading. A few days > > ago, I > > tried to rebase the original "parallel workers" patc

Re: [WIP] Allow pg_upgrade to copy segments of the same relfilenode in parallel

2022-03-28 Thread Justin Pryzby
On Sun, Mar 27, 2022 at 11:07:27AM -0500, Jaime Casanova wrote: > > > It lacks documentation and I need help with WIN32 part of it, I created > > > this new mail to put the patch on the next commitfest. > > > > The patch currently fails on cfbot due to warnings, likely related due to > > the > >

Re: refactoring basebackup.c (zstd workers)

2022-03-28 Thread Justin Pryzby
:workers=N where WHERE is "client" or "server" and N is an integer. When compression is performed on the server side, this will spawn threads inside the PostgreSQL backend. While there is almost no PostgreSQL server code which is thread-safe, the threads here are

Re: [PATCH] Add extra statistics to explain for Nested Loop

2022-03-28 Thread Justin Pryzby
> > +static void show_loop_info(Instrumentation *instrument, bool isworker, > > + ExplainState *es); > > > > I think this should be done as a separate refactoring commit. Right - the 0001 patch I sent seems independently beneficial, and makes the changes in 0002 more appar

Re: refactoring basebackup.c (zstd workers)

2022-03-28 Thread Justin Pryzby
On Mon, Mar 28, 2022 at 05:39:31PM -0400, Robert Haas wrote: > On Mon, Mar 28, 2022 at 4:53 PM Justin Pryzby wrote: > > I suggest to write it differently, as in 0002. > > That doesn't seem better to me. What's the argument for it? I find this much easier to understand:

Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)

2022-03-28 Thread Justin Pryzby
On Sat, Mar 26, 2022 at 08:23:54PM +0900, Michael Paquier wrote: > On Wed, Mar 23, 2022 at 03:17:35PM +0900, Michael Paquier wrote: > > FWIW, per my review the bit of the patch set that I found the most > > relevant is the addition of a note in the docs of pg_stat_file() about > > the case where "f

Re: Commitfest Update

2022-03-29 Thread Justin Pryzby
On Thu, Mar 17, 2022 at 02:07:16PM -0400, Greg Stark wrote: > So far this commitfest these 10 patches have been marked committed. > That leaves us with 175 "Needs Review" and 28 "Ready for Comitter" so > quite a ways to go ... If it were me, I'd move these out of the way somehow; WOA/RWF or move t

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-03-29 Thread Justin Pryzby
+ diff = (int32) (vacrel->NewRelfrozenXid - vacrel->relfrozenxid); + Assert(diff > 0); Did you see that this crashed on windows cfbot? https://api.cirrus-ci.com/v1/artifact/task/4592929254670336/log/tmp_check/postmaster.log TRAP: Failed

Re: In-placre persistance change of a relation

2022-03-30 Thread Justin Pryzby
On Tue, Mar 01, 2022 at 02:14:13PM +0900, Kyotaro Horiguchi wrote: > Rebased on a recent xlog refactoring. It'll come as no surprise that this neds to be rebased again. At least a few typos I reported in January aren't fixed. Set to "waiting".

basebackup/lz4 crash

2022-03-30 Thread Justin Pryzby
Forking: <20220316151253.gb28...@telsasoft.com> On Wed, Mar 16, 2022 at 10:12:54AM -0500, Justin Pryzby wrote: > Also, with a partial regression DB, this crashes when writing to stdout. > > $ src/bin/pg_basebackup/pg_basebackup --wal-method fetch -Ft -D - -h /tmp > --no-sync

Re: Printing backtrace of postgres processes

2022-03-30 Thread Justin Pryzby
lead to denial of service. On receipt of the request, at the next CHECK_FOR_INTERRUPTS(), the target backend logs its backtrace at LOG_SERVER_ONLY level, so that the backtrace will appear in the server log but not be sent to the client. Bump catalog version. Authors: Vignesh C, Bharath Rupireddy

Re: explain_regress, explain(MACHINE), and default to explain(BUFFERS) (was: BUFFERS enabled by default in EXPLAIN (ANALYZE))

2022-03-30 Thread Justin Pryzby
On Sat, Feb 26, 2022 at 03:07:20PM -0600, Justin Pryzby wrote: > Rebased over ebf6c5249b7db525e59563fb149642665c88f747. > It looks like that patch handles only query_id, and this patch also tries to > handle a bunch of other stuff. > > If it's helpful, feel free to kick this

Re: refactoring basebackup.c (zstd workers)

2022-03-30 Thread Justin Pryzby
On Wed, Mar 30, 2022 at 04:14:47PM -0400, Tom Lane wrote: > Robert Haas writes: > >> Maybe if I just put that last sentence into the comment it's clear enough? > > > Done that way, since I thought it was better to fix the bug than wait > > for more feedback on the wording. We can still adjust the

Re: In-placre persistance change of a relation

2022-03-30 Thread Justin Pryzby
On Thu, Mar 31, 2022 at 01:58:45PM +0900, Kyotaro Horiguchi wrote: > Thanks! Version 20 is attached. The patch failed an all CI tasks, and seems to have caused the macos task to hang. http://cfbot.cputube.org/kyotaro-horiguchi.html Would you send a fixed patch, or remove this thread from the CFB

Re: Add non-blocking version of PQcancel

2022-03-30 Thread Justin Pryzby
Note that the patch is still variously failing in cirrus. https://cirrus-ci.com/github/postgresql-cfbot/postgresql/commitfest/37/3511 You may already know that it's possible to trigger the cirrus ci tasks using a github branch. See src/tools/ci/README.

Re: CLUSTER on partitioned index

2022-03-31 Thread Justin Pryzby
On Wed, Mar 30, 2022 at 10:51:43PM +0200, Alvaro Herrera wrote: > On 2022-Feb-23, Justin Pryzby wrote: > > > I hope that Alvaro will comment on the simplified patches. If multiple > > people > > think the patch isn't worth it, feel free to close it. But I don'

Re: CLUSTER on partitioned index

2022-03-31 Thread Justin Pryzby
On Thu, Mar 31, 2022 at 12:54:36PM +0200, Alvaro Herrera wrote: > I realized after posting that we used to allow clustering toast tables, > but after my changes we no longer do. (Justin's version had a > RELKIND_HAS_STORAGE test here instead, which seemed a little too lax.) I > don't know why we a

Re: head fails to build on SLES 12 (wal_compression=zstd)

2022-03-31 Thread Justin Pryzby
On Thu, Mar 31, 2022 at 03:38:39PM +0100, Devrim Gündüz wrote: > On Thu, 2022-03-31 at 10:26 -0400, Tom Lane wrote: > > Devrim =?ISO-8859-1?Q?G=FCnd=FCz?= writes: > > > Latest snapshot tarball fails to build on SLES 12.5, which uses GCC > > > 4.8-8. Build log is attached. > > > > Hmm, what versio

Re: head fails to build on SLES 12 (wal_compression=zstd)

2022-03-31 Thread Justin Pryzby
On Thu, Mar 31, 2022 at 12:26:46PM -0400, Robert Haas wrote: > On Thu, Mar 31, 2022 at 11:44 AM Tom Lane wrote: > > Justin Pryzby writes: > > > Possible responses look like: > > > - Use 0 which also means "default" (need to verify that works across

Re: head fails to build on SLES 12 (wal_compression=zstd)

2022-03-31 Thread Justin Pryzby
On Thu, Mar 31, 2022 at 11:44:40AM -0400, Tom Lane wrote: > Justin Pryzby writes: > > Possible responses look like: > > - Use 0 which also means "default" (need to verify that works across > > versions); > > - Or #ifndef ZSTD_CLEVEL_DEFAULT #define ZSTD_CL

Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)

2022-03-31 Thread Justin Pryzby
On Mon, Mar 14, 2022 at 09:37:25PM -0500, Justin Pryzby wrote: > The original, minimal goal of this patch was to show shared tempdirs in > pg_ls_tmpfile() - rather than hiding them misleadingly as currently happens. > 20200310183037.ga29...@telsasoft.com > 20200313131232.go29...@t

Re: Rewriting the test of pg_upgrade as a TAP test - take three - remastered set

2022-03-31 Thread Justin Pryzby
On Thu, Mar 31, 2022 at 08:42:41PM -0700, Noah Misch wrote: > On Fri, Apr 01, 2022 at 10:16:48AM +0900, Michael Paquier wrote: > > On Thu, Mar 31, 2022 at 09:49:50AM -0400, Tom Lane wrote: > > > Well, let's go ahead with it and see what happens. If it's too > > > much of a mess we can always rever

Re: Use generation context to speed up tuplesorts

2022-04-01 Thread Justin Pryzby
SpaceType; +/* Bitwise option flags for tuple sorts */ +#define TUPLESORT_NONE 0 + +/* specifies whether non-sequential access to the sort result is required */ +#define TUPLESORT_RANDOMACCESS (1 << 0) + typedef struct TuplesortInstrumentation { TuplesortMethod sortMethod; /* sort algorithm

Re: Rewriting the test of pg_upgrade as a TAP test - take three - remastered set

2022-04-01 Thread Justin Pryzby
On Fri, Apr 01, 2022 at 08:53:10PM +0900, Michael Paquier wrote: > On Fri, Apr 01, 2022 at 03:01:38PM +0900, Michael Paquier wrote: > > On Thu, Mar 31, 2022 at 10:51:59PM -0500, Justin Pryzby wrote: > >> Is diff -q defined somewhere ? I can't find it in postgres sources nor

Re: [Proposal] vacuumdb --schema only

2022-04-01 Thread Justin Pryzby
On Wed, Mar 30, 2022 at 02:22:58PM -0700, Nathan Bossart wrote: > I'm personally -1 for the --exclude-schema option. I don't see any > existing "exclude" options in vacuumdb, and the uses for such an option > seem rather limited. If we can point to specific use-cases for this > option, I might be

Re: PostgreSQL shutdown modes

2022-04-01 Thread Justin Pryzby
Isn't this missing support in pg_dumb ?

Re: [PATCH] Add extra statistics to explain for Nested Loop

2022-04-02 Thread Justin Pryzby
nitialization. > > I am convinced that the latest changes have affected the patch in a positive > way. I'll be pleased to hear your thoughts on this. >From 0dec500a0ed934d5d2038cb087ba6a605cafcdef Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Thu, 15 Apr 2021 11:55:09 -0500 Subjec

Re: A qsort template

2022-04-02 Thread Justin Pryzby
On Sat, Apr 02, 2022 at 06:41:30PM +0700, John Naylor wrote: > On Sat, Apr 2, 2022 at 5:27 PM Thomas Munro wrote: > > Reproduced locally, using the same few lines from the cluster.sql > > test. I'll try to dig more tomorrow... > > Thanks! Unfortunately I can't reproduce locally with clang 13/gcc

Re: CLUSTER on partitioned index

2022-04-04 Thread Justin Pryzby
On Sat, Apr 02, 2022 at 07:11:47PM +0200, Alvaro Herrera wrote: > Thanks, pushed. Thank you for revisiting it, and thanks to Zhihong Yu for earlier review. I'll look into your outstanding questions later this week. -- Justin

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2022-04-06 Thread Justin Pryzby
"direct" path. A future patch adding wrappers for smgrwrite(), smgrextend(), and smgrimmedsync() would provide a good location to call pgstat_inc_ioop() for unbuffered IO and avoid regressions for future users of these functions. Author: Melanie Plageman Reviewed-by: Justin Pryzby D

Re: shared-memory based stats collector - v70

2022-04-06 Thread Justin Pryzby
On Wed, Apr 06, 2022 at 12:27:34PM -0700, Andres Freund wrote: > > > + next use of statistical information will cause a new snapshot to be > > > built > > > + or accessed statistics to be cached. > > > > I believe this should be an "and", not an "or". (next access builds both a > > new snapsh

Re: REINDEX blocks virtually any queries but some prepared queries.

2022-04-07 Thread Justin Pryzby
On Thu, Apr 07, 2022 at 01:37:57PM +0200, Frédéric Yhuel wrote: > Maybe something along this line? (patch attached) Some language fixes. I didn't verify the behavior, but +1 to document the practical consequences. I guess this is why someone invented REINDEX CONCURRENTLY. > From 4930bb8de182b7822

Re: WIP: WAL prefetch (another approach)

2022-04-07 Thread Justin Pryzby
y, I reported a similar issue last year. On Thu, Apr 08, 2021 at 10:37:04PM -0500, Justin Pryzby wrote: > --- a/doc/src/sgml/wal.sgml > +++ b/doc/src/sgml/wal.sgml > @@ -816,9 +816,7 @@ > prefetching mechanism is most likely to be effective on systems > with full_page_writes s

Re: How about a psql backslash command to show GUCs?

2022-04-07 Thread Justin Pryzby
On Wed, Apr 06, 2022 at 11:02:54PM -0400, Tom Lane wrote: > "Jonathan S. Katz" writes: > > +1 for \dconf > > Here's a draft patch using \dconf. No tests or docs yet. The patch as written is a thin layer around pg_settings. SHOW and current_setting() translate to human units, which is particula

Re: logical decoding and replication of sequences

2022-04-07 Thread Justin Pryzby
Some typos I found before the patch was reverted. diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml index a6ea6ff3fcf..d4bd8d41c4b 100644 --- a/doc/src/sgml/logicaldecoding.sgml +++ b/doc/src/sgml/logicaldecoding.sgml @@ -834,9 +834,8 @@ typedef void (*LogicalDeco

Re: Collecting statistics about contents of JSONB columns

2022-04-07 Thread Justin Pryzby
I noticed some typos. diff --git a/src/backend/utils/adt/jsonb_selfuncs.c b/src/backend/utils/adt/jsonb_selfuncs.c index f5520f88a1d..d98cd7020a1 100644 --- a/src/backend/utils/adt/jsonb_selfuncs.c +++ b/src/backend/utils/adt/jsonb_selfuncs.c @@ -1342,7 +1342,7 @@ jsonSelectivityContains(JsonStat

Re: [Proposal] vacuumdb --schema only

2022-04-07 Thread Justin Pryzby
On Wed, Apr 06, 2022 at 07:43:42PM +0200, Gilles Darold wrote: > Thanks for the review, all these changes are available in new version v6 > of the patch and attached here. This is failing in CI (except on macos, which is strangely passing). http://cfbot.cputube.org/gilles-darold.html https://api.

Re: pg14 psql broke \d datname.nspname.relname

2022-04-07 Thread Justin Pryzby
On Thu, Apr 07, 2022 at 10:26:18PM -0400, Robert Haas wrote: > + pg_log_error("improper relation name (too many dotted names): > %s", pattern); > > Come to think of it, maybe the error text there could stand some > bikeshedding, but AFAICS AFAICT the error text deliberately matches

Re: SQL/JSON: functions

2022-04-08 Thread Justin Pryzby
On Thu, Mar 31, 2022 at 04:25:58PM -0400, Andrew Dunstan wrote: > No code chunks left, only a documentation patch which should land Documentation review for a6baa4bad. > Construct a JSON the provided strings: a JSON what ? *from* the provided strings ? > Construct a JSON from the provided value

Re: Defer selection of asynchronous subplans until the executor initialization stage

2022-04-08 Thread Justin Pryzby
On Wed, Apr 06, 2022 at 03:58:29PM +0900, Etsuro Fujita wrote: > I have committed the patch after modifying it as such. (I think we > can improve these later, if necessary.) This patch seems to be causing the planner to crash. Here's a query reduced from sqlsmith. | explain SELECT 1 FROM informa

Re: Mingw task for Cirrus CI

2022-04-08 Thread Justin Pryzby
On Thu, Apr 07, 2022 at 10:10:21AM -0700, Andres Freund wrote: > Hi, > > On 2022-04-06 11:03:37 -0400, Andrew Dunstan wrote: > > On 3/30/22 20:26, Andres Freund wrote: > > > Could you try using dash to invoke configure here, and whether it makes > > > configure faster? > > I got weird failures re

Re: avoid multiple hard links to same WAL file after a crash

2022-04-08 Thread Justin Pryzby
On Fri, Apr 08, 2022 at 09:00:36PM -0400, Robert Haas wrote: > > I think there might be another problem. The man page for rename() seems to > > indicate that overwriting an existing file also introduces a window where > > the old and new path are hard links to the same file. This isn't a problem

Re: PostgreSQL commitfest: 2022-09-01

2022-04-08 Thread Justin Pryzby
On Fri, Apr 08, 2022 at 12:55:04PM +, webmas...@postgresql.org wrote: > The following patches that you follow, directly or indirectly, > have received updates in the PostgreSQL commitfest app: > > > Function to log backtrace of postgres processes > https://commitfest.postgresql.org/38/2863/ >

cfbot requests

2022-04-08 Thread Justin Pryzby
I mentioned most/all of these ideas for cfbot at some point. I'm writing them now so other people know about them and they're in once place. - Keep the original patch series and commit messages, rather than squishing them into a single commit with cfbot's own commit messages. Maybe append an em

Re: Schema variables - new implementation for Postgres 15+1

2022-04-10 Thread Justin Pryzby
On Sun, Apr 10, 2022 at 08:30:39PM +0200, Pavel Stehule wrote: > I am sending fresh rebased patch + separation to more patches. This split > is initial, and can be changed later The 0001 patch requires this, but it's not included until 0003. src/include/commands/session_variable.h Each patch shou

typos

2022-04-10 Thread Justin Pryzby
In docs and comments. Mostly for v15. Maybe Fabien will want to comment on the pgbench one. >From b1fb6d9bae1b0d69a30d59e65ed4f5a10f4141da Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Tue, 25 Jan 2022 10:47:55 -0600 Subject: [PATCH 01/19] doc: database SYSTEM si

Re: Defer selection of asynchronous subplans until the executor initialization stage

2022-04-10 Thread Justin Pryzby
On Sun, Apr 10, 2022 at 07:43:48PM +0900, Etsuro Fujita wrote: > On Sat, Apr 9, 2022 at 1:58 AM Etsuro Fujita wrote: > > On Fri, Apr 8, 2022 at 9:43 PM Justin Pryzby wrote: > > > This patch seems to be causing the planner to crash. > > > Here'

Re: typos

2022-04-11 Thread Justin Pryzby
r Masahiko may want to comment on 0012 (doc review: Add ALTER SUBSCRIPTION ... SKIP). >From 98778834b8c762a6cd76d8680191a7c866397d8b Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Wed, 16 Feb 2022 21:07:53 -0600 Subject: [PATCH 01/13] doc: Remove 'synchronized' from --n

Re: [PATCH] Add extra statistics to explain for Nested Loop

2022-04-11 Thread Justin Pryzby
On Tue, Apr 05, 2022 at 05:14:09PM -0400, Greg Stark wrote: > This is not passing regression tests due to some details of the plan > output - marking Waiting on Author: It's unstable due to parallel workers. I'm not sure what the usual workarounds here. Maybe set parallel_leader_participation=no f

Re: CLUSTER on partitioned index

2022-04-11 Thread Justin Pryzby
er;". Since clustering on a partitioned table wasn't supported before, nothing weird will happen to someone who upgrades to v15 unless they elect to use the new feature. As this seems to be POLA, it doesn't even need to be documented. ? >From 19c02209dfbcbf73494e1e6d3ca0db50f64dc5

Re: SQL/JSON: functions

2022-04-11 Thread Justin Pryzby
On Mon, Apr 11, 2022 at 11:54:11AM -0400, Andrew Dunstan wrote: > >> BTW, the documentation references look a little like OIDs... > >> Does someone already have an SNMP-based doc browser ? > >> | For details, see Section 9.16.3.4.2. > > > > I already had a couple of these items on my list but I ran

Re: CLUSTER on partitioned index

2022-04-13 Thread Justin Pryzby
On Wed, Apr 13, 2022 at 03:50:15PM +0900, Michael Paquier wrote: > > > That dates to a556549d7 (see also cbe24a6dd8 for an earlier commit in > > CLUSTER > > itself). The reason was to avoid blocking if an unprivileged user runs > > VACUUM > > FULL which would try to lock things (including share

Re: should frontend tools use syncfs() ?

2022-04-13 Thread Justin Pryzby
On Thu, Sep 30, 2021 at 12:49:36PM +0900, Michael Paquier wrote: > On Wed, Sep 29, 2021 at 07:43:41PM -0500, Justin Pryzby wrote: > > Forking this thread in which Thomas implemented syncfs for the startup > > process > > (61752afb2). > > https://www.postgresql.or

Re: typos

2022-04-13 Thread Justin Pryzby
ew ones match what I pushed. There were no changes - I had resent the patches that removed blank lines so it was apparent that they were "outstanding" / under discussion. There's (only) a few remaining. >From 543b9d77763da814cf1a99938252eb16a0a7d131 Mon Sep 17 00:00:00 2001

Re: TRAP: FailedAssertion("HaveRegisteredOrActiveSnapshot()", File: "toast_internals.c", Line: 670, PID: 19403)

2022-04-13 Thread Justin Pryzby
On Wed, Mar 30, 2022 at 05:58:24PM +0900, Kyotaro Horiguchi wrote: > I'm not still confident on this, but it should be better than the v1. +Andres as this seems to be related to 277692220. I added it as an Opened Item.

Re: typos

2022-04-13 Thread Justin Pryzby
On Thu, Apr 14, 2022 at 09:39:42AM +1200, David Rowley wrote: > On Thu, 14 Apr 2022 at 05:40, Justin Pryzby wrote: > > There's (only) a few remaining. > > I've pushed 0001 and 0002 of the 3rd batch of patches. I left 0003 as Thanks > I just didn't feel it wa

Re: fix cost subqueryscan wrong parallel cost

2022-04-15 Thread Justin Pryzby
On Fri, Apr 15, 2022 at 05:16:44PM +0800, Richard Guo wrote: > Not related to this topic but I noticed another problem from the plan. > Note the first Sort node which is to unique-ify the result of the UNION. > Why cannot we re-arrange the sort keys from (a, b, c) to (a, c, b) so > that we can avoi

Re: convert libpq uri-regress tests to tap test

2022-04-16 Thread Justin Pryzby
it's of any interest, I had done this differently a while back. This probably doesn't apply except on top of some other patches, but you get the idea. commit 923f8a1c2cbea35cb01d1599caa2a81e3186181c Author: Justin Pryzby Date: Mon Feb 28 01:31:10 2022 -0600 f! ci-os-only:

Re: Fwd: Declarative partitioning and partition pruning/check (+postgis)

2022-04-19 Thread Justin Pryzby
On Tue, Apr 19, 2022 at 02:39:12PM +0200, Mats Taraldsvik wrote: > I'm re-trying this email here, as there were no answers in the psql-general > list. Hope that's ok. (please cc me when answering as I'm not subscribed > (yet) ) -hackers is for development and bug reports, so this isn't the right p

Re: pg14 psql broke \d datname.nspname.relname

2022-04-19 Thread Justin Pryzby
On Tue, Apr 19, 2022 at 10:00:01AM -0400, Robert Haas wrote: > (A) This is a new feature. Wait for v16. > (B) This is a bug fix. Commit it now and back-patch to v14. > (C) This is a cleanup that is OK to put into v15 even after feature > freeze but since it is a behavior change we shouldn't back-pa

Re: [Proposal] vacuumdb --schema only

2022-04-20 Thread Justin Pryzby
On Wed, Apr 20, 2022 at 10:38:46AM -0700, Nathan Bossart wrote: > > +void > > +check_objfilter(VacObjectFilter curr_objfilter, VacObjectFilter > > curr_option) > > +{ > > + switch (curr_option) > > + { > > + case OBJFILTER_NONE: > > + break; > > + case OBJ

Re: typos

2022-04-20 Thread Justin Pryzby
On Wed, Apr 20, 2022 at 11:32:08PM +0200, Alvaro Herrera wrote: > On 2022-Apr-19, Alvaro Herrera wrote: > > > I propose we standardize on Zstd everywhere. > > Users can look it up if they're really interested. > > So the attached. > > There are other uses of zstd, but those are referring to > t

cirrus: run macos with COPY_PARSE_PLAN_TREES etc

2022-04-27 Thread Justin Pryzby
;t seem to be an issue, at least for those three defines. What about adding RELCACHE_FORCE_RELEASE, too ? Even with that, macos is only ~1min slower. https://cirrus-ci.com/task/5456727205216256 commit 53480b8db63b5cd2476142e28ed3f9fe8480f9f3 Author: Justin Pryzby Date: Thu Apr 14

XX000: tuple concurrently deleted during DROP STATISTICS

2023-11-08 Thread Justin Pryzby
I found this in our logs, and reproduced it under v11-v16. CREATE TABLE t(a int, b int); INSERT INTO t SELECT generate_series(1,999); CREATE STATISTICS t_stats ON a,b FROM t; while :; do psql postgres -qtxc "ANALYZE t"; done & while :; do psql postgres -qtxc "begin; DROP STATISTICS t_stats"; done

Re: ALTER TABLE uses a bistate but not for toast tables

2023-11-16 Thread Justin Pryzby
@cfbot: rebased >From a9bb61421c24bd3273a8519362febb4073c297a1 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Tue, 21 Jun 2022 22:28:06 -0500 Subject: [PATCH v4] WIP: use BulkInsertState for toast tuples, too DONE: ALTER, CLUSTER TODO: copyto, copyfrom? slot_getsomeat

processes stuck in shutdown following OOM/recovery

2023-11-30 Thread Justin Pryzby
If postgres starts, and one of its children is immediately killed, and the cluster is also told to stop, then, instead, the whole system gets wedged. $ initdb -D ./pgdev.dat1 $ pg_ctl -D ./pgdev.dat1 start -o '-c port=5678' $ kill -9 2524495; sleep 0.05; pg_ctl -D ./pgdev.dat1 stop -m fast # 25244

cannot abort transaction 2737414167, it was already committed

2023-12-27 Thread Justin Pryzby
We had this: < 2023-12-25 04:06:20.062 MST telsasoft >ERROR: could not open file "pg_tblspc/16395/PG_16_202307071/16384/121010871": Input/output error < 2023-12-25 04:06:20.062 MST telsasoft >STATEMENT: commit < 2023-12-25 04:06:20.062 MST telsasoft >WARNING: AbortTransaction while in COMMIT

Re: cannot abort transaction 2737414167, it was already committed

2023-12-27 Thread Justin Pryzby
On Thu, Dec 28, 2023 at 11:33:16AM +1300, Thomas Munro wrote: > I guess the large object usage isn't directly relevant (that module's > EOXact stuff seems to be finished before TRANS_COMMIT, but I don't > know that code well). Everything later is supposed to be about > closing/releasing/cleaning u

Re: warn if GUC set to an invalid shared library

2024-01-07 Thread Justin Pryzby
On Fri, Jul 22, 2022 at 03:26:47PM -0400, Tom Lane wrote: > Hmph. I wonder if we shouldn't change that, because it's a lie. > The value isn't actually coming from the config file, at least > not yet. On Thu, Jul 06, 2023 at 03:15:20PM -0500, Justin Pryzby wrote: > On

Re: cannot abort transaction 2737414167, it was already committed

2024-01-07 Thread Justin Pryzby
On Wed, Dec 27, 2023 at 09:02:25AM -0600, Justin Pryzby wrote: > We had this: > > < 2023-12-25 04:06:20.062 MST telsasoft >ERROR: could not open file > "pg_tblspc/16395/PG_16_202307071/16384/121010871": Input/output error > < 2023-12-25 04:06:20.062 MST telsaso

Re: should frontend tools use syncfs() ?

2023-09-01 Thread Justin Pryzby
> + if (!user_opts.sync_method) > + user_opts.sync_method = pg_strdup("fsync"); why pstrdup? > +parse_sync_method(const char *optarg, SyncMethod *sync_method) > +{ > + if (strcmp(optarg, "fsync") == 0) > + *sync_method = SYNC_METHOD_FSYNC; > +#ifdef HAVE_SYNCFS > +

<    7   8   9   10   11   12   13   14   15   16   >