Re: please update ps display for recovery checkpoint

2020-12-13 Thread Justin Pryzby
On Mon, Dec 14, 2020 at 12:01:33PM +0900, Michael Paquier wrote: > On Sat, Dec 12, 2020 at 12:41:25AM +, Bossart, Nathan wrote: > > On 12/11/20, 4:00 PM, "Michael Paquier" wrote: > >> My counter-proposal is like the attached, with the set/reset part not > >> reversed this time, and the code in

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-12-14 Thread Justin Pryzby
On Sat, Dec 12, 2020 at 01:45:26PM -0600, Justin Pryzby wrote: > On Sat, Dec 12, 2020 at 09:20:35AM +0100, Peter Eisentraut wrote: > > On 2020-12-11 21:27, Alvaro Herrera wrote: > > > By the way-- What did you think of the idea of explictly marking the > > > types us

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-12-15 Thread Justin Pryzby
On Mon, Dec 14, 2020 at 06:14:17PM -0600, Justin Pryzby wrote: > On Sat, Dec 12, 2020 at 01:45:26PM -0600, Justin Pryzby wrote: > > On Sat, Dec 12, 2020 at 09:20:35AM +0100, Peter Eisentraut wrote: > > > On 2020-12-11 21:27, Alvaro Herrera wrote: > > > > By the wa

Re: pg_upgrade test for binary compatibility of core data types

2020-12-16 Thread Justin Pryzby
On Sun, Dec 06, 2020 at 12:02:48PM -0600, Justin Pryzby wrote: > I meant to notice if the binary format is accidentally changed again, which > was > what happened here: > 7c15cef86 Base information_schema.sql_identifier domain on name, not varchar. > > I added a table to the r

[PATCH] nbtree: Do not show debugmessage if deduplication is disabled

2020-12-16 Thread Justin Pryzby
Even though the message literally says whether the index "can safely" or "cannot" use deduplication, the function specifically avoids the debug message for system columns, so I think it also makes sense to hide it when deduplication is turned off. diff --git a/src/backend/access/nbtree/nbtutils.c

Re: New Table Access Methods for Multi and Single Inserts

2020-12-16 Thread Justin Pryzby
Typos: + * 1) Specify is_multi as true, then multi insert state is allcoated. => allocated + * dropped, short-lived memory context is delted and mistate is freed up. => deleted + * 2) Currently, GetTupleSize() handles the existing heap, buffer, minmal and => minimal + /* Mulit insert state

Re: New Table Access Methods for Multi and Single Inserts

2020-12-17 Thread Justin Pryzby
On Thu, Dec 17, 2020 at 04:35:33PM +0530, Bharath Rupireddy wrote: > > You made the v2 insert interface a requirement for all table AMs. > > Should it be optional, and fall back to simple inserts if not implemented ? > > I tried to implement the APIs mentioned by Andreas here in [1]. I just > used

Re: allow to \dtS+ pg_toast.*

2020-12-17 Thread Justin Pryzby
Thanks for looking On Thu, Dec 17, 2020 at 04:16:52PM +0100, Laurenz Albe wrote: > On Mon, 2020-11-30 at 10:54 -0600, Justin Pryzby wrote: > > This makes toast tables a bit less special and easier to inspect. > > > > postgres=# \dtS+ pg_toast.pg_toast_2619 > > pg_to

Re: allow to \dtS+ pg_toast.*

2020-12-18 Thread Justin Pryzby
On Fri, Dec 18, 2020 at 12:43:07PM +0100, Laurenz Albe wrote: > On Fri, 2020-12-18 at 00:58 -0600, Justin Pryzby wrote: > > On Thu, Dec 17, 2020 at 04:16:52PM +0100, Laurenz Albe wrote: > > > On Mon, 2020-11-30 at 10:54 -0600, Justin Pryzby wrote: > > > > This makes t

Re: New Table Access Methods for Multi and Single Inserts

2020-12-18 Thread Justin Pryzby
On Fri, Dec 18, 2020 at 07:39:14AM +0530, Bharath Rupireddy wrote: > On Fri, Dec 18, 2020 at 2:14 AM Justin Pryzby wrote: > > Are you thinking that TableInsertState would eventually have additional > > attributes which would apply to other tableams, but not to heap ? Is > &

Re: terminate called after throwing an instance of 'std::bad_alloc'

2020-12-18 Thread Justin Pryzby
dres Freund wrote: > Hi, > > Sorry for the second send of this email, but somehow I managed to mangle > the headers in the last version I sent. > > On 2020-10-03 19:01:27 -0700, Andres Freund wrote: > > On 2020-10-03 18:30:46 -0500, Justin Pryzby wrote: > > > On Sa

v10 release notes for extended stats

2020-12-19 Thread Justin Pryzby
2017-03-24 [7b504eb28] Implement multivariate n-distinct coefficients 2017-04-05 [2686ee1b7] Collect and use multi-column dependency stats 2017-05-12 [bc085205c] Change CREATE STATISTICS syntax The existing notes say: |Add multi-column optimizer statistics to compute the correlation ratio and num

Re: New Table Access Methods for Multi and Single Inserts

2020-12-20 Thread Justin Pryzby
On Fri, Dec 18, 2020 at 11:54:39AM -0600, Justin Pryzby wrote: > On Fri, Dec 18, 2020 at 07:39:14AM +0530, Bharath Rupireddy wrote: > > On Fri, Dec 18, 2020 at 2:14 AM Justin Pryzby wrote: > > > Are you thinking that TableInsertState would eventually have additional > >

Re: zstd compression for pg_dump

2020-12-21 Thread Justin Pryzby
On Mon, Dec 21, 2020 at 03:02:40PM -0500, Tom Lane wrote: > Justin Pryzby writes: > > I found that our largest tables are 40% smaller and 20% faster to pipe > > pg_dump -Fc -Z0 |zstd relative to native zlib > > The patch might be a tad smaller if you hadn't included a

doc review for v14

2020-12-21 Thread Justin Pryzby
As I did last 2 years, I reviewed docs for v14... This year I've started early, since it takes more than a little effort and it's not much fun to argue the change in each individual hunk. -- Justin Pryzby System Administrator Telsasoft +1-952-707-

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-12-22 Thread Justin Pryzby
On Tue, Dec 22, 2020 at 03:47:57PM +0900, Michael Paquier wrote: > On Wed, Dec 16, 2020 at 10:01:11AM +0900, Michael Paquier wrote: > > On Tue, Dec 15, 2020 at 09:45:17PM -0300, Alvaro Herrera wrote: > > > I don't like this idea too much, because adding an option causes an ABI > > > break. I don't

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-12-22 Thread Justin Pryzby
On Tue, Dec 22, 2020 at 06:57:41PM +0900, Michael Paquier wrote: > On Tue, Dec 22, 2020 at 02:32:05AM -0600, Justin Pryzby wrote: > > Also, this one is going to be subsumed by ExecReindex(), so the palloc will > > go > > away (otherwise I would ask to pass it in from the call

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-12-22 Thread Justin Pryzby
On Tue, Dec 22, 2020 at 03:22:19PM -0800, Zhihong Yu wrote: > Justin: > For reindex_index() : > > + if (options->tablespaceOid == MyDatabaseTableSpace) > + options->tablespaceOid = InvalidOid; > ... > + oldTablespaceOid = iRel->rd_rel->reltablespace; > + if (set_tablespace && > +

Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)

2020-12-23 Thread Justin Pryzby
Maybe this patch needs to wait on somebody fixing our lack of real lstat() on > Windows. I think only the "top" patches depend on lstat (for the "type" column and recursion, to avoid loops). The initial patches are independently useful, and resolve the original issue of

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-12-23 Thread Justin Pryzby
On Wed, Dec 23, 2020 at 07:22:05PM -0300, Alvaro Herrera wrote: > Also: it seems a bit weird to me to put the flags inside the options > struct. I would keep them separate -- so initially the options struct > would only have the tablespace OID, on API cleanliness grounds: I don't see why they'd b

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2020-12-23 Thread Justin Pryzby
On Wed, Dec 23, 2020 at 09:14:18PM -0300, Alvaro Herrera wrote: > On 2020-Dec-23, Justin Pryzby wrote: > > > This was getting ugly: > > > > extern void reindex_index(Oid indexId, bool skip_constraint_checks, > > char relpersistence, int opt

Re: New Table Access Methods for Multi and Single Inserts

2020-12-24 Thread Justin Pryzby
On Thu, Dec 24, 2020 at 05:48:42AM +0530, Bharath Rupireddy wrote: > I'm not posting the updated 0002 to 0004 patches, I plan to do so > after a couple of reviews happen on the design of the APIs in 0001. > > Thoughts? Are you familiar with this work ? https://commitfest.postgresql.org/31/2717/

Re: Add table access method as an option to pgbench

2020-12-24 Thread Justin Pryzby
> --- a/doc/src/sgml/ref/pgbench.sgml > +++ b/doc/src/sgml/ref/pgbench.sgml > @@ -359,6 +359,16 @@ pgbench options > d > > > > + > + > --table-access-method=TABLEAM > + > + > +Create tables using the specified table access method, rather than > t

Re: Add table access method as an option to pgbench

2020-12-27 Thread Justin Pryzby
On Sun, Dec 27, 2020 at 09:14:53AM -0400, Fabien COELHO wrote: > > src/test/regress/sql/create_am.sql:CREATE ACCESS METHOD heap2 TYPE TABLE > > HANDLER heap_tableam_handler; > > ... > > src/test/regress/sql/create_am.sql:DROP ACCESS METHOD heap2; > > > Or maybe using SET default_tablespace instea

Re: pg_upgrade test for binary compatibility of core data types

2020-12-27 Thread Justin Pryzby
On Wed, Dec 16, 2020 at 11:22:23AM -0600, Justin Pryzby wrote: > On Sun, Dec 06, 2020 at 12:02:48PM -0600, Justin Pryzby wrote: > > I meant to notice if the binary format is accidentally changed again, which > > was > > what happened here: > > 7c15cef86 Base inform

Re: doc review for v14

2020-12-27 Thread Justin Pryzby
backpatched. doc/src/sgml/ref/pg_dump.sgml doc/src/sgml/sources.sgml doc/src/sgml/cube.sgml? doc/src/sgml/func.sgml? -- Justin >From 9c24fa421e423edb29fc866a70e935843dab2804 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Sat, 14 Nov 2020 23:09:21 -0600 Subject: [PATCH 1/7] typos in master

Re: create table like: ACCESS METHOD

2020-12-29 Thread Justin Pryzby
On Fri, Dec 25, 2020 at 03:41:46PM +0900, Michael Paquier wrote: > On Wed, Dec 09, 2020 at 02:13:29PM -0600, Justin Pryzby wrote: > > I thought this was a good idea, but didn't hear back when I raised it > > before. > > > > Failing to preserve access method i

Re: pglz compression performance, take two

2020-12-29 Thread Justin Pryzby
On Sat, Dec 26, 2020 at 12:06:59PM +0500, Andrey Borodin wrote: > > 12 дек. 2020 г., в 22:47, Andrey Borodin написал(а): > I've cleaned up comments, checked that memory alignment stuff actually make > sense for 32-bit ARM (according to Godbolt) and did some more code cleanup. > PFA v2 patch.

Re: adding wait_start column to pg_locks

2021-01-01 Thread Justin Pryzby
On Tue, Dec 15, 2020 at 11:47:23AM +0900, torikoshia wrote: > So I'm now thinking about adding a new column in pg_locks which > keeps the time at which locks started waiting. > > Attached a patch. This is failing make check-world, would you send an updated patch ? I added you as an author so it

Re: doc review for v14

2021-01-02 Thread Justin Pryzby
On Sun, Jan 03, 2021 at 03:10:54PM +0900, Michael Paquier wrote: > On Tue, Dec 29, 2020 at 06:22:43PM +0900, Michael Paquier wrote: > > Yes, I have done an extra effort on those fixes where needed. On top > > of that, I have included catalogs.sgml, pgstatstatements.sgml, > > explain.sgml, pg_verif

Re: [HACKERS] Custom compression methods

2021-01-03 Thread Justin Pryzby
The most recent patch doesn't compile --without-lz4: compress_lz4.c:191:17: error: ‘lz4_cmcheck’ undeclared here (not in a function) .datum_check = lz4_cmcheck, ... And fails pg_upgrade check, apparently losing track of the compression (?) CREATE TABLE public.cmdata2 ( -f1 text COMPRESSIO

Re: zstd compression for pg_dump

2021-01-03 Thread Justin Pryzby
On Mon, Dec 21, 2020 at 01:49:24PM -0600, Justin Pryzby wrote: > a big disadvantage of piping through zstd is that it's not identified as a > PGDMP file, and, /usr/bin/file on centos7 fails to even identify zstd by its > magic number.. Other reasons are that pg_dump |zstd >out

Re: zstd compression for pg_dump

2021-01-03 Thread Justin Pryzby
On Mon, Jan 04, 2021 at 11:04:57AM +0500, Andrey Borodin wrote: > > 4 янв. 2021 г., в 07:53, Justin Pryzby написал(а): > > Note, there's currently several "compression" patches in CF app. This patch > > seems to be independent of the others, but probably shoul

Re: PoC/WIP: Extended statistics on expressions

2021-01-04 Thread Justin Pryzby
On Mon, Jan 04, 2021 at 03:34:08PM +, Dean Rasheed wrote: > * I'm not sure I understand the need for 0001. Wasn't there an earlier > version of this patch that just did it by re-populating the type > array, but which still had it as an array rather than turning it into > a list? Making it a lis

[PATCH]: Allow errors in parameter values to be reported during the BIND phase itself..

2021-01-04 Thread Justin Pryzby
e. Think 900 int/smallint columns, or less-wide tables being inserted multiple rows at a time with VALUES(),(),()... Of course, this isn't as good as showing the column name, so I might pursue Tom's suggestion for that at some point. -- Justin >From 40dd798d8518b4d113dbf70c37687c87bf

Re: Online checksums patch - once again

2021-01-05 Thread Justin Pryzby
On Tue, Jan 05, 2021 at 09:29:31PM +0100, Michael Banck wrote: > > @@ -4938,13 +4949,299 @@ GetMockAuthenticationNonce(void) > > /* > > - * Are checksums enabled for data pages? > > + * DataChecksumsNeedWrite > > + * Returns whether data checksums must be written or not > > + * > > + * Are

Re: allow to \dtS+ pg_toast.*

2021-01-05 Thread Justin Pryzby
On Tue, Jan 05, 2021 at 06:46:01PM -0500, Tom Lane wrote: > Laurenz Albe writes: > > On Fri, 2020-12-18 at 11:33 -0600, Justin Pryzby wrote: > >>>> This makes toast tables a bit less special and easier to inspect. > > Pushed, except for Thank you -- Justin

Re: [PATCH] Simple progress reporting for COPY command

2021-01-06 Thread Justin Pryzby
ed properly, and so on. Some more doc fixes. >From 0616f448057905ab9b3218ebddfdd3af52e62bac Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Wed, 6 Jan 2021 19:08:11 -0600 Subject: [PATCH] doc review: COPY progress: 8a4f618e7ae3cb11b0b37d0f06f05c8ff905833f --- doc/src/sgml/monitoring.sgml | 10 +- 1 fi

Re: PoC/WIP: Extended statistics on expressions

2021-01-07 Thread Justin Pryzby
On Fri, Jan 08, 2021 at 01:57:29AM +0100, Tomas Vondra wrote: > Attached is a patch fixing most of the issues. There are a couple > exceptions: In the docs: + — at the cost that its schema must be extended whenever the structure

Re: libpq compression

2021-01-09 Thread Justin Pryzby
do not allow compression from LOCAL connection" or "do not allow compression on encrypted connection". Maybe this would be somehow integrated into pg_hba. But maybe it's not needed (a separate user could be used to allow/disallow compression). -- Justin >From 5680eae56

Re: [HACKERS] Custom compression methods

2021-01-10 Thread Justin Pryzby
On Mon, Jan 04, 2021 at 04:57:16PM +0530, Dilip Kumar wrote: > On Mon, Jan 4, 2021 at 6:52 AM Justin Pryzby wrote: > > And fails pg_upgrade check, apparently losing track of the compression (?) > > > > CREATE TABLE public.cmdata2 ( > > -f1 text COMPRESSION lz4 >

Re: ALTER TABLE .. DETACH PARTITION CONCURRENTLY

2021-01-10 Thread Justin Pryzby
n't had time to investigate this problem yet. I guess it's because you commited the txn and released lock in the middle of the command. -- Justin >From e18c11fd5bcc4f5cd981a3219383265b55974f34 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Sun, 10 Jan 2021 15:41:43 -0600 Subj

Re: pg13.2: invalid memory alloc request size NNNN

2021-02-12 Thread Justin Pryzby
On Thu, Feb 11, 2021 at 07:48:37PM -0600, Justin Pryzby wrote: > #3 0x009fb149 in text_to_cstring (t=0x2aaae8023010) at varlena.c:212 > 212 result = (char *) palloc(len + 1); > > (gdb) l > 207 /* must cast away the const, unfortunately */ > 20

Re: pg13.2: invalid memory alloc request size NNNN

2021-02-12 Thread Justin Pryzby
On Fri, Feb 12, 2021 at 06:44:54PM +0100, Tomas Vondra wrote: > > (gdb) p len > > $1 = -4 > > > > This VM had some issue early today and I killed the VM, causing PG to > > execute > > recovery. I'm tentatively blaming that on zfs, so this could conceivably > > be a > > data error (although reco

Re: Improvements and additions to COPY progress reporting

2021-02-12 Thread Justin Pryzby
--- a/doc/src/sgml/ref/analyze.sgml +++ b/doc/src/sgml/ref/analyze.sgml

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-02-14 Thread Justin Pryzby
On Tue, Feb 09, 2021 at 04:35:03AM +, tsunakawa.ta...@fujitsu.com wrote: > Rebased to HEAD with the following modifications. It passes make check in > the top directory and contrib/postgres_fdw. > That said, with the reviews from some people and good performance results, I > think this can b

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2021-02-14 Thread Justin Pryzby
get_rel_name(relationOid; - >From b4347c18bc732d30295c065ef71edaac65e68fe6 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Sun, 14 Feb 2021 19:49:52 -0600 Subject: [PATCH] Dead code: REINDEX (CONCURRENTLY, TABLESPACE ..): c

Re: CREATE INDEX CONCURRENTLY on partitioned index

2021-02-15 Thread Justin Pryzby
On Mon, Feb 15, 2021 at 10:06:47PM +0300, Anastasia Lubennikova wrote: > On 28.01.2021 17:30, Justin Pryzby wrote: > > On Thu, Jan 28, 2021 at 09:51:51PM +0900, Masahiko Sawada wrote: > > > On Mon, Nov 30, 2020 at 5:22 AM Justin Pryzby > > > wrote: > > >

progress reporting for partitioned REINDEX

2021-02-15 Thread Justin Pryzby
ag, like: params->options & REINDEXOPT_REPORT_PROGRESS_AM I don't understand why e66bcfb4c added multiple calls to pgstat_progress_start_command(). -- Justin >From b04d46958b16bc07ecd6c2f07220599ecb304b4d Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Mon, 15 Feb 2021 14:12:32 -0600

Re: progress reporting for partitioned REINDEX

2021-02-16 Thread Justin Pryzby
On Wed, Feb 17, 2021 at 02:55:04PM +0900, Michael Paquier wrote: > I see no bug here. pg_stat_progress_create_index includes partitions_{done,total} for CREATE INDEX p, so isn't it strange if it wouldn't do likewise for REINDEX INDEX p ? -- Justin

Re: progress reporting for partitioned REINDEX

2021-02-17 Thread Justin Pryzby
On Wed, Feb 17, 2021 at 03:36:20PM +0900, Michael Paquier wrote: > On Wed, Feb 17, 2021 at 12:10:43AM -0600, Justin Pryzby wrote: > > On Wed, Feb 17, 2021 at 02:55:04PM +0900, Michael Paquier wrote: > >> I see no bug here. > > > > pg_stat_progress_create_index includ

Re: progress reporting for partitioned REINDEX

2021-02-18 Thread Justin Pryzby
On Fri, Feb 19, 2021 at 03:06:04PM +0900, Michael Paquier wrote: > On Thu, Feb 18, 2021 at 02:17:00PM +0900, Michael Paquier wrote: > > I have no issues with documenting more precisely on which commands > > partitions_total and partitions_done apply currently, by citing the > > commands where these

Re: progress reporting for partitioned REINDEX

2021-02-19 Thread Justin Pryzby
On Sat, Feb 20, 2021 at 10:37:08AM +0900, Michael Paquier wrote: > > Also, I noticed that vacuum recurses into partition heirarchies since v10, > > but > > pg_stat_progress_vacuum also doesn't show anything about the parent table or > > the progress of recursing through the hierarchy. > > Yeah, t

Re: compression libraries and CF bot

2021-02-20 Thread Justin Pryzby
On Wed, Jan 20, 2021 at 10:29:05AM +1300, Thomas Munro wrote: > I have added lz4 to the FreeBSD and Ubuntu build tasks, so we'll see > if that helps at the next periodic build or when a new patch is > posted. It's failing on Windows because there is no HAVE_LIBLZ4 in > Solution.pm, and I don't kno

Re: Fallback table AM for relkinds without storage

2021-02-21 Thread Justin Pryzby
On Mon, Feb 15, 2021 at 04:21:38PM +0900, Michael Paquier wrote: > On Tue, Feb 09, 2021 at 04:27:34PM +0900, Michael Paquier wrote: > > Putting sanity checks within all the table_* functions of tableam.h > > would not be a good idea, as nothing prevents the call of what's > > stored in rel->rd_tabl

Re: How to customize postgres for sharing read-only tables in multiple data-dirs between servers

2021-02-21 Thread Justin Pryzby
On Tue, Feb 16, 2021 at 01:49:02PM +, Guttman, Maoz wrote: > Hi, > > Problem statement: > I have to develop a solution in which a single source populates a table. Once > the table is populated, it is considered as read-only and then we run many > read-only queries on it. > Such read-only tab

Re: [HACKERS] GSoC 2017: Foreign Key Arrays

2021-02-21 Thread Justin Pryzby
On Tue, Feb 16, 2021 at 12:07:10PM +0200, Mark Rofail wrote: ... There's some errors in the latest patch: http://cfbot.cputube.org/mark-rofail.html gram.y:16933:20: error: invalid operands to binary expression ('List' (aka 'struct List') and 'void *') Assert(**reftypes != NULL); Did yo

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2021-02-21 Thread Justin Pryzby
On Tue, Feb 16, 2021 at 11:15:51PM +1300, David Rowley wrote: > To summarise here, the planner performance gets a fair bit worse with > the patched code. With master, summing the average planning time over > each of the queries resulted in a total planning time of 765.7 ms. > After patching, that w

Re: should INSERT SELECT use a BulkInsertState?

2021-02-21 Thread Justin Pryzby
ce this should maybe be built on top of the table-am patch (2871). Have fun :) -- Justin >From e2b93b3b3aaa32f680193b42d91a80bab40768a4 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Fri, 8 May 2020 02:17:32 -0500 Subject: [PATCH v10 1/4] INSERT SELECT to use BulkInsertState and mult

Re: [PATCH]: Allow errors in parameter values to be reported during the BIND phase itself..

2021-02-21 Thread Justin Pryzby
On Mon, Jan 04, 2021 at 11:09:39AM -0600, Justin Pryzby wrote: > For example: > > $ python3.5 -c "import pg; db=pg.DB(); q = db.query(\"SET > log_parameter_max_length_on_error=-1;\"); db.prepare('p', 'SELECT > \$1::smallint'); db.query_prepared

Re: doc review for v14

2021-02-22 Thread Justin Pryzby
Rebased, with a few additions. >From 27656ceb05d495300d86df40a1d25b0b454fa18b Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Fri, 8 Jan 2021 13:09:55 -0600 Subject: [PATCH 01/21] doc: pageinspect d6061f83a166b015657fda8623c704fcb86930e9 backpatch to 9.6? --- doc/src/sgml/pageinspect.s

Re: [PATCH]: Allow errors in parameter values to be reported during the BIND phase itself..

2021-02-22 Thread Justin Pryzby
On Sun, Feb 21, 2021 at 11:05:26PM -0600, Justin Pryzby wrote: > On Mon, Jan 04, 2021 at 11:09:39AM -0600, Justin Pryzby wrote: > > For example: > > > > $ python3.5 -c "import pg; db=pg.DB(); q = db.query(\"SET > > log_parameter_max_length_on_error=-1;

Re: [PATCH]: Allow errors in parameter values to be reported during the BIND phase itself..

2021-02-23 Thread Justin Pryzby
On Mon, Feb 22, 2021 at 11:15:25AM -0600, Justin Pryzby wrote: > On Sun, Feb 21, 2021 at 11:05:26PM -0600, Justin Pryzby wrote: > > On Mon, Jan 04, 2021 at 11:09:39AM -0600, Justin Pryzby wrote: > > > For example: > > > > > > $ python3.5 -c "

Re: doc review for v14

2021-02-23 Thread Justin Pryzby
On Wed, Feb 24, 2021 at 04:18:51PM +0900, Michael Paquier wrote: > On Mon, Feb 22, 2021 at 02:03:45AM -0600, Justin Pryzby wrote: > > Rebased, with a few additions. > > Thanks. I have done a pass through this series, and applied most of > this stuff with a backpatch fo

Re: Improvements and additions to COPY progress reporting

2021-02-23 Thread Justin Pryzby
On Sun, Feb 21, 2021 at 08:10:09PM +0100, Matthias van de Meent wrote: > Subject: [PATCH v9 1/3] Add progress-reported components for COPY progress > reporting > /* Increment amount of processed tuples and update the > progress */ > /* Increment amount of processed tu

Re: Postgresql network transmission overhead

2021-02-24 Thread Justin Pryzby
On Wed, Feb 24, 2021 at 09:14:19AM +0100, Michael J. Baars wrote: > I've written this function to insert several rows at once, and noticed a > certain postgresql overhead as you can see from the log file. A lot more data > than the > user data is actually sent over the net. This has a certain not

Re: 64-bit XIDs in deleted nbtree pages

2021-02-24 Thread Justin Pryzby
uples, so B-tree index [-scan-]{+scans+} at the cleanup stage [-can only-]{+cannot+} be skipped [-if the second and-] [-subsequent VACUUM cycles detect-]{+until after a vacuum cycle+} {+ which detects+} no dead tuples. >From 44cf90c5b06fb21c2c4d379568b9fe

Re: CREATE INDEX CONCURRENTLY on partitioned index

2021-02-26 Thread Justin Pryzby
ut that - it's already documented in general, for nonpartitioned indexes. -- Justin >From fb60da3c0fac8f1699a6caeea57476770c66576d Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Sat, 6 Jun 2020 17:42:23 -0500 Subject: [PATCH 1/5] Allow CREATE INDEX CONCURRENTLY on partitioned table

Re: [HACKERS] Custom compression methods

2021-02-27 Thread Justin Pryzby
> Subject: [PATCH v28 3/4] Add default_toast_compression GUC This part isn't working. My first patch worked somewhat better: due to doing strcmp() with the default GUC, it avoided using the cached AM OID. (But it would've failed with more than 2 AMs, since the cache wasn't invalidated, since I c

Re: Allow matching whole DN from a client certificate

2021-02-27 Thread Justin Pryzby
On Sat, Jan 30, 2021 at 04:18:12PM -0500, Andrew Dunstan wrote: > @@ -610,6 +610,19 @@ hostnogssenc database > user the verification of client certificates with any authentication > method that supports hostssl entries. > > + > + On any record using client cer

Re: Extending range type operators to cope with elements

2021-02-27 Thread Justin Pryzby
On Fri, Oct 30, 2020 at 11:08:19PM +0100, Tomas Vondra wrote: > Hi, > > + > + > +anyelement >> > anyrange > +boolean > + > + > +Is the element strictly right of the element? > + should say "of the range" ? > +++ b/src/backend/utils/adt/r

Re: [HACKERS] Custom compression methods

2021-02-27 Thread Justin Pryzby
On my PC, this new change is causing a test failure: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; - substr - - 01234567890123456789012345678901234567890123456789 -(1 row) - +ERROR: compressed lz4 data i

[PATCH] refactor ATExec{En,Dis}ableRowSecurity

2021-02-28 Thread Justin Pryzby
tablecmds.c is 17k lines long, this makes it ~30 lines shorter. >From 2e9500227d45142eb00e9e1ebee001642a834518 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Sat, 27 Feb 2021 20:39:10 -0600 Subject: [PATCH 1/2] Refactor ATExec{En,Dis}ableRowSecurity in the style

alter table set TABLE ACCESS METHOD

2021-02-28 Thread Justin Pryzby
t seems like a minor addition, but may not be for v14. https://www.postgresql.org/message-id/20190818193533.gl11...@telsasoft.com On Sun, Aug 18, 2019 at 02:35:33PM -0500, Justin Pryzby wrote: > . What do you think about pg_restore --no-tableam; similar to >--no-tablespaces,

Re: Different compression methods for FPI

2021-02-28 Thread Justin Pryzby
On Sat, Feb 27, 2021 at 12:43:52PM +0500, Andrey Borodin wrote: > So I think it worth to propose a patch to make wal_compression_method = > {"pglz", "lz4", "zlib"}. Probably, "zstd" can be added to the list. > Attached is a draft taking CompressionId from "custom compression methods" > patch and

Re: doc review for v14

2021-02-28 Thread Justin Pryzby
ml/wal.sgml | 18 +- I'm suggesting to either revert that part, or apply these more polished changes in 0002. -- Justin >From 03d014809720d90ba43c780cb34fc82dd7173c8d Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Fri, 26 Feb 202

Re: alter table set TABLE ACCESS METHOD

2021-02-28 Thread Justin Pryzby
On Mon, Mar 01, 2021 at 11:16:36AM +0900, Michael Paquier wrote: > On Sun, Feb 28, 2021 at 04:25:30PM -0600, Justin Pryzby wrote: > > I called this "set TABLE access method" rather than just "set access method" > > for the reasons given on the LIKE thread: > &

Re: doc review for v14

2021-02-28 Thread Justin Pryzby
On Mon, Mar 01, 2021 at 01:11:10PM +0900, Michael Paquier wrote: > On Sun, Feb 28, 2021 at 06:46:47PM -0600, Justin Pryzby wrote: > > It looks like you applied 0010...but I agree that it's not an improvement. > > It > > appears that's something I intended to go b

Re: Different compression methods for FPI

2021-02-28 Thread Justin Pryzby
On Mon, Mar 01, 2021 at 01:57:12PM +0900, Michael Paquier wrote: > On Sun, Feb 28, 2021 at 05:08:17PM -0600, Justin Pryzby wrote: > > Does this need to patch ./configure{,.ac} and Solution.pm for HAVE_LIBLZ4 ? > > I suggest to also include an 0002 patch (not for commit) which change

Re: [HACKERS] Custom compression methods

2021-02-28 Thread Justin Pryzby
On Mon, Mar 01, 2021 at 10:32:23AM +0530, Dilip Kumar wrote: > On Sun, Feb 28, 2021 at 9:48 AM Justin Pryzby wrote: > > > > On my PC, this new change is causing a test failure: > > > > SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; > &g

Re: Table AM modifications to accept column projection lists

2021-03-01 Thread Justin Pryzby
On Thu, Dec 31, 2020 at 01:02:24PM -0800, Soumyadeep Chakraborty wrote: > Hey Masahiko, > > I added it to the Jan CF (https://commitfest.postgresql.org/31/2922/). > > PFA a rebased version against latest head. Thanks for working on this. - pgstat_progress_update_param(PROGRESS_AN

Re: 2019-03 CF now in progress

2021-03-01 Thread Justin Pryzby
Could I suggest to update the CF APP to allow: | Target version: 15 Also, I wanted to suggest that toward the end of the devel cycle, that committers set/unset target version to allow more focused review effort. And so it's not a total surprise when something isn't progressed. And as a simple way

Re: [PATCH] Automatic HASH and LIST partition creation

2021-03-02 Thread Justin Pryzby
https://commitfest.postgresql.org/32/2694/ I don't know what committers will say, but I think that "ALTER TABLE" might be the essential thing for this patch to support, not "CREATE". (This is similar to ALTER..SET STATISTICS, which is not allowed in CREATE.) The reason is that ALTER is what's im

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-03 Thread Justin Pryzby
e parts as my "bulk insert" patch: https://commitfest.postgresql.org/32/2553/ -- Justin >From f7bb368963f5808bc5126f179b78507ca52b9cd2 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Wed, 24 Feb 2021 02:23:17 -0600 Subject: [PATCH] language fixen --- contrib/postgres_fdw/postgres_fdw.

Re: [HACKERS] GSoC 2017: Foreign Key Arrays

2021-03-03 Thread Justin Pryzby
On Wed, Mar 03, 2021 at 11:31:49PM +0200, Mark Rofail wrote: > This is just a rebase patch since the patch is no longer applicable to the > current master. It doesn't just rebase: it also removes the test which was failing on windows CI: --- Try using the indexable operator -SELECT * FROM FKTABLE

Re: PATCH: Batch/pipelining support for libpq

2021-03-03 Thread Justin Pryzby
I'm proposing some minor changes. -- Justin >From 35ed4dc1fc770834972396f7eeed142f6dabee88 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Wed, 3 Mar 2021 16:36:40 -0600 Subject: [PATCH 2/2] doc review --- doc/src/sgml/libpq.sgml| 67 +--

Re: Improvements and additions to COPY progress reporting

2021-03-04 Thread Justin Pryzby
On Thu, Mar 04, 2021 at 05:19:18PM +0100, Matthias van de Meent wrote: > On Thu, 4 Mar 2021 at 13:36, Bharath Rupireddy > wrote: > > > > + > > +Each backend running VACUUM without the > > +FULL option will report its progress in the > > +pg_stat_progress_vacuum view. Backends runni

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2021-03-04 Thread Justin Pryzby
On Thu, Mar 04, 2021 at 12:01:37PM -0300, Alvaro Herrera wrote: > Apparently, the archives system or the commitfest system is not picking > up new messages to the thread, so the CF app is trying to apply a > very old patch version. I'm not sure what's up with that. Thomas, any > clues on where to

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-04 Thread Justin Pryzby
I think this change to the regression tests is suspicous: -CONTEXT: remote SQL command: INSERT INTO public.loc2(f1, f2) VALUES ($1, $2) -COPY rem2, line 1: "-1 xyzzy" +CONTEXT: COPY loc2, line 1: "-1 xyzzy" +remote SQL command: COPY public.loc2(f1, f2) FROM STDIN +COPY rem2, line 2 I thi

Re: PoC/WIP: Extended statistics on expressions

2021-03-04 Thread Justin Pryzby
On Mon, Jan 04, 2021 at 09:45:24AM -0600, Justin Pryzby wrote: > On Mon, Jan 04, 2021 at 03:34:08PM +, Dean Rasheed wrote: > > * I'm not sure I understand the need for 0001. Wasn't there an earlier > > version of this patch that just did it by re-populating the type &

Re: [PATCH] remove deprecated v8.2 containment operators

2021-03-04 Thread Justin Pryzby
On Thu, Mar 04, 2021 at 08:58:39PM -0500, Tom Lane wrote: > Justin Pryzby writes: > > [ 0001-remove-deprecated-v8.2-containment-operators.patch ] > > I'm confused by why this patch is only dropping the operators' > opclass-membership links. Don't we want to ac

Re: [HACKERS] Custom compression methods

2021-03-06 Thread Justin Pryzby
On Sat, Mar 06, 2021 at 08:59:16PM +0530, Dilip Kumar wrote: > - Alter table set compression, will not rewrite the old data, so only > the new tuple will be compressed with the new compression method. > - No preserve. +1, this simplifies things. If someone *wants* to rewrite the table, they can V

Re: CLUSTER on partitioned index

2021-03-06 Thread Justin Pryzby
On Wed, Feb 10, 2021 at 02:04:58PM -0600, Justin Pryzby wrote: > On Sat, Feb 06, 2021 at 08:45:49AM -0600, Justin Pryzby wrote: > > On Mon, Jan 18, 2021 at 12:34:59PM -0600, Justin Pryzby wrote: > > > On Sat, Nov 28, 2020 at 08:03:02PM -0600, Justin Pryzby wrote: > > > &

Re: [HACKERS] Custom compression methods

2021-03-06 Thread Justin Pryzby
On Sun, Mar 07, 2021 at 12:16:41PM +0530, Dilip Kumar wrote: > > If I pg_upgrade from an binary with-lz4 to one without-lz4, it fails > > while restoring the schema, after running check, which is bad: > > | pg_restore: error: could not execute query: ERROR: not built with lz4 > > support > > |CRE

Re: [HACKERS] Custom compression methods

2021-03-07 Thread Justin Pryzby
On Sun, Mar 07, 2021 at 01:36:50PM +0530, Dilip Kumar wrote: > On Sun, Mar 7, 2021 at 12:47 PM Justin Pryzby wrote: > > > IMHO we can always allow creating the table with lz4 and only error > > > out when we really need to compress/decompress the data. I like this > &g

Re: alter table set TABLE ACCESS METHOD

2021-03-07 Thread Justin Pryzby
On Mon, Mar 01, 2021 at 11:16:36AM +0900, Michael Paquier wrote: > On Sun, Feb 28, 2021 at 04:25:30PM -0600, Justin Pryzby wrote: > > I called this "set TABLE access method" rather than just "set access method" > > for the reasons given on the LIKE thread: > &

Re: [HACKERS] Custom compression methods

2021-03-08 Thread Justin Pryzby
On Sun, Mar 07, 2021 at 06:04:41PM +0530, Dilip Kumar wrote: > On Sun, Mar 7, 2021 at 2:19 PM Justin Pryzby wrote: > > > > Earlier in this thread, I suggested to implement an option to pg_restore to > > avoid outputting compression, in order to allow restoring with a differen

Re: [HACKERS] Custom compression methods

2021-03-08 Thread Justin Pryzby
On Mon, Mar 08, 2021 at 03:26:04PM -0500, Robert Haas wrote: > On Mon, Mar 8, 2021 at 5:02 AM Dilip Kumar wrote: > > So now only pending point is, how do we handle the upgrade when you > > are upgrading from --with-lz4 to --without-lz4 binary and a couple of > > options discussed here are > > a)

Re: [HACKERS] Custom compression methods

2021-03-08 Thread Justin Pryzby
On Mon, Mar 08, 2021 at 03:32:39PM +0530, Dilip Kumar wrote: > On Sun, Mar 7, 2021 at 1:27 AM Justin Pryzby wrote: > > > > On Sat, Mar 06, 2021 at 08:59:16PM +0530, Dilip Kumar wrote: > > > - Alter table set compression, will not rewrite the old data, so only >

Re: [HACKERS] Custom compression methods

2021-03-08 Thread Justin Pryzby
On Tue, Mar 09, 2021 at 01:04:10PM +0530, Dilip Kumar wrote: > On Tue, Mar 9, 2021 at 2:45 AM Robert Haas wrote: > > > > On Mon, Mar 8, 2021 at 3:59 PM Justin Pryzby wrote: > > > > It would be nice to have a way to force > > > > anything compressed with

<    1   2   3   4   5   6   7   8   9   10   >