Re: [HACKERS] [PATCH] Generic type subscripting

2020-09-09 Thread Justin Pryzby
On Wed, Aug 05, 2020 at 04:04:22PM +0200, Dmitry Dolgov wrote: > > On Sun, Aug 02, 2020 at 12:50:12PM +0200, Pavel Stehule wrote: > > > > > > > Maybe this could be salvaged by flushing 0005 in its current form and > > > > having the jsonb subscript executor do something like "if the current > > > >

Re: v13: show extended stats target in \d

2020-09-09 Thread Justin Pryzby
.postgresql.org/docs/13/app-psql.html As for the discussion about a separator, I think maybe a comma is enough. I doubt anyone is going to think that you can get a valid command by prefixing this by "CREATE STATISTICS". Actually, it didn't even occur to me it was valid command with

Re: v13: show extended stats target in \d

2020-09-09 Thread Justin Pryzby
On Wed, Sep 09, 2020 at 07:22:30PM -0300, Alvaro Herrera wrote: > On 2020-Sep-09, Justin Pryzby wrote: > > > As for the discussion about a separator, I think maybe a comma is enough. I > > doubt anyone is going to think that you can get a valid command by prefixing > > th

Re: please update ps display for recovery checkpoint

2020-09-09 Thread Justin Pryzby
On Mon, Aug 31, 2020 at 03:52:44PM +0900, Michael Paquier wrote: > On Thu, Aug 20, 2020 at 05:09:05PM +0900, Michael Paquier wrote: > > That could be helpful. Wouldn't it be better to use "end-of-recovery > > checkpoint" instead? That's the common wording in the code comments. > > > > I don't se

Re: Fix for parallel BTree initialization bug

2020-09-09 Thread Justin Pryzby
On Tue, Sep 08, 2020 at 06:25:03PM +, Jameson, Hunter 'James' wrote: > Hi, I ran across a small (but annoying) bug in initializing parallel BTree > scans, which causes the parallel-scan state machine to get confused. The fix > is one line; the description is a bit longer— What postgres versi

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-09-10 Thread Justin Pryzby
On Thu, Sep 10, 2020 at 11:21:02AM -0400, Robert Haas wrote: > On Fri, Aug 28, 2020 at 5:55 AM Ashutosh Sharma wrote: > > Please have a look into the attached patch for the changes and let me know > > for any other concerns. Thank you. > > I have committed this version. Thanks ; I marked it as

Re: PG 13 release notes, first draft

2020-09-10 Thread Justin Pryzby
On Mon, Sep 07, 2020 at 08:40:26AM -0500, Justin Pryzby wrote: Rebasing onto 3965de54e718600a4703233936e56a3202caf73f, I'm left with: diff --git a/doc/src/sgml/release-13.sgml b/doc/src/sgml/release-13.sgml index 8fffc6fe0a..69d143e10c 100644 --- a/doc/src/sgml/release-13.sgml +++ b/doc/src

Re: Fix for parallel BTree initialization bug

2020-09-10 Thread Justin Pryzby
Against all odds, I was able to reproduce this. begin; CREATE TABLE t AS SELECT generate_series(1,99)i; ALTER TABLE t SET (parallel_workers=2, autovacuum_enabled=off); CREATE INDEX ON t(i); commit; SET parallel_leader_participation=off; SET min_parallel_table_scan_size=0; SET enable_bitmapsc

Re: Reword docs of feature "Remove temporary files after backend crash"

2021-10-11 Thread Justin Pryzby
On Mon, Oct 11, 2021 at 12:50:28PM +0530, Bharath Rupireddy wrote: > I noticed another thing that the remove_temp_files_after_crash is > categorized as DEVELOPER_OPTIONS, shouldn't it be under > RESOURCES_DISK? See here: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=797b0fc0b078c7

pg14 psql broke \d datname.nspname.relname

2021-10-11 Thread Justin Pryzby
This commit broke psql \d datname.nspname.relname commit 2c8726c4b0a496608919d1f78a5abc8c9b6e0868 Author: Robert Haas Date: Wed Feb 3 13:19:41 2021 -0500 Factor pattern-construction logic out of processSQLNamePattern. ... patternToSQLRegex is a little more general than what is required

Re: pg14 psql broke \d datname.nspname.relname

2021-10-11 Thread Justin Pryzby
On Mon, Oct 11, 2021 at 02:47:59PM -0700, Mark Dilger wrote: > > |$ LD_LIBRARY_PATH=tmp_install/usr/local/pgsql/lib/ src/bin/psql/psql -h > > /tmp regression > > |psql (15devel) > > |Type "help" for help. > > |regression=# \d regresion.public.bit_defaults > > |Did not find any relation named "regr

Re: pg14 psql broke \d datname.nspname.relname

2021-10-12 Thread Justin Pryzby
I understand Tom's position to be that the behavior should be changed back, since it was 1) unintentional; and 2) breaks legitimate use (when the datname matches current_database). I think there's an easy answer here that would satisfy everyone; two patches: 0001 to fix the unintentional behavior

Re: pg14 psql broke \d datname.nspname.relname

2021-10-13 Thread Justin Pryzby
On Wed, Oct 13, 2021 at 12:46:27PM -0400, Robert Haas wrote: > On Tue, Oct 12, 2021 at 12:57 PM Justin Pryzby wrote: > > I would prefer if it errored if the datname didn't match the current > > database. > > After all, it would've helped me to avoid making a con

Re: prion failed with ERROR: missing chunk number 0 for toast value 14334 in pg_toast_2619

2021-10-17 Thread Justin Pryzby
On Sun, Aug 15, 2021 at 09:44:55AM -0500, Justin Pryzby wrote: > On Sun, May 16, 2021 at 04:23:02PM -0400, Tom Lane wrote: > > 1. Fix FullXidRelativeTo to be a little less trusting. It'd > > probably be sane to make it return FirstNormalTransactionId > > when it

Re: prion failed with ERROR: missing chunk number 0 for toast value 14334 in pg_toast_2619

2021-10-17 Thread Justin Pryzby
On Sun, Oct 17, 2021 at 04:43:15PM -0500, Justin Pryzby wrote: > On Sun, Aug 15, 2021 at 09:44:55AM -0500, Justin Pryzby wrote: > > On Sun, May 16, 2021 at 04:23:02PM -0400, Tom Lane wrote: > > > 1. Fix FullXidRelativeTo to be a little less trusting. It'd > > > pr

Re: Assorted improvements in pg_dump

2021-10-21 Thread Justin Pryzby
On Wed, Oct 20, 2021 at 05:14:45PM -0400, Tom Lane wrote: > Lastly, patch 0003 addresses the concern I raised at [3] that it's > unsafe to call pg_get_partkeydef() and pg_get_expr(relpartbound) > in getTables(). Looking closer I realized that we can't cast > pg_class.reloftype to regtype at that p

Re: Improve logging when using Huge Pages

2021-10-21 Thread Justin Pryzby
+ ereport(LOG, (errmsg("Anonymous shared memory was allocated %s huge pages.", with_hugepages ? "with" : "without"))); You shouldn't break a sentence into pieces like this, since it breaks translation. You don't want an untranslated "without" to appear in the middle of the translat

Re: logical decoding/replication: new functions pg_ls_logicaldir and pg_ls_replslotdir

2021-10-22 Thread Justin Pryzby
On Fri, Oct 22, 2021 at 04:18:04PM +0530, Bharath Rupireddy wrote: > On Fri, Oct 22, 2021 at 3:18 PM Amit Kapila wrote: > > On Fri, Oct 8, 2021 at 4:39 PM Bharath Rupireddy > > wrote: > > > > > > At times, users want to know what are the files (snapshot and mapping > > > files) that are availabl

Re: Assorted improvements in pg_dump

2021-10-24 Thread Justin Pryzby
On Sun, Oct 24, 2021 at 05:10:55PM -0400, Tom Lane wrote: > 0003 is the same except I added a missing free(). > > 0004 is a new patch based on an idea from Andres Freund [1]: > in the functions that repetitively issue the same query against > different tables, issue just one query and use a WHERE

Re: pg_dump versus ancient server versions

2021-10-25 Thread Justin Pryzby
On Mon, Oct 25, 2021 at 11:38:51AM -0400, Tom Lane wrote: > Andrew Dunstan writes: > > On 10/25/21 10:23, Tom Lane wrote: > >> (Hmmm ... but disk space could > >> become a problem, particularly on older machines with not so much > >> disk. Do we really need to maintain a separate checkout for eac

Re: Isn't it better with "autovacuum worker...." instead of "worker took too long to start; canceled" specific to "auto

2021-10-27 Thread Justin Pryzby
On Wed, Oct 27, 2021 at 07:05:10PM +, Bossart, Nathan wrote: > On 10/27/21, 9:29 AM, "Bharath Rupireddy" > wrote: > > Is there a specific reason that we have a generic WARNING "worker took > > too long to start; canceled" for an autovacuum worker? Isn't it better > > with "autovacuum worker t

Re: Improve logging when using Huge Pages

2021-10-28 Thread Justin Pryzby
Hi, On Wed, Oct 27, 2021 at 06:39:46AM +, Shinoda, Noriyoshi (PN Japan FSIP) wrote: > Thank you for your comment. > The attached patch stops message splitting. > This patch also limits the timing of message output when huge_pages = try and > HugePages is not used. Thanks for updating the pa

add more support for PG_DIAG_COLUMN_NAME

2021-10-28 Thread Justin Pryzby
The docs have said this for 8 years: | "As of PostgreSQL 9.3, complete coverage exists only for errors in SQLSTATE class 23 (integrity constraint violation), but this is likely to be expanded in future." Is there any appetite for a patch like this one to improve that ? One might argue that COLU

Re: Vulnerability identified with Postgres 13.4 for Windows

2021-10-30 Thread Justin Pryzby
On Fri, Oct 29, 2021 at 10:40:06AM +, Joel Mariadasan (jomariad) wrote: > Hi, > > The scanning tool used by our organization has detected the presence of > vulnerable libxml version in the latest Postgres 13.4 release for windows > (Zip version). > > Detected by Automated Scanning tool: > l

Re: should we enable log_checkpoints out of the box?

2021-11-02 Thread Justin Pryzby
one's cluster. Note my somewhat recent message at https://www.postgresql.org/message-id/20210615161830.gq31...@telsasoft.com: On Tue, Jun 15, 2021 at 11:18:30AM -0500, Justin Pryzby wrote: > I propose to change some defaults: > > log_autovacuum_min_duration = 0 > log_checkpo

Re: make tuplestore helper function

2021-11-02 Thread Justin Pryzby
Several places have a conditional value for the first argument (randomAccess), but your patch changes the behavior to a constant "true". I didn't review the patch beyond that. > @@ -740,18 +724,14 @@ pg_prepared_statement(PG_FUNCTION_ARGS) > - tupstore = > - tuplestore_begin_heap(

Re: should we enable log_checkpoints out of the box?

2021-11-03 Thread Justin Pryzby
On Tue, Nov 02, 2021 at 08:02:41PM -0400, Tom Lane wrote: > I'm still of the position that the default ought to be that a > normally-functioning server generates no ongoing log output. > Only people who have got Nagios watching their logs, or some > such setup, are going to want anything different.

Re: extended stats on partitioned tables

2021-11-03 Thread Justin Pryzby
On Wed, Nov 03, 2021 at 11:48:44PM +0100, Tomas Vondra wrote: > On 10/8/21 12:45 AM, Justin Pryzby wrote: > > On Thu, Oct 07, 2021 at 03:26:46PM -0500, Jaime Casanova wrote: > >> On Sun, Sep 26, 2021 at 03:25:50PM -0500, Justin Pryzby wrote: > >>> On Sat, Sep 25, 20

Re: extended stats on partitioned tables

2021-11-03 Thread Justin Pryzby
atistics. This works for column stats the way I proposed for extended stats: child stats are never removed, neither when the only child is dropped, nor when re-running ANALYZE (actually, that part is odd). I can stop sending patches if it makes it hard to reconcile, but I wanted to put it "on pa

Re: [sqlsmith] Failed assertion in brin_minmax_multi_distance_float4 on REL_14_STABLE

2021-11-04 Thread Justin Pryzby
On Thu, Nov 04, 2021 at 09:46:49AM +0100, Andreas Seltenreich wrote: > sqlsmith triggers the following assertion when testing REL_14_STABLE: > > TRAP: FailedAssertion("a1 <= a2", File: "brin_minmax_multi.c", Line: > 1879, PID: 631814) > > I can reproduce it with the following query on a fres

Re: jsonb crash

2021-11-05 Thread Justin Pryzby
On Tue, Oct 26, 2021 at 07:07:01PM +1300, David Rowley wrote: > Does anyone have any thoughts on the proposed fixes? I don't have any thoughts, but I want to be sure it isn't forgotten. -- Justin

Re: Draft release notes for next week's releases

2021-11-05 Thread Justin Pryzby
On Fri, Nov 05, 2021 at 08:27:49PM -0400, Tom Lane wrote: > First draft is up at > https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=39387228c8b7043d168bada0c64e9f59e83285f5 > As usual, please send any comments/corrections by Sunday. + one more row than requested, so tha it c

Re: Schema variables - new implementation for Postgres 15

2021-11-06 Thread Justin Pryzby
On Sat, Nov 06, 2021 at 04:45:19AM +0100, Pavel Stehule wrote: > st 3. 11. 2021 v 14:05 odesílatel Tomas Vondra > napsal: > > 1) Not sure why we need to call this "schema variables". Most objects > > are placed in a schema, and we don't say "schema tables" for example. > > And it's CREATE VARIABL

Re: Parallel Full Hash Join

2021-11-06 Thread Justin Pryzby
> Rebased patches attached. I will change status back to "Ready for Committer" The CI showed a crash on freebsd, which I reproduced. https://cirrus-ci.com/task/5203060415791104 The crash is evidenced in 0001 - but only ~15% of the time. I think it's the same thing which was committed and then re

Re: pg_upgrade test for binary compatibility of core data types

2021-11-07 Thread Justin Pryzby
+AND NOT EXISTS (SELECT 1 FROM pg_attribute a WHERE a.atttypid=t.oid AND a.attnum>0 AND a.attrelid='manytypes'::regclass); > 0004 is something I'd like to get done on HEAD to eas

Re: Schema variables - new implementation for Postgres 15

2021-11-08 Thread Justin Pryzby
On Sun, Nov 07, 2021 at 10:14:00PM +0100, Tomas Vondra wrote: > I'd probably vote for "session variables". We can call it local/global > session variables in the future, if we end up implementing that. By chance, I ran into this pre-existing use of the phrase "session variable". introduced since 8

Re: make tuplestore helper function

2021-11-08 Thread Justin Pryzby
On Mon, Nov 08, 2021 at 02:52:28PM -0500, Melanie Plageman wrote: > On Tue, Nov 2, 2021 at 4:23 PM Justin Pryzby wrote: > > > > Several places have a conditional value for the first argument > > (randomAccess), > > but your patch changes the behavior to a consta

Re: 2021-11-11 release announcement draft

2021-11-09 Thread Justin Pryzby
On Tue, Nov 09, 2021 at 11:25:37AM -0500, Jonathan S. Katz wrote: > * Fix is when creating a new range type with `CREATE TYPE` that could cause > problems for later event triggers or subsequent executions of the `CREATE > TYPE` > command. I don't know what "is when" means, but it seems wrong. >

Re: terminate called after throwing an instance of 'std::bad_alloc'

2021-11-10 Thread Justin Pryzby
Thread starting here: https://www.postgresql.org/message-id/20201001021609.GC8476%40telsasoft.com On Fri, Dec 18, 2020 at 05:56:07PM -0600, Justin Pryzby wrote: > I'm 99% sure the "bad_alloc" is from LLVM. It happened multiple times on > different servers (running a similar

SIGABRT causes messages at LOG but not PANIC

2021-11-10 Thread Justin Pryzby
postgres=# SELECT log_time , database, user_name, error_severity sev, left(message,99) FROM postgres_log_2021_11_10_0800 WHERE log_time BETWEEN '2021-11-10 08:57' AND '2021-11-10 08:58' AND database IS NULL; log_time | database | user_name | sev |

Re: Patch abstracts in the Commitfest app

2021-11-12 Thread Justin Pryzby
On Fri, Nov 12, 2021 at 01:51:28PM +0100, Daniel Gustafsson wrote: > While reading through and working with the hundreds of patches in the CF app a > small feature/process request struck me: it would be really helpful if the > patch had a brief abstract outlining what it aims to add or fix (or summ

Re: Parallel Append can break run-time partition pruning

2021-11-12 Thread Justin Pryzby
On Mon, Nov 02, 2020 at 01:50:57PM +1300, David Rowley wrote: > On Tue, 27 Oct 2020 at 19:40, Amit Langote wrote: > > Some comments: > > Thanks for having a look at this. > > I've made some adjustments to those comments and pushed. commit a929e17e5 doesn't appear in the v14 release notes, but I

Re: RFC: Logging plan of the running query

2021-11-12 Thread Justin Pryzby
On Wed, Oct 13, 2021 at 05:28:30PM +0300, Ekaterina Sokolova wrote: > Hi, hackers! > > • pg_query_state is contrib with 2 patches for core (I hope someday > Community will support adding this patches to PostgreSQL). It contains I reviewed this version of the patch - I have some language fixes

Re: support for MERGE

2021-11-13 Thread Justin Pryzby
> > Adding to commitfest. I reviewed the documentation to learn about the feature, and fixed some typos. +notpatch. -- Justin >From 2f3d2465a93ee6207b2afdab8787cf2eaa2c0bb4 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Sat, 13 Nov 2021 12:11:46 -0600 Subject: [PATCH] f!typos --- do

Re: support for MERGE

2021-11-13 Thread Justin Pryzby
hours with no apparent issue - good. It seems like there's an opened question about how RULES should be handled? -- Justin >From 119c6c7342ff9410424e94d60dc11b3a5e9d98c9 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Sat, 13 Nov 2021 15:47:12 -0600 Subject: [PATCH] f!merge code comme

Re: Add psql command to list constraints

2021-11-14 Thread Justin Pryzby
Hi, On Mon, Nov 15, 2021 at 10:38:55AM +0900, Tatsuro Yamada wrote: > postgres=# \dco > List of constsraints > Schema | Name | Definition > | Table > +-+-

Re: BUFFERS enabled by default in EXPLAIN (ANALYZE)

2021-11-15 Thread Justin Pryzby
h is set by regress.c. I'm interested to hear from a reviewer if this is worth pursing like this. -- Justin >From cdf439d9ccdec0f26bdd6b86f9f27bd4e55e5b9e Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Sat, 22 Feb 2020 21:17:10 -0600 Subject: [PATCH 1/3] Add explain(REGRESS)... This i

Re: Printing backtrace of postgres processes

2021-11-15 Thread Justin Pryzby
On Mon, Nov 15, 2021 at 09:12:49PM +0530, vignesh C wrote: > The idea here is to implement & expose pg_print_backtrace function, internally This patch is closely related to this one https://commitfest.postgresql.org/35/3142/ | Logging plan of the currently running query I suggest to review that p

Re: Schema variables - new implementation for Postgres 15

2021-11-15 Thread Justin Pryzby
On Mon, Nov 15, 2021 at 09:00:13PM +0100, Pavel Stehule wrote: > Thank you for review and fixes, I try to complete some version for next > work, and looks so your patch 0001 is broken > > gedit reports to me broken unicode \A0\A0\A0\A0\A0 > > my last patch has 276KB and your patch has 293KB? On

Re: Commitfest 2021-11 Patch Triage - Part 2

2021-11-15 Thread Justin Pryzby
On Mon, Nov 15, 2021 at 03:23:17PM -0500, Robert Haas wrote: > On Mon, Nov 15, 2021 at 2:51 PM Stephen Frost wrote: > > I get that just compressing the entire stream is simpler and easier and > > such, but it's surely cheaper and more efficient to not decompress and > > then recompress data that's

Re: Write visibility map during CLUSTER/VACUUM FULL

2021-11-15 Thread Justin Pryzby
On Sun, Aug 29, 2021 at 07:26:42PM -0500, Justin Pryzby wrote: > On Mon, Jun 28, 2021 at 11:22:01AM +0300, Anna Akenteva wrote: > > On 2019-11-29 05:32, Michael Paquier wrote: > > > These comments are unanswered for more than 2 months, so I am marking > > > this entr

Re: Add psql command to list constraints

2021-11-15 Thread Justin Pryzby
Thanks for updating the patch :) > Currently, DBAs need the table name to see the constraint information. Or, they can query pg_constraint or information_schema: check_constraints, table_constraints. > - success = listConversions(pattern, > show_verbose, show_system)

Re: Should we improve "PID XXXX is not a PostgreSQL server process" warning for pg_terminate_backend(<>)?

2021-11-17 Thread Justin Pryzby
On Wed, Nov 17, 2021 at 03:59:59PM -0300, Euler Taveira wrote: > On Mon, Nov 15, 2021, at 4:27 AM, Bharath Rupireddy wrote: > > As there is some interest shown in this thread at [1], I'm attaching a > > new v3 patch here. Please review it. > I took a look at this patch. I have a few comments. > >

Re: pg_upgrade parallelism

2021-11-17 Thread Justin Pryzby
On Wed, Nov 17, 2021 at 02:44:52PM -0500, Jaime Casanova wrote: > Hi, > > Currently docs about pg_upgrade says: > > """ > > The --jobs option allows multiple CPU cores to be used > for copying/linking of files and to dump and reload database schemas > in parallel; a good plac

Re: logical decoding/replication: new functions pg_ls_logicaldir and pg_ls_replslotdir

2021-11-17 Thread Justin Pryzby
On Wed, Nov 17, 2021 at 06:46:47PM +, Bossart, Nathan wrote: > On 10/30/21, 2:36 AM, "Bharath Rupireddy" > wrote: > > I've added 3 functions pg_ls_logicalsnapdir, pg_ls_logicalmapdir, > > pg_ls_replslotdir, and attached the patch. The sample output looks > > like [1]. Please review it further

Re: pg_upgrade test for binary compatibility of core data types

2021-11-17 Thread Justin Pryzby
On Wed, Nov 17, 2021 at 04:01:19PM +0900, Michael Paquier wrote: > On Sun, Nov 07, 2021 at 01:22:00PM -0600, Justin Pryzby wrote: > > That may be good enough for test.sh, but if the kludges were moved to a .sql > > script which was also run by the buildfarm (in stead of its hardcod

Re: Printing backtrace of postgres processes

2021-11-18 Thread Justin Pryzby
e(pg_backend_pid()); + +CREATE ROLE regress_log_backtrace; + +SELECT has_function_privilege('regress_log_backtrace', + 'pg_log_backtrace(integer)', 'EXECUTE'); -- no + +GRANT EXECUTE ON FUNCTION pg_log_backtrace(integer) + TO regress_log_backtrace; + +SELECT

Re: Should rename "startup process" to something else?

2021-11-18 Thread Justin Pryzby
On Thu, Nov 18, 2021 at 12:24:14PM -0500, Tom Lane wrote: > Robert Haas writes: > > On Thu, Nov 18, 2021 at 11:05 AM Tom Lane wrote: > >> Yeah, given current usage it would be better to call it the "recovery > >> process". However, I'm feeling dubious that it's worth the cost to > >> change. Th

Re: make tuplestore helper function

2021-11-18 Thread Justin Pryzby
On Thu, Nov 18, 2021 at 12:59:03PM -0500, Melanie Plageman wrote: > On Mon, Nov 8, 2021 at 3:13 PM Justin Pryzby wrote: > > On Mon, Nov 08, 2021 at 02:52:28PM -0500, Melanie Plageman wrote: > > > On Tue, Nov 2, 2021 at 4:23 PM Justin Pryzby wrote: > > > >

Re: terminate called after throwing an instance of 'std::bad_alloc' (llvmjit)

2021-11-18 Thread Justin Pryzby
On Wed, Nov 10, 2021 at 09:56:44AM -0600, Justin Pryzby wrote: > Thread starting here: > https://www.postgresql.org/message-id/20201001021609.GC8476%40telsasoft.com > > On Fri, Dec 18, 2020 at 05:56:07PM -0600, Justin Pryzby wrote: > > I'm 99% sure the "bad_alloc

Re: Feature Proposal: Connection Pool Optimization - Change the Connection User

2021-11-20 Thread Justin Pryzby
On Sun, Nov 21, 2021 at 03:11:03AM +1100, Todd Hubers wrote: > I have just joined to start a community consultation process for a > proposal. I just finished the proposal document, I spent time writing a > Problem and Solution section, and I have done quite a bit of upfront > exploration of the cod

Re: Parallel Full Hash Join

2021-11-20 Thread Justin Pryzby
On Wed, Nov 17, 2021 at 01:45:06PM -0500, Melanie Plageman wrote: > On Sat, Nov 6, 2021 at 11:04 PM Justin Pryzby wrote: > > > > > Rebased patches attached. I will change status back to "Ready for > > > Committer" > > > > The CI showed a crash o

Re: using extended statistics to improve join estimates

2021-11-21 Thread Justin Pryzby
Your regression tests include two errors, which appear to be accidental, and fixing the error shows that this case is being estimated poorly. +-- try combining with single-column (and single-expression) statistics +DROP STATISTICS join_test_2; +ERROR: statistics object "join_test_2" does not exis

Re: [PATCH] Add extra statistics to explain for Nested Loop

2021-11-21 Thread Justin Pryzby
bbe5f48ae9695cb89de4 (an issue I reported the first time I was looking at this patch). I think the non-text format timing stuff needs to be within "if (timing)". I'm curious to hear what you and others think of the refactoring. It'd be nice if there's a good way

Re: pg_upgrade parallelism

2021-11-23 Thread Justin Pryzby
On Tue, Nov 23, 2021 at 06:54:03PM +, Jacob Champion wrote: > On Wed, 2021-11-17 at 14:34 -0600, Justin Pryzby wrote: > > On Wed, Nov 17, 2021 at 02:44:52PM -0500, Jaime Casanova wrote: > > > > > > - why we read()/write() at all? is not a faster way of copying t

Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)

2021-11-23 Thread Justin Pryzby
t's wanted, but not usually what "ls" would show), nor ready to implement recurse. As before: On Tue, Apr 06, 2021 at 11:01:31AM -0500, Justin Pryzby wrote: > The first handful of patches address the original issue, and I think could be > "ready": > > $ git log

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-11-24 Thread Justin Pryzby
Thanks for working on this. I was just trying to find something like "pg_stat_checkpointer". You wrote beentry++ at the start of two loops, but I think that's wrong; it should be at the end, as in the rest of the file (or as a loop increment). BackendStatusArray[0] is actually used (even though i

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-11-26 Thread Justin Pryzby
On Wed, Nov 24, 2021 at 07:15:59PM -0600, Justin Pryzby wrote: > There's extraneous blank lines in these functions: > > +pgstat_sum_io_path_ops > +pgstat_report_live_backend_io_path_ops > +pgstat_recv_resetsharedcounter > +GetIOPathDesc > +StrategyRejectBuf

Re: Unifying VACUUM VERBOSE and log_autovacuum_min_duration output

2021-11-26 Thread Justin Pryzby
On Fri, Nov 26, 2021 at 12:37:32PM -0800, Peter Geoghegan wrote: > My preferred approach to this is simple: redefine VACUUM VERBOSE to > not show incremental output, which seems rather unhelpful anyway. > I don't think that we need to keep the getrusage() stuff at all, though. +1 > * VACUUM VERB

Re: Enforce work_mem per worker

2021-11-27 Thread Justin Pryzby
On Sat, Nov 27, 2021 at 04:33:07PM +, Arne Roland wrote: > Hello! > > Since I used a lot of my time chasing short lived processes eating away big > chunks of memory in recent weeks, I am wondering about a decent way to go > about this. > The problem I am facing essentially relates to the fac

GUC flags

2021-11-28 Thread Justin Pryzby
ar to GUC categories fixed in a55a98477. Tom was of the impression that there's more loose ends on that thread. https://www.postgresql.org/message-id/flat/16997-ff16127f6e0d1...@postgresql.org -- Justin >From b52bbf317126dbd75a2f8e98ceec4f3beb66f572 Mon Sep 17 00:00:00 2001 From: Justin

Re: Enforce work_mem per worker

2021-11-29 Thread Justin Pryzby
On Mon, Nov 29, 2021 at 02:01:35PM +, Arne Roland wrote: > But my main goal is something else. I can't explain my clients, why a chanced > statistics due to autovacuum suddenly leads to oom. They would be right to > question postgres qualification for any serious production system. What vers

Re: Lots of memory allocated when reassigning Large Objects

2021-11-29 Thread Justin Pryzby
On Mon, Nov 29, 2021 at 01:49:24PM +0100, Guillaume Lelarge wrote: > One of our customers had a very bad issue while trying to reassign objects > from user A to user B. He had a lot of them, and the backend got very > hungry for memory. It finally all went down when the linux kernel decided > to ki

Re: Lots of memory allocated when reassigning Large Objects

2021-11-29 Thread Justin Pryzby
On Mon, Nov 29, 2021 at 01:40:31PM -0500, Tom Lane wrote: > DROP OWNED BY likely has similar issues. I tried a few more commands but found no significant issue. IMO if you have 100k tables, then you can afford 1GB RAM. SELECT format('CREATE TABLE t%s()', a) FROM generate_series(1,)a\gexec SET

Re: Unifying VACUUM VERBOSE and log_autovacuum_min_duration output

2021-11-29 Thread Justin Pryzby
I think the 2nd chunk here could say "if (instrument)" like the first: > @@ -482,8 +480,10 @@ heap_vacuum_rel(Relation rel, VacuumParams *params, > TransactionId FreezeLimit; > MultiXactId MultiXactCutoff; > > - /* measure elapsed time iff autovacuum logging requires it */

Re: GUC flags

2021-11-30 Thread Justin Pryzby
On Tue, Nov 30, 2021 at 03:36:45PM +0900, Michael Paquier wrote: > - gettext_noop("Forces a switch to the next WAL file if a > " > - "new file has not been started > within N seconds."), > + gettext_noop("Sets th

Re: should we document an example to set multiple libraries in shared_preload_libraries?

2021-12-01 Thread Justin Pryzby
On Wed, Dec 01, 2021 at 04:20:52PM +0530, Bharath Rupireddy wrote: > It seems like users can try different ways to set multiple values for > shared_preload_libraries GUC even after reading the documentation > [1]), something like: ... > ALTER SYSTEM SET shared_preload_libraries = > "auth_delay,pg_

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-12-01 Thread Justin Pryzby
On Wed, Dec 01, 2021 at 05:00:14PM -0500, Melanie Plageman wrote: > > Also: > > src/include/miscadmin.h:#define BACKEND_NUM_TYPES (B_LOGGER + 1) > > > > I think it's wrong to say NUM_TYPES = B_LOGGER + 1 (which would suggest > > using > > lessthan-or-equal instead of lessthan as you are). > > > >

Re: BUFFERS enabled by default in EXPLAIN (ANALYZE)

2021-12-01 Thread Justin Pryzby
On Mon, Nov 15, 2021 at 01:09:54PM -0600, Justin Pryzby wrote: > Some time ago, I had a few relevant patches: > 1) add explain(REGRESS) which is shorthand for (BUFFERS OFF, TIMING OFF, > COSTS OFF, SUMMARY OFF) > 2) add explain(MACHINE) which elides machine-specific output from expla

Re: Index scan prefetch?

2018-03-26 Thread Justin Pryzby
On Mon, Mar 26, 2018 at 12:43:02PM +0300, Konstantin Knizhnik wrote: > Hi, hackers. > > I was faced with the following bad performance use case with Postgres: there > is a huge append-only table with serial key (ID) > which is permanently appended using multithreaded pgloader. I think this could

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-03-28 Thread Justin Pryzby
On Thu, Mar 29, 2018 at 11:30:59AM +0900, Michael Paquier wrote: > On Tue, Mar 27, 2018 at 11:53:08PM -0400, Tom Lane wrote: > > Craig Ringer writes: > >> TL;DR: Pg should PANIC on fsync() EIO return. > > > > Surely you jest. > > Any callers of pg_fsync in the backend code are careful enough to

bulk typos

2018-03-31 Thread Justin Pryzby
I needed another distraction so bulk-checked for typos, limited to comments in *.[ch]. I'm not passionate about this, but it serves the purpose of reducing the overhead of fixing them individually. Also I heard something here recently about ugly languages.. time find . -name '*.c' -print0 |xargs

open/lseek overhead with many partitions (including with "faster partitioning pruning")

2018-04-03 Thread Justin Pryzby
TLDR: even with "faster pruning" patch, pg11dev open()+lseek() every file backing every table being queried, even for those partitions eventually "excluded". One of our customers (with the largest number of child tables, currently of relkind='r') takes a long time to plan query, and I wondered whe

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-09 Thread Justin Pryzby
On Mon, Apr 09, 2018 at 09:31:56AM +0800, Craig Ringer wrote: > You could make the argument that it's OK to forget if the entire file > system goes away. But actually, why is that ok? I was going to say that it'd be okay to clear error flag on umount, since any opened files would prevent unmountin

Re: Should we add GUCs to allow partition pruning to be disabled?

2018-04-17 Thread Justin Pryzby
On Wed, Apr 18, 2018 at 12:07:18PM +1200, David Rowley wrote: > In PG10 the planner's partition pruning could be disabled by changing > the constraint_exclusion GUC to off. This is still the case for PG11, > but only for UPDATE and DELETE queries. There is currently no way to > disable partition p

Re: [patch] BUG #15005: ANALYZE can make pg_class.reltuples inaccurate.

2018-04-23 Thread Justin Pryzby
Just want to add for the archive that I happened to run across what appears to be a 7-year old report of (I think) both of these vacuum/analyze bugs: Re: [patch] BUG #15005: ANALYZE can make pg_class.reltuples inaccurate.

tab complete for explain SETTINGS

2019-09-26 Thread Justin Pryzby
Here's to hoping this is the worst omission in v12. Justin >From e21f58504e5006de9766fe586550b59167e00ffd Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Thu, 26 Sep 2019 21:12:26 -0500 Subject: [PATCH v1] tab completion for explain (SETTINGS) m

v12 relnotes: alter system tables

2019-09-27 Thread Justin Pryzby
https://www.postgresql.org/docs/12/release-12.html |Allow modifications of system catalogs' options using ALTER TABLE (Peter Eisentraut) |Modifications of catalogs' reloptions and autovacuum settings are now supported. I wonder if that should say: "... WHEN ALLOW_SYSTEM_TABLE_MODS IS ENABLED."

default partitions can be partitioned and have default partitions?

2019-09-28 Thread Justin Pryzby
postgres=# CREATE TABLE t(i int)PARTITION BY RANGE(i); CREATE TABLE postgres=# CREATE TABLE t0 PARTITION OF t DEFAULT PARTITION BY RANGE(i); CREATE TABLE postgres=# CREATE TABLE t00 PARTITION OF t0 DEFAULT; -- oh yes CREATE TABLE ... Not sure how it could be useful to partition default into subpar

typo: postGER

2019-09-29 Thread Justin Pryzby
$ git grep Postger src/backend/po/tr.po:"Bu durum, sistemin semaphore set (SEMMNI) veya semaphore (SEMMNS) sayı sınırlaması aşmasında meydana gelmektedir. Belirtilen parametrelerin değerleri yükseltmelisiniz. Başka seçeneğiniz ise PostgerSQL sisteminin semaphore tütekitimini max_connections pa

format of pg_upgrade loadable_libraries warning

2019-10-02 Thread Justin Pryzby
ibdir/pgfincore": No such file or directory |Database: postgres |Database: too I think the list of databases should be formatted to indicate its association with the preceding error by indentation and verbage, or larger refactoring to present in a list, like: "Databases with library which fa

consider including server_version in explain(settings)

2019-10-03 Thread Justin Pryzby
explain(SETTINGS) was implemented to show relevant settings for which an odd value could affect a query but could be forgotten during troubleshooting. This is a "concept" patch to show the version, which is frequently requested on -performance list and other support requests. If someone sends exp

Re: format of pg_upgrade loadable_libraries warning

2019-10-04 Thread Justin Pryzby
On Fri, Oct 04, 2019 at 05:37:46PM -0400, Bruce Momjian wrote: > On Wed, Oct 2, 2019 at 12:23:37PM -0500, Justin Pryzby wrote: > > Regarding the previous thread and commit here: > > https://www.postgresql.org/message-id/flat/20180713162815.GA3835%40momjian.us > > https://git.

v12.0: ERROR: could not find pathkey item to sort

2019-10-11 Thread Justin Pryzby
I've reduced the failing query as much as possible to this: -- This is necessary to fail: SET enable_nestloop=off; SELECT * FROM (SELECT start_time, t1.site_id FROM pgw_kpi_view t1 -- Apparently the where clause is necessary to fail... WHERE (start_time>='2019-10-1

Re: v12.0: ERROR: could not find pathkey item to sort

2019-10-11 Thread Justin Pryzby
On Fri, Oct 11, 2019 at 10:48:37AM -0400, Tom Lane wrote: > Justin Pryzby writes: > > The view is actually a join of two relkind=p partitioned tables (which I > > will acknowledge probably performs poorly). > > Could you provide a self-contained test case please? Working

v12.0 ERROR: trying to store a heap tuple into wrong type of slot

2019-10-11 Thread Justin Pryzby
I'm not sure why we have that index, and my script probably should have known to choose a better one to cluster on, but still.. ts=# CLUSTER huawei_m2000_config_enodebcell_enodeb USING huawei_m2000_config_enodebcell_enodeb_coalesce_idx ; DEBUG: 0: building index "pg_toast_1840151315_index" o

v12.0: segfault in reindex CONCURRENTLY

2019-10-11 Thread Justin Pryzby
One of our servers crashed last night like this: < 2019-10-10 22:31:02.186 EDT postgres >STATEMENT: REINDEX INDEX CONCURRENTLY child.eric_umts_rnc_utrancell_hsdsch_eul_201910_site_idx < 2019-10-10 22:31:02.399 EDT >LOG: server process (PID 29857) was terminated by signal 11: Segmentation faul

Re: v12.0: ERROR: could not find pathkey item to sort

2019-10-12 Thread Justin Pryzby
On Fri, Oct 11, 2019 at 10:48:37AM -0400, Tom Lane wrote: > Could you provide a self-contained test case please? SET enable_partitionwise_aggregate = 'on'; SET enable_partitionwise_join = 'on'; SET max_parallel_workers_per_gather=0; -- maybe not important but explain(settings) suggests I should in

v12.0: reindex CONCURRENTLY: lock ShareUpdateExclusiveLock on object 14185/39327/0 is already held

2019-10-12 Thread Justin Pryzby
I ran into this while trying to trigger the previously-reported segfault. CREATE TABLE t(i) AS SELECT * FROM generate_series(1,9); CREATE INDEX ON t(i); [pryzbyj@database ~]$ for i in `seq 1 9`; do PGOPTIONS='-cstatement_timeout=9' psql postgres --host /tmp --port 5678 -c "REINDEX INDEX CONCURR

Re: v12.0: segfault in reindex CONCURRENTLY

2019-10-13 Thread Justin Pryzby
Resending this message, which didn't make it to the list when I sent it earlier. (And, notified -www). On Sun, Oct 13, 2019 at 06:06:43PM +0900, Michael Paquier wrote: > On Fri, Oct 11, 2019 at 07:44:46PM -0500, Justin Pryzby wrote: > > Unfortunately, there was no core file, and I&

<    2   3   4   5   6   7   8   9   10   11   >