Re: maintenance_work_mem used by Vacuum

2019-10-15 Thread Amit Kapila
On Wed, Oct 16, 2019 at 7:20 AM Masahiko Sawada wrote: > > On Sat, Oct 12, 2019 at 8:45 PM Amit Kapila wrote: > > > > On Sat, Oct 12, 2019 at 10:49 AM Masahiko Sawada > > wrote: > > > > > > On Fri, Oct 11, 2019 at 5:13 PM Amit Kapila > > > wrote: > > > > > > > > That's right, but OTOH, if the

RE: v12 and pg_restore -f-

2019-10-15 Thread imai.yoshik...@fujitsu.com
Hi, On Sun, Oct 6, 2019 at 7:09 PM, Justin Pryzby wrote: > I saw this and updated our scripts with pg_restore -f- > https://www.postgresql.org/docs/12/release-12.html > |In pg_restore, require specification of -f - to send the dump contents to > standard output (Euler Taveira) > |Previously, this

Re: Non-Active links being referred in our source code

2019-10-15 Thread Michael Paquier
On Mon, Oct 14, 2019 at 09:48:12PM +0530, vignesh C wrote: > About pg_crc.h, I have made the changes with the correct links. > The patch for the same is attached. Confirmed, so applied. Thanks, Vignesh. -- Michael signature.asc Description: PGP signature

Re: ERROR: multixact X from before cutoff Y found to be still running

2019-10-15 Thread Thomas Munro
On Wed, Sep 18, 2019 at 8:11 AM Bossart, Nathan wrote: > Thanks for the detailed background information. FWIW I am now in > favor of the v2 patch. Here's a version with a proposed commit message and a comment. Please let me know if I credited things to the right people! 0001-Fix-bug-that-coul

RE: Copy data to DSA area

2019-10-15 Thread ideriha.take...@fujitsu.com
Hi, >ShmZoneContext for SharedPlan and SharedRelCache is not implemented but I'm >going to do it following your points. After looking into existing code, I'm thinking Generation Memory Context seems to have the similar purpose. So I'll implement ShmZoneContext by reference it. Generation context

Re: [HACKERS] Block level parallel vacuum

2019-10-15 Thread Amit Kapila
On Wed, Oct 16, 2019 at 6:50 AM Masahiko Sawada wrote: > > On Tue, Oct 15, 2019 at 6:33 PM Amit Kapila wrote: > > > > Attached updated patch set. 0001 patch introduces new index AM field > amcanparallelvacuum. All index AMs except for gist sets true for now. > 0002 patch incorporated the all comm

Re: Questions/Observations related to Gist vacuum

2019-10-15 Thread Dilip Kumar
On Tue, Oct 15, 2019 at 7:13 PM Heikki Linnakangas wrote: > > On 15/10/2019 09:37, Amit Kapila wrote: > > While reviewing a parallel vacuum patch [1], we noticed a few things > > about $SUBJECT implemented in commit - > > 7df159a620b760e289f1795b13542ed1b3e13b87. > > > > 1. A new memory context Gi

Re: let's make the list of reportable GUCs configurable (was Re: Add %r substitution for psql prompts to show recovery status)

2019-10-15 Thread Dave Cramer
On Sat, 12 Oct 2019 at 05:05, Tom Lane wrote: > Andres Freund writes: > > On 2019-10-11 16:30:17 -0400, Robert Haas wrote: > >> But, if it does need to be changed, it seems like a terrible idea to > >> allow it to be done via SQL. Otherwise, the user can break the driver > >> by using SQL to set

Re: pgbench - extend initialization phase control

2019-10-15 Thread btendouan
Here is rebase v3. Hi, Thanks for your new patch. Failed regression test. It's necessary to change the first a in “allowed step characters are” to uppercase A in the regression test of 002_pgbench_no_server.pl. The behavior of "g" is different between v12 and the patche, and backward c

Re: Collation versioning

2019-10-15 Thread Thomas Munro
On Wed, Oct 16, 2019 at 5:33 PM Thomas Munro wrote: > On Tue, Oct 15, 2019 at 5:39 PM Thomas Munro wrote: > > Here's a version with a small note added to the documentation. I'm > > planning to commit this tomorrow. > > Done. The buildfarm is telling me that I didn't test this with the full set

Re: Collation versioning

2019-10-15 Thread Thomas Munro
On Tue, Oct 15, 2019 at 5:39 PM Thomas Munro wrote: > Here's a version with a small note added to the documentation. I'm > planning to commit this tomorrow. Done. It's not much, but it's a start. Some things to do: * handle default collation (probably comes with CF entry 2256?) * preserve ver

Re: Ordering of header file inclusion

2019-10-15 Thread vignesh C
On Wed, Oct 16, 2019 at 8:10 AM Amit Kapila wrote: > > Thanks for working on this. I will look into this in the coming few > days or during next CF. Can you please register it for the next CF > (https://commitfest.postgresql.org/25/)? > Thanks, I have added it to the commitfest. Regards, Vignes

RE: Copy data to DSA area

2019-10-15 Thread ideriha.take...@fujitsu.com
Hi, Sorry for waiting. >Thomas Munro wrote: >>What do you think about the following? Even though I know you want to >>start with much simpler kinds of cache, I'm looking ahead to the lofty >>end-goal of having a shared plan cache. No doubt, that involves >>solving many other problems that don

Re: Ordering of header file inclusion

2019-10-15 Thread Amit Kapila
On Tue, Oct 15, 2019 at 10:57 PM vignesh C wrote: > > On Wed, Oct 9, 2019 at 11:37 AM Amit Kapila wrote: > > > > On Tue, Oct 8, 2019 at 8:19 PM Tom Lane wrote: > > > > > > Amit Kapila writes: > > > > On Wed, Oct 2, 2019 at 2:57 PM vignesh C wrote: > > > >> I noticed that some of the header fil

Re: maintenance_work_mem used by Vacuum

2019-10-15 Thread Masahiko Sawada
On Sat, Oct 12, 2019 at 8:45 PM Amit Kapila wrote: > > On Sat, Oct 12, 2019 at 10:49 AM Masahiko Sawada > wrote: > > > > On Fri, Oct 11, 2019 at 5:13 PM Amit Kapila wrote: > > > > > > That's right, but OTOH, if the user specifies gin_pending_list_limit > > > as an option during Create Index wit

Understanding TupleQueue impact and overheads?

2019-10-15 Thread Tom Mercha
I have been looking at PostgreSQL's Tuple Queue (/include/executor/tqueue.h) which provides functionality for queuing tuples between processes through shm_mq. I am still familiarising myself with the bigger picture and TupTableStores. I can see that a copy (not a reference) of a HeapTuple (obta

Re: [HACKERS] Block level parallel vacuum

2019-10-15 Thread Masahiko Sawada
On Tue, Oct 15, 2019 at 6:33 PM Amit Kapila wrote: > > On Tue, Oct 15, 2019 at 1:26 PM Masahiko Sawada wrote: > > > > On Tue, Oct 15, 2019 at 4:15 PM Masahiko Sawada > > wrote: > > > > > > > > > If we avoid postponing deleting empty pages till the cleanup phase, > > > > > > then we don't have t

Re: BRIN index which is much faster never chosen by planner

2019-10-15 Thread David Rowley
On Wed, 16 Oct 2019 at 11:40, Justin Pryzby wrote: > It didn't occur to me at the time, but that would also allow > creating numerous, partial BRIN indices, each of which would have separate > correlation computed over just their "restricted range", which *might* also > handle your case, depending

Re: BRIN index which is much faster never chosen by planner

2019-10-15 Thread David Rowley
On Wed, 16 Oct 2019 at 05:05, Jeremy Finzel wrote: > But perhaps it would be worth exploring if there could be more detailed stats > on physical vs logical correlation, such as when ANALYZE takes its samples, > noting physical locations as well as logical values, and allowing the > correlation

Re: BRIN index which is much faster never chosen by planner

2019-10-15 Thread Justin Pryzby
This reminds me of an issue I reported several years ago where Btree index scans were chosen over seq scan of a large, INSERT-only table due to very high correlation, but performed poorly. I concluded that use of the the high "large scale" correlation on a large 50+GB table caused the planner to f

Re: v12.0 ERROR: trying to store a heap tuple into wrong type of slot

2019-10-15 Thread Justin Pryzby
On Tue, Oct 15, 2019 at 01:50:09PM -0700, Andres Freund wrote: > On 2019-10-13 07:51:06 -0700, Andres Freund wrote: > > On 2019-10-11 16:03:20 -0500, Justin Pryzby wrote: > > > ts=# CLUSTER huawei_m2000_config_enodebcell_enodeb USING > > > huawei_m2000_config_enodebcell_enodeb_coalesce_idx ; > > T

Re: v12.0 ERROR: trying to store a heap tuple into wrong type of slot

2019-10-15 Thread Andres Freund
Hi, On 2019-10-13 07:51:06 -0700, Andres Freund wrote: > On 2019-10-11 16:03:20 -0500, Justin Pryzby wrote: > > I'm not sure why we have that index, and my script probably should have > > known > > to choose a better one to cluster on, but still.. > > > > ts=# CLUSTER huawei_m2000_config_enodebce

Re: BRIN index which is much faster never chosen by planner

2019-10-15 Thread Michael Lewis
Thanks for closing the loop on the data correlation question. I've been playing with BRIN indexes on a log table of sorts and this thread helped clear up some of the behavior I have been seeing. I am curious, would a partial btree index fit your needs? Perhaps the maintenance overhead is too signi

v12.0: interrupt reindex CONCURRENTLY: ccold: ERROR: could not find tuple for parent of relation ...

2019-10-15 Thread Justin Pryzby
On a badly-overloaded VM, we hit the previously-reported segfault in progress reporting. This left around some *ccold indices. I tried to drop them but: sentinel=# DROP INDEX child.alarms_null_alarm_id_idx1_ccold; -- child.alarms_null_alarm_time_idx_ccold; -- alarms_null_alarm_id_idx_ccold; ERR

Re: BRIN index which is much faster never chosen by planner

2019-10-15 Thread Jeremy Finzel
Thank you for the thorough and thoughtful reply! Please see below. On Mon, Oct 14, 2019 at 3:48 PM David Rowley wrote: > Another thing which you might want to look at is the correlation > column in the pg_stats view for the rec_insert_time column. Previous > to 7e534adcd, BRIN index were costed

Re: Questions/Observations related to Gist vacuum

2019-10-15 Thread Heikki Linnakangas
On 15/10/2019 09:37, Amit Kapila wrote: While reviewing a parallel vacuum patch [1], we noticed a few things about $SUBJECT implemented in commit - 7df159a620b760e289f1795b13542ed1b3e13b87. 1. A new memory context GistBulkDeleteResult->page_set_context has been introduced, but it doesn't seem to

Re: Fix most -Wundef warnings

2019-10-15 Thread Andrew Gierth
> "Mark" == Mark Dilger writes: >> +#ifdef HSTORE_IS_HSTORE_NEW Mark> Checking the current sources, git history, and various older Mark> commits, I did not find where HSTORE_IS_HSTORE_NEW was ever Mark> defined. In contrib/hstore, it never was. The current version of contrib/hstore had

Re: tuplesort test coverage

2019-10-15 Thread Peter Geoghegan
On Sun, Oct 13, 2019 at 3:41 PM Andres Freund wrote: > - cluster for expression indexes (line 935) We've never had coverage of this, but perhaps that can be added now. > - sorts exceeding INT_MAX / 2 memory (line 1337), but that seems hard to > test realistically I don't think that that can b

Re: Zedstore - compressed in-core columnar storage

2019-10-15 Thread Ashutosh Sharma
Hi, I got chance to spend some time looking into the recent changes done in the zedstore code, basically the functions for packing datums into the attribute streams and handling attribute leaf pages. I didn't find any issues but there are some minor comments that I found when reviewing. I have wor

Re: configure fails for perl check on CentOS8

2019-10-15 Thread Kyotaro Horiguchi
Hi. Sorry for the delay. At Thu, 10 Oct 2019 11:51:21 -0400, Tom Lane wrote in > Andrew Dunstan writes: > > On 10/10/19 1:46 AM, Kyotaro Horiguchi wrote: > >> Hello, While I'm moving to CentOS8 environment, I got stuck at > >> ./configure with the following error. > >> configure: error: libperl

Re: [HACKERS] Block level parallel vacuum

2019-10-15 Thread Amit Kapila
On Tue, Oct 15, 2019 at 1:26 PM Masahiko Sawada wrote: > > On Tue, Oct 15, 2019 at 4:15 PM Masahiko Sawada wrote: > > > > > > > If we avoid postponing deleting empty pages till the cleanup phase, > > > > > then we don't have the problem for gist indexes. > > > > > > > > Yes. But considering your

Re: ProcArrayGroupClearXid() compare-exchange style

2019-10-15 Thread Amit Kapila
On Tue, Oct 15, 2019 at 9:23 AM Noah Misch wrote: > > ProcArrayGroupClearXid() has this: > > while (true) > { > nextidx = > pg_atomic_read_u32(&procglobal->procArrayGroupFirst); > > ... > > if > (pg_atomic_compare_exchange_u32(&proc

Re: [HACKERS] Block level parallel vacuum

2019-10-15 Thread Dilip Kumar
On Tue, Oct 15, 2019 at 12:25 PM Amit Kapila wrote: > > Right, apart from some functions for memory allocation/estimation and > stats copy, we might need something like amcanparallelvacuum, so that > index methods can have the option to not participate in parallel > vacuum due to reasons similar

Re: [HACKERS] Block level parallel vacuum

2019-10-15 Thread Masahiko Sawada
On Tue, Oct 15, 2019 at 4:15 PM Masahiko Sawada wrote: > > On Tue, Oct 15, 2019 at 3:55 PM Amit Kapila wrote: > > > > On Tue, Oct 15, 2019 at 10:34 AM Masahiko Sawada > > wrote: > > > > > > On Mon, Oct 14, 2019 at 6:37 PM Amit Kapila > > > wrote: > > > > > > > > > > > 3. Do we really need to

Re: Columns correlation and adaptive query optimization

2019-10-15 Thread Konstantin Knizhnik
On 15.10.2019 1:20, legrand legrand wrote: Hello Konstantin, What you have proposed regarding join_selectivity and multicolumn statistics is a very good new ! Regarding your auto_explain modification, maybe an "advisor" mode would also be helpfull (with auto_explain_add_statistics_threshold=-

Re: [HACKERS] Block level parallel vacuum

2019-10-15 Thread Masahiko Sawada
On Tue, Oct 15, 2019 at 3:55 PM Amit Kapila wrote: > > On Tue, Oct 15, 2019 at 10:34 AM Masahiko Sawada > wrote: > > > > On Mon, Oct 14, 2019 at 6:37 PM Amit Kapila wrote: > > > > > > > > 3. Do we really need to give the responsibility of deleting empty > > > pages (gistvacuum_delete_empty_page

Clean up MinGW def file generation

2019-10-15 Thread Peter Eisentraut
I was mystified by this comment in Makefile.shlib: # We need several not-quite-identical variants of .DEF files to build # DLLs for Windows. These are made from the single source file # exports.txt. Since we can't assume that Windows boxes will have # sed, the .DEF files are always built and inc