Re: [HACKERS] Refactor handling of database attributes between pg_dump and pg_dumpall

2017-03-21 Thread Haribabu Kommi
Because of this refactor handing of database objects between pg_dump and pg_dumpall, the latest pg_dump tap tests are failing in the following scenarios. 1. CREATE DATABASE postgres Before this patch, the pg_dump uses to dump the CREATE DATABASE command of postgres but not by pg_dumpall. During t

[HACKERS] Partition-wise aggregation/grouping

2017-03-21 Thread Jeevan Chalke
Hi all, Declarative partitioning is supported in PostgreSQL 10 and work is already in progress to support partition-wise joins. Here is a proposal for partition-wise aggregation/grouping. Our initial performance measurement has shown 7 times performance when partitions are on foreign servers and

Re: [HACKERS] btree_gin and btree_gist for enums

2017-03-21 Thread Anastasia Lubennikova
28.02.2017 00:22, Andrew Dunstan: OK, here's the whole series of patches. Patch 1 adds the CallerFInfoFunctionCall{1,2} functions. Patch 2 adds btree_gist support for their use for non-varlena types Patch 3 does the same for varlena types (Not required for patch 4, but better to be consistent,

Re: [HACKERS] Proposal: GetOldestXminExtend for ignoring arbitrary vacuum flags

2017-03-21 Thread Haribabu Kommi
On Tue, Mar 21, 2017 at 3:16 PM, Seki, Eiji wrote: > > > Thank you for you review. > > I reflected your comment and attach the updated patch. Thanks for the updated patch. +/* Use these flags in GetOldestXmin as "flags" */ How about some thing like the following. /* Use the following flags as

[HACKERS] segfault in hot standby for hash indexes

2017-03-21 Thread Jeff Janes
Against an unmodified HEAD (17fa3e8), I got a segfault in the hot standby. Using the attached files, I start the test case like this: nice sh do_nocrash_sr.sh >& do_nocrash_sr.err & And start the replica like: rm -r /tmp/data2_replica/ ; psql -p 9876 -c "select pg_create_physical_replication_sl

Re: [HACKERS] Partition-wise aggregation/grouping

2017-03-21 Thread Antonin Houska
Jeevan Chalke wrote: > Declarative partitioning is supported in PostgreSQL 10 and work is already in > progress to support partition-wise joins. Here is a proposal for > partition-wise > aggregation/grouping. Our initial performance measurement has shown 7 times > performance when partitions are

Re: [HACKERS] segfault in hot standby for hash indexes

2017-03-21 Thread Amit Kapila
On Tue, Mar 21, 2017 at 1:28 PM, Jeff Janes wrote: > Against an unmodified HEAD (17fa3e8), I got a segfault in the hot standby. > I think I see the problem in hash_xlog_vacuum_get_latestRemovedXid(). It seems to me that we are using different block_id for registering the deleted items in xlog XLO

[HACKERS] Implementing delete in columnar store fdw

2017-03-21 Thread sri harsha
Hello, I want to implement delete functionality for a column store fdw in postgres. It is similar to file_fdw. I want to use the “AddForeignUpdateTargets” function to implement this , but the junk filter shouldn’t be a column present in the table . Is it possible to add a Expr/Var to the ju

Re: [HACKERS] Asymmetry between parent and child wrt "false" quals

2017-03-21 Thread Amit Langote
On 2017/03/21 14:59, Ashutosh Bapat wrote: > When I run a query like below on a child-less table, the plan comes out to be > > explain verbose SELECT * FROM uprt1_l WHERE a = 1 AND a = 2; > QUERY PLAN > -

Re: [HACKERS] [PATCH]: fix bug in SP-GiST box_ops

2017-03-21 Thread Anastasia Lubennikova
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:not tested As I can see, this bugfix was already discussed and reviewed. All

Re: [HACKERS] Other formats in pset like markdown, rst, mediawiki

2017-03-21 Thread Ideriha, Takeshi
Hi, I like your suggestion and took a look at your patch though I’m not the expert about psql. I like the idea taking advantage of linestyle utilities to implement rst and markdown format efficiently instead of newly developing pset format things. But I'm thinking two comments below needs change

Re: [HACKERS] extended statistics: n-distinct

2017-03-21 Thread Kyotaro HORIGUCHI
Thank you for finishing this. At Mon, 20 Mar 2017 16:02:20 -0300, Alvaro Herrera wrote in <20170320190220.ixlaueanxegqd5gr@alvherre.pgsql> > Here is a closer to final version of the multivariate statistics series, > last posted at > https://www.postgresql.org/message-id/20170316222033.ncdi7nidah

Re: [HACKERS] Partitioned tables and relfilenode

2017-03-21 Thread Amit Langote
On 2017/03/21 1:16, Robert Haas wrote: > On Fri, Mar 17, 2017 at 4:57 AM, Amit Langote > wrote: >>> Yes, but on the flip side, you're having to add code in a lot of >>> places -- I think I counted 7 -- where you turn around and ignore >>> those AppendRelInfos. >> >> Perhaps you were looking at the

Re: [HACKERS] postgres_fdw: support parameterized foreign joins

2017-03-21 Thread Etsuro Fujita
On 2017/03/16 22:23, Arthur Zakirov wrote: 2017-02-27 12:40 GMT+03:00 Etsuro Fujita : I'd like to propose to support parameterized foreign joins. Attached is a patch for that, which has been created on top of [1]. Can you rebase the patch? It is not applied now. Ok, will do. Thanks for th

Re: [HACKERS] postgres_fdw bug in 9.6

2017-03-21 Thread Etsuro Fujita
On 2017/03/17 0:37, David Steele wrote: This patch does not apply cleanly at cccbdde: Marked "Waiting on Author". Ok, I'll update the patch. One thing I'd like to revise in addition to that is (1) add to JoinPathExtraData a flag member to indicate whether to give the FDW a chance to consi

Re: [HACKERS] Push down more full joins in postgres_fdw

2017-03-21 Thread Etsuro Fujita
On 2017/03/17 2:35, Robert Haas wrote: And ... I don't see anything to complain about, so, committed. Thanks for committing, Robert! Thanks for reviewing, Ashutosh and David! Best regards, Etsuro Fujita -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] Other formats in pset like markdown, rst, mediawiki

2017-03-21 Thread Pavel Stehule
2017-03-21 9:59 GMT+01:00 Ideriha, Takeshi : > Hi, > > I like your suggestion and took a look at your patch though I’m not the > expert about psql. > > > > I like the idea taking advantage of linestyle utilities > to implement rst and markdown format efficiently instead of newly > developing pset

Re: [HACKERS] Other formats in pset like markdown, rst, mediawiki

2017-03-21 Thread Jan Michálek
2017-03-21 9:59 GMT+01:00 Ideriha, Takeshi : > Hi, > > I like your suggestion and took a look at your patch though I’m not the > expert about psql. > > > > I like the idea taking advantage of linestyle utilities > > to implement rst and markdown format efficiently instead of newly > developing pse

Re: [HACKERS] WIP: Covering + unique indexes.

2017-03-21 Thread Anastasia Lubennikova
Patch rebased to the current master is in attachments. -- Anastasia Lubennikova Postgres Professional: http://www.postgrespro.com The Russian Postgres Company commit 497d52b713dd8f926b465ddf22f21db7229b12e3 Author: Anastasia Date: Tue Mar 21 12:58:13 2017 +0300 include_columns_10.0_v4.pa

Re: [HACKERS] Other formats in pset like markdown, rst, mediawiki

2017-03-21 Thread Pavel Stehule
2017-03-21 10:59 GMT+01:00 Jan Michálek : > > > 2017-03-21 9:59 GMT+01:00 Ideriha, Takeshi >: > >> Hi, >> >> I like your suggestion and took a look at your patch though I’m not the >> expert about psql. >> >> >> >> I like the idea taking advantage of linestyle utilities >> >> to implement rst and

Re: [HACKERS] Enabling parallelism for queries coming from SQL or other PL functions

2017-03-21 Thread Rafia Sabih
On Wed, Mar 15, 2017 at 8:55 PM, Robert Haas wrote: > Note this: > > if (completed || !fcache->returnsSet) > postquel_end(es); > > When the SQL function doesn't return a set, then we can allow > parallelism even when lazyEval is set, because we'll only call > ExecutorStart() on

Re: [HACKERS] pageinspect and hash indexes

2017-03-21 Thread Amit Kapila
On Tue, Mar 21, 2017 at 10:15 AM, Ashutosh Sharma wrote: >> >> I think it is not just happening for freed overflow but also for newly >> allocated bucket page. It would be good if we could mark freed >> overflow page as UNUSED page rather than just initialising it's header >> portion and leaving

Re: [HACKERS] Partition-wise join for join between (declaratively) partitioned tables

2017-03-21 Thread Rajkumar Raghuwanshi
> On Mon, Mar 20, 2017 at 1:19 PM, Ashutosh Bapat > wrote: I have created some test to cover partition wise joins with postgres_fdw, also verified make check. patch attached. Thanks & Regards, Rajkumar Raghuwanshi QMG, EnterpriseDB Corporation diff --git a/contrib/postgres_fdw/expected/postgres_

Re: [HACKERS] Logical Replication and Character encoding

2017-03-21 Thread Kyotaro HORIGUCHI
At Mon, 6 Mar 2017 11:13:48 +0100, Petr Jelinek wrote in > >>> Well the length is necessary to be able to add binary format support in > >>> future so it's definitely not an omission. > >> > >> Right. So it appears the right function to use here is > >> pq_sendcountedtext(). However, this send

Re: [HACKERS] Logical Replication and Character encoding

2017-03-21 Thread Kyotaro HORIGUCHI
Mmm. I shot the previous mail halfway. At Mon, 6 Mar 2017 11:13:48 +0100, Petr Jelinek wrote in > > By the way, I noticed that postmaster launches logical > > replication launcher even if wal_level < logical. Is it right? > > Yes, that came up couple of times in various threads. The logical >

Re: [HACKERS] Enabling parallelism for queries coming from SQL or other PL functions

2017-03-21 Thread Dilip Kumar
On Tue, Mar 21, 2017 at 3:36 PM, Rafia Sabih wrote: > On Wed, Mar 15, 2017 at 8:55 PM, Robert Haas wrote: >> Note this: >> >> if (completed || !fcache->returnsSet) >> postquel_end(es); >> >> When the SQL function doesn't return a set, then we can allow >> parallelism even when

Re: [HACKERS] Other formats in pset like markdown, rst, mediawiki

2017-03-21 Thread Jan Michálek
2017-03-21 11:01 GMT+01:00 Pavel Stehule : > > > 2017-03-21 10:59 GMT+01:00 Jan Michálek : > >> >> >> 2017-03-21 9:59 GMT+01:00 Ideriha, Takeshi > m>: >> >>> Hi, >>> >>> I like your suggestion and took a look at your patch though I’m not the >>> expert about psql. >>> >>> >>> >>> I like the idea t

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-21 Thread Amit Kapila
On Fri, Mar 10, 2017 at 11:37 PM, Alvaro Herrera wrote: > Robert Haas wrote: >> On Wed, Mar 8, 2017 at 2:30 PM, Alvaro Herrera >> wrote: >> > Not really -- it's a bit slower actually in a synthetic case measuring >> > exactly the slowed-down case. See >> > https://www.postgresql.org/message-id/

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-21 Thread Amit Kapila
On Thu, Mar 9, 2017 at 8:43 AM, Robert Haas wrote: > On Wed, Mar 8, 2017 at 2:30 PM, Alvaro Herrera > wrote: >> Not really -- it's a bit slower actually in a synthetic case measuring >> exactly the slowed-down case. See >> https://www.postgresql.org/message-id/cad__ougk12zqmwwjzim-yyud1y8jmmy6x

Re: [HACKERS] segfault in hot standby for hash indexes

2017-03-21 Thread Ashutosh Sharma
Hi Jeff, On Tue, Mar 21, 2017 at 1:54 PM, Amit Kapila wrote: > On Tue, Mar 21, 2017 at 1:28 PM, Jeff Janes wrote: >> Against an unmodified HEAD (17fa3e8), I got a segfault in the hot standby. >> > > I think I see the problem in hash_xlog_vacuum_get_latestRemovedXid(). > It seems to me that we ar

[HACKERS] Problem in Parallel Bitmap Heap Scan?

2017-03-21 Thread Thomas Munro
Hi, I noticed a failure in the inet.sql test while running the regression tests with parallelism cranked up, and can reproduce it interactively as follows. After an spgist index is created and the plan changes to the one shown below, the query returns no rows. regression=# set force_parallel_mod

[HACKERS] postgres_fdw: correct regression test for parameterized scan for foreign table

2017-03-21 Thread Etsuro Fujita
Hi, While working on adding support for parameterized foreign joins to postgres_fdw, I noticed that commit e4106b2528727c4b48639c0e12bf2f70a766b910 forgot to modify a test query for execution of a parameterized foreign scan for a foreign table: --- parameterized remote path +-- parameteri

Re: [HACKERS] Create replication slot in pg_basebackup if requested and not yet present

2017-03-21 Thread Michael Banck
Hi, Am Dienstag, den 21.03.2017, 11:03 +0900 schrieb Michael Paquier: > On Tue, Mar 21, 2017 at 1:32 AM, Michael Banck > wrote: > /* > + * Try to create a permanent replication slot if one is specified. Do > + * not error out if the slot already exists, other errors are already > +

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-21 Thread Robert Haas
On Tue, Mar 21, 2017 at 6:56 AM, Amit Kapila wrote: >> Hmm, that test case isn't all that synthetic. It's just a single >> column bulk update, which isn't anything all that crazy, and 5-10% >> isn't nothing. >> >> I'm kinda surprised it made that much difference, though. >> > > I think it is beca

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-03-21 Thread Thomas Munro
Hi, Here is a new version of the patch series addressing complaints from Rafia, Peter, Andres and Robert -- see below. First, two changes not already covered in this thread: 1. Today Robert asked me a question off-list that I hadn't previously considered: since I am sharing tuples between backe

Re: [HACKERS] Removing binaries

2017-03-21 Thread Robert Haas
On Mon, Mar 20, 2017 at 6:15 PM, David Steele wrote: > On 3/20/17 3:40 PM, Jan de Visser wrote: >> On Monday, March 20, 2017 3:30:49 PM EDT Robert Haas wrote: >>> That would annoy me, because I use these constantly. I also think >>> that they solve a problem for users, which is this: >>> >>> [rha

Re: [HACKERS] Create replication slot in pg_basebackup if requested and not yet present

2017-03-21 Thread Michael Banck
Am Dienstag, den 21.03.2017, 12:52 +0100 schrieb Michael Banck: > New patches attached. On second though, there was a superfluous whitespace change in t/010_pg_basebackup.pl, and I've moved the check-for-hex regex fix to the second patch as it's cosmetic and not related to changing the --slot crea

Re: [HACKERS] BRIN cost estimate

2017-03-21 Thread David Rowley
On 17 March 2017 at 23:50, Emre Hasegeli wrote: >> 1. >> >> + Assert(nnumbers == 1); >> >> I think its a bad idea to Assert() this. The stat tuple can come from >> a plugin which could do anything. Seems like if we need to be certain >> of that then it should be an elog(ERROR), maybe mention that

Re: [HACKERS] Problem in Parallel Bitmap Heap Scan?

2017-03-21 Thread Dilip Kumar
On Tue, Mar 21, 2017 at 4:47 PM, Thomas Munro wrote: > I noticed a failure in the inet.sql test while running the regression > tests with parallelism cranked up, and can reproduce it interactively > as follows. After an spgist index is created and the plan changes to > the one shown below, the qu

Re: [HACKERS] ANALYZE command progress checker

2017-03-21 Thread Haribabu Kommi
On Tue, Mar 21, 2017 at 3:41 PM, vinayak wrote: > Thank you for testing the patch on Windows platform. > > Thanks for the updated patch. It works good for a normal relation. But for a relation that contains child tables, the PROGRESS_ANALYZE_NUM_ROWS_SAMPLED produces wrong results. How about a

Re: [HACKERS] BRIN cost estimate

2017-03-21 Thread Emre Hasegeli
> Not sure what you mean here. I'm not speaking of the brin index am, I > mean the get_index_stats_hook call which you've added. I see. Actually this part was from Alvaro. I haven't noticed the get_index_stats_hook call before, but it is still the same coding as btcostestimate(). btcostestimate

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-21 Thread Pavan Deolasee
On Tue, Mar 21, 2017 at 5:34 PM, Robert Haas wrote: > On Tue, Mar 21, 2017 at 6:56 AM, Amit Kapila > wrote: > >> Hmm, that test case isn't all that synthetic. It's just a single > >> column bulk update, which isn't anything all that crazy, and 5-10% > >> isn't nothing. > >> > >> I'm kinda surpr

[HACKERS] Implementing delete in columnar store fdw

2017-03-21 Thread Hari Krishnan
Hello, I want to implement delete functionality for a column store fdw in postgres. It is similar to file_fdw. I want to use the “AddForeignUpdateTargets” function to implement this , but the junk filter shouldn’t be a column present in the table . Is it possible to add a Expr/Var to t

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2017-03-21 Thread Amit Kapila
On Mon, Mar 20, 2017 at 8:27 AM, Robert Haas wrote: > On Fri, Mar 17, 2017 at 2:30 AM, Amit Kapila wrote: >>> I was wondering about doing an explicit test: if the XID being >>> committed matches the one in the PGPROC, and nsubxids matches, and the >>> actual list of XIDs matches, then apply the o

Re: [HACKERS] increasing the default WAL segment size

2017-03-21 Thread Beena Emerson
PFA an updated patch. This fixes an issue reported by Tushar internally. Since the patch changes the way min and max wal_size is stored internally from segment count to size in kb, it limited the maximum size of min and max_wal_size to 2GB in 32 bit systems. The minimum required segment is 2 and

Re: [HACKERS] increasing the default WAL segment size

2017-03-21 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > On Mon, Mar 20, 2017 at 7:23 PM, David Steele wrote: > > With 16MB WAL segments the filename neatly aligns with the LSN. For > > example: > > > > WAL FILE 0001000100FE = LSN 1/FE00 > > > > This no longer holds true with this

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-21 Thread Robert Haas
On Tue, Mar 21, 2017 at 8:41 AM, Pavan Deolasee wrote: >> Yeah. So what's the deal with this? Is somebody working on figuring >> out a different approach that would reduce this overhead? Are we >> going to defer WARM to v11? Or is the intent to just ignore the 5-10% >> slowdown on a single col

Re: [HACKERS] [PATCH]: fix bug in SP-GiST box_ops

2017-03-21 Thread Teodor Sigaev
Thank you, pushed. I just make test table permanent. Anastasia Lubennikova wrote: The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-21 Thread Alvaro Herrera
Amit Kapila wrote: > I think it is because heap_getattr() is not that cheap. We have > noticed the similar problem during development of scan key push down > work [1]. One possibility to reduce the cost of that is to use whole tuple deform instead of repeated individual heap_getattr() calls. Si

Re: [HACKERS] exposing wait events for non-backends (was: Tracking wait event for latches)

2017-03-21 Thread Kuntal Ghosh
On Tue, Mar 21, 2017 at 10:52 AM, Michael Paquier wrote: Thank you for the review. > Unfortunately this is true only for background workers that connect to > a database. And this would break for bgworkers that do not do that. > The point to fix is here: > + if (MyBackendId != InvalidBackendId)

Re: [HACKERS] Partitioned tables and relfilenode

2017-03-21 Thread Robert Haas
On Tue, Mar 21, 2017 at 5:05 AM, Amit Langote wrote: > Attached updated patches. Committed 0001 after removing a comma. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make cha

Re: [HACKERS] Freeze on Cygwin w/ concurrency

2017-03-21 Thread Robert Haas
On Mon, Mar 20, 2017 at 11:47 PM, Noah Misch wrote: > "pgbench -i -s 50; pgbench -S -j2 -c16 -T900 -P5" freezes consistently on > Cygwin 2.2.1 and Cygwin 2.6.0. (I suspect most other versions are affected.) > I've pinged[1] the Cygwin bug thread with some additional detail. Ouch. -- Robert Haa

[HACKERS] Multiple false-positive warnings from Valgrind

2017-03-21 Thread Aleksander Alekseev
Hello. I need a little help. Recently I've decided to run PostgreSQL under Valgrind according to wiki description [1]. Lots of warnings are generated [2] but it is my understanding that all of them are false-positive. For instance I've found these two reports particularly interesting: ``` ==00:0

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-21 Thread Amit Kapila
On Tue, Mar 21, 2017 at 6:55 PM, Robert Haas wrote: > On Tue, Mar 21, 2017 at 8:41 AM, Pavan Deolasee > wrote: >>> Yeah. So what's the deal with this? Is somebody working on figuring >>> out a different approach that would reduce this overhead? Are we >>> going to defer WARM to v11? Or is the

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-21 Thread Robert Haas
On Tue, Mar 21, 2017 at 10:01 AM, Amit Kapila wrote: >> I think that very wide columns and highly indexed tables are not >> particularly unrealistic, nor do I think updating all the rows is >> particularly unrealistic. Sure, it's not everything, but it's >> something. Now, I would agree that all

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-21 Thread Pavan Deolasee
On Tue, Mar 21, 2017 at 6:55 PM, Robert Haas wrote: > > I think that very wide columns and highly indexed tables are not > particularly unrealistic, nor do I think updating all the rows is > particularly unrealistic. Ok. But those who update 10M rows in a single transaction, would they really n

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-21 Thread Alvaro Herrera
Robert Haas wrote: > On Tue, Mar 21, 2017 at 10:01 AM, Amit Kapila wrote: > > Sure, we can try that. I think we need to try it with > > synchronous_commit = off, otherwise, WAL writes completely overshadows > > everything. > > synchronous_commit = off is a much more realistic scenario than fsyn

Re: [HACKERS] Logical decoding on standby

2017-03-21 Thread Simon Riggs
On 21 March 2017 at 02:21, Craig Ringer wrote: > On 20 March 2017 at 17:33, Andres Freund wrote: > >>> Subject: [PATCH 2/3] Follow timeline switches in logical decoding >> >> FWIW, the title doesn't really seem accurate to me. > > Yeah, it's not really at the logical decoding layer at all. > > "T

Re: [HACKERS] Partition-wise join for join between (declaratively) partitioned tables

2017-03-21 Thread Ashutosh Bapat
Thanks Rajkumar. Added those in the latest set of patches. On Tue, Mar 21, 2017 at 3:52 PM, Rajkumar Raghuwanshi wrote: >> On Mon, Mar 20, 2017 at 1:19 PM, Ashutosh Bapat >> wrote: > > I have created some test to cover partition wise joins with > postgres_fdw, also verified make check. > patch a

Re: [HACKERS] Removing binaries

2017-03-21 Thread David G. Johnston
On Tue, Mar 21, 2017 at 5:12 AM, Robert Haas wrote: > > Here's another idea: what if we always created the default database at > initdb time? For example, if I initdb as rhaas, maybe it should > create an "rhaas" database for me, so that this works: > > initdb > pg_ctl start > psql > > I think a

Re: [HACKERS] Partition-wise join for join between (declaratively) partitioned tables

2017-03-21 Thread Robert Haas
On Tue, Mar 21, 2017 at 7:41 AM, Ashutosh Bapat wrote: > On Mon, Mar 20, 2017 at 10:17 PM, Ashutosh Bapat > wrote: >>> >>> On a further testing of this patch I find another case when it is >>> showing regression, the time taken with patch is around 160 secs and >>> without it is 125 secs. >>> Ano

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-21 Thread Robert Haas
On Tue, Mar 21, 2017 at 10:21 AM, Alvaro Herrera wrote: > Robert Haas wrote: >> On Tue, Mar 21, 2017 at 10:01 AM, Amit Kapila >> wrote: > >> > Sure, we can try that. I think we need to try it with >> > synchronous_commit = off, otherwise, WAL writes completely overshadows >> > everything. >> >>

Re: [HACKERS] increasing the default WAL segment size

2017-03-21 Thread David Steele
On 3/21/17 9:04 AM, Stephen Frost wrote: Robert, * Robert Haas (robertmh...@gmail.com) wrote: On Mon, Mar 20, 2017 at 7:23 PM, David Steele wrote: With 16MB WAL segments the filename neatly aligns with the LSN. For example: WAL FILE 0001000100FE = LSN 1/FE00 This no longer

Re: [HACKERS] Removing binaries

2017-03-21 Thread David Steele
On 3/21/17 10:30 AM, David G. Johnston wrote: On Tue, Mar 21, 2017 at 5:12 AM, Robert Haas mailto:robertmh...@gmail.com>>wrote: Here's another idea: what if we always created the default database at initdb time? For example, if I initdb as rhaas, maybe it should create an "rhaas" d

Re: [HACKERS] Removing binaries

2017-03-21 Thread Robert Haas
On Tue, Mar 21, 2017 at 10:43 AM, David Steele wrote: > I suppose it would be too big a change to have psql try the username and > then fallback to postgres on failure? It's not so much that the change would be too big as that the resulting semantics would be confusing, at least IMHO. Imagine:

Re: [HACKERS] [COMMITTERS] pgsql: Add missing support for new node fields

2017-03-21 Thread Tom Lane
Andres Freund writes: > On 2017-03-21 07:22:57 +0100, Fabien COELHO wrote: >> I've been annoyed by these stupid functions and forgetting to update them >> since I run into them while trying to fix an issue in pg_stat_statement some >> time ago. >> >> I've started to develop a perl script to gener

Re: [HACKERS] [COMMITTERS] pgsql: Add missing support for new node fields

2017-03-21 Thread Robert Haas
On Tue, Mar 21, 2017 at 10:56 AM, Tom Lane wrote: > Andres Freund writes: >> On 2017-03-21 07:22:57 +0100, Fabien COELHO wrote: >>> I've been annoyed by these stupid functions and forgetting to update them >>> since I run into them while trying to fix an issue in pg_stat_statement some >>> time a

Re: [HACKERS] WIP: Faster Expression Processing v4

2017-03-21 Thread Tom Lane
Andres Freund writes: > On 2017-03-20 16:06:27 -0400, Tom Lane wrote: >> ... is there a reason why resultnum for EEOP_ASSIGN_* steps is declared >> size_t and not just int? Since it's an array index, and one that >> certainly can't be bigger than AttrNumber, that seems rather confusing. > Not th

Re: [HACKERS] Review: GIN non-intrusive vacuum of posting tree

2017-03-21 Thread Teodor Sigaev
Thank you for your suggestions, do not hesitate to ask any questions, concurrency and GIN both are very interesting topics. I had a look on patch and found some issue. Look at ginvacuum.c around line 387, function ginVacuumPostingTreeLeaves(): /* * All subtree is empty - just

Re: [HACKERS] [PATCH] kNN for SP-GiST

2017-03-21 Thread David Steele
Hi Nikita, On 3/9/17 8:52 AM, Alexander Korotkov wrote: I take a look to this patchset. My first notes are following. This thread has been idle for quite a while. Please respond and/or post a new patch by 2017-03-24 00:00 AoE (UTC-12) or this submission will be marked "Returned with Feedb

Re: [HACKERS] Potential data loss of 2PC files

2017-03-21 Thread Teodor Sigaev
If that can happen, don't we have the same problem in many other places? Like, all the SLRUs? They don't fsync the directory either. Right, pg_commit_ts and pg_clog enter in this category. Implemented as attached. Is unlink() guaranteed to be durable, without fsyncing the directory? If not, t

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2017-03-21 Thread Robert Haas
On Sun, Mar 19, 2017 at 9:03 PM, Peter Geoghegan wrote: > On Sun, Mar 12, 2017 at 3:05 PM, Peter Geoghegan wrote: >> I attach my V9 of the patch. I came up some stuff for the design of >> resource management that I think meets every design goal that we have >> for shared/unified BufFiles: > > Com

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-21 Thread Andres Freund
On 2017-03-21 08:04:11 -0400, Robert Haas wrote: > On Tue, Mar 21, 2017 at 6:56 AM, Amit Kapila wrote: > >> Hmm, that test case isn't all that synthetic. It's just a single > >> column bulk update, which isn't anything all that crazy, and 5-10% > >> isn't nothing. > >> > >> I'm kinda surprised it

Re: [HACKERS] Partitioned tables and relfilenode

2017-03-21 Thread Simon Riggs
On 16 March 2017 at 10:03, Amit Langote wrote: > On 2017/03/15 7:09, Robert Haas wrote: >> I think that eliding the Append node when there's only one child may >> be unsafe in the case where the child's attribute numbers are >> different from the parent's attribute numbers. I remember Tom making

Re: [HACKERS] [COMMITTERS] pgsql: Add missing support for new node fields

2017-03-21 Thread Pavel Stehule
> > IMHO, what would be a lot more useful than something that generates > {read,equal,copy,out}funcs.c automatically would be something that > just checks them for trivial errors of omission. For example, if you > read a list of structure members from the appropriate header files and > cross-check

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-21 Thread Andres Freund
On 2017-03-21 19:49:07 +0530, Pavan Deolasee wrote: > On Tue, Mar 21, 2017 at 6:55 PM, Robert Haas wrote: > > > > > I think that very wide columns and highly indexed tables are not > > particularly unrealistic, nor do I think updating all the rows is > > particularly unrealistic. > > > Ok. But

Re: [HACKERS] segfault in hot standby for hash indexes

2017-03-21 Thread Jeff Janes
On Tue, Mar 21, 2017 at 4:00 AM, Ashutosh Sharma wrote: > Hi Jeff, > > On Tue, Mar 21, 2017 at 1:54 PM, Amit Kapila > wrote: > > On Tue, Mar 21, 2017 at 1:28 PM, Jeff Janes > wrote: > >> Against an unmodified HEAD (17fa3e8), I got a segfault in the hot > standby. > >> > > > > I think I see the

Re: [HACKERS] Partitioned tables and relfilenode

2017-03-21 Thread Robert Haas
On Tue, Mar 21, 2017 at 12:19 PM, Simon Riggs wrote: > On 16 March 2017 at 10:03, Amit Langote wrote: >> On 2017/03/15 7:09, Robert Haas wrote: > >>> I think that eliding the Append node when there's only one child may >>> be unsafe in the case where the child's attribute numbers are >>> differen

Re: [HACKERS] Do we create a new roadmap page for development?

2017-03-21 Thread Robert Haas
On Tue, Mar 21, 2017 at 2:36 AM, Tsunakawa, Takayuki wrote: > I'd like to share our roadmap for PostgreSQL development, as other companies > and individuals do in the following page. But this page is for PostgreSQL 10. > > PostgreSQL10 Roadmap > https://wiki.postgresql.org/wiki/PostgreSQL10_Road

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-21 Thread Bruce Momjian
On Tue, Mar 21, 2017 at 09:25:49AM -0400, Robert Haas wrote: > On Tue, Mar 21, 2017 at 8:41 AM, Pavan Deolasee > > TBH I see many artificial scenarios here. It will be very useful if he can > > rerun the query with some of these restrictions lifted. I'm all for > > addressing whatever we can, but I

Re: [HACKERS] Automatic cleanup of oldest WAL segments with pg_receivexlog

2017-03-21 Thread David Steele
Hi Michael, On 3/10/17 9:15 AM, Peter Eisentraut wrote: On 3/9/17 17:03, Michael Paquier wrote: Having something like --limit-retained-segments partially addresses it, as long as there is a way to define an automatic mode, based on statvfs() obviously. But that is not portable/usable enough,

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-21 Thread Robert Haas
On Tue, Mar 21, 2017 at 12:49 PM, Bruce Momjian wrote: > On Tue, Mar 21, 2017 at 09:25:49AM -0400, Robert Haas wrote: >> On Tue, Mar 21, 2017 at 8:41 AM, Pavan Deolasee >> > TBH I see many artificial scenarios here. It will be very useful if he can >> > rerun the query with some of these restricti

Re: [HACKERS] make async slave to wait for lsn to be replayed

2017-03-21 Thread David Steele
Hi Ivan, On 3/12/17 10:20 PM, Thomas Munro wrote: On Fri, Mar 10, 2017 at 1:49 AM, Ivan Kartyshov wrote: Here I attached rebased patch waitlsn_10dev_v3 (core feature) I will leave the choice of implementation (core/contrib) to the discretion of the community. Will be glad to hear your suggest

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-21 Thread Peter Geoghegan
On Tue, Mar 21, 2017 at 10:04 AM, Robert Haas wrote: > I think that's a good question. I previously expressed similar > concerns. On the one hand, it's hard to ignore the fact that, in the > cases where this wins, it already buys us a lot of performance > improvement. On the other hand, as you

Re: [HACKERS] PassDownLimitBound for ForeignScan/CustomScan [take-2]

2017-03-21 Thread David Steele
Hi, On 3/13/17 3:25 AM, Jeevan Chalke wrote: I have reviewed this patch further and here are my comments: This thread has been idle for over a week. Please respond and/or post a new patch by 2017-03-24 00:00 AoE (UTC-12) or this submission will be marked "Returned with Feedback". Thanks,

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-21 Thread Robert Haas
On Tue, Mar 21, 2017 at 1:08 PM, Peter Geoghegan wrote: > On Tue, Mar 21, 2017 at 10:04 AM, Robert Haas wrote: >> I think that's a good question. I previously expressed similar >> concerns. On the one hand, it's hard to ignore the fact that, in the >> cases where this wins, it already buys us a

Re: [HACKERS] free space map and visibility map

2017-03-21 Thread Masahiko Sawada
On Mon, Mar 20, 2017 at 11:28 PM, Robert Haas wrote: > On Sat, Mar 18, 2017 at 5:42 PM, Jeff Janes wrote: >> Isn't HEAP2_CLEAN only issued before an intended HOT update? (Which then >> can't leave the block as all visible or all frozen). I think the issue is >> here is HEAP2_VISIBLE or HEAP2_FR

Re: [HACKERS] [PATCH] Generic type subscripting

2017-03-21 Thread David Steele
Hi Dmitry, On 3/14/17 7:10 PM, Tom Lane wrote: Dmitry Dolgov <9erthali...@gmail.com> writes: [ generic_type_subscription_v7.patch ] I looked through this a bit. This thread has been idle for over a week. Please respond and/or post a new patch by 2017-03-24 00:00 AoE (UTC-12) or this submi

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-21 Thread Bruce Momjian
On Tue, Mar 21, 2017 at 01:04:14PM -0400, Robert Haas wrote: > > I know we have talked about it, but not recently, and if everyone else > > is fine with it, I am too, but I have to ask these questions. > > I think that's a good question. I previously expressed similar > concerns. On the one hand

Re: [HACKERS] make async slave to wait for lsn to be replayed

2017-03-21 Thread Robert Haas
On Sun, Mar 12, 2017 at 10:20 PM, Thomas Munro wrote: > Maybe someone can think of a clever way for an extension to insert a > wait for a user-supplied LSN *before* acquiring a snapshot so it can > work for the higher levels, or maybe the hooks should go into core > PostgreSQL so that the extensio

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-03-21 Thread Bruce Momjian
On Tue, Mar 21, 2017 at 01:14:00PM -0400, Robert Haas wrote: > On Tue, Mar 21, 2017 at 1:08 PM, Peter Geoghegan wrote: > > On Tue, Mar 21, 2017 at 10:04 AM, Robert Haas wrote: > >> I think that's a good question. I previously expressed similar > >> concerns. On the one hand, it's hard to ignore

Re: [HACKERS] Partitioned tables and relfilenode

2017-03-21 Thread Simon Riggs
On 21 March 2017 at 16:33, Robert Haas wrote: > On Tue, Mar 21, 2017 at 12:19 PM, Simon Riggs wrote: >> On 16 March 2017 at 10:03, Amit Langote >> wrote: >>> On 2017/03/15 7:09, Robert Haas wrote: >> I think that eliding the Append node when there's only one child may be unsafe in the

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-03-21 Thread Masahiko Sawada
On Wed, Mar 15, 2017 at 4:50 PM, Amit Kapila wrote: > On Thu, Mar 9, 2017 at 10:21 PM, Masahiko Sawada > wrote: >> On Wed, Mar 8, 2017 at 1:43 AM, Peter Geoghegan wrote: >>> On Sat, Mar 4, 2017 at 1:30 AM, Amit Kapila wrote: > While I can't see this explained anywhere, I'm > pretty sur

Re: [HACKERS] GUC for cleanup indexes threshold.

2017-03-21 Thread David Steele
Hi, On 3/15/17 9:50 AM, Amit Kapila wrote: What about if somebody does manual vacuum and there are no garbage tuples to clean, won't in that case also you want to avoid skipping the lazy_cleanup_index? Another option could be to skip updating the relfrozenxid if we have skipped the index clean

Re: [HACKERS] Measuring replay lag

2017-03-21 Thread David Steele
Hi Thomas, On 3/15/17 8:38 PM, Simon Riggs wrote: On 16 March 2017 at 08:02, Thomas Munro wrote: I agree that these states exist, but we disagree on what 'lag' really means, or, rather, which of several plausible definitions would be the most useful here. My proposal is that the *_lag column

Re: [HACKERS] PATCH: recursive json_populate_record()

2017-03-21 Thread David Steele
On 3/16/17 11:54 AM, David Steele wrote: On 2/1/17 12:53 AM, Michael Paquier wrote: On Thu, Jan 26, 2017 at 6:49 AM, Tom Lane wrote: Nikita Glukhov writes: On 25.01.2017 23:58, Tom Lane wrote: I think you need to take a second look at the code you're producing and realize that it's not so c

[HACKERS] Re: [PATCH] guc-ify the formerly hard-coded MAX_SEND_SIZE to max_wal_send

2017-03-21 Thread David Steele
On 3/16/17 11:56 AM, David Steele wrote: My recommendation is that we mark this patch "Returned with Feedback" to allow you time to test and refine the patch. You can resubmit once it is ready. This submission has been marked "Returned with Feedback". Please feel free to resubmit to a futur

Re: [HACKERS] [PATCH] Transaction traceability - txid_status(bigint)

2017-03-21 Thread Robert Haas
On Mon, Mar 20, 2017 at 1:38 AM, Craig Ringer wrote: > On 14 March 2017 at 19:57, Robert Haas wrote: >> On Mon, Mar 13, 2017 at 10:22 PM, Craig Ringer wrote: >>> I'll introduce a new LWLock, ClogTruncationLock, which will be held >>> from when we advance the new clogOldestXid field through to wh

Re: [HACKERS] logical replication apply to run with sync commit off by default

2017-03-21 Thread Robert Haas
On Mon, Mar 20, 2017 at 7:56 PM, Petr Jelinek wrote: > On 18/03/17 13:31, Petr Jelinek wrote: >> On 07/03/17 06:23, Petr Jelinek wrote: >>> there has been discussion at the logical replication initial copy thread >>> [1] about making apply work with sync commit off by default for >>> performance r

Re: [HACKERS] Review: GIN non-intrusive vacuum of posting tree

2017-03-21 Thread Andrew Borodin
Hi, Teodor! 2017-03-21 20:32 GMT+05:00 Teodor Sigaev : > I had a look on patch That's great, thanks! > > /* > * All subtree is empty - just return TRUE to indicate that parent > must > * do a cleanup. Unless we are ROOT an there is way to go upper. > */ > >

  1   2   3   >