Re: [HACKERS] Streaming replication for psycopg2

2015-10-04 Thread Craig Ringer
On 30 June 2015 at 22:42, Shulgin, Oleksandr wrote: > On Thu, Jun 4, 2015 at 5:49 PM, Shulgin, Oleksandr > wrote: >> >> On Tue, Jun 2, 2015 at 2:23 PM, Shulgin, Oleksandr >> wrote: >> > >> > I've submitted a patch to psycopg2 to support streaming replication >> > protocol (COPY_BOTH): https://gi

Re: [HACKERS] [PATCH] postgres_fdw extension support

2015-10-04 Thread Michael Paquier
On Sun, Oct 4, 2015 at 11:40 AM, Paul Ramsey wrote: > I put all changes relative to your review here if you want a nice colorized > place to check > > https://github.com/pramsey/postgres/commit/ed33e7489601e659f436d6afda3cce28304eba50 -/* updatable is available on both server and table */

Re: [HACKERS] ALTER TABLE behind-the-scenes effects' CONTEXT

2015-10-04 Thread Pavel Stehule
2015-10-05 0:08 GMT+02:00 Marko Tiikkaja : > Hi, > > In the past I've found the error message in cases such as this somewhat > less helpful than it could be: > > =# CREATE TABLE qqq (a int); > =# CREATE UNIQUE INDEX IF NOT EXISTS qqq_a_idx ON qqq(a); > =# ALTER TABLE qqq ALTER COLUMN a TYPE json U

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2015-10-04 Thread Amit Kapila
On Mon, Oct 5, 2015 at 6:34 AM, Jeff Janes wrote: > On Fri, Sep 11, 2015 at 8:01 PM, Amit Kapila > wrote: >> >> >> If I am not wrong we need 1048576 number of transactions difference >> for each record to make each CLOG access a disk access, so if we >> increment XID counter by 100, then probabl

Re: [HACKERS] Confusing remark about UPSERT in fdwhandler.sgml

2015-10-04 Thread Etsuro Fujita
On 2015/10/03 5:57, Robert Haas wrote: On Fri, Oct 2, 2015 at 4:04 AM, Peter Geoghegan wrote: On Fri, Oct 2, 2015 at 1:00 AM, Etsuro Fujita wrote: ISTM that the sentence "as remote constraints are not locally known" is somewhat confusing, because check constrains on remote tables can be defin

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

2015-10-04 Thread Jeff Janes
On Fri, Sep 11, 2015 at 8:01 PM, Amit Kapila wrote: > On Fri, Sep 11, 2015 at 9:21 PM, Robert Haas > wrote: > > > > On Fri, Sep 11, 2015 at 10:31 AM, Amit Kapila > wrote: > > > > Could you perhaps try to create a testcase where xids are accessed > that > > > > are so far apart on average that t

Re: [HACKERS] Less than ideal error reporting in pg_stat_statements

2015-10-04 Thread Peter Geoghegan
On Wed, Sep 23, 2015 at 1:41 PM, Jim Nasby wrote: > max was set to 1. I don't know about average query text size, but the > command that was causing the error was a very large number of individual > INSERT ... VALUES statements all in one command. > > The machine had plenty of free memory and

Re: [HACKERS] CustomScan support on readfuncs.c

2015-10-04 Thread Kouhei Kaigai
> -Original Message- > From: Robert Haas [mailto:robertmh...@gmail.com] > Sent: Saturday, October 03, 2015 5:44 AM > To: Kaigai Kouhei(海外 浩平) > Cc: Amit Kapila; Andres Freund; pgsql-hackers > Subject: ##freemail## Re: [HACKERS] CustomScan support on readfuncs.c > > On Tue, Sep 29, 2015 at

Re: [HACKERS] debbugs

2015-10-04 Thread Stephen Frost
Andrew, * Andrew Dunstan (and...@dunslane.net) wrote: > Could someone please point me at the documentation that says how to > stand up and administer an instance of debbugs? If it doesn't exist > I would say that is a very heavy mark against it. If it does, it's > well enough hidden that a quick u

Re: [HACKERS] No Issue Tracker - Say it Ain't So!]

2015-10-04 Thread Nathan Wagner
On Sun, Oct 04, 2015 at 04:30:49PM -0700, Josh Berkus wrote: > That would be the key part, wouldn't it? Nice that you have [code to > store and parse email messages]. Yeah. It actually made most of the work pretty easy. It's available with a bunch of other code at https://pd.if.org/git/ if anyo

[HACKERS] debbugs

2015-10-04 Thread Andrew Dunstan
Could someone please point me at the documentation that says how to stand up and administer an instance of debbugs? If it doesn't exist I would say that is a very heavy mark against it. If it does, it's well enough hidden that a quick use of Google doesn't make it stand out, which doesn't fill

Re: [HACKERS] No Issue Tracker - Say it Ain't So!]

2015-10-04 Thread Josh Berkus
On 10/04/2015 03:42 PM, Nathan Wagner wrote: > I downloaded the archives for pgsql-bugs, and fed them into a database. This > part was easy, since I have already written a pg backed usenet server and had > the code hand for storing and parsing out bits of rfc 2822 messages. That would be the key

Re: [HACKERS] Less than ideal error reporting in pg_stat_statements

2015-10-04 Thread Peter Geoghegan
On Sun, Oct 4, 2015 at 3:16 PM, Tom Lane wrote: > Peter Geoghegan writes: >> To be clear: I wasn't sure why you though I falsely count entries with >> dropped texts within entry_dealloc(). > > In the existing^H^H^Hprevious code, dropped-text entries would essentially > act as length-zero summands

Re: [HACKERS] Less than ideal error reporting in pg_stat_statements

2015-10-04 Thread Peter Geoghegan
On Sun, Oct 4, 2015 at 3:10 PM, Tom Lane wrote: >> That seems perfectly reasonable, yes. Should I leave that to you? > > After a closer look I decided that wasn't reasonable at all. Discounting > sticky texts would then mean that after a GC cycle, we might still think > the query texts file is bl

Re: [HACKERS] Less than ideal error reporting in pg_stat_statements

2015-10-04 Thread Tom Lane
Andrew Dunstan writes: > Sorry, I'm a bit late to this party. Does what you have committed mean > people are less likely to see "Out of Memory" coming from > pg_stat_statements? If not, what can be done about them short of a > restart? And what bad effects follow from an event generating them?

Re: [HACKERS] Less than ideal error reporting in pg_stat_statements

2015-10-04 Thread Andrew Dunstan
On 10/04/2015 06:16 PM, Tom Lane wrote: Peter Geoghegan writes: To be clear: I wasn't sure why you though I falsely count entries with dropped texts within entry_dealloc(). In the existing^H^H^Hprevious code, dropped-text entries would essentially act as length-zero summands in the average c

Re: [HACKERS] No Issue Tracker - Say it Ain`t So!]

2015-10-04 Thread Greg Sabino Mullane
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 > Comments are welcome, and no, I don't really expect that this will be what > gets > adopted, mainly I wanted to show that we can probably just build something > rather effective off our existing infrastructure +1, good job. > The bugs have

Re: [HACKERS] No Issue Tracker - Say it Ain't So!]

2015-10-04 Thread Nathan Wagner
I don't have the original message for this thread, so I arbitrarily picked a message to reply to. Since what has been asked for is a bug *tracker*, and we already have a bugs mailing list, I put together something. I downloaded the archives for pgsql-bugs, and fed them into a database. This part

Re: [HACKERS] Less than ideal error reporting in pg_stat_statements

2015-10-04 Thread Tom Lane
Peter Geoghegan writes: > To be clear: I wasn't sure why you though I falsely count entries with > dropped texts within entry_dealloc(). In the existing^H^H^Hprevious code, dropped-text entries would essentially act as length-zero summands in the average calculation, whereas I think we agree that

Re: [HACKERS] Less than ideal error reporting in pg_stat_statements

2015-10-04 Thread Tom Lane
Peter Geoghegan writes: > On Sun, Oct 4, 2015 at 1:01 PM, Tom Lane wrote: >> Hm. The problem I've got with this is that then mean_query_len means >> something significantly different after entry_dealloc than it does >> after gc_texts. >> >> I'd be okay with changing *both* of those functions to

[HACKERS] ALTER TABLE behind-the-scenes effects' CONTEXT

2015-10-04 Thread Marko Tiikkaja
Hi, In the past I've found the error message in cases such as this somewhat less helpful than it could be: =# CREATE TABLE qqq (a int); =# CREATE UNIQUE INDEX IF NOT EXISTS qqq_a_idx ON qqq(a); =# ALTER TABLE qqq ALTER COLUMN a TYPE json USING NULL; ERROR: data type json has no default operat

Re: [HACKERS] Less than ideal error reporting in pg_stat_statements

2015-10-04 Thread Peter Geoghegan
On Sun, Oct 4, 2015 at 1:12 PM, Peter Geoghegan wrote: > On Sun, Oct 4, 2015 at 1:01 PM, Tom Lane wrote: >> Ah, right, sorry. I meant to make its result match what gc_texts would >> get, by not falsely counting entries with dropped texts. That's not >> what you have in your patch but it seems l

Re: [HACKERS] Less than ideal error reporting in pg_stat_statements

2015-10-04 Thread Tom Lane
Peter Geoghegan writes: > On Sun, Oct 4, 2015 at 1:01 PM, Tom Lane wrote: >> Hm. The problem I've got with this is that then mean_query_len means >> something significantly different after entry_dealloc than it does >> after gc_texts. >> >> I'd be okay with changing *both* of those functions to

Re: [HACKERS] Less than ideal error reporting in pg_stat_statements

2015-10-04 Thread Peter Geoghegan
On Sun, Oct 4, 2015 at 1:01 PM, Tom Lane wrote: > Ah, right, sorry. I meant to make its result match what gc_texts would > get, by not falsely counting entries with dropped texts. That's not > what you have in your patch but it seems like an easy enough fix. I'm trying to make mean_query_len re

Re: [HACKERS] Potential GIN vacuum bug

2015-10-04 Thread Jeff Janes
On Thu, Sep 3, 2015 at 10:42 AM, Jeff Janes wrote: > On Mon, Aug 31, 2015 at 12:10 AM, Jeff Janes wrote: > >> On Sun, Aug 30, 2015 at 3:57 PM, Tom Lane wrote: >> >>> Jeff Janes writes: >>> > On Sun, Aug 30, 2015 at 11:11 AM, Tom Lane wrote: >>> >> > But we would still have to deal with the >>

Re: [HACKERS] check fails on Fedora 23

2015-10-04 Thread Andrew Dunstan
On 10/04/2015 12:52 PM, Pavel Stehule wrote: Isn't this arguably a Fedora regression? What did they change in F23 to make it fail? I note that F23 is still in Beta. It is working on F22 - so it is looking as regression in some fedora components. can somebody repeat check

Re: [HACKERS] Less than ideal error reporting in pg_stat_statements

2015-10-04 Thread Tom Lane
Peter Geoghegan writes: > I'm not clear on what you actually propose to do to "make > entry_dealloc's recomputation of mean_query_len sane", but I think you > are talking about something distinct from what I've proposed Ah, right, sorry. I meant to make its result match what gc_texts would get,

Re: [HACKERS] Less than ideal error reporting in pg_stat_statements

2015-10-04 Thread Peter Geoghegan
On Sun, Oct 4, 2015 at 9:27 AM, Tom Lane wrote: > Hm. I'm unconvinced by the aspects of this that involve using > mean_query_len as a filter on which texts will be accepted. While that's > not necessarily bad in the abstract, there are way too many implementation > artifacts here that will resul

Re: [HACKERS] about fsync in CLOG buffer write

2015-10-04 Thread Andres Freund
On 2015-10-04 12:14:05 -0700, Jeff Janes wrote: > My (naive) expectation is that no additional locking is needed. > > Once we decide to consult the clog, we already know the transaction is no > longer in progress, so it can't be in-flight to change that clog entry we > care about because it was re

Re: [HACKERS] about fsync in CLOG buffer write

2015-10-04 Thread Jeff Janes
On Sat, Sep 12, 2015 at 5:21 PM, Andres Freund wrote: > On September 12, 2015 5:18:28 PM PDT, Jeff Janes > wrote: > >On Wed, Sep 2, 2015 at 5:32 AM, Andres Freund > >wrote: > > > >> On 2015-09-10 19:39:59 +0800, 张广舟(明虚) wrote: > >> > We found there is a fsync call when CLOG buffer > >> > is wri

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-10-04 Thread Tom Lane
Tomas Vondra writes: > Anyway, I think you're right we're going in circles here. I think we > both presented all the arguments we had and we still disagree. I'm not > going to continue with this - I'm unlikely to win an argument against > two committers if that didn't happen until now. Thanks f

Re: [HACKERS] check fails on Fedora 23

2015-10-04 Thread Pavel Stehule
> Isn't this arguably a Fedora regression? What did they change in F23 to >> make it fail? I note that F23 is still in Beta. >> > It is working on F22 - so it is looking as regression in some fedora components. can somebody repeat check on FC23? Regards Pavel

Re: [HACKERS] Odd query execution behavior with extended protocol

2015-10-04 Thread Tom Lane
Shay Rojansky writes: >> Try adding a sync before the second execute. > I tried inserting a Sync right before the second Execute, this caused an > error with the message 'portal "MQ1" does not exist'. > This seems like problematic behavior on its own, regardless of my issues > here (Sync shouldn'

Re: [HACKERS] Odd query execution behavior with extended protocol

2015-10-04 Thread Tom Lane
Shay Rojansky writes: >> To my mind there is not a lot of value in performing Bind until you >> are ready to do Execute. The only reason the operations are separated >> in the protocol is so that you can do multiple Executes with a row limit >> on each one, to retrieve a large query result in chu

Re: [HACKERS] Less than ideal error reporting in pg_stat_statements

2015-10-04 Thread Tom Lane
Peter Geoghegan writes: > Attached, revised patch deals with the issues around removing the > query text file when garbage collection encounters trouble. There is > no reason to be optimistic about any error within gc_qtexts() not > recurring during a future garbage collection. OOM might be an > e

Re: [HACKERS] check fails on Fedora 23

2015-10-04 Thread Pavel Stehule
2015-10-04 17:52 GMT+02:00 Andrew Dunstan : > > > On 10/04/2015 11:35 AM, Pavel Stehule wrote: > >> >> >> >> > fails on assert >> >> Works for me ... what locale/collation are you running in? >> >> >> LANG=cs_CZ.UTF-8 >> >> >> it depends on locale - it is workin

Re: [HACKERS] check fails on Fedora 23

2015-10-04 Thread Andrew Dunstan
On 10/04/2015 11:35 AM, Pavel Stehule wrote: > fails on assert Works for me ... what locale/collation are you running in? LANG=cs_CZ.UTF-8 it depends on locale - it is working with C or en_US.UTF-8, but doesn't work with Czech locale and fails w

Re: [HACKERS] Odd query execution behavior with extended protocol

2015-10-04 Thread Shay Rojansky
> > I'm fairly sure that the query snapshot is established at Bind time, > which means that this SELECT will run with a snapshot that indeed > does not see the effects of the UPDATE. > > To my mind there is not a lot of value in performing Bind until you > are ready to do Execute. The only reason

Re: [HACKERS] Odd query execution behavior with extended protocol

2015-10-04 Thread Shay Rojansky
> > Try adding a sync before the second execute. > I tried inserting a Sync right before the second Execute, this caused an error with the message 'portal "MQ1" does not exist'. This seems like problematic behavior on its own, regardless of my issues here (Sync shouldn't be causing an implicit clo

Re: [HACKERS] check fails on Fedora 23

2015-10-04 Thread Pavel Stehule
>>> > fails on assert >>> >>> Works for me ... what locale/collation are you running in? >>> >> >> LANG=cs_CZ.UTF-8 >> > > it depends on locale - it is working with C or en_US.UTF-8, but doesn't > work with Czech locale > and fails with Hungarian locales too > > Pavel > > >> >> Regards >> >> Pav

Re: [HACKERS] check fails on Fedora 23

2015-10-04 Thread Pavel Stehule
2015-10-04 17:07 GMT+02:00 Pavel Stehule : > > > 2015-10-04 16:37 GMT+02:00 Tom Lane : > >> Pavel Stehule writes: >> > I am testing PostgreSQL (master) on Fedora 23. The query >> >> > ELECT p1.oid, p1.proname, p2.oid, p2.proname >> > FROM pg_proc AS p1, pg_proc AS p2 >> > WHERE p1.oid < p2.oid AN

Re: [HACKERS] check fails on Fedora 23

2015-10-04 Thread Pavel Stehule
2015-10-04 16:37 GMT+02:00 Tom Lane : > Pavel Stehule writes: > > I am testing PostgreSQL (master) on Fedora 23. The query > > > ELECT p1.oid, p1.proname, p2.oid, p2.proname > > FROM pg_proc AS p1, pg_proc AS p2 > > WHERE p1.oid < p2.oid AND > > p1.prosrc = p2.prosrc AND > > p1.prolang =

Re: [HACKERS] Odd query execution behavior with extended protocol

2015-10-04 Thread Tom Lane
Shay Rojansky writes: > Npgsql supports sending multiple SQL statements in a single packet via the > extended protocol. This works fine, but when the second query SELECTs a > value modified by the first's UPDATE, I'm getting a result as if the UPDATE > hasn't yet occurred. > The exact messages se

Re: [HACKERS] check fails on Fedora 23

2015-10-04 Thread Tom Lane
Pavel Stehule writes: > I am testing PostgreSQL (master) on Fedora 23. The query > ELECT p1.oid, p1.proname, p2.oid, p2.proname > FROM pg_proc AS p1, pg_proc AS p2 > WHERE p1.oid < p2.oid AND > p1.prosrc = p2.prosrc AND > p1.prolang = 12 AND p2.prolang = 12 AND > (p1.proisagg = false

Re: [HACKERS] row_security GUC, BYPASSRLS

2015-10-04 Thread Stephen Frost
* Noah Misch (n...@leadboat.com) wrote: > On Mon, Sep 28, 2015 at 05:13:56PM -0400, Stephen Frost wrote: > > * Noah Misch (n...@leadboat.com) wrote: > > > In schema reviews, I will raise a red flag for use of this feature; the > > > best > > > designs will instead use additional roles. I forecast

Re: [HACKERS] Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

2015-10-04 Thread Amir Rohan
On 10/04/2015 04:29 PM, Andres Freund wrote: > On October 4, 2015 3:27:00 PM GMT+02:00, Amir Rohan > wrote: > >> Perhaps it would help a little if you posted the latest patch here as >> well? So that at least the app picks it up again. > > You can as additional threads in the cf app. > Done,

Re: [HACKERS] Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

2015-10-04 Thread Andres Freund
On October 4, 2015 3:27:00 PM GMT+02:00, Amir Rohan wrote: >Perhaps it would help a little if you posted the latest patch here as >well? So that at least the app picks it up again. You can as additional threads in the cf app. -- Please excuse brevity and formatting - I am writing this on my mo

Re: [HACKERS] WIP: Rework access method interface

2015-10-04 Thread Amit Kapila
On Sat, Oct 3, 2015 at 5:07 PM, Petr Jelinek wrote: > On 2015-10-03 08:27, Amit Kapila wrote: > >> On Fri, Oct 2, 2015 at 8:14 PM, Alexander Korotkov >> mailto:a.korot...@postgrespro.ru>> wrote: >> > >> > >> > I agree about staying with one SQL-visible function. >> > > Okay, this does not nece

[HACKERS] Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

2015-10-04 Thread Amir Rohan
On 10/02/2015 03:33 PM, Michael Paquier wrote: > > Michael, I'm afraid my email bungling has damaged your thread. I didn't include an "In-reply-To" header when I posted: trinity-b4a8035d-59af-4c42-a37e-258f0f28e44a-1443795007012@3capp-mailcom-lxa08. And we subsequently had our discussion over t

Re: [HACKERS] Odd query execution behavior with extended protocol

2015-10-04 Thread Andres Freund
On October 4, 2015 2:50:10 PM GMT+02:00, Shay Rojansky wrote: >> >> > Npgsql supports sending multiple SQL statements in a single packet >via >> the extended protocol. This works fine, but when the second query >SELECTs a >> value modified by the first's UPDATE, I'm getting a result as if the >> >

Re: [HACKERS] Odd query execution behavior with extended protocol

2015-10-04 Thread Shay Rojansky
> > > Npgsql supports sending multiple SQL statements in a single packet via > the extended protocol. This works fine, but when the second query SELECTs a > value modified by the first's UPDATE, I'm getting a result as if the > > UPDATE hasn't yet occurred. > > Looks like the first updating stateme

[HACKERS] Re: In-core regression tests for replication, cascading, archiving, PITR, etc.

2015-10-04 Thread Michael Paquier
On Sat, Oct 3, 2015 at 10:47 PM, Michael Paquier wrote: > On Sat, Oct 3, 2015 at 9:50 PM, Amir Rohan wrote: Block until recovery is finished, before testing. eliminate races, and avoid the stupid sleep(3) I used. >>> >>> TODO > > Well. I just recalled this item in the list of things you m

Re: [HACKERS] check fails on Fedora 23

2015-10-04 Thread Pavel Stehule
#15 0x00469376 in main (argc=8, argv=0x16a45e0) at main.c:223 >> >> Linux yen 4.2.1-300.fc23.x86_64+debug #1 SMP Mon Sep 21 21:58:30 UTC 2015 >> x86_64 x86_64 x86_64 GNU/Linux >> gcc (GCC) 5.1.1 20150618 (Red Hat 5.1.1-4) >> >> Postgres 9.4.4 is working well >> > > configured with defaults

Re: [HACKERS] check fails on Fedora 23

2015-10-04 Thread Pavel Stehule
2015-10-04 10:50 GMT+02:00 Pavel Stehule : > Hi > > I am testing PostgreSQL (master) on Fedora 23. The query > > ELECT p1.oid, p1.proname, p2.oid, p2.proname > FROM pg_proc AS p1, pg_proc AS p2 > WHERE p1.oid < p2.oid AND > p1.prosrc = p2.prosrc AND > p1.prolang = 12 AND p2.prolang = 12 AN

[HACKERS] check fails on Fedora 23

2015-10-04 Thread Pavel Stehule
Hi I am testing PostgreSQL (master) on Fedora 23. The query ELECT p1.oid, p1.proname, p2.oid, p2.proname FROM pg_proc AS p1, pg_proc AS p2 WHERE p1.oid < p2.oid AND p1.prosrc = p2.prosrc AND p1.prolang = 12 AND p2.prolang = 12 AND (p1.proisagg = false OR p2.proisagg = false) AND (