Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Hannu Krosing
On 01/14/2014 03:44 AM, Dave Chinner wrote: > On Tue, Jan 14, 2014 at 02:26:25AM +0100, Andres Freund wrote: >> On 2014-01-13 17:13:51 -0800, James Bottomley wrote: >>> a file into a user provided buffer, thus obtaining a page cache entry >>> and a copy in their userspace buffer, then insert the pa

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-14 Thread David Rowley
On Tue, Jan 14, 2014 at 2:29 PM, Tom Lane wrote: > Florian Pflug writes: > > I think it'd be worthwile to get this into 9.4, if that's still an > option, > > even if we only support COUNT. > > My thought is > > (1) we can certainly implement inverse transitions for COUNT() and the > integer vari

Re: [HACKERS] Add CREATE support to event triggers

2014-01-14 Thread Pavel Stehule
Hello 2014/1/13 Alvaro Herrera > Alvaro Herrera escribió: > > > In an event trigger, the function > pg_event_trigger_get_creation_commands() > > returns the following JSON blob: > > After playing with this for a while, I realized something that must have > seemed quite obvious to those paying a

Re: [HACKERS] Proposal: variant of regclass

2014-01-14 Thread Tatsuo Ishii
> On Tue, Jan 14, 2014 at 4:28 PM, Yugo Nagata wrote: >> Here is the patch to implement to_regclass, to_regproc, to_regoper, >> and to_regtype. They are new functions similar to regclass, regproc, >> regoper, and regtype except that if requested object is not found, >> returns InvalidOid, rather t

Re: [HACKERS] Capturing EXPLAIN ANALYZE for a query without discarding the normal result set

2014-01-14 Thread Marti Raudsepp
On Tue, Jan 14, 2014 at 5:13 AM, Marti Raudsepp wrote: > You can use the auto_explain contrib module I just remembered that there's also the pg_stat_plans extension, which is closer to what you asked: https://github.com/2ndQuadrant/pg_stat_plans . This one you'll have to build yourself (it's not

Re: [HACKERS] [Lsf-pc] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Hannu Krosing
On 01/13/2014 11:22 PM, James Bottomley wrote: > >> The less exciting, more conservative option would be to add kernel >> interfaces to teach Postgres about things like raid geometries. Then >> Postgres could use directio and decide to do prefetching based on the >> raid geometry, how much availabl

Re: [HACKERS] plpgsql.consistent_into

2014-01-14 Thread Pavel Stehule
>> I am thinking so GUC and plpgsql option can live together. If you like to >> accent a some behave, then you can use a plpgsql option. On second hand, I >> would to use a some functionality, that is safe, but I don't would to dirty >> source code by using repeated options. But I have to check (an

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-14 Thread Heikki Linnakangas
On 01/13/2014 07:07 PM, Alexander Korotkov wrote: I've fixed this bug and many other bug. Now patch passes test suite that I've used earlier. The results are so: Operations time: event | period ---+- index_build | 00:01:47.539

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Claudio Freire
On Tue, Jan 14, 2014 at 5:08 AM, Hannu Krosing wrote: > Again, as said above the linux file system is doing fine. What we > want is a few ways to interact with it to let it do even better when > working with postgresql by telling it some stuff it otherwise would > have to second guess and by somet

Re: [HACKERS] GIN improvements part 1: additional information

2014-01-14 Thread Alexander Korotkov
On Tue, Jan 14, 2014 at 12:34 PM, Heikki Linnakangas < hlinnakan...@vmware.com> wrote: > On 01/13/2014 07:07 PM, Alexander Korotkov wrote: > >> I've fixed this bug and many other bug. Now patch passes test suite that >> I've used earlier. The results are so: >> >> Operations time: >> eve

Re: [HACKERS] UNION ALL on partitioned tables won't use indices.

2014-01-14 Thread Kyotaro HORIGUCHI
This is cont'd from previous CF3. You'll see the overview and the discussion since in the thread begins from there. The issue ramains as of current 9.4dev head. http://www.postgresql.org/message-id/20131024.193953.233464126.horiguchi.kyot...@lab.ntt.co.jp The issue in brief is that UNION ALL on

Re: [HACKERS] Get more from indices.

2014-01-14 Thread Kyotaro HORIGUCHI
Hello, since CF4 is already closed but this patch ramains marked as 'Ready for Committer', please let me re-post the latest version for CF4 to get rid of vanishing :-p > tgl> But aside from hasty typos, > > Oops! I've picked up wrong node. It always denies pathkeys extension. > > | !IsA(member,

Re: [HACKERS] Using indices for UNION.

2014-01-14 Thread Kyotaro HORIGUCHI
This is cont'd from CF3. http://www.postgresql.org/message-id/20131122.165927.27412386.horiguchi.kyot...@lab.ntt.co.jp The issue in brief is that UNION is never flattened differently to UNION ALL so UNION cannot make use of index scan even if usable. This patch flattens UNION likewise currently

Re: [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Heikki Linnakangas
On 01/14/2014 12:26 AM, Mel Gorman wrote: On Mon, Jan 13, 2014 at 03:15:16PM -0500, Robert Haas wrote: The other thing that comes to mind is the kernel's caching behavior. We've talked a lot over the years about the difficulties of getting the kernel to write data out when we want it to and to n

Re: [HACKERS] Using indices for UNION.

2014-01-14 Thread Kyotaro HORIGUCHI
Sorry, I missed to attach file. > This is cont'd from CF3. > > http://www.postgresql.org/message-id/20131122.165927.27412386.horiguchi.kyot...@lab.ntt.co.jp > > The issue in brief is that UNION is never flattened differently > to UNION ALL so UNION cannot make use of index scan even if > usable.

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-14 Thread David Rowley
On Tue, Jan 14, 2014 at 9:09 PM, David Rowley wrote: > I think unless anyone has some objections I'm going to remove the inverse > transition for SUM(numeric) and modify the documents to tell the user how > to build their own FAST_SUM(numeric) using the built in functions to do it. > I'm starting

Re: [HACKERS] plpgsql.consistent_into

2014-01-14 Thread Pavel Stehule
2014/1/14 Florian Pflug > On Jan14, 2014, at 00:52 , Marko Tiikkaja wrote: > > When I've worked with PL/PgSQL, this has been a source of a few bugs that > > would have been noticed during testing if the behaviour of INTO wasn't as > > dangerous as it is right now. > > The question is, how many b

Re: [HACKERS] inherit support for foreign tables

2014-01-14 Thread Shigeru Hanada
2013/11/18 Tom Lane : > Robert Haas writes: >> I think it's been previously proposed that we have some version of a >> CHECK constraint that effectively acts as an assertion for query >> optimization purposes, but isn't actually enforced by the system. I >> can see that being useful in a variety

[HACKERS] Trigger information for auto_explain.

2014-01-14 Thread Kyotaro HORIGUCHI
Hello, Now explain can show trigger statistics (from when?). =# create table t (a int, b int); =# create or replace function hoge() returns trigger as 'begin new.b = new.a; return new; end;' language plpgsql; =# create trigger ins_hoge before insert or update on t for each row execute procedure

Re: [HACKERS] Case sensitive mode in windows build option

2014-01-14 Thread Dilip kumar
On 01/14/2014 11:25 AM Craig Ringer Wrote, > > As per current behavior if user want to build in debug mode in > > windows, then he need to give debug in capital letters (DEBUG), > > > > I think many user will always make mistake in giving this option, in > > my opinion we can make it case insensit

Re: [HACKERS] Case sensitive mode in windows build option

2014-01-14 Thread Craig Ringer
On 01/14/2014 05:35 PM, Dilip kumar wrote: > On 01/14/2014 11:25 AM Craig Ringer Wrote, > >>> As per current behavior if user want to build in debug mode in >>> windows, then he need to give debug in capital letters (DEBUG), >>> >>> I think many user will always make mistake in giving this option,

Re: [HACKERS] nested hstore patch

2014-01-14 Thread Oleg Bartunov
Erik, thanks for docs fixes, we have even more :) Oleg On Tue, Jan 14, 2014 at 4:18 AM, Erik Rijkers wrote: > On Mon, January 13, 2014 18:30, Andrew Dunstan wrote: >> >> >> On 01/13/2014 11:16 AM, Oleg Bartunov wrote: >>> Andrew, >>> >>> did you run perl script ? Actually, I found, that operato

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-14 Thread David Rowley
On Tue, Jan 14, 2014 at 9:09 PM, David Rowley wrote: > I think unless anyone has some objections I'm going to remove the inverse > transition for SUM(numeric) and modify the documents to tell the user how > to build their own FAST_SUM(numeric) using the built in functions to do it. > I'm starting

[HACKERS] Inheritance and indexes

2014-01-14 Thread knizhnik
From PostgreSQL manual: "A serious limitation of the inheritance feature is that indexes (including unique constraints) and foreign key constraints only apply to single tables, not to their inheritance children." But is it possible to use index for derived table at all? Why sequential search i

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-14 Thread Peter Geoghegan
On Mon, Jan 13, 2014 at 6:45 PM, Peter Geoghegan wrote: > + uint32 > + SpeculativeInsertionIsInProgress(TransactionId xid, RelFileNode rel, > ItemPointer tid) > + { For the purposes of preventing unprincipled deadlocking, commenting out the following (the only caller of the above) has no immediat

Re: [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Mel Gorman
On Mon, Jan 13, 2014 at 03:24:38PM -0800, Josh Berkus wrote: > On 01/13/2014 02:26 PM, Mel Gorman wrote: > > Really? > > > > zone_reclaim_mode is often a complete disaster unless the workload is > > partitioned to fit within NUMA nodes. On older kernels enabling it would > > sometimes cause massiv

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-14 Thread Heikki Linnakangas
On 01/14/2014 12:20 PM, Peter Geoghegan wrote: I think that the prevention of unprincipled deadlocking is all down to this immediately prior piece of code, at least in those test cases: ! /* !* in insertion by other. !* !

Re: [HACKERS] plpgsql.consistent_into

2014-01-14 Thread Marko Tiikkaja
On 1/14/14 10:16 AM, Pavel Stehule wrote: 2014/1/14 Florian Pflug So if we really want to change this, I think we need to have a LANGUAGE_VERSION attribute on functions. Each time a major postgres release changes the behaviour of one of the procedural languages, we'd increment that language's

Re: [HACKERS] INSERT...ON DUPLICATE KEY LOCK FOR UPDATE

2014-01-14 Thread Heikki Linnakangas
On 01/14/2014 12:44 AM, Peter Geoghegan wrote: On Mon, Jan 13, 2014 at 12:58 PM, Heikki Linnakangas wrote: Well, even if you don't agree that locking all the conflicting rows for update is sensible, it's still perfectly sensible to return the rejected rows to the user. For example, you're inser

Re: [HACKERS] PostgreSQL Service on Windows does not start if data directory given is relative path

2014-01-14 Thread Rajeev rastogi
On Tue, Jan 12, 2014 David Rowley wrote: >>I have found a case that PostgreSQL as win32 service does not start, if the >>data directory given as relative path. >>Error observed in this case is: >>"The PostgreSQL on Local Computer started and >> then stopped". >>Th

Re: [HACKERS] plpgsql.consistent_into

2014-01-14 Thread Marti Raudsepp
I've always hated INTO in procedures since it makes the code harder to follow and has very different behavior on the SQL level, in addition to the multi-row problem you bring up. If we can make assignment syntax more versatile and eventually replace INTO, then that solves multiple problems in the l

Re: [HACKERS] Inheritance and indexes

2014-01-14 Thread Marti Raudsepp
On Tue, Jan 14, 2014 at 12:07 PM, knizhnik wrote: > But is it possible to use index for derived table at all? Yes, the planner will do an index scan when it makes sense. > Why sequential search is used for derived table in the example below: > insert into derived_table values (2,2); > create in

Re: [HACKERS] plpgsql.consistent_into

2014-01-14 Thread Marko Tiikkaja
On 1/14/14 12:28 PM, Marti Raudsepp wrote: I've always hated INTO in procedures since it makes the code harder to follow and has very different behavior on the SQL level, in addition to the multi-row problem you bring up. If we can make assignment syntax more versatile and eventually replace INTO

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2014-01-14 Thread KONDO Mitsumasa
Hi, I fix and submit this patch in CF4. In my past patch, it is significant bug which is mistaken caluculation of offset in posix_fadvise():-( However it works well without problem in pgbench. Because pgbench transactions are always random access... And I test my patch in DBT-2 benchmark. Resul

Re: [HACKERS] plpgsql.consistent_into

2014-01-14 Thread Pavel Stehule
2014/1/14 Marko Tiikkaja > On 1/14/14 12:28 PM, Marti Raudsepp wrote: > >> I've always hated INTO in procedures since it makes the code harder to >> follow and has very different behavior on the SQL level, in addition >> to the multi-row problem you bring up. If we can make assignment >> syntax m

[HACKERS] Extending BASE_BACKUP in replication protocol: incremental backup and backup format

2014-01-14 Thread Michael Paquier
Hi all, As of today, replication protocol has a command called BASE_BACKUP to allow a client connecting with the replication protocol to retrieve a full backup from server through a connection stream. The description of its current options are here: http://www.postgresql.org/docs/9.3/static/protoc

Re: [HACKERS] Soften pg_[start|stop]_backup to allow them on a standby?

2014-01-14 Thread Andres Freund
Hi, On 2014-01-14 12:31:09 +0900, Michael Paquier wrote: > Currently, pg_start_backup and pg_stop_backup cannot run on a standby > because it is not possible to write a backup_label file to disk, > because of the nature of a standby server preventing to write any data > in its PGDATA. Is this thou

Re: [HACKERS] Extending BASE_BACKUP in replication protocol: incremental backup and backup format

2014-01-14 Thread Andres Freund
Hi, On 2014-01-14 21:47:43 +0900, Michael Paquier wrote: > I would like to propose the following things to extend BASE_BACKUP to > retrieve a backup from a stream: > - Addition of an option FORMAT, to control the output format of > backup, with possible options as 'plain' and 'tar'. Default is tar

Re: [HACKERS] Extending BASE_BACKUP in replication protocol: incremental backup and backup format

2014-01-14 Thread Heikki Linnakangas
On 01/14/2014 02:47 PM, Michael Paquier wrote: I would like to propose the following things to extend BASE_BACKUP to retrieve a backup from a stream: - Addition of an option FORMAT, to control the output format of backup, with possible options as 'plain' and 'tar'. Default is tar for backward com

Re: [HACKERS] Extending BASE_BACKUP in replication protocol: incremental backup and backup format

2014-01-14 Thread Magnus Hagander
On Tue, Jan 14, 2014 at 1:47 PM, Michael Paquier wrote: > Hi all, > > As of today, replication protocol has a command called BASE_BACKUP to > allow a client connecting with the replication protocol to retrieve a > full backup from server through a connection stream. The description > of its curren

Re: [HACKERS] Extending BASE_BACKUP in replication protocol: incremental backup and backup format

2014-01-14 Thread Michael Paquier
On Tue, Jan 14, 2014 at 10:01 PM, Heikki Linnakangas wrote: >> - Addition of an option called INCREMENTAL to send an incremental >> backup to the client. This option uses as input an LSN, and sends back >> to client relation pages (in the shape of reduced relation files) that >> are newer than the

Re: [HACKERS] Extending BASE_BACKUP in replication protocol: incremental backup and backup format

2014-01-14 Thread Andres Freund
On 2014-01-14 14:12:46 +0100, Magnus Hagander wrote: > Either way - if we can do this in a safe way, it sounds like a good idea. > It would be sort of like rsync, except relying on the fact that we can look > at the LSN and don't have to compare the actual files, right? Which is an advantage, yes.

Re: [HACKERS] Extending BASE_BACKUP in replication protocol: incremental backup and backup format

2014-01-14 Thread Michael Paquier
On Tue, Jan 14, 2014 at 10:12 PM, Magnus Hagander wrote: > On Tue, Jan 14, 2014 at 1:47 PM, Michael Paquier > wrote: >> >> Hi all, >> >> As of today, replication protocol has a command called BASE_BACKUP to >> allow a client connecting with the replication protocol to retrieve a >> full backup fr

Re: [HACKERS] Extending BASE_BACKUP in replication protocol: incremental backup and backup format

2014-01-14 Thread Magnus Hagander
On Tue, Jan 14, 2014 at 2:18 PM, Andres Freund wrote: > On 2014-01-14 14:12:46 +0100, Magnus Hagander wrote: > > Either way - if we can do this in a safe way, it sounds like a good idea. > > It would be sort of like rsync, except relying on the fact that we can > look > > at the LSN and don't have

Re: [HACKERS] Extending BASE_BACKUP in replication protocol: incremental backup and backup format

2014-01-14 Thread Andres Freund
On 2014-01-14 14:40:46 +0100, Magnus Hagander wrote: > On Tue, Jan 14, 2014 at 2:18 PM, Andres Freund wrote: > > > On 2014-01-14 14:12:46 +0100, Magnus Hagander wrote: > > > Either way - if we can do this in a safe way, it sounds like a good idea. > > > It would be sort of like rsync, except relyi

Re: [HACKERS] Extending BASE_BACKUP in replication protocol: incremental backup and backup format

2014-01-14 Thread Magnus Hagander
On Tue, Jan 14, 2014 at 2:16 PM, Michael Paquier wrote: > On Tue, Jan 14, 2014 at 10:01 PM, Heikki Linnakangas > wrote: > >> - Addition of an option called INCREMENTAL to send an incremental > >> backup to the client. This option uses as input an LSN, and sends back > >> to client relation pages

Re: [HACKERS] Extending BASE_BACKUP in replication protocol: incremental backup and backup format

2014-01-14 Thread Magnus Hagander
On Tue, Jan 14, 2014 at 2:41 PM, Andres Freund wrote: > On 2014-01-14 14:40:46 +0100, Magnus Hagander wrote: > > On Tue, Jan 14, 2014 at 2:18 PM, Andres Freund >wrote: > > > > > On 2014-01-14 14:12:46 +0100, Magnus Hagander wrote: > > > > Either way - if we can do this in a safe way, it sounds li

Re: [HACKERS] Extending BASE_BACKUP in replication protocol: incremental backup and backup format

2014-01-14 Thread Andres Freund
On 2014-01-14 14:42:36 +0100, Magnus Hagander wrote: > On Tue, Jan 14, 2014 at 2:41 PM, Andres Freund wrote: > > > On 2014-01-14 14:40:46 +0100, Magnus Hagander wrote: > > > On Tue, Jan 14, 2014 at 2:18 PM, Andres Freund > >wrote: > > > > > > > On 2014-01-14 14:12:46 +0100, Magnus Hagander wrote:

Re: [HACKERS] [PATCH] Filter error log statements by sqlstate

2014-01-14 Thread Oskari Saarenmaa
On Tue, Jan 14, 2014 at 12:22:30PM +0530, Jeevan Chalke wrote: > On Mon, Jan 13, 2014 at 4:30 PM, Oskari Saarenmaa wrote: > > On 13/01/14 10:26, Jeevan Chalke wrote: > > > > > 1. Documentation is missing and thus becomes difficult to understand > > > what exactly you are trying to do. Or in other

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2014-01-14 Thread Claudio Freire
On Tue, Jan 14, 2014 at 8:58 AM, KONDO Mitsumasa wrote: > > In my past patch, it is significant bug which is mistaken caluculation of > offset in posix_fadvise():-( However it works well without problem in > pgbench. > Because pgbench transactions are always random access... Did you notice any d

Re: [HACKERS] Optimize kernel readahead using buffer access strategy

2014-01-14 Thread Claudio Freire
On Tue, Jan 14, 2014 at 8:58 AM, KONDO Mitsumasa wrote: > > In my past patch, it is significant bug which is mistaken caluculation of > offset in posix_fadvise():-( However it works well without problem in > pgbench. > Because pgbench transactions are always random access... Did you notice any d

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Hannu Krosing
On 01/14/2014 09:39 AM, Claudio Freire wrote: > On Tue, Jan 14, 2014 at 5:08 AM, Hannu Krosing wrote: >> Again, as said above the linux file system is doing fine. What we >> want is a few ways to interact with it to let it do even better when >> working with postgresql by telling it some stuff it

Re: [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Robert Haas
On Mon, Jan 13, 2014 at 5:26 PM, Mel Gorman wrote: >> Amen to that. Actually, I think NUMA can be (mostly?) fixed by >> setting zone_reclaim_mode; is there some other problem besides that? > > Really? > > zone_reclaim_mode is often a complete disaster unless the workload is > partitioned to fit w

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Kevin Grittner
First off, I want to give a +1 on everything in the recent posts from Heikki and Hannu. Jan Kara wrote: > Now the aging of pages marked as volatile as it is currently > implemented needn't be perfect for your needs but you still have > time to influence what gets implemented... Actually develope

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Claudio Freire
On Tue, Jan 14, 2014 at 11:39 AM, Hannu Krosing wrote: > On 01/14/2014 09:39 AM, Claudio Freire wrote: >> On Tue, Jan 14, 2014 at 5:08 AM, Hannu Krosing wrote: >>> Again, as said above the linux file system is doing fine. What we >>> want is a few ways to interact with it to let it do even better

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Robert Haas
On Tue, Jan 14, 2014 at 3:39 AM, Claudio Freire wrote: > On Tue, Jan 14, 2014 at 5:08 AM, Hannu Krosing wrote: >> Again, as said above the linux file system is doing fine. What we >> want is a few ways to interact with it to let it do even better when >> working with postgresql by telling it some

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Robert Haas
On Tue, Jan 14, 2014 at 5:00 AM, Jan Kara wrote: > I thought that instead of injecting pages into pagecache for aging as you > describe in 3), you would mark pages as volatile (i.e. for reclaim by > kernel) through vrange() syscall. Next time you need the page, you check > whether the kernel recla

Re: [HACKERS] plpgsql.consistent_into

2014-01-14 Thread Marko Tiikkaja
On 1/14/14 1:28 PM, Pavel Stehule wrote: I prefer subquery only syntax - a := (some) or (a,b,c) = (some a,b,c) with possible enhancing for statements with RETURNING a advance is compatibility with DB2 (SQL/PSM) syntax - and this code is written now - it is done in my sql/psm implementation Are

Re: [HACKERS] extension_control_path

2014-01-14 Thread Tom Lane
Dimitri Fontaine writes: > Please find attached to this email a patch implementing a new GUC that > allows users to setup a list of path where PostgreSQL will search for > the extension control files at CREATE EXTENSION time. Why is that a good idea? It's certainly not going to simplify DBAs' li

Re: [HACKERS] GIN improvements part2: fast scan

2014-01-14 Thread Alexander Korotkov
On Thu, Nov 21, 2013 at 12:14 AM, Alexander Korotkov wrote: > On Wed, Nov 20, 2013 at 3:06 AM, Alexander Korotkov > wrote: > >> On Fri, Nov 15, 2013 at 11:19 AM, Alexander Korotkov < >> aekorot...@gmail.com> wrote: >> >>> On Fri, Nov 15, 2013 at 12:34 AM, Heikki Linnakangas < >>> hlinnakan...@vmw

Re: [HACKERS] plpgsql.consistent_into

2014-01-14 Thread Pavel Stehule
Hello 2014/1/14 Marko Tiikkaja > On 1/14/14 1:28 PM, Pavel Stehule wrote: > >> I prefer subquery only syntax - a := (some) or (a,b,c) = (some a,b,c) with >> possible enhancing for statements with RETURNING >> >> a advance is compatibility with DB2 (SQL/PSM) syntax - and this code is >> written

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Tom Lane
James Bottomley writes: > The current mechanism for coherency between a userspace cache and the > in-kernel page cache is mmap ... that's the only way you get the same > page in both currently. Right. > glibc used to have an implementation of read/write in terms of mmap, so > it should be possib

Re: [HACKERS] PoC: Partial sort

2014-01-14 Thread Alexander Korotkov
Hi! On Tue, Jan 14, 2014 at 12:54 AM, Marti Raudsepp wrote: > First, thanks a lot for working on this feature. This PostgreSQL > shortcoming crops up in all the time in web applications that implement > paging by multiple sorted columns. > Thanks! I've been trying it out in a few situations. I

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Claudio Freire
On Tue, Jan 14, 2014 at 12:42 PM, Trond Myklebust wrote: >> James Bottomley writes: >>> The current mechanism for coherency between a userspace cache and the >>> in-kernel page cache is mmap ... that's the only way you get the same >>> page in both currently. >> >> Right. >> >>> glibc used to hav

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Tom Lane
Trond Myklebust writes: > On Jan 14, 2014, at 10:39, Tom Lane wrote: >> "Don't be aggressive" isn't good enough. The prohibition on early write >> has to be absolute, because writing a dirty page before we've done >> whatever else we need to do results in a corrupt database. It has to >> be tre

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Jan Kara
On Tue 14-01-14 09:08:40, Hannu Krosing wrote: > >>> Effectively you end up with buffered read/write that's also mapped into > >>> the page cache. It's a pretty awful way to hack around mmap. > >> Well, the problem is that you can't really use mmap() for the things we > >> do. Postgres' durability

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Jan Kara
On Tue 14-01-14 11:11:28, Heikki Linnakangas wrote: > On 01/14/2014 12:26 AM, Mel Gorman wrote: > >On Mon, Jan 13, 2014 at 03:15:16PM -0500, Robert Haas wrote: > >>The other thing that comes to mind is the kernel's caching behavior. > >>We've talked a lot over the years about the difficulties of ge

Re: [HACKERS] [Lsf-pc] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Dave Chinner
On Mon, Jan 13, 2014 at 09:29:02PM +, Greg Stark wrote: > On Mon, Jan 13, 2014 at 9:12 PM, Andres Freund wrote: > > For one, postgres doesn't use mmap for files (and can't without major > > new interfaces). Frequently mmap()/madvise()/munmap()ing 8kb chunks has > > horrible consequences for pe

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Dave Chinner
On Mon, Jan 13, 2014 at 03:24:38PM -0800, Josh Berkus wrote: > On 01/13/2014 02:26 PM, Mel Gorman wrote: > > Really? > > > > zone_reclaim_mode is often a complete disaster unless the workload is > > partitioned to fit within NUMA nodes. On older kernels enabling it would > > sometimes cause massiv

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Dave Chinner
On Tue, Jan 14, 2014 at 02:26:25AM +0100, Andres Freund wrote: > On 2014-01-13 17:13:51 -0800, James Bottomley wrote: > > a file into a user provided buffer, thus obtaining a page cache entry > > and a copy in their userspace buffer, then insert the page of the user > > buffer back into the page ca

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread James Bottomley
On Mon, 2014-01-13 at 19:48 -0500, Trond Myklebust wrote: > On Jan 13, 2014, at 19:03, Hannu Krosing wrote: > > > On 01/13/2014 09:53 PM, Trond Myklebust wrote: > >> On Jan 13, 2014, at 15:40, Andres Freund wrote: > >> > >>> On 2014-01-13 15:15:16 -0500, Robert Haas wrote: > On Mon, Jan 13

Re: [HACKERS] Extending BASE_BACKUP in replication protocol: incremental backup and backup format

2014-01-14 Thread Magnus Hagander
On Jan 14, 2014 2:44 PM, "Andres Freund" wrote: > > On 2014-01-14 14:42:36 +0100, Magnus Hagander wrote: > > On Tue, Jan 14, 2014 at 2:41 PM, Andres Freund wrote: > > > > > On 2014-01-14 14:40:46 +0100, Magnus Hagander wrote: > > > > On Tue, Jan 14, 2014 at 2:18 PM, Andres Freund < and...@2ndquadr

Re: [HACKERS] extension_control_path

2014-01-14 Thread Dimitri Fontaine
Tom Lane writes: > Why is that a good idea? It's certainly not going to simplify DBAs' > lives, more the reverse. ("This dump won't reload." "Uh, where did > you get that extension from?" "Ummm...") The latest users for the feature are the Red Hat team working on Open Shift where they want to h

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread James Bottomley
On Tue, 2014-01-14 at 15:39 +0100, Hannu Krosing wrote: > On 01/14/2014 09:39 AM, Claudio Freire wrote: > > On Tue, Jan 14, 2014 at 5:08 AM, Hannu Krosing > > wrote: > >> Again, as said above the linux file system is doing fine. What we > >> want is a few ways to interact with it to let it do eve

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Trond Myklebust
On Jan 14, 2014, at 10:39, Tom Lane wrote: > James Bottomley writes: >> The current mechanism for coherency between a userspace cache and the >> in-kernel page cache is mmap ... that's the only way you get the same >> page in both currently. > > Right. > >> glibc used to have an implementatio

Re: [HACKERS] extension_control_path

2014-01-14 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: > Dimitri Fontaine writes: > > Please find attached to this email a patch implementing a new GUC that > > allows users to setup a list of path where PostgreSQL will search for > > the extension control files at CREATE EXTENSION time. > > Why is that a good i

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-01-14 Thread Florian Pflug
On Jan14, 2014, at 11:06 , David Rowley wrote: > Here's a patch which removes sum(numeric) and changes the documents a little > to remove a reference to using sum(numeric) to workaround the fact that > there's no inverse transitions for sum(float). I also made a small change in > the aggregates

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Robert Haas
On Tue, Jan 14, 2014 at 11:44 AM, James Bottomley wrote: > No, I'm sorry, that's never going to be possible. No user space > application has all the facts. If we give you an interface to force > unconditional holding of dirty pages in core you'll livelock the system > eventually because you made

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Claudio Freire
On Tue, Jan 14, 2014 at 1:48 PM, Robert Haas wrote: > On Tue, Jan 14, 2014 at 11:44 AM, James Bottomley > wrote: >> No, I'm sorry, that's never going to be possible. No user space >> application has all the facts. If we give you an interface to force >> unconditional holding of dirty pages in c

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Heikki Linnakangas
On 01/14/2014 06:08 PM, Tom Lane wrote: Trond Myklebust writes: On Jan 14, 2014, at 10:39, Tom Lane wrote: "Don't be aggressive" isn't good enough. The prohibition on early write has to be absolute, because writing a dirty page before we've done whatever else we need to do results in a corru

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Robert Haas
On Tue, Jan 14, 2014 at 11:57 AM, James Bottomley wrote: > On Tue, 2014-01-14 at 11:48 -0500, Robert Haas wrote: >> On Tue, Jan 14, 2014 at 11:44 AM, James Bottomley >> wrote: >> > No, I'm sorry, that's never going to be possible. No user space >> > application has all the facts. If we give you

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Robert Haas
On Tue, Jan 14, 2014 at 12:12 PM, Robert Haas wrote: > In terms of avoiding double-buffering, here's my thought after reading > what's been written so far. Suppose we read a page into our buffer > pool. Until the page is clean, it would be ideal for the mapping to Correction: "For so long as th

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Claudio Freire
On Tue, Jan 14, 2014 at 2:12 PM, Robert Haas wrote: > > In terms of avoiding double-buffering, here's my thought after reading > what's been written so far. Suppose we read a page into our buffer > pool. Until the page is clean, it would be ideal for the mapping to > be shared between the buffer

Re: [HACKERS] plpgsql.consistent_into

2014-01-14 Thread Tom Lane
Marko Tiikkaja writes: > On 1/14/14 12:28 PM, Marti Raudsepp wrote: >> Now, another question is whether it's possible to make the syntax >> work. Is this an assignment from the result of a subquery, or is it a >> query by itself? >> a = (SELECT foo FROM table); > That looks like a scalar subquery

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Robert Haas
On Tue, Jan 14, 2014 at 12:15 PM, Claudio Freire wrote: > On Tue, Jan 14, 2014 at 2:12 PM, Robert Haas wrote: >> In terms of avoiding double-buffering, here's my thought after reading >> what's been written so far. Suppose we read a page into our buffer >> pool. Until the page is clean, it woul

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Kevin Grittner
Claudio Freire wrote: > Robert Haas wrote: >> James Bottomley wrote: >>> I don't understand why this has to be absolute: if you advise >>> us to hold the pages dirty and we do up until it becomes a >>> choice to hold on to the pages or to thrash the system into a >>> livelock, why would you eve

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Hannu Krosing
On 01/14/2014 05:44 PM, James Bottomley wrote: > On Tue, 2014-01-14 at 10:39 -0500, Tom Lane wrote: >> James Bottomley writes: >>> The current mechanism for coherency between a userspace cache and the >>> in-kernel page cache is mmap ... that's the only way you get the same >>> page in both curren

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Kevin Grittner
James Bottomley wrote: > you mean the order of write out, if we have to do it, is > important.  In the rest of the kernel, we do this with barriers > which causes ordered grouping of I/O chunks.  If we could force a > similar ordering in the writeout code, is that enough? Unless it can be betwee

[HACKERS] Exposing currentTransactionWALVolume

2014-01-14 Thread Simon Riggs
Short patch to expose a function GetCurrentTransactionWALVolume() that gives the total number of bytes written to WAL by current transaction. User interface to this information discussed on separate thread, so that we don't check the baby out with the bathwater when people discuss UI pros and cons

Re: [HACKERS] shared memory message queues

2014-01-14 Thread Robert Haas
On Mon, Dec 23, 2013 at 12:46 PM, Robert Haas wrote: > Oh, dear. That's rather embarrassing. > > Incremental (incremental-shm-mq.patch) and full (shm-mq-v3.patch) > patches attached. OK, I have pushed the patches in this stack. I'm not sure we quite concluded the review back-and-forth but nobod

Re: [HACKERS] extension_control_path

2014-01-14 Thread Tom Lane
Dimitri Fontaine writes: > Tom Lane writes: >> Why is that a good idea? It's certainly not going to simplify DBAs' >> lives, more the reverse. ("This dump won't reload." "Uh, where did >> you get that extension from?" "Ummm...") > The latest users for the feature are the Red Hat team working o

Re: [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Jeff Janes
On Mon, Jan 13, 2014 at 2:36 PM, Mel Gorman wrote: > On Mon, Jan 13, 2014 at 06:27:03PM -0200, Claudio Freire wrote: > > On Mon, Jan 13, 2014 at 5:23 PM, Jim Nasby wrote: > > > On 1/13/14, 2:19 PM, Claudio Freire wrote: > > >> > > >> On Mon, Jan 13, 2014 at 5:15 PM, Robert Haas > > >> wrote: >

Re: [HACKERS] ALTER TABLE lock strength reduction patch is unsafe

2014-01-14 Thread Simon Riggs
On 7 July 2013 14:24, Simon Riggs wrote: > On 3 January 2012 18:42, Tom Lane wrote: >> I wrote: Another point that requires some thought is that switching SnapshotNow to be MVCC-based will presumably result in a noticeable increase in each backend's rate of wanting to acquire snaps

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Robert Haas
On Tue, Jan 14, 2014 at 12:20 PM, James Bottomley wrote: > On Tue, 2014-01-14 at 15:15 -0200, Claudio Freire wrote: >> On Tue, Jan 14, 2014 at 2:12 PM, Robert Haas wrote: >> > In terms of avoiding double-buffering, here's my thought after reading >> > what's been written so far. Suppose we read

Re: [HACKERS] shared memory message queues

2014-01-14 Thread Thom Brown
On 14 January 2014 17:29, Robert Haas wrote: > On Mon, Dec 23, 2013 at 12:46 PM, Robert Haas wrote: >> Oh, dear. That's rather embarrassing. >> >> Incremental (incremental-shm-mq.patch) and full (shm-mq-v3.patch) >> patches attached. > > OK, I have pushed the patches in this stack. I'm not sure

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread Tom Lane
Robert Haas writes: > On Tue, Jan 14, 2014 at 11:57 AM, James Bottomley > wrote: >> No, I do ... you mean the order of write out, if we have to do it, is >> important. In the rest of the kernel, we do this with barriers which >> causes ordered grouping of I/O chunks. If we could force a similar

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread James Bottomley
On Tue, 2014-01-14 at 11:48 -0500, Robert Haas wrote: > On Tue, Jan 14, 2014 at 11:44 AM, James Bottomley > wrote: > > No, I'm sorry, that's never going to be possible. No user space > > application has all the facts. If we give you an interface to force > > unconditional holding of dirty pages

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread James Bottomley
On Tue, 2014-01-14 at 15:15 -0200, Claudio Freire wrote: > On Tue, Jan 14, 2014 at 2:12 PM, Robert Haas wrote: > > > > In terms of avoiding double-buffering, here's my thought after reading > > what's been written so far. Suppose we read a page into our buffer > > pool. Until the page is clean,

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-14 Thread James Bottomley
On Tue, 2014-01-14 at 10:39 -0500, Tom Lane wrote: > James Bottomley writes: > > The current mechanism for coherency between a userspace cache and the > > in-kernel page cache is mmap ... that's the only way you get the same > > page in both currently. > > Right. > > > glibc used to have an impl

Re: [HACKERS] plpgsql.consistent_into

2014-01-14 Thread Marko Tiikkaja
On 1/14/14, 6:15 PM, Tom Lane wrote: Marko Tiikkaja writes: How about: (a) = SELECT 1; (a, b) = SELECT 1, 2; (a, b) = INSERT INTO foo RETURNING col1, col2; Same semantics: TOO_MANY_ROWS on rows > 1, sets FOUND and row_count. AFAICT this can be parsed unambiguously, too, and w

  1   2   >