Re: [HACKERS] Patch to add a primary key using an existing index

2010-11-28 Thread David Fetter
On Sun, Nov 28, 2010 at 08:40:08PM -0500, Robert Haas wrote: > On Sun, Nov 28, 2010 at 8:06 PM, Itagaki Takahiro > wrote: > > On Fri, Nov 26, 2010 at 05:58, Steve Singer wrote: > >> The attached version of the patch gets your regression tests to > >> pass. I'm going to mark this as ready for a c

Re: [HACKERS] pg_execute_from_file review

2010-11-28 Thread Itagaki Takahiro
On Fri, Nov 26, 2010 at 06:24, Dimitri Fontaine wrote: > Thanks for your review. Please find attached a revised patch where I've > changed the internals of the function so that it's split in two and that > the opr_sanity check passes, per comments from David Wheeler and Tom Lane. I have some comm

[HACKERS] On-the-fly index tuple deletion vs. hot_standby

2010-11-28 Thread Noah Misch
I have a hot_standby system and use it to bear the load of various reporting queries that take 15-60 minutes each. In an effort to avoid long pauses in recovery, I set a vacuum_defer_cleanup_age constituting roughly three hours of the master's transactions. Even so, I kept seeing recovery pause f

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Tom Lane
Robert Haas writes: > Yeah, very true. What's a bit frustrating about the whole thing is > that we spend a lot of time pulling data into the caches that's > basically static and never likely to change anywhere, ever. True. I wonder if we could do something like the relcache init file for the ca

Re: [HACKERS] [GENERAL] column-level update privs + lock table

2010-11-28 Thread KaiGai Kohei
(2010/11/29 10:43), Robert Haas wrote: > 2010/11/28 KaiGai Kohei: >>> I'm not totally convinced that this is the correct behavior. It seems >>> a bit surprising that UPDATE privilege on a single column is enough to >>> lock out all SELECT activity from the table. It's actually a bit >>> surprisin

Re: [HACKERS] contrib: auth_delay module

2010-11-28 Thread Robert Haas
On Sun, Nov 28, 2010 at 7:45 PM, Fujii Masao wrote: > Thanks. I found the typo: I only have one? :-) Thanks, fixed. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make change

Re: [HACKERS] Feature request: INSERT .... ON DUPLICATE UPDATE ....

2010-11-28 Thread Robert Haas
On Sun, Nov 28, 2010 at 7:44 PM, Yourfriend wrote: > In our company,  we use both PostgreSQL and MySQL, our developers include me > think that the "INSERT ... ON DUPLICATE UPDATE " clause is a much more user > friendly function,so, would you please add this liked function in > PostgreSQL, I know w

Re: [HACKERS] [GENERAL] column-level update privs + lock table

2010-11-28 Thread Robert Haas
2010/11/28 KaiGai Kohei : >> I'm not totally convinced that this is the correct behavior.  It seems >> a bit surprising that UPDATE privilege on a single column is enough to >> lock out all SELECT activity from the table.  It's actually a bit >> surprising that even full-table UPDATE privileges are

Re: [HACKERS] Patch to add a primary key using an existing index

2010-11-28 Thread Robert Haas
On Sun, Nov 28, 2010 at 8:06 PM, Itagaki Takahiro wrote: > On Fri, Nov 26, 2010 at 05:58, Steve Singer wrote: >> The attached version of the patch gets your regression tests to pass. >> I'm going to mark this as ready for a committer. > > I think we need more discussions about the syntax: >  ALTE

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Robert Haas
On Sun, Nov 28, 2010 at 7:15 PM, Tom Lane wrote: > Robert Haas writes: >> One possible way to get a real speedup here would be to look for ways >> to trim the number of catcaches. > > BTW, it's not going to help to remove catcaches that have a small > initial size, as the pg_am cache certainly do

Re: [HACKERS] Assertion failure on hot standby

2010-11-28 Thread Simon Riggs
On Fri, 2010-11-26 at 01:11 -0500, Tom Lane wrote: > Simon Riggs writes: > > That would mean running GetCurrentTransactionId() inside LockAcquire() > > > if (lockmode >= AccessExclusiveLock && > > locktag->locktag_type == LOCKTAG_RELATION && > > !RecoveryInProgress()) > > (void) GetCu

Re: [HACKERS] Patch to add a primary key using an existing index

2010-11-28 Thread Itagaki Takahiro
On Fri, Nov 26, 2010 at 05:58, Steve Singer wrote: > The attached version of the patch gets your regression tests to pass. > I'm going to mark this as ready for a committer. I think we need more discussions about the syntax: ALTER TABLE table_name ADD PRIMARY KEY (...) WITH (INDEX='index_name')

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Tom Lane
Greg Stark writes: > On Mon, Nov 29, 2010 at 12:33 AM, Tom Lane wrote: >> Another question that would be worth asking here is whether the >> hand-baked MemSet macro still outruns memset on modern architectures. >> I think it's been quite a few years since that was last tested. > I know glibc has

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Greg Stark
On Mon, Nov 29, 2010 at 12:33 AM, Tom Lane wrote: > The most portable way to do that would be to use calloc insted of malloc, > and hope that libc is smart enough to provide freshly-mapped space. > It would be good to look and see whether glibc actually does so, > of course.  If not we might end u

Re: [HACKERS] Report: Linux huge pages with Postgres

2010-11-28 Thread Tom Lane
Greg Stark writes: > On Mon, Nov 29, 2010 at 12:12 AM, Tom Lane wrote: >> Really you do want to scrape the value. > Couldn't we just round the shared memory allocation down to a multiple > of 4MB? That would handle all older architectures where the size is > 2MB or 4MB. Rounding *down* will not

Re: [HACKERS] contrib: auth_delay module

2010-11-28 Thread Fujii Masao
On Sat, Nov 27, 2010 at 9:25 PM, Robert Haas wrote: > On Thu, Nov 25, 2010 at 1:18 AM, KaiGai Kohei wrote: >> The attached patch is revised version. >> >> - Logging part within auth_delay was removed. This module now focuses on >>  injection of a few seconds delay on authentication failed. >> - D

[HACKERS] Feature request: INSERT .... ON DUPLICATE UPDATE ....

2010-11-28 Thread Yourfriend
Dear all, In our company, we use both PostgreSQL and MySQL, our developers include me think that the "INSERT ... ON DUPLICATE UPDATE " clause is a much more user friendly function,so, would you please add this liked function in PostgreSQL, I know we can write PROCEDURE or RULE to solve this probl

Re: [HACKERS] Report: Linux huge pages with Postgres

2010-11-28 Thread Greg Stark
On Mon, Nov 29, 2010 at 12:12 AM, Tom Lane wrote: >> I would expect that you can just iterate through the size possibilities >> pretty quickly and just use the first one that works -- no /proc >> groveling. > > It's not really that easy, because (at least on the kernel version I > tested) it's not

Re: [HACKERS] Rethinking representation of sort/hash semantics in queries and plans

2010-11-28 Thread Tom Lane
I wrote: > Now, this loss of flexibility doesn't particularly bother me, because > I know of no existing or contemplated btree-substitute access methods. > If one did appear on the horizon, there are a couple of ways we could > fix the problem, the cleanest being to let a non-btree opfamily declare

Re: [HACKERS] [PATCH] Return command tag 'REPLACE X' for CREATE OR REPLACE statements.

2010-11-28 Thread Tom Lane
Robert Haas writes: > I think more expessive command tags are in general a good thing. The > idea that this particular change would be useful primarily for humans > examining the psql output seems a bit weak to me, but I can easily see > it being useful for programs. Right now a program has no r

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Tom Lane
BTW, this might be premature to mention pending some tests about mapping versus zeroing overhead, but it strikes me that there's more than one way to skin a cat. I still think the idea of statically allocated space sucks. But what if we rearranged things so that palloc0 doesn't consist of palloc-

Re: [HACKERS] [PATCH] Return command tag 'REPLACE X' for CREATE OR REPLACE statements.

2010-11-28 Thread Robert Haas
On Sun, Nov 28, 2010 at 11:29 AM, Tom Lane wrote: > Marti Raudsepp writes: >> This patch returns command tag "CREATE X" or "REPLACE X" for >> LANGAUGE/VIEW/RULE/FUNCTION. This is done by passing completionTag to >> from ProcessUtility to more functions, and adding a 'bool *didUpdate' >> argument

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Tom Lane
Robert Haas writes: > One possible way to get a real speedup here would be to look for ways > to trim the number of catcaches. BTW, it's not going to help to remove catcaches that have a small initial size, as the pg_am cache certainly does. If the bucket zeroing cost is really something to mini

Re: [HACKERS] Report: Linux huge pages with Postgres

2010-11-28 Thread Tom Lane
Kenneth Marshall writes: > On Sat, Nov 27, 2010 at 02:27:12PM -0500, Tom Lane wrote: >> ... A bigger problem is that the shmem request size must be a >> multiple of the system's hugepage size, which is *not* a constant >> even though the test patch just uses 2MB as the assumed value. For a >> pro

Re: [HACKERS] contrib: auth_delay module

2010-11-28 Thread Robert Haas
On Sun, Nov 28, 2010 at 7:10 PM, Jeff Janes wrote: > Oh, I wasn't complaining.  I think that having max_connections be > charged for the duration even if the socket is dropped is the only > reasonable thing to do, and wanted to verify that it did happen. > Otherwise the module wouldn't do a very g

Re: [HACKERS] contrib: auth_delay module

2010-11-28 Thread Jeff Janes
On Sun, Nov 28, 2010 at 3:57 PM, Robert Haas wrote: > On Sun, Nov 28, 2010 at 5:41 PM, Jeff Janes wrote: >> On Sun, Nov 28, 2010 at 5:38 AM, Robert Haas wrote: >>> On Sat, Nov 27, 2010 at 2:44 PM, Jeff Janes wrote: >> I haven' t thought of a way to test this, so I guess I'll just ask.

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Robert Haas
On Sun, Nov 28, 2010 at 6:41 PM, Tom Lane wrote: > Robert Haas writes: >> After our recent conversation >> about KNNGIST, it occurred to me to wonder whether there's really any >> point in pretending that a user can usefully add an AM, both due to >> hard-wired planner knowledge and due to lack o

Re: [HACKERS] contrib: auth_delay module

2010-11-28 Thread Robert Haas
On Sun, Nov 28, 2010 at 5:41 PM, Jeff Janes wrote: > On Sun, Nov 28, 2010 at 5:38 AM, Robert Haas wrote: >> On Sat, Nov 27, 2010 at 2:44 PM, Jeff Janes wrote: > >>> I haven' t thought of a way to test this, so I guess I'll just ask. >>> If the attacking client just waits a few milliseconds for a

Re: [HACKERS] [GENERAL] column-level update privs + lock table

2010-11-28 Thread KaiGai Kohei
(2010/11/27 9:11), Robert Haas wrote: > 2010/11/25 KaiGai Kohei: >> (2010/10/16 4:49), Josh Kupershmidt wrote: >>> [Moving to -hackers] >>> >>> On Fri, Oct 15, 2010 at 3:43 AM, Simon Riggs >>> wrote: On Mon, 2010-10-11 at 09:41 -0400, Josh Kupershmidt wrote: > On Thu, Oct 7, 2010 at 7:

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Tom Lane
Robert Haas writes: > After our recent conversation > about KNNGIST, it occurred to me to wonder whether there's really any > point in pretending that a user can usefully add an AM, both due to > hard-wired planner knowledge and due to lack of any sort of extensible > XLOG support. If not, we cou

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Robert Haas
On Sun, Nov 28, 2010 at 3:53 PM, Tom Lane wrote: > Robert Haas writes: >> The more general issue here is what to do about our >> high backend startup costs.  Beyond trying to recycle backends for new >> connections, as I've previous proposed and with all the problems it >> entails, the only thing

Re: [HACKERS] contrib: auth_delay module

2010-11-28 Thread Jeff Janes
On Sun, Nov 28, 2010 at 5:38 AM, Robert Haas wrote: > On Sat, Nov 27, 2010 at 2:44 PM, Jeff Janes wrote: >> I haven' t thought of a way to test this, so I guess I'll just ask. >> If the attacking client just waits a few milliseconds for a response >> and then drops the socket, opening a new one,

Re: [HACKERS] Report: Linux huge pages with Postgres

2010-11-28 Thread Kenneth Marshall
On Sat, Nov 27, 2010 at 02:27:12PM -0500, Tom Lane wrote: > We've gotten a few inquiries about whether Postgres can use "huge pages" > under Linux. In principle that should be more efficient for large shmem > regions, since fewer TLB entries are needed to support the address > space. I spent a bi

Re: [HACKERS] Rethinking representation of sort/hash semantics in queries and plans

2010-11-28 Thread Tom Lane
I wrote: > (For some extra amusement, trace through where > build_index_pathkeys' data comes from...) While I don't propose to implement right away the whole SortGroupClause and plan tree modification sketched above, I did look into fixing build_index_pathkeys so that it doesn't uselessly convert

[HACKERS] SSI using rw-conflict lists

2010-11-28 Thread Kevin Grittner
Well, it's been a productive holiday weekend. I've completed the switch of the SSI implementation from one conflict pointer in and one conflict pointer out per transaction to a list of conflicts between transactions. This eliminated all false positives in my dtester suite. The only test which ha

Re: [HACKERS] [GENERAL] column-level update privs + lock table

2010-11-28 Thread Robert Haas
On Sun, Nov 28, 2010 at 11:35 AM, Simon Riggs wrote: > On Fri, 2010-11-26 at 19:11 -0500, Robert Haas wrote: >> 2010/11/25 KaiGai Kohei : >> > (2010/10/16 4:49), Josh Kupershmidt wrote: >> >> [Moving to -hackers] >> >> >> >> On Fri, Oct 15, 2010 at 3:43 AM, Simon Riggs   >> >> wrote: >> >>> On Mon

Re: [HACKERS] Rethinking representation of sort/hash semantics in queries and plans

2010-11-28 Thread Tom Lane
Dimitri Fontaine writes: > Tom Lane writes: >> So to fix these problems we'd need to replace sort operator OIDs in >> SortGroupClause and plan nodes with those three items. Obviously, this >> would be slightly bulkier, but the extra cost added to copying parse and >> plan trees should be tiny co

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Tom Lane
Robert Haas writes: > The more general issue here is what to do about our > high backend startup costs. Beyond trying to recycle backends for new > connections, as I've previous proposed and with all the problems it > entails, the only thing that looks promising here is to try to somehow > cut do

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Bruce Momjian
Robert Haas wrote: > On Sat, Nov 27, 2010 at 11:18 PM, Bruce Momjian wrote: > > Not sure that information moves us forward. ?If the postmaster cleared > > the memory, we would have COW in the child and probably be even slower. > > Well, we can determine the answers to these questions empirically.

Re: [HACKERS] Rethinking representation of sort/hash semantics in queries and plans

2010-11-28 Thread Dimitri Fontaine
Tom Lane writes: > If you look closely at what we're doing with sort operators > (get_ordering_op_properties pretty much encapsulates this), it turns out > that a sort operator is shorthand for three pieces of information: > > 1. btree opfamily OID > 2. specific input datatype for the opfamily > 3

Re: [HACKERS] Report: Linux huge pages with Postgres

2010-11-28 Thread Martijn van Oosterhout
On Sun, Nov 28, 2010 at 02:32:04PM -0500, Tom Lane wrote: > > Sure, but 4MB of memory is enough to require 1000 TLB entries, which is > > more than enough to blow the TLB even on a Nehalem. > > That can't possibly be right. I'm sure the chip designers have heard of > programs using more than 4MB.

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Robert Haas
On Sun, Nov 28, 2010 at 11:41 AM, Tom Lane wrote: > Robert Haas writes: >> On Sat, Nov 27, 2010 at 11:18 PM, Bruce Momjian wrote: >>> Not sure that information moves us forward.  If the postmaster cleared >>> the memory, we would have COW in the child and probably be even slower. > >> Well, we c

Re: [HACKERS] Report: Linux huge pages with Postgres

2010-11-28 Thread Tom Lane
Simon Riggs writes: > On Sun, 2010-11-28 at 12:04 -0500, Tom Lane wrote: >> There's no exposed API for causing a process's regular memory to become >> hugepages. > We could make all the palloc stuff into shared memory also ("private" > shared memory that is). We're not likely to run out of 64-bit

Re: [HACKERS] Report: Linux huge pages with Postgres

2010-11-28 Thread Simon Riggs
On Sun, 2010-11-28 at 12:04 -0500, Tom Lane wrote: > Simon Riggs writes: > > On Sat, 2010-11-27 at 14:27 -0500, Tom Lane wrote: > >> This is discouraging; it certainly doesn't make me want to expend the > >> effort to develop a production patch. > > > Perhaps. > > > Why do this only for shared m

Re: [HACKERS] Report: Linux huge pages with Postgres

2010-11-28 Thread Tom Lane
Simon Riggs writes: > On Sat, 2010-11-27 at 14:27 -0500, Tom Lane wrote: >> This is discouraging; it certainly doesn't make me want to expend the >> effort to develop a production patch. > Perhaps. > Why do this only for shared memory? There's no exposed API for causing a process's regular memo

Re: [HACKERS] Report: Linux huge pages with Postgres

2010-11-28 Thread Simon Riggs
On Sat, 2010-11-27 at 14:27 -0500, Tom Lane wrote: > This is discouraging; it certainly doesn't make me want to expend the > effort to develop a production patch. Perhaps. Why do this only for shared memory? Surely the majority of memory accesses are to private memory, so being able to allocate

Re: [HACKERS] PLy_malloc and plperl mallocs

2010-11-28 Thread Tom Lane
=?UTF-8?B?SmFuIFVyYmHFhHNraQ==?= writes: > I'll do that for PL/Python for now. While on the topic of needless FATAL > errors, if you try to create a Python 3 function in a session that > already loaded Python 2, you get a FATAL error with an errhint of > Start a new session to use a different Pyt

Re: [HACKERS] profiling connection overhead

2010-11-28 Thread Tom Lane
Robert Haas writes: > On Sat, Nov 27, 2010 at 11:18 PM, Bruce Momjian wrote: >> Not sure that information moves us forward.  If the postmaster cleared >> the memory, we would have COW in the child and probably be even slower. > Well, we can determine the answers to these questions empirically.

Re: [HACKERS] ECPG question about PREPARE and EXECUTE

2010-11-28 Thread Michael Meskes
On Wed, Nov 10, 2010 at 11:44:52AM +0100, Boszormenyi Zoltan wrote: > a question came to us in the form of a code example, > which I shortened. Say, we have this structure: > ... > Any comment on why it isn't done? Missing feature. Originally the pure text based statement copying wasn't able to co

Re: [HACKERS] [GENERAL] column-level update privs + lock table

2010-11-28 Thread Simon Riggs
On Fri, 2010-11-26 at 19:11 -0500, Robert Haas wrote: > 2010/11/25 KaiGai Kohei : > > (2010/10/16 4:49), Josh Kupershmidt wrote: > >> [Moving to -hackers] > >> > >> On Fri, Oct 15, 2010 at 3:43 AM, Simon Riggs wrote: > >>> On Mon, 2010-10-11 at 09:41 -0400, Josh Kupershmidt wrote: > On Thu, O

Re: [HACKERS] [PATCH] Return command tag 'REPLACE X' for CREATE OR REPLACE statements.

2010-11-28 Thread Tom Lane
Marti Raudsepp writes: > This patch returns command tag "CREATE X" or "REPLACE X" for > LANGAUGE/VIEW/RULE/FUNCTION. This is done by passing completionTag to > from ProcessUtility to more functions, and adding a 'bool *didUpdate' > argument to some lower-level functions. I'm not sure if passing ba

Re: [HACKERS] [GENERAL] column-level update privs + lock table

2010-11-28 Thread Josh Kupershmidt
On Fri, Nov 26, 2010 at 7:11 PM, Robert Haas wrote: > I'm not totally convinced that this is the correct behavior.  It seems > a bit surprising that UPDATE privilege on a single column is enough to > lock out all SELECT activity from the table.  It's actually a bit > surprising that even full-tabl

[HACKERS] [PATCH] Return command tag 'REPLACE X' for CREATE OR REPLACE statements.

2010-11-28 Thread Marti Raudsepp
Hi list, Often enough when developing PostgreSQL views and functions, I have pasted the CREATE OR REPLACE commands into the wrong window/shell and ran them there without realizing that I'm creating a function in the wrong database, instead of replacing. Currently psql does not provide any feedback

Re: [HACKERS] Rethinking representation of sort/hash semantics in queries and plans

2010-11-28 Thread Martijn van Oosterhout
On Sat, Nov 27, 2010 at 10:02:33PM -0500, Tom Lane wrote: > In recent discussions of the plan-tree representation for KNNGIST index > scans, there seemed to be agreement that it'd be a good idea to explicitly > represent the expected sort ordering of the output. While thinking about > how best to

Re: [HACKERS] PLy_malloc and plperl mallocs

2010-11-28 Thread Jan Urbański
On 28/11/10 05:23, Andrew Dunstan wrote: > > > On 11/27/2010 10:28 PM, Tom Lane wrote: >> =?UTF-8?B?SmFuIFVyYmHFhHNraQ==?= writes: >>> I noticed that PL/Python uses a simple wrapper around malloc that does >>> ereport(FATAL) if malloc returns NULL. I find it a bit harsh, don't we >>> normally do

Re: [HACKERS] PROPOSAL of xmlvalidate

2010-11-28 Thread Andrew Dunstan
On 11/28/2010 05:33 AM, Tomáš Pospíšil wrote: Hi, I am working on patch adding xmlvalidate() functionality. LibXML 2.7.7 improved DTD, XSD, Relax-NG validation, so using that. I have idea of creating system table for holding DTDs, XSDs, Relax-NGs (similar as on ORACLE). Is that good idea? I

Re: [HACKERS] contrib: auth_delay module

2010-11-28 Thread Robert Haas
On Sat, Nov 27, 2010 at 2:44 PM, Jeff Janes wrote: > On Thu, Nov 4, 2010 at 6:35 AM, Stephen Frost wrote: >> * Jan Urbański (wulc...@wulczer.org) wrote: >>> On 04/11/10 14:09, Robert Haas wrote: >>> > Hmm, I wonder how useful this is given that restriction. >>> >>> As KaiGai mentined, it's more t

Re: [HACKERS] ALTER OBJECT any_name SET SCHEMA name

2010-11-28 Thread Robert Haas
On Sat, Nov 27, 2010 at 2:17 PM, Dimitri Fontaine wrote: > Thanks! > > The _oid variants will have to re-appear in the "alter extension set > schema" patch, which is the last of the series. Meanwhile, I will have > to merge head with the current extension patch (already overdue for a > new version

[HACKERS] PROPOSAL of xmlvalidate

2010-11-28 Thread Tomáš Pospíšil
Hi, I am working on patch adding xmlvalidate() functionality. LibXML 2.7.7 improved DTD, XSD, Relax-NG validation, so using that. I have idea of creating system table for holding DTDs, XSDs, Relax-NGs (similar as on ORACLE). Is that good idea? If so, how to implement that table? pg_attribute a