Re: [HACKERS] Uh, I change my mind about commit_delay + commit_siblings (sort of)

2012-05-28 Thread Heikki Linnakangas
On 29.05.2012 04:18, Peter Geoghegan wrote: The attached very simple patch moves the commit_delay + commit_siblings sleep into XLogFlush, where the leader alone sleeps. This appears to be a much more effective site for a delay. Benchmark results, with and without a delay of 3000ms (commit_siblin

Re: [HACKERS] libpq URL syntax vs SQLAlchemy

2012-05-28 Thread Alex
Peter Eisentraut writes: > On mån, 2012-05-14 at 18:16 +0300, Alex Shulgin wrote: >> Upon closer inspection of the issue I came to believe that the proper >> fix is to drop support for special treatment of "host part" starting >> with slash altogether. >> >> Attached is a patch to do that. > >

[HACKERS] AbortOutOfAnyTransaction is a few bricks shy of a load

2012-05-28 Thread Tom Lane
While poking at the slow-relcache-rebuild issue recently reported by Jeff Frost and Greg Mullane, I had set up a test case that used a small function to create and drop enough temp tables to provoke an sinval reset. I had also modified sinval.c so that it reported sinval reset processing as elog(L

Re: [HACKERS] Patch: add conversion from pg_wchar to multibyte

2012-05-28 Thread Tatsuo Ishii
> On Tue, May 22, 2012 at 3:27 PM, Tatsuo Ishii wrote: > >> > Thanks for your comments. They clarify a lot. >> > But I still don't realize how can we distinguish IS_LCPRV2 and IS_LC2? >> > Isn't it possible for them to produce same pg_wchar? >> >> If LB is in 0x90 - 0x99 range, then they are LC2.

[HACKERS] Uh, I change my mind about commit_delay + commit_siblings (sort of)

2012-05-28 Thread Peter Geoghegan
The attached very simple patch moves the commit_delay + commit_siblings sleep into XLogFlush, where the leader alone sleeps. This appears to be a much more effective site for a delay. Benchmark results, with and without a delay of 3000ms (commit_siblings is 0 in both cases) are available from: ht

Re: [HACKERS] pg_basebackup --xlog compatibility break

2012-05-28 Thread Fujii Masao
On Tue, May 29, 2012 at 5:38 AM, Magnus Hagander wrote: > On Mon, May 28, 2012 at 10:11 PM, Peter Eisentraut wrote: >> In 9.1, the pg_basebackup option --xlog takes no argument.  In 9.2, it >> takes a required argument.  I think such compatibility breaks should be >> avoided, especially in client

Re: [HACKERS] Bogus nestloop rows estimate in 8.4.7

2012-05-28 Thread Tom Lane
Marti Raudsepp writes: > On Mon, May 28, 2012 at 11:23 PM, Tom Lane wrote: >> However, the error in your original example is far too large to be >> explained by that, so I think it was tripping over something different. > Good point. But I generated a bigger data set with the above test case > a

Re: [HACKERS] Bogus nestloop rows estimate in 8.4.7

2012-05-28 Thread Marti Raudsepp
On Mon, May 28, 2012 at 11:23 PM, Tom Lane wrote: > However, the error in your original example is far too large to be > explained by that, so I think it was tripping over something different. Good point. But I generated a bigger data set with the above test case and it gets progressively worse w

Re: [HACKERS] FDW / list of needed columns, WHERE conditions (in PlanForeignScan)

2012-05-28 Thread Tom Lane
Tomas Vondra writes: > I'm writing my first FDW, and I need to get the list of columns I > actually need to fetch when planning the query. I do want to fetch only > the columns that are actually needed, not all of them. reltargetlist and attr_needed only tell you about columns the scan has to *ou

[HACKERS] FDW / list of needed columns, WHERE conditions (in PlanForeignScan)

2012-05-28 Thread Tomas Vondra
Hi, I'm writing my first FDW, and I need to get the list of columns I actually need to fetch when planning the query. I do want to fetch only the columns that are actually needed, not all of them. Initially I've used RelOptInfo->reltargetlist but it seems it does not cover cases like SELECT 1

Re: [HACKERS] pg_basebackup --xlog compatibility break

2012-05-28 Thread Magnus Hagander
On Mon, May 28, 2012 at 10:11 PM, Peter Eisentraut wrote: > In 9.1, the pg_basebackup option --xlog takes no argument.  In 9.2, it > takes a required argument.  I think such compatibility breaks should be > avoided, especially in client-side programs.  Now you can't write a > script running pg_bas

Re: [HACKERS] Bogus nestloop rows estimate in 8.4.7

2012-05-28 Thread Tom Lane
Marti Raudsepp writes: > On Mon, May 28, 2012 at 10:32 PM, Marti Raudsepp wrote: >> There was a similar case in 9.0.4 with WHERE i=1, but that has been >> fixed in 9.0.7 > Oh, it's been fixed in 9.0.7, but apparently not in 8.4.11; the empty > parent tables are confusing the estimate: Hmm ... w

[HACKERS] pg_basebackup --xlog compatibility break

2012-05-28 Thread Peter Eisentraut
In 9.1, the pg_basebackup option --xlog takes no argument. In 9.2, it takes a required argument. I think such compatibility breaks should be avoided, especially in client-side programs. Now you can't write a script running pg_basebackup that works with 9.1 and 9.2, if you need to include the WAL

Re: [HACKERS] pg_upgrade libraries check

2012-05-28 Thread Robert Haas
On Sun, May 27, 2012 at 11:31 AM, Tom Lane wrote: > I don't recall exactly what problems drove us to make pg_upgrade do > what it does with extensions, but we need a different fix for them. Well, you need pg_upgrade to preserve the OIDs of objects that are part of extensions just as you do for an

Re: [HACKERS] Per-Database Roles

2012-05-28 Thread Robert Haas
On Fri, May 25, 2012 at 11:12 PM, Bruce Momjian wrote: > On Fri, May 25, 2012 at 10:34:54PM -0400, Stephen Frost wrote: >> * Robert Haas (robertmh...@gmail.com) wrote: >> > On Thu, May 24, 2012 at 6:21 PM, Bruce Momjian wrote: >> > > Yes, pre-1996.  I think the fact that authentication/user names

Re: [HACKERS] Per-Database Roles

2012-05-28 Thread Robert Haas
On Sun, May 27, 2012 at 2:53 PM, Peter Eisentraut wrote: > On tis, 2012-05-22 at 10:19 -0400, Robert Haas wrote: >> I think we should have made roles and tablespaces database >> objects rather than shared objects, > > User names are global objects in the SQL standard, which is part of the > reason

Re: [HACKERS] How could we make it simple to access the log as a table?

2012-05-28 Thread Robert Haas
On Mon, May 28, 2012 at 2:21 PM, Christopher Browne wrote: >> Yeah, I agree.  I think what is missing here is something that can be read >> (and maybe indexed?) like a table, but written by a pretty dumb process.   >> It's not terribly workable to have PG log to PG, because there are too many >>

Re: [HACKERS] libpq URL syntax vs SQLAlchemy

2012-05-28 Thread Peter Eisentraut
On mån, 2012-05-14 at 18:16 +0300, Alex Shulgin wrote: > Upon closer inspection of the issue I came to believe that the proper > fix is to drop support for special treatment of "host part" starting > with slash altogether. > > Attached is a patch to do that. Committed. I also updated the docume

Re: [HACKERS] Bogus nestloop rows estimate in 8.4.7

2012-05-28 Thread Marti Raudsepp
On Mon, May 28, 2012 at 10:32 PM, Marti Raudsepp wrote: > There was a similar case in 9.0.4 with WHERE i=1, but that has been > fixed in 9.0.7 Oh, it's been fixed in 9.0.7, but apparently not in 8.4.11; the empty parent tables are confusing the estimate: explain select * from a_parent join b_par

Re: [HACKERS] Bogus nestloop rows estimate in 8.4.7

2012-05-28 Thread Marti Raudsepp
On Mon, May 28, 2012 at 8:56 PM, Tom Lane wrote: > Also, what do you have constraint_exclusion set to? The only sane value, "partition" > This sounds familiar, but a quick trawl through the commit logs didn't > immediately turn up any related-looking patches.  Can you put together > a self-conta

[HACKERS] Upcoming back-branch PG releases

2012-05-28 Thread Tom Lane
A minor security issue (CVE-2012-2143) has been found in code that Postgres shares with several other projects. After some discussion it was decided that the issue is not of enough severity to justify exactly coordinated security releases, which would be quite difficult anyway given the projects'

Re: [HACKERS] How could we make it simple to access the log as a table?

2012-05-28 Thread Christopher Browne
On Mon, May 28, 2012 at 1:45 PM, Robert Haas wrote: > On May 28, 2012, at 11:57 AM, Christopher Browne wrote: >> 2.  Ask Syslog >> >> My favorite way to configure *my* PG instances (e.g. - those that I >> use for testing) is for them to forward messages to syslog.  That way >> they, and my Slony

Re: [HACKERS] Bogus nestloop rows estimate in 8.4.7

2012-05-28 Thread Tom Lane
Marti Raudsepp writes: > This bug isn't causing me any immediate problems -- the plan works out > well regardless -- but PostgreSQL 8.4.7 is somehow overestimating the > number of rows coming from a nestloop join, when joining 2 large > partitioned tables. This sounds familiar, but a quick trawl

Re: [HACKERS] How could we make it simple to access the log as a table?

2012-05-28 Thread Robert Haas
On May 28, 2012, at 11:57 AM, Christopher Browne wrote: > All interesting... > > We've got several models as to how logs are likely to be captured, > which mean that it'll be difficult to have one uniform answer. > > 1. An easy traditional default is to capture logs in a log directory. > > An

Re: [HACKERS] WalSndWakeup() and synchronous_commit=off

2012-05-28 Thread Tom Lane
Andres Freund writes: > Does anybody have a better idea than to either call WalSndWakeup() at > essentially the wrong places or calling it inside a critical section? > Tom, what danger do you see from calling it in a critical section? My concern was basically that it might throw an error. Look

[HACKERS] Re: [GENERAL] Attempting to do a rolling move to 9.2Beta (as a slave) fails

2012-05-28 Thread Karl Denninger
On 5/28/2012 11:44 AM, Tom Lane wrote: > Karl Denninger writes: >> I am attempting to validate the path forward to 9.2, and thus tried the >> following: >> 1. Build 9.2Beta1; all fine. >> 2. Run a pg_basebackup from the current master machine (running 9.1) to >> a new directory on the slave machin

Re: [HACKERS] proclock table corrupted

2012-05-28 Thread Tom Lane
Harshitha S writes: > Sorry, the OS is WindRiver Linux. What I would suspect first is issues with the MIPS spinlock assembly code (look into s_lock.h) not being portable to your platform. That code hasn't been tested on very many machines, I suspect. It's not impossible that it doesn't work at

Re: [HACKERS] [GENERAL] Attempting to do a rolling move to 9.2Beta (as a slave) fails

2012-05-28 Thread Tom Lane
Karl Denninger writes: > I am attempting to validate the path forward to 9.2, and thus tried the > following: > 1. Build 9.2Beta1; all fine. > 2. Run a pg_basebackup from the current master machine (running 9.1) to > a new directory on the slave machine, using the 9.2Beta1 pg_basebackup > execut

Re: [HACKERS] pg_upgrade libraries check

2012-05-28 Thread Tom Lane
Dimitri Fontaine writes: > I have some plans that we will be discussing later in the new dev cycle > and that would impact such a method if we're to follow them. To better > solve both the per-system (not even cluster) and per-database extension > versions and the inline/os-packaged extension disc

Re: [HACKERS] pg_upgrade libraries check

2012-05-28 Thread Dimitri Fontaine
Tom Lane writes: > Well, the scheme I had in mind would require pg_upgrade to verify that > the new cluster contains an extension control file for each extension in > the old cluster (which is something it really oughta do anyway, if it > doesn't already). After that, though, it ought not be look

Re: [HACKERS] How could we make it simple to access the log as a table?

2012-05-28 Thread Christopher Browne
On Mon, May 28, 2012 at 11:39 AM, Dimitri Fontaine wrote: > Stephen Frost writes: >> This is really where I was hoping to eventually get to with the logging >> changes that have been discussed over the past couple of years.  We need >> to have a mechanism to allow logging to different places, bas

Re: [HACKERS] How could we make it simple to access the log as a table?

2012-05-28 Thread Dimitri Fontaine
Stephen Frost writes: > This is really where I was hoping to eventually get to with the logging > changes that have been discussed over the past couple of years. We need > to have a mechanism to allow logging to different places, based on > information included in the log message and/or context,

Re: [HACKERS] Backends stalled in 'startup' state: index corruption

2012-05-28 Thread Tom Lane
Greg Sabino Mullane writes: > On Sun, May 27, 2012 at 05:44:15PM -0700, Jeff Frost wrote: >> On May 27, 2012, at 12:53 PM, Tom Lane wrote: >>> occurring, they'd take long enough to expose the process to sinval >>> overrun even with not-very-high DDL rates. >> As it turns out, there are quite a fe

Re: [HACKERS] WalSndWakeup() and synchronous_commit=off

2012-05-28 Thread Andres Freund
On Tuesday, May 15, 2012 05:30:27 PM Andres Freund wrote: > On Monday, May 14, 2012 07:55:32 PM Fujii Masao wrote: > > On Mon, May 14, 2012 at 6:32 PM, Andres Freund > > wrote: > > > On Friday, May 11, 2012 08:45:23 PM Tom Lane wrote: > > >> Andres Freund writes: > > >> > Its the only place thou

[HACKERS] Function call hierarchy/path since getting the buffer until access its data

2012-05-28 Thread Waldecir Faria
Good morning, I am doing a study about buffer management to improve the performance of one program that does heavy I/O operations. After looking and reading from different softwares' source codes/texts one friend suggested me to take a look at the PostgreSQL code. I already took a look at the P

Re: [HACKERS] [RFC] Interface of Row Level Security

2012-05-28 Thread Florian Pflug
On May28, 2012, at 02:46 , Noah Misch wrote: > On Thu, May 24, 2012 at 07:31:37PM +0200, Florian Pflug wrote: >> Since the security barrier flag carries a potentially hefty performance >> penalty, I think it should be optional. Application which don't allow >> SQL-level access to the database might

Re: [HACKERS] pg_stat_statements temporary file

2012-05-28 Thread Andres Freund
On Friday, May 25, 2012 05:19:28 PM Tom Lane wrote: > Andres Freund writes: > > On Friday, May 25, 2012 04:03:49 PM Peter Geoghegan wrote: > >> Where do you suggest the file be written to? > > > > One could argue stats_temp_directory would be the correct place. > > No, that would be exactly the

Re: [HACKERS] Backends stalled in 'startup' state: index corruption

2012-05-28 Thread Greg Sabino Mullane
On Sun, May 27, 2012 at 05:44:15PM -0700, Jeff Frost wrote: > On May 27, 2012, at 12:53 PM, Tom Lane wrote: > > occurring, they'd take long enough to expose the process to sinval > > overrun even with not-very-high DDL rates. > As it turns out, there are quite a few temporary tables created. For t

Re: [HACKERS] Bogus nestloop rows estimate in 8.4.7

2012-05-28 Thread Marti Raudsepp
On Mon, May 28, 2012 at 10:45 AM, Marti Raudsepp wrote: > Query: > SELECT '2012-05-28T09:00:00', count(*), > uniq(sort(array_agg(visitor_id))), banner_id, client_body_id, > partner_body_id, space_id, campaign_id, evt_type_id FROM stats_request > WHERE stats_request.request_time >= '2012-05-28T09:0

[HACKERS] Bogus nestloop rows estimate in 8.4.7

2012-05-28 Thread Marti Raudsepp
Hi list, This bug isn't causing me any immediate problems -- the plan works out well regardless -- but PostgreSQL 8.4.7 is somehow overestimating the number of rows coming from a nestloop join, when joining 2 large partitioned tables. Maybe it's been fixed in more recent versions, sadly it's an EO