Re: [HACKERS] Uppercase tab completion keywords in psql?

2012-05-30 Thread Devrim GÜNDÜZ
Hi, On Wed, 2012-05-30 at 22:03 -0400, Bruce Momjian wrote: > A control variable was added in this commit: > > commit db84ba65ab5c0ad0b34d68ab5a687bc5f4ca3ba6 > Author: Peter Eisentraut Thanks Bruce, apparently I missed it. Regards, -- Devrim GÜNDÜZ Principal Systems Engineer

Re: [HACKERS] remembering locks and dynahash.c

2012-05-30 Thread Tom Lane
Jeff Janes writes: > I am working on making resource owner remember a limited number of > locks, so it can reassign them more efficiently. Check, per previous discussion. > Currently I'm having resowner.c remember the LOCALLOCKTAG, because I > thought that that was the only handle I could use.

Re: [HACKERS] Bug in new buffering GiST build code

2012-05-30 Thread Tom Lane
Alexander Korotkov writes: > I've one note not directly related to buffering build. While I debugging > buffering GiST index build, backend was frequently crashed. After recovery > partially built index file was remain. Do we have some tool to detect such > "dead" files? If not, probably we need s

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-30 Thread Tatsuo Ishii
> I'm not excited by this patch. It dodges the O(N^2) lock behavior for > the initial phase of acquiring the locks, but it does nothing for the > lock-related slowdown occurring in all pg_dump's subsequent commands. > I think we really need to get in the server-side fix that Jeff Janes is > workin

Re: [HACKERS] FailedAssertion("!(PrivateRefCount[i] == 0)", File: "bufmgr.c", Line: 1741

2012-05-30 Thread Erik Rijkers
On Thu, May 31, 2012 03:30, Robert Haas wrote: > On Wed, May 30, 2012 at 6:00 PM, Erik Rijkers wrote: >> directory >> 2012-05-30 23:40:57.909 CEST 3909 CONTEXT:  writing block 5152 of relation >> base/21268/26569 >>        xlog redo multi-insert (init): rel 1663/21268/26581; blk 3852; 35 >> tupl

Re: [HACKERS] WIP: 2d-mapping based indexing for ranges

2012-05-30 Thread Jeff Davis
On Mon, 2012-05-28 at 23:42 +0400, Alexander Korotkov wrote: > Hackers, > > > attached patch implements another approach to indexing of ranges: > mapping lower and upper bounds as 2d-coordinates and using same > indexing approaches as for 2d points. Patch provides range_ops2 > operator class whic

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-30 Thread Tom Lane
Stephen Frost writes: > The current situation where the client-to-server latency accounts for > multiple minutes of time is just ridiculous, however, so I feel we need > some form of this patch, even if the server side is magically made much > faster. The constant back-and-forth isn't cheap. No,

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-30 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: > Tatsuo Ishii writes: > > Shall I commit to master and all supported branches? > > I'm not excited by this patch. It dodges the O(N^2) lock behavior for > the initial phase of acquiring the locks, but it does nothing for the > lock-related slowdown occurri

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-30 Thread Tom Lane
Tatsuo Ishii writes: >> Ok, I modified the part of pg_dump where tremendous number of LOCK >> TABLE are issued. I replace them with single LOCK TABLE with multiple >> tables. With 100k tables LOCK statements took 13 minutes in total, now >> it only takes 3 seconds. Comments? > Shall I commit to m

Re: [HACKERS] temporal support patch

2012-05-30 Thread Jeff Davis
On Wed, 2012-05-16 at 23:14 +0200, Miroslav Šimulčík wrote: > Hi all, > > > as a part of my master's thesis I have created temporal support patch > for PostgreSQL. It enables the creation of special temporal tables > with entries versioning. Modifying operations (UPDATE, DELETE, > TRUNCATE) on th

Re: [HACKERS] Synchronized scans versus relcache reinitialization

2012-05-30 Thread Tom Lane
Jeff Davis writes: > On Sat, 2012-05-26 at 15:14 -0400, Tom Lane wrote: >> 3. Having now spent a good deal of time poking at this, I think that the >> syncscan logic is in need of more tuning, and I am wondering whether we >> should even have it turned on by default. It appears to be totally >> u

Re: [HACKERS] Synchronized scans versus relcache reinitialization

2012-05-30 Thread Jeff Davis
On Sat, 2012-05-26 at 15:14 -0400, Tom Lane wrote: > 3. Having now spent a good deal of time poking at this, I think that the > syncscan logic is in need of more tuning, and I am wondering whether we > should even have it turned on by default. It appears to be totally > useless for fully-cached-in

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Jeff Janes
On Wed, May 30, 2012 at 7:00 PM, Stephen Frost wrote: > Robert, > > * Robert Haas (robertmh...@gmail.com) wrote: >> On Wed, May 30, 2012 at 9:10 PM, Sergey Koposov >> wrote: >> > I understand the need of significant locking when there concurrent writes, >> > but not when there only reads. But I'

Re: [HACKERS] Uppercase tab completion keywords in psql?

2012-05-30 Thread Bruce Momjian
On Thu, May 31, 2012 at 04:06:50AM +0300, Devrim Gunduz wrote: > Hi, > > On Mon, 2012-05-07 at 13:22 -0400, Robert Haas wrote: > > On Sat, May 5, 2012 at 9:03 AM, Bruce Momjian > > wrote: > > > On Fri, May 04, 2012 at 08:46:28PM +0300, Peter Eisentraut wrote: > > >> On tor, 2012-05-03 at 15:47 -0

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: > On Wed, May 30, 2012 at 9:10 PM, Sergey Koposov wrote: > > I understand the need of significant locking when there concurrent writes, > > but not when there only reads. But I'm not a RDBMS expert, so that's maybe > > that's misunderstanding o

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Florian Pflug
On May31, 2012, at 02:26 , Sergey Koposov wrote: > On Thu, 31 May 2012, Florian Pflug wrote: >> Wait, so performance *increased* by spreading the backends out over as many >> dies as possible, not by using as few as possible? That'd be exactly the >> opposite of what I'd have expected. (I'm assum

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Robert Haas
On Wed, May 30, 2012 at 9:10 PM, Sergey Koposov wrote: > I understand the need of significant locking when there concurrent writes, > but not when there only reads. But I'm not a RDBMS expert, so that's maybe > that's misunderstanding on my side. If we knew in advance that no writes would come al

Re: [HACKERS] FailedAssertion("!(PrivateRefCount[i] == 0)", File: "bufmgr.c", Line: 1741

2012-05-30 Thread Robert Haas
On Wed, May 30, 2012 at 6:00 PM, Erik Rijkers wrote: > (I double-checked that I got your latest commit in) > > I'm afraid it's not yet resolved; the sync-slave still crashes almost > immediately: > > master logfile says: > 2012-05-30 23:30:07.846 CEST 3918 LOG:  standby wal_receiver_01 is now the

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Sergey Koposov
On Wed, 30 May 2012, Jeff Janes wrote: But the question now is whether there is a *PG* problem here or not, or is it Intel's or Linux's problem ? Because still the slowdown was caused by locking. If there wouldn't be locking there wouldn't be any problems (as demonstrated a while ago by just cat

Re: [HACKERS] Uppercase tab completion keywords in psql?

2012-05-30 Thread Devrim GÜNDÜZ
Hi, On Mon, 2012-05-07 at 13:22 -0400, Robert Haas wrote: > On Sat, May 5, 2012 at 9:03 AM, Bruce Momjian > wrote: > > On Fri, May 04, 2012 at 08:46:28PM +0300, Peter Eisentraut wrote: > >> On tor, 2012-05-03 at 15:47 -0400, Bruce Momjian wrote: > >> > Peter, where are we on this? > >> > >> I had

Re: [HACKERS] Not quite a security hole: CREATE LANGUAGE for non-superusers

2012-05-30 Thread Tom Lane
Noah Misch writes: > I wondered about ALTER FUNCTION SET guc = '...' and tried to test it: > CREATE FUNCTION f(out ret text) RETURNS text LANGUAGE plpgsql AS > 'BEGIN ret := current_setting(''work_mem''); END'; > ALTER FUNCTION plpgsql_call_handler() SET work_mem = '2MB'

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Stephen Frost
* Sergey Koposov (kopo...@ast.cam.ac.uk) wrote: > I did a specific test with just 6 threads (== number of cores per cpu) > and ran it on a single phys cpu, it took ~ 12 seconds for each thread, > and when I tried to spread it across 4 cpus it took 7-9 seconds per > thread. But all these numbers are

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Stephen Frost
Sergey, all, * Sergey Koposov (kopo...@ast.cam.ac.uk) wrote: > I did a specific test with just 6 threads (== number of cores per cpu) > and ran it on a single phys cpu, it took ~ 12 seconds for each thread, > and when I tried to spread it across 4 cpus it took 7-9 seconds per > thread. But all the

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Jeff Janes
On Wed, May 30, 2012 at 4:16 PM, Sergey Koposov wrote: > But the question now is whether there is a *PG* problem here or not, or is > it Intel's or Linux's problem ? Because still the slowdown was caused by > locking. If there wouldn't be locking there wouldn't be any problems (as > demonstrated

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Sergey Koposov
On Thu, 31 May 2012, Florian Pflug wrote: Wait, so performance *increased* by spreading the backends out over as many dies as possible, not by using as few as possible? That'd be exactly the opposite of what I'd have expected. (I'm assuming that cores on one die have ascending ids on linux. If

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-30 Thread Tatsuo Ishii
>> Yeah, Jeff's experiments indicated that the remaining bottleneck is lock >> management in the server. What I fixed so far on the pg_dump side >> should be enough to let partial dumps run at reasonable speed even if >> the whole database contains many tables. But if psql is taking >> AccessShar

Re: [HACKERS] Figuring out shared buffer pressure

2012-05-30 Thread Jeff Janes
On Wed, May 30, 2012 at 2:55 PM, Bruce Momjian wrote: > On Wed, May 30, 2012 at 11:51:23AM -0700, Jeff Janes wrote: >> On Wed, May 30, 2012 at 11:23 AM, Bruce Momjian wrote: >> > On Wed, May 30, 2012 at 11:06:45AM -0700, Jeff Janes wrote: >> >> On Wed, May 30, 2012 at 10:57 AM, Bruce Momjian wro

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Florian Pflug
On May31, 2012, at 01:16 , Sergey Koposov wrote: > On Wed, 30 May 2012, Florian Pflug wrote: >> >> I wonder if the huge variance could be caused by non-uniform synchronization >> costs across different cores. That's not all that unlikely, because at least >> some cache levels (L2 and/or L3, I th

Re: [HACKERS] WalSndWakeup() and synchronous_commit=off

2012-05-30 Thread Andres Freund
On Thursday, May 31, 2012 01:33:33 AM Fujii Masao wrote: > On Wed, May 30, 2012 at 9:46 PM, Andres Freund wrote: > > On Tuesday, May 29, 2012 08:42:43 PM Andres Freund wrote: > >> Patch attached. > > > > Imo this patch should be backported to 9.1, 9.0 doesn't use latches and > > does not do expl

Re: [HACKERS] Not quite a security hole: CREATE LANGUAGE for non-superusers

2012-05-30 Thread Noah Misch
On Wed, May 30, 2012 at 12:02:06PM -0400, Tom Lane wrote: > One thing the owner *can* do is use ALTER FUNCTION to change secondary > properties of the function, such as strictness, volatility, SECURITY > DEFINER, etc. So far as I can see, none of these properties are examined > for a PL support fu

Re: [HACKERS] We're not lax enough about maximum time zone offset from UTC

2012-05-30 Thread Tom Lane
"David E. Wheeler" writes: > On May 30, 2012, at 3:10 PM, Tom Lane wrote: >> However, as pointed out by Patric, if you dump and restore an old >> timestamptz value in one of these zones, it will fail to restore because >> of the sanity check. I think therefore that we'd better enlarge the >> allo

Re: [HACKERS] WalSndWakeup() and synchronous_commit=off

2012-05-30 Thread Fujii Masao
On Wed, May 30, 2012 at 9:46 PM, Andres Freund wrote: > On Tuesday, May 29, 2012 08:42:43 PM Andres Freund wrote: >> Patch attached. > Imo this patch should be backported to 9.1, 9.0 doesn't use latches and does > not do explicit wakeup of the sender so its not applicable there. > > I can prepare

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Sergey Koposov
On Wed, 30 May 2012, Florian Pflug wrote: I wonder if the huge variance could be caused by non-uniform synchronization costs across different cores. That's not all that unlikely, because at least some cache levels (L2 and/or L3, I think) are usually shared between all cores on a single die. T

Re: [HACKERS] We're not lax enough about maximum time zone offset from UTC

2012-05-30 Thread David E. Wheeler
On May 30, 2012, at 3:10 PM, Tom Lane wrote: > However, as pointed out by Patric, if you dump and restore an old > timestamptz value in one of these zones, it will fail to restore because > of the sanity check. I think therefore that we'd better enlarge the > allowed range to 15:59:59 either way.

Re: [HACKERS] Early hint bit setting

2012-05-30 Thread Ants Aasma
On Thu, May 31, 2012 at 1:01 AM, Merlin Moncure wrote: > I think this is a really neat idea, and could solve a lot of problems. >  Since you don't have to do any clog checks (you know when you commit) > -- i think it's a win all around -- so much so that it might be worth > seeing the worst case l

[HACKERS] patch: avoid heavyweight locking on hash metapage

2012-05-30 Thread Robert Haas
I developed the attached patch to avoid taking a heavyweight lock on the metapage of a hash index. Instead, an exclusive buffer content lock is viewed as sufficient permission to modify the metapage, and a shared buffer content lock is used when such modifications need to be prevented. For the mo

[HACKERS] We're not lax enough about maximum time zone offset from UTC

2012-05-30 Thread Tom Lane
Currently, our datetime input code thinks that any UTC offset of more than 14:59:59 either way from Greenwich must be a mistake. However, after seeing Patric Bechtel's recent bug report, I went trolling in the Olson timezone files to see what are the largest offsets used there. I found three entri

Re: [HACKERS] Early hint bit setting

2012-05-30 Thread Merlin Moncure
On Wed, May 30, 2012 at 4:42 PM, Ants Aasma wrote: > I was thinking about what is the earliest time where we could set hint > bits. This would be just after the commit has been made visible. When > the transaction completes and commit confirmation is sent to the > client the backend will usually g

Re: [HACKERS] FailedAssertion("!(PrivateRefCount[i] == 0)", File: "bufmgr.c", Line: 1741

2012-05-30 Thread Erik Rijkers
On Wed, May 30, 2012 22:25, Robert Haas wrote: > On Wed, May 30, 2012 at 2:52 PM, Robert Haas wrote: >> On Wed, May 30, 2012 at 1:47 PM, Robert Haas wrote: The process holding the AccessExclusiveLock is the startup process. It's holding the lock on behalf of the transaction in the maste

Re: [HACKERS] Figuring out shared buffer pressure

2012-05-30 Thread Bruce Momjian
On Wed, May 30, 2012 at 11:51:23AM -0700, Jeff Janes wrote: > On Wed, May 30, 2012 at 11:23 AM, Bruce Momjian wrote: > > On Wed, May 30, 2012 at 11:06:45AM -0700, Jeff Janes wrote: > >> On Wed, May 30, 2012 at 10:57 AM, Bruce Momjian wrote: > >> > On Wed, May 30, 2012 at 10:38:10AM -0700, Jeff Ja

[HACKERS] Early hint bit setting

2012-05-30 Thread Ants Aasma
I was thinking about what is the earliest time where we could set hint bits. This would be just after the commit has been made visible. When the transaction completes and commit confirmation is sent to the client the backend will usually go to sleep waiting on the network socket waiting for further

Re: [HACKERS] Fake async rep target

2012-05-30 Thread Florian Pflug
On May30, 2012, at 22:28 , james wrote: > Well, I was assuming that there was some intelligence in the receiver that > could effectively parse this for the application; are you suggesting that is > effectively binary deltas to apply to raw pages? In parts. The log that is streamed to replication

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Florian Pflug
On May30, 2012, at 22:07 , Sergey Koposov wrote: > If I restart the db the timings do not change significantly. There is always > some variation which I don't really understand, e.g. the parallel runs > sometimes > take 18s, or 25 seconds, or 30 seconds per thread. So there is something else > a

Re: [HACKERS] GiST subsplit question

2012-05-30 Thread Alexander Korotkov
On Wed, May 30, 2012 at 11:21 PM, Jeff Davis wrote: > I looked for the follow-up commit to support subsplit in the contrib > modules, figuring that would answer some questions, but I couldn't find > it. > > The part that's confusing me is that the commit message says: "pickSplit > should set spl_

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Sergey Koposov
On Wed, 30 May 2012, Merlin Moncure wrote: How big is idt_match?  What if you drop all indexes on idt_match, encouraging all the backends to do hash joins against it, which occur in local memory and so don't have contention? You just missed his post -- it's only 3G. can you run your 'small'

Re: [HACKERS] Fake async rep target

2012-05-30 Thread james
Well, I was assuming that there was some intelligence in the receiver that could effectively parse this for the application; are you suggesting that is effectively binary deltas to apply to raw pages? Certainly, Sybase rep server works by creating function calls or SQL updates (depending on ho

Re: [HACKERS] Fake async rep target

2012-05-30 Thread james
Well, I was assuming that there was some intelligence in the receiver that could effectively parse this for the application; are you suggesting that is effectively binary deltas to apply to raw pages? Certainly, Sybase rep server works by creating function calls or SQL updates (depending on ho

Re: [HACKERS] Advisory locks seem rather broken

2012-05-30 Thread Robert Haas
On Fri, May 4, 2012 at 9:17 AM, Robert Haas wrote: > On Thu, May 3, 2012 at 5:18 PM, Tom Lane wrote: >> ... btw, it appears to me that the "fast path" patch has broken things >> rather badly in LockReleaseAll.  AFAICS it's not honoring either the >> lockmethodid restriction nor the allLocks restr

Re: [HACKERS] FailedAssertion("!(PrivateRefCount[i] == 0)", File: "bufmgr.c", Line: 1741

2012-05-30 Thread Robert Haas
On Wed, May 30, 2012 at 2:52 PM, Robert Haas wrote: > On Wed, May 30, 2012 at 1:47 PM, Robert Haas wrote: >>> The process holding the AccessExclusiveLock is the startup process. It's >>> holding the lock on behalf of the transaction in the master. But something's >>> wrong, and the AccessExclusiv

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Merlin Moncure
On Wed, May 30, 2012 at 3:15 PM, Jeff Janes wrote: > On Wed, May 30, 2012 at 11:45 AM, Sergey Koposov > wrote: >> On Wed, 30 May 2012, Merlin Moncure wrote: >>> >>> >>> Hm, why aren't we getting a IOS?  Just for kicks (assuming this is >>> test data), can we drop the index on just transitid, lea

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Jeff Janes
On Wed, May 30, 2012 at 11:45 AM, Sergey Koposov wrote: > On Wed, 30 May 2012, Merlin Moncure wrote: >> >> >> Hm, why aren't we getting a IOS?  Just for kicks (assuming this is >> test data), can we drop the index on just transitid, leaving the index >> on transitid, healpixid?    Is enable_indexo

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Sergey Koposov
On Wed, 30 May 2012, Merlin Moncure wrote: hurk -- ISTM that since IOS is masikng the heap lookups, there must be contention on the index itself? Does this working set fit in shared memory? If so, what happens when you do a database restart and repeat the IOS test? The dataset fits well in

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Merlin Moncure
On Wed, May 30, 2012 at 1:45 PM, Sergey Koposov wrote: > On Wed, 30 May 2012, Merlin Moncure wrote: >> >> >> Hm, why aren't we getting a IOS?  Just for kicks (assuming this is >> test data), can we drop the index on just transitid, leaving the index >> on transitid, healpixid?    Is enable_indexon

Re: [HACKERS] [RFC] Interface of Row Level Security

2012-05-30 Thread Kohei KaiGai
2012/5/29 Robert Haas : > On Fri, May 25, 2012 at 5:08 PM, Kohei KaiGai wrote: I think it is a good idea not to apply RLS when current user has superuser privilege from perspective of security model consistency, but it is inconsistent to check privileges underlying tables. >>> >>> S

[HACKERS] GiST subsplit question

2012-05-30 Thread Jeff Davis
http://archives.postgresql.org/message-id/CAEsn3ybKcnFno_tGQDJ=afhir2xd9ka1fqt5cqxxyu3wz_h...@mail.gmail.com I was trying to answer that question on -general, and I found it a little more challenging than I expected. There was a previous discussion here: http://archives.postgresql.org/pgsql-

[HACKERS] pg_restore logging inconsistency

2012-05-30 Thread Josh Kupershmidt
Hi all, Bosco Rama recently complained[1] about not seeing a message printed by pg_restore for each LO to be restored. The culprit seems to be the different level passed to ahlog() for this status message: pg_backup_archiver.c: ahlog(AH, 2, "restoring large object with OID %u\n", oid); pg_back

Re: [HACKERS] FailedAssertion("!(PrivateRefCount[i] == 0)", File: "bufmgr.c", Line: 1741

2012-05-30 Thread Robert Haas
On Wed, May 30, 2012 at 1:47 PM, Robert Haas wrote: >> The process holding the AccessExclusiveLock is the startup process. It's >> holding the lock on behalf of the transaction in the master. But something's >> wrong, and the AccessExclusiveLock doesn't stop a regular backend from >> acquiring the

Re: [HACKERS] Figuring out shared buffer pressure

2012-05-30 Thread Jeff Janes
On Wed, May 30, 2012 at 11:23 AM, Bruce Momjian wrote: > On Wed, May 30, 2012 at 11:06:45AM -0700, Jeff Janes wrote: >> On Wed, May 30, 2012 at 10:57 AM, Bruce Momjian wrote: >> > On Wed, May 30, 2012 at 10:38:10AM -0700, Jeff Janes wrote: >> >> >> >> Isn't that what the buffers_alloc from pg_sta

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Sergey Koposov
On Wed, 30 May 2012, Merlin Moncure wrote: Hm, why aren't we getting a IOS? Just for kicks (assuming this is test data), can we drop the index on just transitid, leaving the index on transitid, healpixid?Is enable_indexonlyscan on? Has idt_match been vacuumed? What kind of plan do you get

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Merlin Moncure
On Wed, May 30, 2012 at 12:11 PM, Robert Haas wrote: > On Wed, May 30, 2012 at 12:58 PM, Sergey Koposov > wrote: >> Here is the one to one comparison of the 'bogged' ** >> >>  QUERY PLAN >> --

Re: [HACKERS] Figuring out shared buffer pressure

2012-05-30 Thread Bruce Momjian
On Wed, May 30, 2012 at 11:06:45AM -0700, Jeff Janes wrote: > On Wed, May 30, 2012 at 10:57 AM, Bruce Momjian wrote: > > On Wed, May 30, 2012 at 10:38:10AM -0700, Jeff Janes wrote: > >> On Wed, May 30, 2012 at 9:56 AM, Bruce Momjian wrote: > >> > As part of a blog, I started looking at how a user

Re: [HACKERS] Figuring out shared buffer pressure

2012-05-30 Thread Jeff Janes
On Wed, May 30, 2012 at 10:57 AM, Bruce Momjian wrote: > On Wed, May 30, 2012 at 10:38:10AM -0700, Jeff Janes wrote: >> On Wed, May 30, 2012 at 9:56 AM, Bruce Momjian wrote: >> > As part of a blog, I started looking at how a user could measure the >> > pressure on shared buffers, e.g. how much ar

Re: [HACKERS] Figuring out shared buffer pressure

2012-05-30 Thread Bruce Momjian
On Wed, May 30, 2012 at 10:38:10AM -0700, Jeff Janes wrote: > On Wed, May 30, 2012 at 9:56 AM, Bruce Momjian wrote: > > As part of a blog, I started looking at how a user could measure the > > pressure on shared buffers, e.g. how much are they being used, recycled, > > etc. > > > > They way you no

Re: [HACKERS] FailedAssertion("!(PrivateRefCount[i] == 0)", File: "bufmgr.c", Line: 1741

2012-05-30 Thread Robert Haas
On Wed, May 30, 2012 at 1:07 PM, Heikki Linnakangas wrote: > There's something wrong with the way AccessExclusiveLocks work on a standby. > I did "begin; truncate foo; -- leave the xact open" in the master, and > waited until the xlog records are shipped to the standby. Then I did this in > the st

Re: [HACKERS] Figuring out shared buffer pressure

2012-05-30 Thread Jeff Janes
On Wed, May 30, 2012 at 9:56 AM, Bruce Momjian wrote: > As part of a blog, I started looking at how a user could measure the > pressure on shared buffers, e.g. how much are they being used, recycled, > etc. > > They way you normally do it on older operating systems is to see how > many buffers on

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Robert Haas
On Wed, May 30, 2012 at 12:58 PM, Sergey Koposov wrote: > Here is the one to one comparison of the 'bogged' ** > >  QUERY PLAN > -

Re: [HACKERS] FailedAssertion("!(PrivateRefCount[i] == 0)", File: "bufmgr.c", Line: 1741

2012-05-30 Thread Heikki Linnakangas
On 26.05.2012 12:21, Erik Rijkers wrote: But when that if-block is added the client crashes after a while (sometimes almost immediately; it never survives longer then 20 minutes): 2012-05-26 10:44:22.617 CEST 10274 ERROR: could not fsync file "base/21268/32807": No such file or directory 2012

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Sergey Koposov
On Wed, 30 May 2012, Merlin Moncure wrote: 1. Can we see an explain analyze during a 'bogged' case? Here is the one to one comparison of the 'bogged' ** QUERY PLAN

[HACKERS] Figuring out shared buffer pressure

2012-05-30 Thread Bruce Momjian
As part of a blog, I started looking at how a user could measure the pressure on shared buffers, e.g. how much are they being used, recycled, etc. They way you normally do it on older operating systems is to see how many buffers on the free list (about to be reused) are reclaimed as needed --- tha

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-05-30 Thread Ants Aasma
On Wed, May 30, 2012 at 2:10 PM, Heikki Linnakangas wrote: > Also, I wonder if DropRelFileNodeBuffers() could scan the pool without > grabbing the spinlocks on every buffer? It could do an unlocked test first, > and only grab the spinlock on buffers that need to be dropped. The scanning of buffer

Re: [HACKERS] too low NAPTIME_PER_CYCLE /too many wakeups in walreceiver.c

2012-05-30 Thread Fujii Masao
On Wed, May 30, 2012 at 9:57 PM, Andres Freund wrote: > Hi, > > Currently the walreceiver wakeups NAPTIME_PER_CYCLE=100 miliseconds in idle > state. This is rather frequent. I don't really see any reason to do so. > A nice fix would be to latchify that with WaitLatchOrSocket + a SetLatch in > the

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Merlin Moncure
On Wed, May 30, 2012 at 10:42 AM, Sergey Koposov wrote: > Here is the actual explain analyze of the query on the smaller dataset > which I have been using for the recent testing. > > test=# explain analyze create table _tmp0 as select * from > >  ( select *, >        (select healpixid from idt_mat

Re: [HACKERS] Uh, I change my mind about commit_delay + commit_siblings (sort of)

2012-05-30 Thread Peter Geoghegan
On 30 May 2012 15:25, Simon Riggs wrote: >> 1. It seems wrong to do it in xact_redo_commit_internal().  It won't >> matter if commit_siblings>0 since there won't be any other backends >> with transaction IDs anyway, but if commit_siblings==0 then we'll >> sleep for no possible benefit. > > Agreed

Re: [HACKERS] Re: [COMMITTERS] pgsql: Send new protocol keepalive messages to standby servers.

2012-05-30 Thread Fujii Masao
On Wed, May 30, 2012 at 4:34 AM, Simon Riggs wrote: > On 24 May 2012 21:11, Robert Haas wrote: >> On Thu, May 24, 2012 at 2:52 PM, Tom Lane wrote: >>> Robert Haas writes: On Wed, May 23, 2012 at 2:28 PM, Fujii Masao wrote: > Is there plan to implement such external functions before 9.

[HACKERS] Not quite a security hole: CREATE LANGUAGE for non-superusers

2012-05-30 Thread Tom Lane
We allow non-superuser database owners to execute CREATE LANGUAGE for a trusted language (one marked as tmpldbacreate in pg_pltemplate). Currently, the C-language support functions for the language end up owned by that non-superuser. This is on the hairy edge of being a security hole, since genera

[HACKERS] remembering locks and dynahash.c

2012-05-30 Thread Jeff Janes
Currently the resource owner does not remember what locks it holds. When a resource owner wants to release its locks or reassign them to its parent, it just digs through the backends entire LockMethodLocalHash table. When that table is very large, but the current owner owns only a small fraction o

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Sergey Koposov
Here is the actual explain analyze of the query on the smaller dataset which I have been using for the recent testing. test=# explain analyze create table _tmp0 as select * from ( select *, (select healpixid from idt_match as m where m.transitid=o.transitid) as x from id

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-05-30 Thread Jeff Janes
On Wed, May 30, 2012 at 4:10 AM, Heikki Linnakangas wrote: > On 30.05.2012 03:40, Sergey Koposov wrote: >> >> I was running some tests on PG9.2beta where I'm creating and dropping >> large number of tables (~ 2). >> >> And I noticed that table dropping was extremely slow -- e.g. like half a >>

Re: [HACKERS] Function call hierarchy/path since getting the buffer until access its data

2012-05-30 Thread Robert Haas
On Wed, May 30, 2012 at 9:37 AM, Waldecir Faria wrote: > Thank you for the reply Robert. I think I am getting the idea about reading > buffers but I am confused about the writing part, can you give me a function > name where it does some write operations like creating a table or inserting > a tupl

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Merlin Moncure
On Sun, May 27, 2012 at 1:45 PM, Sergey Koposov wrote: > Hi, > > I did another test using the same data and the same code, which I've > provided before and the performance of the single thread seems to be > degrading quadratically with the number of threads. > > Here are the results: > Nthreads Ti

Re: [HACKERS] Uh, I change my mind about commit_delay + commit_siblings (sort of)

2012-05-30 Thread Simon Riggs
On 30 May 2012 13:24, Robert Haas wrote: > OK, but there are a lot of places where we call XLogFlush(), and it's > far from obvious that it's a win to do this in all of those cases.  At > least, nobody's done that analysis.  XLogFlush is called from: > > - WriteTruncateXlogRec(), which is called

Re: [HACKERS] How could we make it simple to access the log as a table?

2012-05-30 Thread Marc Mamin
> The CSV format is both rich and > machine-parseable (good start!) but it takes an unreasonable amount of > work to make it usefully queryable. We need something that looks more > like a big red button. Hello, The Pg logs consist of a rich soup with many different information kind. It's comfor

Re: [HACKERS] Function call hierarchy/path since getting the buffer until access its data

2012-05-30 Thread Waldecir Faria
Thank you for the reply Robert. I think I am getting the idea about reading buffers but I am confused about the writing part, can you give me a function name where it does some write operations like creating a table or inserting a tuple for me read as a example. Best Regards , -Waldecir > Da

Re: [HACKERS] hash index concurrency

2012-05-30 Thread Merlin Moncure
On Wed, May 30, 2012 at 3:49 AM, Simon Riggs wrote: > On 30 May 2012 04:54, Robert Haas wrote: > >>> This was a hobby horse of mine a couple of years ago, but I never got >>> much traction.  The main question I have is, what do we even want hash >>> indexes to be?  NBTree is very good, has been e

[HACKERS] too low NAPTIME_PER_CYCLE /too many wakeups in walreceiver.c

2012-05-30 Thread Andres Freund
Hi, Currently the walreceiver wakeups NAPTIME_PER_CYCLE=100 miliseconds in idle state. This is rather frequent. I don't really see any reason to do so. A nice fix would be to latchify that with WaitLatchOrSocket + a SetLatch in the signal handler for shutdown but that seems to be too invasive at

Re: [HACKERS] Uh, I change my mind about commit_delay + commit_siblings (sort of)

2012-05-30 Thread Peter Geoghegan
On 30 May 2012 13:24, Robert Haas wrote: > Most of those actually do look like reasonable places to try to get > grouped flushing behavior, but: > > 1. It seems wrong to do it in xact_redo_commit_internal().  It won't > matter if commit_siblings>0 since there won't be any other backends > with tra

Re: [HACKERS] WalSndWakeup() and synchronous_commit=off

2012-05-30 Thread Andres Freund
On Tuesday, May 29, 2012 08:42:43 PM Andres Freund wrote: > Patch attached. Imo this patch should be backported to 9.1, 9.0 doesn't use latches and does not do explicit wakeup of the sender so its not applicable there. I can prepare a patch for 9.1 if people agree, there has been some amount of

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-05-30 Thread Robert Haas
On Wed, May 30, 2012 at 7:10 AM, Heikki Linnakangas wrote: > So we drop the buffers for each relation fork separately, which means that > we scan the buffer pool four times. Relation forks in 8.4 introduced that > issue, and 9.1 made it worse by adding another fork for unlogged tables. > With some

Re: [HACKERS] Uh, I change my mind about commit_delay + commit_siblings (sort of)

2012-05-30 Thread Robert Haas
On Wed, May 30, 2012 at 4:36 AM, Simon Riggs wrote: > When I read this the first time, I was in full agreement. > > On closer inspection neither point is valid, though both points were > worth considering. > >> Well, consider the one in the background writer, for example.  That's >> just a periodi

Re: [HACKERS] How do I get the name of the relation on which FDW has been called?

2012-05-30 Thread Atri Sharma
On Wed, May 30, 2012 at 5:15 PM, Shigeru HANADA wrote: > Hi Atri, > > (2012/05/30 19:49), Atri Sharma wrote: >> SELECT * FROM table1; >> >> If,for above command,fdw1 is invoked,how do I get the name/Oid of the >> table(i.e. table1) in fdw1? > > For 9.1 and 9.0, you can get foreign table's oid as t

Re: [HACKERS] How do I get the name of the relation on which FDW has been called?

2012-05-30 Thread Shigeru HANADA
Hi Atri, (2012/05/30 19:49), Atri Sharma wrote: > SELECT * FROM table1; > > If,for above command,fdw1 is invoked,how do I get the name/Oid of the > table(i.e. table1) in fdw1? For 9.1 and 9.0, you can get foreign table's oid as the first parameter of PlanForeignScan function. For 9.2, you can g

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-05-30 Thread Heikki Linnakangas
On 30.05.2012 03:40, Sergey Koposov wrote: I was running some tests on PG9.2beta where I'm creating and dropping large number of tables (~ 2). And I noticed that table dropping was extremely slow -- e.g. like half a second per table. ... I also stopped PG with gdb a few times and it was al

[HACKERS] How do I get the name of the relation on which FDW has been called?

2012-05-30 Thread Atri Sharma
SELECT * FROM table1; If,for above command,fdw1 is invoked,how do I get the name/Oid of the table(i.e. table1) in fdw1? Atri -- Regards, Atri l'apprenant -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mail

Re: [HACKERS] Bug in new buffering GiST build code

2012-05-30 Thread Alexander Korotkov
On Wed, May 30, 2012 at 1:01 PM, Heikki Linnakangas < heikki.linnakan...@enterprisedb.com> wrote: > I also spotted and fixed another little oversight: the temporary file > didn't get deleted after the index build. > I've one note not directly related to buffering build. While I debugging bufferin

Re: [HACKERS] Bug in new buffering GiST build code

2012-05-30 Thread Heikki Linnakangas
On 28.05.2012 00:46, Alexander Korotkov wrote: On Sat, May 26, 2012 at 12:33 AM, Heikki Linnakangas< heikki.linnakan...@enterprisedb.com> wrote: Attached is a patch to replace the path stacks with a hash table. With this patch, the index build time in my test case dropped from 59 minutes to ab

Re: [HACKERS] hash index concurrency

2012-05-30 Thread Simon Riggs
On 30 May 2012 04:54, Robert Haas wrote: >> This was a hobby horse of mine a couple of years ago, but I never got >> much traction.  The main question I have is, what do we even want hash >> indexes to be?  NBTree is very good, has been extensively optimized, >> and extensively tested.  If there

Re: [HACKERS] Uh, I change my mind about commit_delay + commit_siblings (sort of)

2012-05-30 Thread Simon Riggs
On 29 May 2012 17:58, Robert Haas wrote: > On Tue, May 29, 2012 at 12:47 PM, Peter Geoghegan > wrote: >> Why do you think that doing this for all XLogFlush() callsites might >> be problematic? > > Well, consider the one in the background writer, for example.  That's > just a periodic flush, so I