Re: [HACKERS] Performance Improvement by reducing WAL for Update Operation

2013-12-05 Thread Amit Kapila
On Fri, Dec 6, 2013 at 12:10 PM, Haribabu kommi wrote: > On 05 December 2013 21:16 Amit Kapila wrote: >> Note - >> a. Performance is data is taken on my laptop, needs to be tested on >> some better m/c b. Attached Patch is just a prototype of chunkwise >> concept, code needs to be improved and dec

Re: [HACKERS] Performance Improvement by reducing WAL for Update Operation

2013-12-05 Thread Haribabu kommi
On 05 December 2013 21:16 Amit Kapila wrote: > Note - > a. Performance is data is taken on my laptop, needs to be tested on > some better m/c b. Attached Patch is just a prototype of chunkwise > concept, code needs to be improved and decode > handling/test is pending. I ran the performance te

Re: [HACKERS] Heavily modified big table bloat even in auto vacuum is running

2013-12-05 Thread Amit Kapila
On Fri, Nov 29, 2013 at 6:55 PM, Haribabu kommi wrote: > On 29 November 2013 12:00 Amit Kapila wrote: >> On Tue, Nov 26, 2013 at 7:26 PM, Haribabu kommi >> wrote: >> Few questions about your latest patch: >> a. Is there any reason why you are doing estimation of dead tuples only >> for Autovacuum

Re: [HACKERS] [PATCH] Add transforms feature

2013-12-05 Thread Peter Eisentraut
On Tue, 2013-11-26 at 18:10 +0100, Dimitri Fontaine wrote: > The problem is installing a set of extensions where some of them are > already using the new transform feature and some of them are not. We > need a way to cater with that, I think. Here is an idea. Add a GUC that basically says somethi

[HACKERS] spinlocks storm bug

2013-12-05 Thread Pavel Stehule
Hello I have a report of critical bug (database is temporary unavailability .. restart is necessary). A customer use: PostgreSQL 9.2.4, 24 CPU 140G RAM SSD disc for all Database is under high load. There is a few databases with very high number of similar simple statements. When application p

Re: [HACKERS] Extra functionality to createuser

2013-12-05 Thread Peter Eisentraut
On Wed, 2013-11-20 at 11:23 -0500, Christopher Browne wrote: > I note that similar (with not quite identical behaviour) issues apply > to the user name. Perhaps the > resolution to this is to leave quoting issues to the administrator. > That simplifies the problem away. How about only one role na

Re: [HACKERS] dblink performance regression

2013-12-05 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 12/05/2013 07:16 PM, Fabrízio de Royes Mello wrote: > Hi Joe, how are you? Hi Fabrizio -- great to hear from you! I'm well. > Well, when Tom sent this email I was reviewing your patch and the > main suggestion is about use of 'pg_encoding_to_char'

Re: [HACKERS] Regression tests failing if not launched on db "regression"

2013-12-05 Thread Michael Paquier
On Fri, Dec 6, 2013 at 12:02 PM, Peter Eisentraut wrote: > On Thu, 2013-12-05 at 17:12 +0900, Michael Paquier wrote: >> IMHO, the regression test suite would gain in consistency and >> portability if we do not refer to data that is database-dependent. > > I once did some similar fixes (e3d9dceef62

Re: [HACKERS] Time-Delayed Standbys

2013-12-05 Thread Fabrízio de Royes Mello
On Tue, Dec 3, 2013 at 5:33 PM, Simon Riggs wrote: > > > - compute recoveryUntilDelayTime in XLOG_XACT_COMMIT and > > XLOG_XACT_COMMIT_COMPACT checks > > Why just those? Why not aborts and restore points also? > I think make no sense execute the delay after aborts and/or restore points, because i

Re: [HACKERS] RFC: programmable file format for postgresql.conf

2013-12-05 Thread Peter Eisentraut
On Thu, 2013-12-05 at 00:51 +0100, Álvaro Hernández Tortosa wrote: > In return for this extra information, we: > > - Provide users with more help (information) to help them configure > postgres (which is no easy task, specially for newcomers). > > - Help and encourage app developers to create bo

Re: [HACKERS] Performance optimization of btree binary search

2013-12-05 Thread Peter Eisentraut
On Thu, 2013-12-05 at 15:29 +0200, Heikki Linnakangas wrote: > It happens to me about 75% of the time when I write a new C function. > These days, I usually realize it pretty quickly. > > I wonder how easy it would be to make the compiler produce a warning > about it. Or issue a warning in Postg

Re: [HACKERS] UNNEST with multiple args, and TABLE with multiple funcs

2013-12-05 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: > Noah Misch writes: > > Two naming proposals, "ROWS FROM" and "TABLE FROM", got an ACK from more > > than > > one person apiece. I move that we settle on "ROWS FROM". > > I'm not sufficiently annoyed by "ROWS FROM" to object. Other opinions? Works well

Re: [HACKERS] UNNEST with multiple args, and TABLE with multiple funcs

2013-12-05 Thread Tom Lane
Noah Misch writes: > Two naming proposals, "ROWS FROM" and "TABLE FROM", got an ACK from more than > one person apiece. I move that we settle on "ROWS FROM". I'm not sufficiently annoyed by "ROWS FROM" to object. Other opinions? regards, tom lane -- Sent via pgsql-ha

Re: [HACKERS] dblink performance regression

2013-12-05 Thread Fabrízio de Royes Mello
On Fri, Dec 6, 2013 at 1:05 AM, Joe Conway wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > On 12/05/2013 06:53 PM, Tom Lane wrote: > > I seem to remember that at some point we realized that the encoding > > ID assignments are part of libpq's ABI and so can't practically be > > change

Re: [HACKERS] WITHIN GROUP patch

2013-12-05 Thread Tom Lane
David Johnston writes: > The original design references the spec as allowing a column reference if it > is a group by key: > Select cume_dist(f1) within group ( order by f1 ) from text_tbl group by f1 > No example was shown where this would be useful...but nonetheless the > comment and error bot

Re: [HACKERS] dblink performance regression

2013-12-05 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 12/05/2013 06:53 PM, Tom Lane wrote: > I seem to remember that at some point we realized that the encoding > ID assignments are part of libpq's ABI and so can't practically be > changed ever, so the above may be moot. Even so, I think it's a > bad

Re: [HACKERS] Regression tests failing if not launched on db "regression"

2013-12-05 Thread Peter Eisentraut
On Thu, 2013-12-05 at 17:12 +0900, Michael Paquier wrote: > IMHO, the regression test suite would gain in consistency and > portability if we do not refer to data that is database-dependent. I once did some similar fixes (e3d9dceef62e072cf9a433ae6c74a1c5a10d94d3) but then didn't pursue it any long

Re: [HACKERS] dblink performance regression

2013-12-05 Thread Tom Lane
Joe Conway writes: > Apparently setting client encoding is an expensive operation. Yeah, it requires sending a command to the remote end. I'm not sure if it'd be a good idea for PQsetClientEncoding to try to skip the operation when the client encoding is already the right thing. The given name

Re: [HACKERS] WITHIN GROUP patch

2013-12-05 Thread David Johnston
Tom Lane-2 wrote > Further questions about WITHIN GROUP: > > I believe that the spec requires that the "direct" arguments of an inverse > or hypothetical-set aggregate must not contain any Vars of the current > query level. They don't manage to say that in plain English, of course, > but in the

[HACKERS] dblink performance regression

2013-12-05 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I was recently approached by someone with a dblink performance regression, going back to somewhere between 8.3 (good) and 8.4 (bad). They were measuring ~8% degradation in dblink connection speed. I was able to replicate on my own hardware with the fol

Re: [HACKERS] ANALYZE sampling is too good

2013-12-05 Thread Peter Geoghegan
On Thu, Dec 5, 2013 at 3:50 PM, Josh Berkus wrote: > There are fairly well researched algorithms for block-based sampling > which estimate for the skew introduced by looking at consecutive rows in > a block. In general, a minimum sample size of 5% is required, and the > error is no worse than our

Re: [HACKERS] WITHIN GROUP patch

2013-12-05 Thread Tom Lane
Further questions about WITHIN GROUP: I believe that the spec requires that the "direct" arguments of an inverse or hypothetical-set aggregate must not contain any Vars of the current query level. They don't manage to say that in plain English, of course, but in the case the behavior is defined

Re: [HACKERS] UNNEST with multiple args, and TABLE with multiple funcs

2013-12-05 Thread Noah Misch
On Tue, Dec 03, 2013 at 02:27:06PM -0500, Tom Lane wrote: > Noah Misch writes: > > On Tue, Dec 03, 2013 at 10:03:32AM -0500, Stephen Frost wrote: > >> Alright, for my 2c, I like having this syntax include 'TABLE' simply > >> because it's what folks coming from Oracle might be looking for. > >> Fol

Re: [HACKERS] ANALYZE sampling is too good

2013-12-05 Thread Claudio Freire
On Tue, Dec 3, 2013 at 8:30 PM, Greg Stark wrote: > Worse, my experience with the posix_fadvise benchmarking is that on > spinning media reading one out of every 16 blocks takes about the same > time as reading them all. Presumably this is because the seek time > between tracks dominates and readi

Re: [HACKERS] Proposal: variant of regclass

2013-12-05 Thread Tom Lane
Tatsuo Ishii writes: > Can I make sure that we want to keep the current behavior: > test=# SELECT 'pg_klass'::regclass; > ERROR: relation "pg_klass" does not exist Yeah, I think the consensus is to not change the behavior of the input functions, just add some new ones.

Re: [HACKERS] ANALYZE sampling is too good

2013-12-05 Thread Josh Berkus
On 12/03/2013 03:30 PM, Greg Stark wrote: > It means if your table is anywhere up to 240MB you're effectively > doing a full table scan and then throwing out nearly all the data > read. There are lots of issues with our random sampling approach for ANALYZE. This is why, back in our Greenplum days

Re: [HACKERS] Proposal: variant of regclass

2013-12-05 Thread Tatsuo Ishii
> I'm getting less enamored of just-change-the-input-behavior myself. > The case that occurred to me is, suppose somebody's got a table containing > a regclass or regproc column, and he dumps and reloads it. If the input > converter silently replaces unknown names by 0, he's at risk of unexpected

Re: [HACKERS] How to detect invisible rows caused by the relfrozenxid bug?

2013-12-05 Thread Omar Kilani
Hi Robert, Sounds good. Is it safe to upgrade to 9.2.6 (we're on 9.2.5) in the mean time, or should we just leave things untouched? We (unknowingly) got hit by the slave replication bug 2 months ago, too. That was fun. :) Omar On Thu, Dec 5, 2013 at 2:48 PM, Robert Treat wrote: > On Thu, Dec 5

Re: [HACKERS] Reference to parent query from ANY sublink

2013-12-05 Thread Antonin Houska
On 10/31/2013 09:37 PM, Antonin Houska wrote: > On 10/31/2013 03:46 PM, Antonin Houska wrote: > I'm not sure if it's legal for the WHERE clause to reference LHS of the > original outer join (a.j). Some more restriction may be needed. I need > to think about it a bit more. For a subquery or sublink

Re: [HACKERS] pg_archivecleanup bug

2013-12-05 Thread Tom Lane
Kevin Grittner writes: > | Applications wishing to check for error situations should set > | errno to 0 before calling readdir(). If errno is set to non-zero > | on return, an error occurred. > So an error in scanning the directory will not be reported; the > cleanup will quietly terminate the WA

Re: [HACKERS] How to detect invisible rows caused by the relfrozenxid bug?

2013-12-05 Thread Robert Treat
On Thu, Dec 5, 2013 at 4:14 PM, Omar Kilani wrote: > Hi, > > How would one go about detecting whether they've lost rows due to the > relfrozenxid? > > Unfortunately running 'SELECT txid_current() < 2^31' on our DB returns > false, and I'm a little bit worried, since we've been seeing some > WeirdS

Re: [HACKERS] Regression tests failing if not launched on db "regression"

2013-12-05 Thread Michael Paquier
> On 2013/12/06, at 3:52, Tom Lane wrote: > > Robert Haas writes: >>> On Thu, Dec 5, 2013 at 9:24 AM, Tom Lane wrote: >>> Michael Paquier writes: It happens that the following regression tests are failing if they are run on a database not named "regression": > >>> This does not

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-05 Thread bricklen
On Thu, Dec 5, 2013 at 12:43 PM, Josh Berkus wrote: > On 12/05/2013 12:41 PM, Robert Haas wrote: > > Do drunks lurch differently in cathedrals than they do elsewhere? > > Yeah, because they lurch from one column to another. > Row by row?

[HACKERS] How to detect invisible rows caused by the relfrozenxid bug?

2013-12-05 Thread Omar Kilani
Hi, How would one go about detecting whether they've lost rows due to the relfrozenxid? Unfortunately running 'SELECT txid_current() < 2^31' on our DB returns false, and I'm a little bit worried, since we've been seeing some WeirdStuff(tm) lately. We're "only" 200M txids or so past 2^31. Thanks

Re: [HACKERS] Dynamic Shared Memory stuff

2013-12-05 Thread Heikki Linnakangas
On 12/05/2013 09:34 PM, Robert Haas wrote: On Thu, Dec 5, 2013 at 11:12 AM, Heikki Linnakangas wrote: One idea is to create the shared memory object with shm_open, and wait until all the worker processes that need it have attached to it. Then, shm_unlink() it, before using it for anything. That

Re: [HACKERS] Proof of concept: standalone backend with full FE/BE protocol

2013-12-05 Thread Heikki Linnakangas
On 12/05/2013 10:37 PM, Robert Haas wrote: On Thu, Dec 5, 2013 at 3:05 PM, Tom Lane wrote: It might be unpleasant to use in some cases, though. Why would there be more than a few cases in the first place? Who is going to use this beyond psql, pg_dump(all), and pg_upgrade, and why? Well, yo

Re: [HACKERS] Re: [BUGS] BUG #7873: pg_restore --clean tries to drop tables that don't exist

2013-12-05 Thread Dean Rasheed
On 5 December 2013 10:06, Pavel Stehule wrote: >> I think #1175 is close to being ready for commit. Pavel, will you >> produce an updated patch based on our last discussion? I'll set this >> patch to waiting on author. > > > I expected so your version was a final. I have no problem to do other > e

Re: [HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?

2013-12-05 Thread Alvaro Herrera
Josh Berkus wrote: > Heck, I'd be happy just to have a class of messages which specifically > means "OMG, there's something wrong with the server", that is, a flag > for messages which only occur when PostgreSQL encounters a bug, data > corrpution, or platform error. Right now, I have to suss tho

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-05 Thread Josh Berkus
On 12/05/2013 12:41 PM, Robert Haas wrote: > On Thu, Dec 5, 2013 at 12:54 PM, Josh Berkus wrote: >> Actually, I've been able to do 35K TPS on commodity hardware on Ubuntu >> 10.04. I have yet to go about 15K on any Ubuntu running a 3.X Kernel. >> The CPU scheduling on 2.6 just seems to be far b

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-05 Thread Robert Haas
On Thu, Dec 5, 2013 at 12:54 PM, Josh Berkus wrote: > Actually, I've been able to do 35K TPS on commodity hardware on Ubuntu > 10.04. I have yet to go about 15K on any Ubuntu running a 3.X Kernel. > The CPU scheduling on 2.6 just seems to be far better tuned, aside from > the IO issues; at 35K

Re: [HACKERS] pg_archivecleanup bug

2013-12-05 Thread Robert Haas
On Thu, Dec 5, 2013 at 3:06 PM, Kevin Grittner wrote: > An EDB customer reported a problem with pg_archivecleanup which I > have looked into and found a likely cause. It is, in any event, a > bug which I think should be fixed. It has to do with our use of > the readdir() function: > > http://pub

Re: [HACKERS] Proof of concept: standalone backend with full FE/BE protocol

2013-12-05 Thread Robert Haas
On Thu, Dec 5, 2013 at 3:05 PM, Tom Lane wrote: > I'm pretty much persuaded by Andres' point that we should not allow a > child process to be launched under a client app without clear permission > from the code of the app (and *not* just some environment variable that > might have been set far awa

Re: [HACKERS] Proof of concept: standalone backend with full FE/BE protocol

2013-12-05 Thread Tom Lane
Robert Haas writes: > Yeah, seriously. I don't understand what the big deal is here. The > right design here is 99.44% clear here, and the committer (presumably > Tom) can handle the other 0.56% however he'd like. Let's do this and > move on. Yeah, but the remaining 0.56% is an important decis

[HACKERS] pg_archivecleanup bug

2013-12-05 Thread Kevin Grittner
An EDB customer reported a problem with pg_archivecleanup which I have looked into and found a likely cause.  It is, in any event, a bug which I think should be fixed.  It has to do with our use of the readdir() function: http://pubs.opengroup.org/onlinepubs/7908799/xsh/readdir_r.html These are t

Re: [HACKERS] Proof of concept: standalone backend with full FE/BE protocol

2013-12-05 Thread Robert Haas
On Thu, Dec 5, 2013 at 11:52 AM, Andres Freund wrote: > On 2013-12-05 11:39:29 -0500, Peter Eisentraut wrote: >> I think this proposal is a bit deadlocked now. >> >> - There are technical concerns about launching a server executable from >> within a client. >> >> - There are conceptual concerns ab

Re: [HACKERS] Dynamic Shared Memory stuff

2013-12-05 Thread Robert Haas
On Thu, Dec 5, 2013 at 11:12 AM, Heikki Linnakangas wrote: > Hmm. Those two use cases are quite different. For message-passing, you want > a lot of small queues, but for parallel sort, you want one huge allocation. > I wonder if we shouldn't even try a one-size-fits-all solution. > > For message-p

Re: [HACKERS] shared memory message queues

2013-12-05 Thread Robert Haas
On Thu, Dec 5, 2013 at 8:29 AM, Andres Freund wrote: >> Patch #1, on-dsm-detach-v1.patch, adds the concept of on_dsm_detach >> hooks >> [snip] >> The part of this patch which I >> suppose will elicit some controversy is that I've had to rearrange >> on_shmem_exit a bit. It turns out that during s

Re: [HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?

2013-12-05 Thread Josh Berkus
On 12/05/2013 10:46 AM, Tom Lane wrote: > Before we could get very far we'd need a better understanding than we have > of what cases a DBA might be interested in. To take the specific example > that started this thread, there wouldn't be a lot of value IMO in a > classification like "connection fa

Re: [HACKERS] Regression tests failing if not launched on db "regression"

2013-12-05 Thread Tom Lane
Robert Haas writes: > On Thu, Dec 5, 2013 at 9:24 AM, Tom Lane wrote: >> Michael Paquier writes: >>> It happens that the following regression tests are failing if they are >>> run on a database not named "regression": >> This does not seem like a bug to me, although maybe we'd better update the

Re: [HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?

2013-12-05 Thread Tom Lane
Josh Berkus writes: > On 12/05/2013 10:21 AM, Stephen Frost wrote: > But ... if we set a firm policy on this, then we could gradually clean > up the error messages piecemeal over the next couple of major versions. > We could also make sure that any new features complied with the > categorization

Re: [HACKERS] Regression tests failing if not launched on db "regression"

2013-12-05 Thread Robert Haas
On Thu, Dec 5, 2013 at 9:24 AM, Tom Lane wrote: > Michael Paquier writes: >> It happens that the following regression tests are failing if they are >> run on a database not named "regression": > > This does not seem like a bug to me, although maybe we'd better update the > documentation to specif

Re: [HACKERS] Problem with displaying "wide" tables in psql

2013-12-05 Thread Robert Haas
On Thu, Dec 5, 2013 at 1:09 AM, Sergey Muraviov wrote: > And my patch affects the row view only. To help us avoid forgetting about this patch, please add it here: https://commitfest.postgresql.org/action/commitfest_view/open -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterpri

Re: [HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?

2013-12-05 Thread Josh Berkus
On 12/05/2013 10:21 AM, Stephen Frost wrote: > * David Johnston (pol...@yahoo.com) wrote: >> ISTM that instituting some level of categorization for messages would be >> helpful. Then logging and reporting frameworks would be able to identify >> and segregate the logs in whatever way they and the c

Re: [HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?

2013-12-05 Thread Stephen Frost
* David Johnston (pol...@yahoo.com) wrote: > ISTM that instituting some level of categorization for messages would be > helpful. Then logging and reporting frameworks would be able to identify > and segregate the logs in whatever way they and the configuration deems > appropriate. I've wanted to

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-05 Thread Claudio Freire
On Thu, Dec 5, 2013 at 1:03 PM, Metin Doslu wrote: >> From what I've seen so far the bigger problem than contention in the >> lwlocks itself, is the spinlock protecting the lwlocks... > > Postgres 9.3.1 also reports spindelay, it seems that there is no contention > on spinlocks. Did you check hu

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-05 Thread Josh Berkus
On 12/05/2013 05:48 AM, Stephen Frost wrote: > * Peter Geoghegan (p...@heroku.com) wrote: >> On Wed, Dec 4, 2013 at 11:07 AM, Josh Berkus wrote: >>> But you know what? 2.6, overall, still performs better than any kernel >>> in the 3.X series, at least for Postgres. >> >> What about the fseek() sc

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-05 Thread Josh Berkus
On 12/05/2013 07:40 AM, Greg Stark wrote: > On Thu, Dec 5, 2013 at 2:54 PM, Claudio Freire wrote: >> That's a bad idea in the current state of affairs. MM files haven't >> been designed for that usage, and getting stable performance out of >> that will be way too difficult. > > I'm talking about

Re: [HACKERS] Proposal: variant of regclass

2013-12-05 Thread David E. Wheeler
On Dec 5, 2013, at 7:52 AM, Tom Lane wrote: > BTW, another arguable advantage of fixing this via new functions is > that users could write equivalent (though no doubt slower) functions > for use in pre-9.4 releases, and thus not need to maintain multiple > versions of app code that relies on this

Re: [HACKERS] Time-Delayed Standbys

2013-12-05 Thread Fabrízio de Royes Mello
On Thu, Dec 5, 2013 at 7:57 AM, Simon Riggs wrote: > On 5 December 2013 08:51, Magnus Hagander wrote: > > > Not recalling the older thread, but it seems the "breaks on clock > drift", I > > think we can fairly easily make that situation "good enough". Just have > > IDENTIFY_SYSTEM return the cur

Re: [HACKERS] Errors on missing pg_subtrans/ files with 9.3

2013-12-05 Thread J Smith
On Thu, Nov 28, 2013 at 5:15 AM, Andres Freund wrote: > Hi, > > Do you still have the core file around? If so could you 'p > *ShmemVariableCache' and 'p *ControlFile'? > So sorry, I didn't see this message until just today. Seems it was accidentally archived before hitting my eyeballs. I see tha

Re: [HACKERS] Proposal: variant of regclass

2013-12-05 Thread Peter Eisentraut
On 12/5/13, 10:08 AM, Tom Lane wrote: > Peter Eisentraut writes: >> We could invent some sneaky syntax variants, like 'pg_klass'::regclass >> errors, but '?pg_klass'::regclass does not. > > Hmm ... cute idea, but shoehorning it into regoperator might be > problematic. You'd have to pick a flag c

Re: [HACKERS] Proof of concept: standalone backend with full FE/BE protocol

2013-12-05 Thread Andres Freund
On 2013-12-05 11:39:29 -0500, Peter Eisentraut wrote: > I think this proposal is a bit deadlocked now. > > - There are technical concerns about launching a server executable from > within a client. > > - There are conceptual concerns about promoting an embedded database mode. > > On the other ha

Re: [HACKERS] Extension Templates S03E11

2013-12-05 Thread Peter Eisentraut
On 12/1/13, 10:47 PM, Stephen Frost wrote: > Having a management system for sets of objects is a *great* idea- and > one which we already have through schemas. What we don't have is any > kind of versioning system built-in or other metadata about it, nor do we > have good tooling which leverages s

Re: [HACKERS] logical changeset generation v6.7

2013-12-05 Thread Andres Freund
Hi, On 2013-12-05 22:03:51 +0900, Kyotaro HORIGUCHI wrote: > > > - You assined HeapTupleGetOid(tuple) into relid to read in > > >several points but no modification. Nevertheless, you left > > >HeapTupleGetOid not replaced there. I think 'relid' just for > > >repeated reading has far s

Re: [HACKERS] Proof of concept: standalone backend with full FE/BE protocol

2013-12-05 Thread Peter Eisentraut
I think this proposal is a bit deadlocked now. - There are technical concerns about launching a server executable from within a client. - There are conceptual concerns about promoting an embedded database mode. On the other hand: - Everyone would like to have a way to use psql (and other basic

[HACKERS] Re: [RFC] Shouldn't we remove annoying FATAL messages from server log?

2013-12-05 Thread David Johnston
Tom Lane-2 wrote > "MauMau" < > maumau307@ > > writes: >> Shouldn't we lower the severity or avoiding those messages to server log? > > No. They are FATAL so far as the individual session is concerned. > Possibly some documentation effort is needed here, but I don't think > any change in the c

Re: [HACKERS] Feature request: Logging SSL connections

2013-12-05 Thread Marko Kreen
On Thu, Dec 05, 2013 at 09:43:31AM -0500, Peter Eisentraut wrote: > On 12/5/13, 8:53 AM, Dr. Andreas Kunert wrote: > > we were really missing the information in our log files if (and which > > of) our users are using SSL during their connections. > > > > The attached patch is a very simple solutio

Re: [HACKERS] Dynamic Shared Memory stuff

2013-12-05 Thread Heikki Linnakangas
On 11/20/2013 09:58 PM, Robert Haas wrote: On Wed, Nov 20, 2013 at 8:32 AM, Heikki Linnakangas wrote: How many allocations? What size will they have have typically, minimum and maximum? The facility is intended to be general, so the answer could vary widely by application. The testing that I

Re: [HACKERS] Performance optimization of btree binary search

2013-12-05 Thread Tom Lane
Andres Freund writes: > On 2013-12-05 10:34:16 -0500, Tom Lane wrote: >> In any case, the number of bugs I can remember that such a thing >> would've prevented is negligible. > Cases talked about upthread, where a plain datatype is returned as a > Datum instead of using FooGetDatum() and the reve

Re: [HACKERS] Performance optimization of btree binary search

2013-12-05 Thread Andres Freund
On 2013-12-05 10:34:16 -0500, Tom Lane wrote: > Andres Freund writes: > > I was actually thinking about making Datum (and some other types we > > have) structs or unions. Currently it's far, far to easy to mix them. We > > throw > > away pretty much all of the little typesafety C has by typedef'i

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-05 Thread Metin Doslu
> From what I've seen so far the bigger problem than contention in the > lwlocks itself, is the spinlock protecting the lwlocks... Postgres 9.3.1 also reports spindelay, it seems that there is no contention on spinlocks. PID 21121 lwlock 0: shacq 0 exacq 33 blk 1 spindelay 0 PID 21121 lwlock 33:

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-05 Thread Tom Lane
Greg Stark writes: > I think the way to use mmap would be to mmap very large chunks, > possibly whole tables. We would need some way to control page flushes > that doesn't involve splitting mappings and can be efficiently > controlled without having the kernel storing arbitrarily large tags on > p

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-05 Thread Metin Doslu
> You tested the correct branch, right? Which commit does "git rev-parse > HEAD" show? I applied last two patches manually on PostgreSQL 9.2 Stable.

Re: [HACKERS] Proposal: variant of regclass

2013-12-05 Thread Tom Lane
Robert Haas writes: >> Another advantage of this approach is that, IIUC, type input functions >> can't return a NULL value. So 'pg_klass'::regclass could return 0, >> but not NULL. On the other hand, toregclass('pg_klass') *could* >> return NULL, which seems conceptually cleaner. BTW, another a

Re: [HACKERS] Proof of concept: standalone backend with full FE/BE protocol

2013-12-05 Thread Peter Eisentraut
On 12/5/13, 6:07 AM, Simon Riggs wrote: > On 5 December 2013 01:55, Peter Eisentraut wrote: >> On Thu, 2013-11-14 at 12:11 +0530, Amit Kapila wrote: >>>If an application wants to allow these connection parameters to be >>> used, it would need to do PQenableStartServer() first. If it doesn't, >

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-05 Thread Andres Freund
On 2013-12-05 17:46:44 +0200, Metin Doslu wrote: > I tried your patches on next link. As you suspect I didn't see any > improvements. I tested it on PostgreSQL 9.2 Stable. You tested the correct branch, right? Which commit does "git rev-parse HEAD" show? But generally, as long as your profile hid

Re: [HACKERS] same-address mappings vs. relative pointers

2013-12-05 Thread Andres Freund
On 2013-12-05 10:17:18 -0500, Robert Haas wrote: > On Thu, Dec 5, 2013 at 9:44 AM, Andres Freund wrote: > >> Why? Lots of people have written lots of programs that do just that. > > > > Well, but we're a database, not a generic programming library ;) > > I think we're arguably both. Fair enough

Re: [HACKERS] Parallel Select query performance and shared buffers

2013-12-05 Thread Metin Doslu
> You could try my lwlock-scalability improvement patches - for some > workloads here, the improvements have been rather noticeable. Which > version are you testing? I tried your patches on next link. As you suspect I didn't see any improvements. I tested it on PostgreSQL 9.2 Stable. http://git.p

Re: [HACKERS] [RFC] Shouldn't we remove annoying FATAL messages from server log?

2013-12-05 Thread Tom Lane
"MauMau" writes: > Shouldn't we lower the severity or avoiding those messages to server log? No. They are FATAL so far as the individual session is concerned. Possibly some documentation effort is needed here, but I don't think any change in the code behavior would be an improvement. > 1. FATA

Re: [HACKERS] [RFC] Shouldn't we remove annoying FATAL messages from server log?

2013-12-05 Thread Peter Eisentraut
On 12/5/13, 10:25 AM, MauMau wrote: > Report these as FATAL to the client because the client wants to know the > reason. But don't output them to server log because they are not > necessary for DBAs Yeah, this is part of a more general problem, which you have characterized correctly: What is fata

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-05 Thread Greg Stark
On Thu, Dec 5, 2013 at 2:54 PM, Claudio Freire wrote: > That's a bad idea in the current state of affairs. MM files haven't > been designed for that usage, and getting stable performance out of > that will be way too difficult. I'm talking about long-term goals here. Either of these two routes wo

Re: [HACKERS] Proposal: variant of regclass

2013-12-05 Thread Tom Lane
Robert Haas writes: > On Thu, Dec 5, 2013 at 9:41 AM, Tom Lane wrote: >> I don't think new types are a good idea. If we are afraid to change >> the behavior of the input converters, what we should do is introduce >> new functions, eg "toregclass(text) returns regclass". > That seems like a pret

Re: [HACKERS] Performance optimization of btree binary search

2013-12-05 Thread Tom Lane
Andres Freund writes: > I was actually thinking about making Datum (and some other types we > have) structs or unions. Currently it's far, far to easy to mix them. We throw > away pretty much all of the little typesafety C has by typedef'ing them > to integral types with lots of autocasting behavi

Re: [HACKERS] shared memory message queues

2013-12-05 Thread Kohei KaiGai
Sorry for my late. I checked the part-3 (shm-mq-v1.patc) portion as below. Your comments towards part-1 and part-2 are reasonable for me, so I have no argue on this portion. Even though shm_mq_create() expects the "address" is aligned, however, no mechanism to ensure. How about to put Assert() he

Re: [HACKERS] Proposal: variant of regclass

2013-12-05 Thread Robert Haas
On Thu, Dec 5, 2013 at 9:41 AM, Tom Lane wrote: > Pavel Golub writes: >> I personally see two approaches: >> 1. Implement GUC variable controling this behaviour per session >> 2. Introduce new safe reg* variables, e.g. "sregclass", "sregtype" etc. > > I don't think new types are a good idea. If

[HACKERS] [RFC] Shouldn't we remove annoying FATAL messages from server log?

2013-12-05 Thread MauMau
Hello, My customers and colleagues sometimes (or often?) ask about the following message: FATAL: the database system is starting up This message is often output dozens of times during a failover or PITR. The users seem to be worried because the message level is FATAL and they don't know w

Re: [HACKERS] same-address mappings vs. relative pointers

2013-12-05 Thread Robert Haas
On Thu, Dec 5, 2013 at 9:44 AM, Andres Freund wrote: >> Why? Lots of people have written lots of programs that do just that. > > Well, but we're a database, not a generic programming library ;) I think we're arguably both. > But what's your alternative if you have a shared_palloc() like thingy?

Re: [HACKERS] Performance optimization of btree binary search

2013-12-05 Thread Andres Freund
On 2013-12-05 10:02:56 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2013-12-05 08:58:55 -0500, Tom Lane wrote: > >> I'm a bit worried that somebody, particularly third-party code, > >> might've sloppily written "return foo" in a V1 function when "return > >> Int32GetDatum(foo)" would be c

Re: [HACKERS] Proposal: variant of regclass

2013-12-05 Thread Tom Lane
Peter Eisentraut writes: > We could invent some sneaky syntax variants, like 'pg_klass'::regclass > errors, but '?pg_klass'::regclass does not. Hmm ... cute idea, but shoehorning it into regoperator might be problematic. You'd have to pick a flag character that wasn't a valid operator character,

Re: [HACKERS] Performance optimization of btree binary search

2013-12-05 Thread Tom Lane
Andres Freund writes: > On 2013-12-05 08:58:55 -0500, Tom Lane wrote: >> I'm a bit worried that somebody, particularly third-party code, >> might've sloppily written "return foo" in a V1 function when "return >> Int32GetDatum(foo)" would be correct. In that case, the resultant Datum >> might have

Re: [HACKERS] same-address mappings vs. relative pointers

2013-12-05 Thread Robert Haas
On Thu, Dec 5, 2013 at 8:57 AM, Heikki Linnakangas wrote: > On 12/05/2013 06:32 AM, Robert Haas wrote: >> During development of the dynamic shared memory facility, Noah and I >> spent a lot of time arguing about whether it was practical to ensure >> that a dynamic shared memory segment got mapped

Re: [HACKERS] Proposal: variant of regclass

2013-12-05 Thread Peter Eisentraut
On 12/5/13, 9:41 AM, Tom Lane wrote: > Pavel Golub writes: >> I personally see two approaches: >> 1. Implement GUC variable controling this behaviour per session >> 2. Introduce new safe reg* variables, e.g. "sregclass", "sregtype" etc. > > I don't think new types are a good idea. If we are afra

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-05 Thread Claudio Freire
On Thu, Dec 5, 2013 at 11:42 AM, Greg Stark wrote: > (b) is the way more interesting research project though. I don't think > anyone's tried it and the kernel interface to provide the kinds of > information Postgres needs requires a lot of thought. If it's done > right then Postgres wouldn't need

Re: [HACKERS] same-address mappings vs. relative pointers

2013-12-05 Thread Andres Freund
On 2013-12-05 15:44:34 +0100, Andres Freund wrote: > On 2013-12-05 07:44:27 -0500, Robert Haas wrote: > > And then I thought, boy, it sucks > > not to be able to declare what kind of a thing we're pointing *at* > > here, but apart from using C++ I see no solution to that problem. I > > guess we co

Re: [HACKERS] Proposal: variant of regclass

2013-12-05 Thread Tom Lane
Pavel Golub writes: > I personally see two approaches: > 1. Implement GUC variable controling this behaviour per session > 2. Introduce new safe reg* variables, e.g. "sregclass", "sregtype" etc. I don't think new types are a good idea. If we are afraid to change the behavior of the input convert

Re: [HACKERS] Why we are going to have to go DirectIO

2013-12-05 Thread Greg Stark
On Thu, Dec 5, 2013 at 8:35 AM, KONDO Mitsumasa wrote: > Yes. And using something efficiently DirectIO is more difficult than > BufferedIO. > If we change write() flag with direct IO in PostgreSQL, it will execute > hardest ugly randomIO. Using DirectIO presumes you're using libaio or threads to

Re: [HACKERS] Feature request: Logging SSL connections

2013-12-05 Thread Peter Eisentraut
On 12/5/13, 8:53 AM, Dr. Andreas Kunert wrote: > we were really missing the information in our log files if (and which > of) our users are using SSL during their connections. > > The attached patch is a very simple solution to this problem - it just > tests if the ssl pointer in Port is null. If n

Re: [HACKERS] same-address mappings vs. relative pointers

2013-12-05 Thread Andres Freund
On 2013-12-05 07:44:27 -0500, Robert Haas wrote: > On Thu, Dec 5, 2013 at 4:56 AM, Andres Freund wrote: > > Hi Robert, > > > > On 2013-12-04 23:32:27 -0500, Robert Haas wrote: > >> But I'm also learning painfully that this kind of thing only goes so > >> far. For example, I spent some time lookin

Re: [HACKERS] More legacy code: pg_ctl

2013-12-05 Thread Peter Eisentraut
On 11/18/13, 8:20 PM, Josh Berkus wrote: > c) that "stop" defaults to "smart" mode, instead of "fast" mode. This has been discussed many times already, so you'd need to check the archives. (I'm not in favor of smart mode either.) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql

Re: [HACKERS] More legacy code: pg_ctl

2013-12-05 Thread Peter Eisentraut
On 11/18/13, 8:09 PM, Josh Berkus wrote: > a) by default, it returns to the caller without waiting for postgres to > actually start/stop/restart. In this mode, it also always returns > success regardless of result. The reason for this is that until sometime recently (PQping) we didn't have a reli

  1   2   >