Re: [HACKERS] logging in high performance systems.

2012-03-15 Thread Robert Haas
On Fri, Mar 9, 2012 at 9:53 AM, Robert Haas wrote: > Tom recently committed something by another author that is along > similar lines to what you have here (I think).  Can you comment on > whether you think more is still needed and what the differences are > between that approach and yours? Heari

Re: [HACKERS] logging in high performance systems.

2012-03-09 Thread Robert Haas
On Wed, Nov 23, 2011 at 9:28 PM, Theo Schlossnagle wrote: > We have a need for logging in systems where it isn't feasible to log > to disk as it negatively impacts performance. > > I'd like to be able to creatively solve this problem without modifying > the core, but today I cannot. > > So... here

Re: [HACKERS] logging in high performance systems.

2012-03-06 Thread Robert Haas
On Tue, Mar 6, 2012 at 2:11 PM, Tom Lane wrote: > Robert Haas writes: >> I would dismissed this out of hand at this if you said it a year ago, >> but I'm older and wiser now.  At some point this cycle, I did some >> benchmarking of the subtransaction abort path, since the slowness of >> things li

Re: [HACKERS] logging in high performance systems.

2012-03-06 Thread Tom Lane
Robert Haas writes: > I would dismissed this out of hand at this if you said it a year ago, > but I'm older and wiser now. At some point this cycle, I did some > benchmarking of the subtransaction abort path, since the slowness of > things like EXCEPTION blocks in PL/pgsql is a known sore point.

Re: [HACKERS] logging in high performance systems.

2012-01-19 Thread Robert Haas
On Mon, Jan 16, 2012 at 3:51 AM, Greg Smith wrote: > There is an important distinction to make here.  Statement logging is one of > the largest producers of logging data you want to worry about optimizing for > on a high performance system.  The decision about whether to log or not may > need to b

Re: [HACKERS] logging in high performance systems.

2012-01-17 Thread Alvaro Herrera
Excerpts from Marti Raudsepp's message of mar ene 17 12:12:50 -0300 2012: > On Thu, Nov 24, 2011 at 04:28, Theo Schlossnagle wrote: > > So... here's my first whack at solving this with some flexibility. > > > > The first thing I did was add hook points where immediate statement > > logging happen

Re: [HACKERS] logging in high performance systems.

2012-01-17 Thread Marti Raudsepp
On Thu, Nov 24, 2011 at 04:28, Theo Schlossnagle wrote: > So... here's my first whack at solving this with some flexibility. > > The first thing I did was add hook points where immediate statement > logging happens "pre_exec" and those that present duration > "post_exec".  These should, with optim

Re: [HACKERS] logging in high performance systems.

2012-01-16 Thread Greg Smith
On 11/23/2011 09:28 PM, Theo Schlossnagle wrote: The first thing I did was add hook points where immediate statement logging happens "pre_exec" and those that present duration "post_exec". These should, with optimization turned on, have only a few instructions of impact when no hooks are registe

Re: [HACKERS] logging in high performance systems.

2011-12-13 Thread Michael Glaesemann
On Dec 13, 2011, at 13:57, Greg Smith wrote: > With this idea still being pretty new, and several of the people popping out > opinions in this thread being local--Theo, Stephen, myself--we've decided to > make our local Baltimore/Washington PUG meeting this month be an excuse to > hash some of

Re: [HACKERS] logging in high performance systems.

2011-12-13 Thread Greg Smith
On 11/24/2011 11:33 AM, Theo Schlossnagle wrote: I see the next steps being: 1) agreeing that a problem exists (I know one does, but I suppose consensus is req'd) 2) agreeing that "hooks" are the right approach, if not propose a different approach. (fwiw, it's incredible common) 3) reworkin

Re: [HACKERS] logging in high performance systems.

2011-12-04 Thread Martin Pihlak
On 11/24/2011 06:33 PM, Theo Schlossnagle wrote: > I see the next steps being: > 1) agreeing that a problem exists (I know one does, but I suppose > consensus is req'd) +1, high volume logging is also a huge problem here at Skype. Some of the issues that immediately come to mind: - extracting us

Re: [HACKERS] logging in high performance systems.

2011-11-27 Thread Stephen Frost
* Theo Schlossnagle (je...@omniti.com) wrote: > I'd like to be able to creatively solve this problem without modifying > the core, but today I cannot. I thought a hook already existed for this (there's already a module which uses a hook to log commands done as a superuser, for example).. Perhaps i

Re: [HACKERS] logging in high performance systems.

2011-11-24 Thread Robert Haas
On Wed, Nov 23, 2011 at 9:28 PM, Theo Schlossnagle wrote: > Thoughts? Feedback? Can you add it to the next CommitFest? https://commitfest.postgresql.org/action/commitfest_view/open -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-ha

Re: [HACKERS] logging in high performance systems.

2011-11-24 Thread Theo Schlossnagle
On Wed, Nov 23, 2011 at 11:45 PM, Greg Smith wrote: > My assumption has been that eventually a lossy logger was going to be > necessary for busier sites, I just haven't been suffering from one enough to > hack on it yet.  If it's possible to work this out in enough detail to > figure out where the

Re: [HACKERS] logging in high performance systems.

2011-11-23 Thread Greg Smith
On 11/23/2011 09:28 PM, Theo Schlossnagle wrote: The second thing I did was write a sample use of those hooks to implement a completely non-blocking fifo logger. (if it would block, it drops the log line). The concept is that we could run this without risk of negative performance impact due to s

[HACKERS] logging in high performance systems.

2011-11-23 Thread Theo Schlossnagle
We have a need for logging in systems where it isn't feasible to log to disk as it negatively impacts performance. I'd like to be able to creatively solve this problem without modifying the core, but today I cannot. So... here's my first whack at solving this with some flexibility. The first thi