Re: [HACKERS] autovacuum stress-testing our system

2012-12-02 Thread Tomas Vondra
On 21.11.2012 19:02, Robert Haas wrote: > On Sun, Nov 18, 2012 at 5:49 PM, Tomas Vondra wrote: >> The two main changes are these: >> >> (1) The stats file is split into a common "db" file, containing all the >> DB Entries, and per-database files with tables/functions. The common >> file is

Re: [HACKERS] autovacuum stress-testing our system

2012-11-21 Thread Robert Haas
On Sun, Nov 18, 2012 at 5:49 PM, Tomas Vondra wrote: > The two main changes are these: > > (1) The stats file is split into a common "db" file, containing all the > DB Entries, and per-database files with tables/functions. The common > file is still called "pgstat.stat", the per-db files h

Re: [HACKERS] autovacuum stress-testing our system

2012-11-18 Thread Tomas Vondra
On 26.9.2012 18:29, Tom Lane wrote: > What seems to me like it could help more is fixing things so that the > autovac launcher needn't even launch a child process for databases that > haven't had any updates lately. I'm not sure how to do that, but it > probably involves getting the stats collecto

Re: [HACKERS] autovacuum stress-testing our system

2012-11-18 Thread Tomas Vondra
Hi! On 26.9.2012 19:18, Jeff Janes wrote: > On Wed, Sep 26, 2012 at 9:29 AM, Tom Lane wrote: >> Alvaro Herrera writes: >>> Excerpts from Euler Taveira's message of mié sep 26 11:53:27 -0300 2012: On 26-09-2012 09:43, Tomas Vondra wrote: > 5) splitting the single stat file into multiple

Re: [HACKERS] autovacuum stress-testing our system

2012-09-27 Thread Simon Riggs
On 27 September 2012 15:57, Alvaro Herrera wrote: > Excerpts from Simon Riggs's message of jue sep 27 06:51:28 -0300 2012: >> On 26 September 2012 15:47, Alvaro Herrera wrote: >> > >> > Really, as far as autovacuum is concerned, it would be much more useful >> > to be able to reliably detect that

Re: [HACKERS] autovacuum stress-testing our system

2012-09-27 Thread Alvaro Herrera
Excerpts from Simon Riggs's message of jue sep 27 06:51:28 -0300 2012: > On 26 September 2012 15:47, Alvaro Herrera wrote: > > > > Really, as far as autovacuum is concerned, it would be much more useful > > to be able to reliably detect that a table has been recently vacuumed, > > without having t

Re: [HACKERS] autovacuum stress-testing our system

2012-09-27 Thread Simon Riggs
On 26 September 2012 15:47, Alvaro Herrera wrote: > > Really, as far as autovacuum is concerned, it would be much more useful > to be able to reliably detect that a table has been recently vacuumed, > without having to request a 10ms-recent pgstat snapshot. That would > greatly reduce the amount

Re: [HACKERS] autovacuum stress-testing our system

2012-09-26 Thread Tomas Vondra
On 26.9.2012 18:14, Jeff Janes wrote: > On Wed, Sep 26, 2012 at 8:25 AM, Tomas Vondra wrote: >> Dne 26.09.2012 16:51, Jeff Janes napsal: >> >> >>> What is generating the endless stream you are seeing is that you have >>> 1000 databases so if naptime is one minute you are vacuuming 16 per >>> secon

Re: [HACKERS] autovacuum stress-testing our system

2012-09-26 Thread Tomas Vondra
On 26.9.2012 18:29, Tom Lane wrote: > Alvaro Herrera writes: >> Excerpts from Euler Taveira's message of miĂŠ sep 26 11:53:27 -0300 2012: >>> On 26-09-2012 09:43, Tomas Vondra wrote: 5) splitting the single stat file into multiple pieces - e.g. per database, written separately, so that t

Re: [HACKERS] autovacuum stress-testing our system

2012-09-26 Thread Jeff Janes
On Wed, Sep 26, 2012 at 9:29 AM, Tom Lane wrote: > Alvaro Herrera writes: >> Excerpts from Euler Taveira's message of mié sep 26 11:53:27 -0300 2012: >>> On 26-09-2012 09:43, Tomas Vondra wrote: 5) splitting the single stat file into multiple pieces - e.g. per database, written separate

Re: [HACKERS] autovacuum stress-testing our system

2012-09-26 Thread Tom Lane
Alvaro Herrera writes: > Excerpts from Euler Taveira's message of mié sep 26 11:53:27 -0300 2012: >> On 26-09-2012 09:43, Tomas Vondra wrote: >>> 5) splitting the single stat file into multiple pieces - e.g. per database, >>> written separately, so that the autovacuum workers don't need to read a

Re: [HACKERS] autovacuum stress-testing our system

2012-09-26 Thread Jeff Janes
On Wed, Sep 26, 2012 at 8:25 AM, Tomas Vondra wrote: > Dne 26.09.2012 16:51, Jeff Janes napsal: > > >> What is generating the endless stream you are seeing is that you have >> 1000 databases so if naptime is one minute you are vacuuming 16 per >> second. Since every database gets a new process, t

Re: [HACKERS] autovacuum stress-testing our system

2012-09-26 Thread Tomas Vondra
Dne 26.09.2012 17:29, Alvaro Herrera napsal: Excerpts from Tomas Vondra's message of mié sep 26 12:25:58 -0300 2012: Dne 26.09.2012 16:51, Jeff Janes napsal: > I think forking it off to to another value would be better. If you > are an autovacuum worker which is just starting up and so getti

Re: [HACKERS] autovacuum stress-testing our system

2012-09-26 Thread Alvaro Herrera
Excerpts from Tomas Vondra's message of mié sep 26 12:25:58 -0300 2012: > Dne 26.09.2012 16:51, Jeff Janes napsal: > > I think forking it off to to another value would be better. If you > > are an autovacuum worker which is just starting up and so getting its > > initial stats, you can tolerate a

Re: [HACKERS] autovacuum stress-testing our system

2012-09-26 Thread Alvaro Herrera
Excerpts from Euler Taveira's message of mié sep 26 11:53:27 -0300 2012: > On 26-09-2012 09:43, Tomas Vondra wrote: > > 5) splitting the single stat file into multiple pieces - e.g. per database, > >written separately, so that the autovacuum workers don't need to read all > >the data even

Re: [HACKERS] autovacuum stress-testing our system

2012-09-26 Thread Tomas Vondra
Dne 26.09.2012 16:51, Jeff Janes napsal: On Wed, Sep 26, 2012 at 5:43 AM, Tomas Vondra wrote: First - our system really is not a "common" one - we do have ~1000 of databases of various size, each containing up to several thousands of tables (several user-defined tables, the rest serve as ca

Re: [HACKERS] autovacuum stress-testing our system

2012-09-26 Thread Euler Taveira
On 26-09-2012 09:43, Tomas Vondra wrote: > I've been struggling with autovacuum generating a lot of I/O and CPU on some > of our > systems - after a night spent analyzing this behavior, I believe the current > autovacuum accidentally behaves a bit like a stress-test in some corner cases > (but > I

Re: [HACKERS] autovacuum stress-testing our system

2012-09-26 Thread Jeff Janes
On Wed, Sep 26, 2012 at 5:43 AM, Tomas Vondra wrote: > First - our system really is not a "common" one - we do have ~1000 of > databases of > various size, each containing up to several thousands of tables (several > user-defined > tables, the rest serve as caches for a reporting application - ye

Re: [HACKERS] autovacuum stress-testing our system

2012-09-26 Thread Alvaro Herrera
Really, as far as autovacuum is concerned, it would be much more useful to be able to reliably detect that a table has been recently vacuumed, without having to request a 10ms-recent pgstat snapshot. That would greatly reduce the amount of time autovac spends on pgstat requests. -- Álvaro Herre