Re: [GENERAL] Slow PITR restore

2007-12-13 Thread Joshua D. Drake
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, 12 Dec 2007 23:08:35 -0800 "Joshua D. Drake" <[EMAIL PROTECTED]> wrote: > Tom Lane wrote: > > "Joshua D. Drake" <[EMAIL PROTECTED]> writes: > >> Tom Lane wrote: > >>> You sure about that? I tested CVS HEAD just now, by setting the > >>> check

Re: [GENERAL] Slow PITR restore

2007-12-13 Thread Alvaro Herrera
Gregory Stark wrote: > "Simon Riggs" <[EMAIL PROTECTED]> writes: > > > It's a good idea, but it will require more complex code. I prefer the > > simpler solution of using more processes to solve the I/O problem. > > Huh, I forgot about that idea. Ironically that was what I suggested when > Heikki

Re: [GENERAL] Slow PITR restore

2007-12-13 Thread Gregory Stark
"Simon Riggs" <[EMAIL PROTECTED]> writes: > On Thu, 2007-12-13 at 06:27 +, Gregory Stark wrote: >> Heikki proposed a while back to use posix_fadvise() when processing logs to >> read-ahead blocks which the recover will need before actually attempting to >> recover them. On a raid array that wo

Re: [GENERAL] Slow PITR restore

2007-12-13 Thread Simon Riggs
On Thu, 2007-12-13 at 06:27 +, Gregory Stark wrote: > "Tom Lane" <[EMAIL PROTECTED]> writes: > > > "Joshua D. Drake" <[EMAIL PROTECTED]> writes: > > > >> Exactly. Which is the point I am making. Five minutes of transactions > >> is nothing (speaking generally).. In short, if we are in recovery

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Joshua D. Drake
Tom Lane wrote: "Joshua D. Drake" <[EMAIL PROTECTED]> writes: Tom Lane wrote: You sure about that? I tested CVS HEAD just now, by setting the checkpoint_ parameters really high, ... And: 2007-12-13 00:55:20 EST LOG: restored log file "000107E1006B" from archive Hmm --

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Tom Lane
"Joshua D. Drake" <[EMAIL PROTECTED]> writes: > Tom Lane wrote: >> You sure about that? I tested CVS HEAD just now, by setting the >> checkpoint_ parameters really high, > ... And: >> 2007-12-13 00:55:20 EST LOG: restored log file >> "000107E1006B" from archive Hmm --- I was t

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Gregory Stark
"Tom Lane" <[EMAIL PROTECTED]> writes: > "Joshua D. Drake" <[EMAIL PROTECTED]> writes: > >> Exactly. Which is the point I am making. Five minutes of transactions >> is nothing (speaking generally).. In short, if we are in recovery, and >> we are not saturated the I/O and at least a single CPU, the

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Joshua D. Drake
Tom Lane wrote: "Joshua D. Drake" <[EMAIL PROTECTED]> writes: Jeff Trout <[EMAIL PROTECTED]> wrote: in this case it was 24hrs of data - about 1500 wal segments. During this time the machine was nearly complete idle and there wasn't very much IO going on (few megs/sec). Exactly. Which is

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Tom Lane
"Joshua D. Drake" <[EMAIL PROTECTED]> writes: > Jeff Trout <[EMAIL PROTECTED]> wrote: >> in this case it was 24hrs of data - about 1500 wal segments. During >> this time the machine was nearly complete idle and there wasn't very >> much IO going on (few megs/sec). > Exactly. Which is the poin

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Joshua D. Drake
Gregory Stark wrote: "Joshua D. Drake" <[EMAIL PROTECTED]> writes: On Wed, 12 Dec 2007 18:02:39 + Gregory Stark <[EMAIL PROTECTED]> wrote: I'm not sure what you guys' expectations are, but if you're restoring 5 minutes worth of database traffic in 8 seconds I wouldn't be complaining. I w

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Gregory Stark
"Joshua D. Drake" <[EMAIL PROTECTED]> writes: > On Wed, 12 Dec 2007 18:02:39 + > Gregory Stark <[EMAIL PROTECTED]> wrote: > >> I'm not sure what you guys' expectations are, but if you're restoring >> 5 minutes worth of database traffic in 8 seconds I wouldn't be >> complaining. > > I would be.

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Joshua D. Drake
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, 12 Dec 2007 13:13:35 -0500 Jeff Trout <[EMAIL PROTECTED]> wrote: > > On Dec 12, 2007, at 1:02 PM, Gregory Stark wrote: > > > > I'm not sure what you guys' expectations are, but if you're > > restoring 5 > > minutes worth of database traffic

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Simon Riggs
On Wed, 2007-12-12 at 12:56 -0500, Tom Lane wrote: > Jeff Trout <[EMAIL PROTECTED]> writes: > > I've seen this on my PITR restores (thankfully, they were for > > fetching some old data, not because we expoded). On a 2.4ghz opteron > > it took 5-50 seconds per wal segment, and there were a LOT

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Simon Riggs
On Wed, 2007-12-12 at 10:08 -0800, Joshua D. Drake wrote: > > Depending on your transaction mix and what percentage of it is > > read-only select queries you might reasonably expect the restore to > > take as long as it took to generate t > > We archive selects? No, but that is exactly his point

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Jeff Trout
On Dec 12, 2007, at 1:02 PM, Gregory Stark wrote: I'm not sure what you guys' expectations are, but if you're restoring 5 minutes worth of database traffic in 8 seconds I wouldn't be complaining. Depending on your transaction mix and what percentage of it is read- only select queries you

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Joshua D. Drake
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, 12 Dec 2007 18:02:39 + Gregory Stark <[EMAIL PROTECTED]> wrote: > I'm not sure what you guys' expectations are, but if you're restoring > 5 minutes worth of database traffic in 8 seconds I wouldn't be > complaining. I would be. This is a

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Gregory Stark
"Jeff Trout" <[EMAIL PROTECTED]> writes: > On Dec 12, 2007, at 11:29 AM, Joshua D. Drake wrote: > >> 8 seconds for a single archive recovery is very slow in consideration of >> this machine. Even single threaded that seems slow. > > I've seen this on my PITR restores (thankfully, they were for f

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Tom Lane
Jeff Trout <[EMAIL PROTECTED]> writes: > I've seen this on my PITR restores (thankfully, they were for > fetching some old data, not because we expoded). On a 2.4ghz opteron > it took 5-50 seconds per wal segment, and there were a LOT of > segments (replay took hours and hours). I asked a f

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Jeff Trout
On Dec 12, 2007, at 11:29 AM, Joshua D. Drake wrote: 8 seconds for a single archive recovery is very slow in consideration of this machine. Even single threaded that seems slow. I've seen this on my PITR restores (thankfully, they were for fetching some old data, not because we expoded).

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Simon Riggs
On Wed, 2007-12-12 at 12:13 -0500, Tom Lane wrote: > Simon Riggs <[EMAIL PROTECTED]> writes: > > On Tue, 2007-12-11 at 22:21 -0500, Tom Lane wrote: > >> Yeah, restoring is known to be less than speedy, because essentially > >> zero optimization work has been done on it. > > > If there was a patch

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Joshua D. Drake
Tom Lane wrote: Simon Riggs <[EMAIL PROTECTED]> writes: On Tue, 2007-12-11 at 22:21 -0500, Tom Lane wrote: Yeah, restoring is known to be less than speedy, because essentially zero optimization work has been done on it. If there was a patch to improve this, would it be applied to 8.3? Good

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Tom Lane
Simon Riggs <[EMAIL PROTECTED]> writes: > On Tue, 2007-12-11 at 22:21 -0500, Tom Lane wrote: >> Yeah, restoring is known to be less than speedy, because essentially >> zero optimization work has been done on it. > If there was a patch to improve this, would it be applied to 8.3? Good grief, no.

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Joshua D. Drake
Simon Riggs wrote: On Wed, 2007-12-12 at 07:36 -0800, Joshua D. Drake wrote: We can't really expect people to use PITR if they new it would take hours to recover even on the size of machine I was working on. That's not true statement in all cases and can often be improved with some monitoring

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Simon Riggs
On Wed, 2007-12-12 at 07:36 -0800, Joshua D. Drake wrote: > We can't really expect people to use PITR if they new > it would take hours to recover even on the size of machine I was working on. That's not true statement in all cases and can often be improved with some monitoring and tuning. Just

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Joshua D. Drake
Simon Riggs wrote: On Tue, 2007-12-11 at 22:21 -0500, Tom Lane wrote: "Joshua D. Drake" <[EMAIL PROTECTED]> writes: ... Now I understand that restoring log files can be slow but this is a big machine. Yeah, restoring is known to be less than speedy, because essentially zero optimization work h

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Simon Riggs
On Tue, 2007-12-11 at 22:21 -0500, Tom Lane wrote: > "Joshua D. Drake" <[EMAIL PROTECTED]> writes: > > ... Now I understand that restoring log files can be slow but this is a big > > machine. > > Yeah, restoring is known to be less than speedy, because essentially > zero optimization work has bee

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Alvaro Herrera
Gregory Stark wrote: > bgwriter is started already when doing recovery, right? Perhaps things could > be helped by telling bgwriter to behave differently during recovery. No. -- Alvaro Herrera http://www.flickr.com/photos/alvherre/ "Amanece.

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Simon Riggs
On Wed, 2007-12-12 at 08:55 +, Gregory Stark wrote: > "Joshua D. Drake" <[EMAIL PROTECTED]> writes: > > > Wow, o.k. well it is something we (the community) really should look at for > > 8.4. I am surprised that it is slower than just walking through the xlogs on > > recovery. I am sure there i

Re: [GENERAL] Slow PITR restore

2007-12-12 Thread Gregory Stark
"Joshua D. Drake" <[EMAIL PROTECTED]> writes: > Wow, o.k. well it is something we (the community) really should look at for > 8.4. I am surprised that it is slower than just walking through the xlogs on > recovery. I am sure there is a reason just surprised. Well in the worst case it has to do ne

Re: [GENERAL] Slow PITR restore

2007-12-11 Thread Joshua D. Drake
Tom Lane wrote: "Joshua D. Drake" <[EMAIL PROTECTED]> writes: ... Now I understand that restoring log files can be slow but this is a big machine. Yeah, restoring is known to be less than speedy, because essentially zero optimization work has been done on it. Heikki has improved matters a bit

Re: [GENERAL] Slow PITR restore

2007-12-11 Thread Tom Lane
"Joshua D. Drake" <[EMAIL PROTECTED]> writes: > ... Now I understand that restoring log files can be slow but this is a big > machine. Yeah, restoring is known to be less than speedy, because essentially zero optimization work has been done on it. Heikki has improved matters a bit in 8.3, but I