Matthew Ahrens wrote:
Joseph Barbey wrote:
Robert Milkowski wrote:
JB> So, normally, when the script runs, all snapshots finish in maybe
a minute
JB> total. However, on Sundays, it continues to take longer and
longer. On
JB> 2/25 it took 30 minutes, and this last Sunday, it took 2:11. The
only
JB> thing special thing about Sunday's snapshots is that they are the
first
JB> ones created since the full backup (using NetBackup) on Saturday.
All
JB> other backups are incrementals.
hmmmmm do you have atime property set to off?
Maybe you spend most of the time in destroying snapshots due to much
larger delta coused by atime updates? You can possibly also gain some
performance by setting atime to off.
Yep, atime is set to off for all pools and filesystems. I looked
through the other possible properties, and nothing really looked like
it would really affect things.
One additional weird thing. My script hits each filesystem
(email-pool/A..Z) individually, so I can run zfs list -t snapshot and
find out how long each snapshot actually takes. Everything runs fine
until I get to around V or (normally) W. Then it can take a couple of
hours on the one FS. After that, the rest go quickly.
So, what operation exactly is taking "a couple of hours on the one FS"?
The only one I can imagine taking more than a minute would be 'zfs
destroy', but even that should be very rare on a snapshot. Is it always
the same FS that takes longer than the rest? Is the pool busy when you
do the slow operation?
I'll need to look at this. I expect the next wait will be again on Sunday.
My first-blush guess is that creating the snapshot itself is actually the
problem, but I'll likely have to wait until next Sunday to be sure. I've
added some debugging to my script, so I can see how long each part takes.
Also, it is always the 'W' filesytems's snapshot that seems to indicate a
long time to be created. Recently it showed up a little bit in the 'V'
filesystem as well.
You should be able to improve performance considerably (~26x) by using
just doing one 'zfs snapshot -r', 'zfs destroy -r', and 'zfs rename -r'.
(rename -r is in progress, should be available in OpenSolaris soon; the
others are in s10u3.)
When I first set this stuff up, we could not use the -r option anywhere, so
I didn't use it in my script. A quick test verifies that I now CAN use it
as indicated above. Once my test on Sunday is done, I'll be using -r in my
script as well.
Also, all 3 pools are still 'formatted' as v2. I'll try upgrading all 3
before Sunday, and see if that helps as well.
--
Joe Barbey IT Services/Network Services
office: (715) 425-4357 Davee Library room 166C
cell: (715) 821-0008 UW - River Falls
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
