-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I would never operate in a manner that only has 5-6 days of old backups. The backups that I am deleting are more than a year old.
On 09/15/2014 02:24 AM, Robert Bell wrote: > Kevin, > > Thanks for the reply and interest in this topic. > > Comments below. > > Regards > > Rob. > >> I did consider that but rejected it for 2 reasons... >> >> 1. Backup run time. We have a 4 hour window to run backups at >> night. Using recycled directories significantly extended the >> backup run time. The deletion time is eliminated but frankly, we >> have the other 20 hours of the day to do deletions. We had to >> give up using - --link-dest when the deletions started to >> actually take that long even though the backups still ran in >> under 4 hours. > > For us, the recycling of old directories significantly shortened > the time to do backups, since the recycled backups have typically > 95% of the files/directories correct (with daily backups and Tower > of Hanoi, half of our recycled backups are only 5 to 6 days old). > > I've just done some tests with a fairly pathological case, all on > one host. > > I set up a source tree 's' with 11111 sub-directories and 10000 > files, and then two destinations: cp -a s d1 cp -afl d1 d2 > > I then did the first test: # rsync to a new directory, followed by > a remove of an old directory." time rsync -a --link-dest=../d2 s/ > d3 time /bin/rm -rf d1 > > > I then scrubbed the lot, set it up again, and did the second test: > mv d1 d3 # rsync to a recycled directory" time rsync -a > --link-dest=../d2 --delete s/ d3 > > I hope I got this right! I've made no effort to circumvent > caching. > > Anyway, here is a table of the average times (seconds) over 5 runs > of each test. > > Real User Sys (User+Sys) test 1 2.454s 0.150s > 2.196s 2.346s test 2 0.392s 0.100s 0.572s 0.672s > ratio 6.3 1.5 3.8 3.5 > > (The User+Sys time is pretty much invariant, even though in earlier > tests the real time suffered major blowouts owing to contention.) > > So, the big difference is that in test 1, the 11111 sub-directories > and 10000 files were created in the destination d3, and then the > same numbers were deleted from the old directory d1. In test 2, > rsync does none of that, but only has to check for differences. > ~40,000 metadata operations avoided on the filesystem in this > case. > > >> >> 2. Metadata history. If there is an existing file in the target >> dir that differs only by metadata (permissions, ownership, >> timestamp) then rsync will simply change that metadata. That >> change affects all instances of that file. Of course this is >> better for storage space as the alternative is storing another >> copy of the file with the different metadata but we decided it >> was better to have that information saved. > Yes. > > > I would love to see someone make a patched version of rsync to > allow callers to select a different behaviour in this case! > > So, if a file has identical content on source and destination but > different metadata, then if --link-dest is in use and the link > count on the destination is > 1, then take a new copy from source > rather than just updating the metadata (the file could be copied on > the destination and then the copy updated with the new metadata and > the old version removed, but this would not be essential - just > perhaps an efficiency gain.) > > Thanks in anticipation! > > > Dr Robert C. Bell HPC National Partnerships | Scientific Computing > Information Management and Technology CSIRO T +61 3 9669 8102 Alt > +61 3 8601 3810 Mob +61 428 108 333 > robert.b...@csiro.au<mailto:robert.b...@csiro.au> | www.csiro.au | > wiki.csiro.au/display/ASC/ Street: CSIRO ASC Level 11, 700 Collins > Street, Docklands Vic 3008, Australia Postal: CSIRO ASC Level 11, > GPO Box 1289, Melbourne Vic 3001, Australia > > PLEASE NOTE The information contained in this email may be > confidential or privileged. Any unauthorised use or disclosure is > prohibited. If you have received this email in error, please > delete it immediately and notify the sender by return email. Thank > you. To the extent permitted by law, CSIRO does not represent, > warrant and/or guarantee that the integrity of this communication > has been maintained or that the communication is free of errors, > virus, interception or interference. > > Please consider the environment before printing this email. - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Florida k...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlQW/8YACgkQVKC1jlbQAQfCIwCdGKm9z00G0Xu4tItwuUlUaLum 8dwAn0sY8qriEJeUsReRlU67GkbA5BRZ =2b6r -----END PGP SIGNATURE----- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
