On Sun, 20 Apr 2008, Peter Tribble wrote: >> >> What is the cause of the "struggling"? Does the backup host run short of >> RAM or CPU? If backups are incremental, is a large portion of time spent >> determining the changes to be backed up? What is the relative cost of many >> small files vs large files? > > It's just the fact that, while the backup completes, it can take over 24 > hours. > Clearly this takes you well over any backup window. It's not so much that the > backup software is defective; it's an indication that traditional notions of > backup need to be rethought.
There is no doubt about that. However, there are organizations with hundreds of terrabytes online and they manage to survive somehow. I receive bug reports from people with 600K files in a single subdirectory. Terrabyte-sized USB drives are available now. When you say that the backup can take over 24 hours, are you talking only about the initial backup, or incrementals as well? > I have one small (200G) filesystem that takes an hour to do an incremental > with no changes. (After a while, it was obvious we don't need to do that > every night.) That is pretty outrageous. It seems that your backup software is suspect since it must be severely assaulting the filesystem. I am using 'rsync' (version 3.0) to do disk-to-disk network backups (with differencing) to a large Firewire type drive and have not noticed any performance issues. I do not have 10 million files though (I have about half of that). Since zfs supports really efficient snapshots, a backup system which is aware of snapshots can take snapshots and then backup safely even if the initial dump takes several days. Really smart software could perform both initial dump and incremental dump simultaneously. The minimum useful incremental backup interval would still be be limited to the time required to do one incremental backup. Bob ====================================== Bob Friesenhahn [EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
