Re: [Dirvish] Dirvish moving forward
I am glad to hear this and would emphasize again what Keith said: Keep It Stable. If a dirvish repository worked 20 years ago, it is now your responsibility to see that even if no one looks at it for the next 50 years, it still 'just works' in 2067. Taking the long view of human history has not been one of the great strengths of a majority of the open source community. Keith: Any time you want to set up for me to pull copies of any of your space archives for inclusion in the NSS archives, just get back to me. I am, as always, swampped. I need being pinged until I get it done. -- +---+ | Dale Amon Immortal Data| | CEO Midland International Air and Space Port| | a...@vnl.com "Data Systems for Deep Space and Time" | +---+ ___ Dirvish mailing list Dirvish@dirvish.org http://www.dirvish.org/mailman/listinfo/dirvish
[Dirvish] Misc old problems with dirvish speed limits
For all I know some fix might be in for these speed limit issues by now, but I will put them out here anyway just in case. 1) The speed-limit does not seem to function at all in a /etc/dirvish/master.conf, only in vault/dirvish/default.conf, ie you can set speed limits for one vault but not a default. 2) The speed-limit value is off by a factor of 10 fro the documentation and from the corresponding rsync parameter: .5 is 50 kbs, not 500kbs 3) Wish list: An rsync speedlimit with times attached would be nice ie use x kbs from hh:mm to hh:mm; use y kbs from hh:mm to hh:mm, etc. Keith: I finally got around to signing on... it only took 2 or 3 years! ___ Dirvish mailing list Dirvish@dirvish.org http://www.dirvish.org/mailman/listinfo/dirvish
Re: [Dirvish] Bare Metal Restore
Some things to beware of. It is likely the partition was not idle at the time of back up so it is not 'quite' a snapshot. Have the new disk mounted on a different machine or boot the old machine off a KNOPPIX disk. You do not want to be running at the same time you are restoring. Probably the way you want to do this is build a new system using the same install CD you used originally so that your backup is essentially just the 'upgrade'. You want to your rsync to not just update, but also delete from the target so that you are mirroring. Even then be prepared to perform some minor Unix magic, just in case. UUID's were already named. Be careful about the restore of /dev as I have sometimes had rsync barf on that; also be wary of files that get linked in a circle. Places where things can get screwed up are probably few, given that the new machine is the same as the old... if not the number of tweaks increases a bit, but you can clear the little things up over time. A new mother board probably means new drivers. There might be old mappings in in the udev sticky rules that could bite you by making your new eth0 be eth2 or some silliness. Good luck, and good hunting. I've rebuilt more than one root partition from scratch and it can be interesting, for small values of interesting... ___ Dirvish mailing list Dirvish@dirvish.org http://www.dirvish.org/mailman/listinfo/dirvish
Re: [Dirvish] Bare Metal Restore
Oh, forgot to ask. If you have grub on your boot sector of the new drive, you should be okay; but if you have lilo, you will have to do the dance of the OS transition to get the restored kernel up in place of the old kernel and thus avoid the even more fun rework of your lilo boot sector. This can be tough to do without being booted on the new root disk, and means the simple rsync mirroring can give you a non-bootable system... until you figure out how to run lilo with your disk mounted by KNOPPIX and get it to do things in the right disk context. Chances are it is grub though, so you won't have this hassle. If you do, there are probably people here who can tell you how to recover. I'd done it but its been a couple years and I'd have to think hard. I believe I worked through chroot from KNOPPIX. Don't take all my comments as a voice of doom. Chances are it will work fine. Just that once in awhile it don't. And you have the luxury that you can wipe and start over as many times as you want. Believe me, that *IS* a luxury! ___ Dirvish mailing list Dirvish@dirvish.org http://www.dirvish.org/mailman/listinfo/dirvish
Re: [Dirvish] Bare Metal Restore
On Sat, Oct 23, 2010 at 10:02:04AM +0100, Jenny Hopkins wrote: > Out of interest, if Rich copies all the dated backup directories to > the new hard drive, will the hard links be preserved between the daily > images that contain the same files or will each file copy as unique? I believe rsync -aHv does the job for transferring dirvish repositories, so I presume it would also work for writing back to disk. tar can also handle hard links and I have not checked cp to see if it has a similar switch. Probably does. ___ Dirvish mailing list Dirvish@dirvish.org http://www.dirvish.org/mailman/listinfo/dirvish
Re: [Dirvish] Backup Drive Failure
I've had USB connected drives appear to be failing but then work fine after powering down and reconnecting them. I've also had a drive that did not have good enough air flow and appeared to have failed... but after I got another cooling fan in the enclosure it worked fine and the 'bad' sectors even went away. ___ Dirvish mailing list Dirvish@dirvish.org http://www.dirvish.org/mailman/listinfo/dirvish
Re: [Dirvish] Backup Drive Failure
On Thu, Nov 04, 2010 at 06:52:48PM +0100, madworm_de.dirv...@spitzenpfeil.org wrote: > 'Self-healing' bad sector errors... but maybe I'm just paranoid. Do SATA > drives do bad sector remapping, and do they do it well? Who knows... I have multiple copies including offsite so losing one won't matter much. I don't believe it was the media, it was the drive electronics. I'm an EE so I see things as ckts :-) ___ Dirvish mailing list Dirvish@dirvish.org http://www.dirvish.org/mailman/listinfo/dirvish
Re: [Dirvish] Backup Drive Failure
On Thu, Nov 04, 2010 at 02:34:30PM -0400, Przemek Klosowski wrote: > > 'Self-healing' bad sector errors... but maybe I'm just paranoid. Do SATA > > drives do bad sector remapping, and do they do it well? Who knows... > > Certainly they do. I recommend checking the SMART firmware status by > running smartctl -a /dev/sdd or skdump /dev/sdd > On more recent kernels they can interrogate most disks across the USB > bus as well as the standard ATA-attached disks. > Look for Reallocated_Sector_Ct and especially Current_Pending_Sector : > the former shows hard errors that were recovered by replacing the bad > sector by a spare; the latter shows presently occurring errors that > could not yet be repaired. Absolutely. And in my particular case, there were not even any errors showing from smartd after the drive was brought back into it's normal temperature operating range and a full badblock check was run on the drive before reusing it. It just makes sense to me that if a device goes a bit outside its thermal range it might show temporary bad results... now if one were to keep it running well over temp, you would eventually see permanent damage. At least that's my story and I'm sticking to it. :-^ I guess it comes from having actually worked with discrete transistors, or maybe due to the mental trauma from the time I picked up a soldering iron by the wrong end one 5am in the morning... ___ Dirvish mailing list Dirvish@dirvish.org http://www.dirvish.org/mailman/listinfo/dirvish
[Dirvish] Problem with .gvfs
Has anyone else run across this headache?
sending incremental file list
rsync: readlink_stat("/home/amon/.gvfs") failed: Permission denied (13)
FATAL I/O ERROR: dying to avoid a --delete-during issue with a pre-3.0.7
receiver.
rsync error: requested action not supported (code 4) at flist.c(1800)
[sender=3.0.7]
I'm doing an rsync from root to back up a number of directories;
however, .gvfs attributes are only visible to the user, not to
root and thus it causes the back up to fail. I'd have to do
a seperate backup for each user to get around this.
Any ideas?
___
Dirvish mailing list
Dirvish@dirvish.org
http://www.dirvish.org/mailman/listinfo/dirvish
Re: [Dirvish] Problem with .gvfs
On Sat, Nov 06, 2010 at 06:29:12PM +, Jenny Hopkins wrote:
> If you search the archives, this issue has come up a few times.
> I solved the problem by adding this line to the exclude section in the
> default.conf for dirvish vault:
>
> exclude:
> /home/{user}/.gvfs
That works. I was curious about the issue of whether
there was anyway to actually back such things up, although
I can get by with not doing so.
___
Dirvish mailing list
Dirvish@dirvish.org
http://www.dirvish.org/mailman/listinfo/dirvish
Re: [Dirvish] Quick help to sort out my schedual...
On Wed, Nov 10, 2010 at 03:22:59PM +0100, Paul Slootman wrote: > On Wed 10 Nov 2010, Don Gould wrote: > > Can someone smarter than me just tell me what I need to do to make it: > > > > m-s - daily - deleted after 2 weeks > > expire-default: +15 days > > sun - keep for 12 months > # MIN HR DOM MON DOW STRFTIME_FMT > * * * * 1 +1 year > > > First and 15th of each month - just keep it, never del > > * * 1,15* * +99 years Since there will only be Sunday backup sets left after two weeks, this last line will save backups when Sunday falls on the 1st or 15th of the month. Perhaps you want 1-7,15-22 so that you will capture the 1st and second Sunday no matter where they fall in the month. Is it your intent is that this system will actually be in service for 100 years? Or that it will be moved to newer hardware as time goes on? I usually make my last line simply save at least one backup for the year with 'never' for expiry. ___ Dirvish mailing list Dirvish@dirvish.org http://www.dirvish.org/mailman/listinfo/dirvish
Re: [Dirvish] Dirvish/rsync thinking partition is 'new'
On Sun, Mar 06, 2011 at 01:35:44AM +0100, Paul Slootman wrote: > On Sat 05 Mar 2011, hanj wrote: > > > I did this and here is a partial output of some of the files: > > > > cd+ home/hanj/.audacious/.thumbs/ > > >f+ home/hanj/.audacious/.thumbs/Classic.png > > > The files on the actual server: > > > > drwxr-xr-x 2 hanj users 240 Apr 6 2007 . > > -rw-r--r-- 1 hanj users 1515 Apr 6 2007 Classic.png > > > Here is a view of the first image on the backup server in the same > > directory: > > > > drwxr-xr-x 2 back users 240 Apr 6 2007 . > > -rw-r--r-- 28 back users 1515 Apr 6 2007 Default.png > > The owner is (has become?) different. Hence dirvish needs to retransmit > those files as the metadata is different. But if the contents has not changed, the actual transmittal should be nothing by a few bytes of directory data here and there. In this case, I would expect (hope!) that the update would run very fast. I do see this as a problem though... does he end up with a (locally copied) version of the file instead of a hard link? Seems a bit on the assinine side to duplicate say, an unchanged 100GB iso image, just because the user 'touch'd, chmod'ed or chown'ed it. Really a rather serious waste of resources. ___ Dirvish mailing list Dirvish@dirvish.org http://www.dirvish.org/mailman/listinfo/dirvish
Re: [Dirvish] Dirvish/rsync thinking partition is 'new'
On Sun, Mar 06, 2011 at 01:28:29PM +0100, Paul Slootman wrote: > Dirvish's mission is to keep a completely correct backup of the source > tree. If attributes have changed, they may have been changed because > things didn't work correctly before, so it's important to have the > latest image reflect those changes. It's also possible that the changes > were in error, hence it's also important to preserve the previous image, > so simply hard-linking the files and updating the owner is wrong. > Unfortunately the only correct solution is therefore to create a new > copy of all those files. Oh futz. Hard links don't allow a new directory entry with different metadata. I had not thought that all the way through. > I don't know whether the COW implementation on top of btrfs might help > in that respect. You could get enormous savings in back up space if these problems could be solved. Another is the case where you move an iso or rename it. Nothing has changed and yet you suck up disk space like mad in your dirvish vault. It's sort of the deduplication problem... if btrfs really does solve that, it could be a very big win. ___ Dirvish mailing list Dirvish@dirvish.org http://www.dirvish.org/mailman/listinfo/dirvish
Re: [Dirvish] Dirvish/rsync thinking partition is 'new'
On Mon, Mar 07, 2011 at 12:47:23PM +0100, Paul Slootman wrote: > On Sun 06 Mar 2011, Dale Amon wrote: > > > > Oh futz. Hard links don't allow a new directory entry with different > > metadata. I had not thought that all the way through. > > I get a sense of misunderstanding here about how files, directories and > inodes really work, so I'll volunteer some info now :) I understand all too well. That's why my forehead is still sore ;-) ___ Dirvish mailing list Dirvish@dirvish.org http://www.dirvish.org/mailman/listinfo/dirvish
Re: [Dirvish] Dirvish/rsync thinking partition is 'new'
On Mon, Mar 07, 2011 at 06:49:05PM +0100, Paul Slootman wrote: > It's just that the phrase "new directory entry with different metadata" > is so wrong :-) Ah, but it is a true statement that hard link design doesn't allow for multiple sources of meta data. ___ Dirvish mailing list Dirvish@dirvish.org http://www.dirvish.org/mailman/listinfo/dirvish
Re: [Dirvish] expiry rules
On Tue, Mar 15, 2011 at 10:27:28PM +0100, Thomas Vander Stichele wrote: > Is there any way to achieve what I want, or is just not how dirvish > works ? I recommend setting your expire rules to keep one 'never' expire per month or whatever interval pleases you. You can always manually rm -rf it if you run short of space. ___ Dirvish mailing list Dirvish@dirvish.org http://www.dirvish.org/mailman/listinfo/dirvish
Re: [Dirvish] Dirvish and Strato Hi-Drive?
On Wed, Jan 04, 2012 at 04:54:08PM +0100, brian wrote: > On 01/04/2012 10:28 AM, madworm_de.dirv...@spitzenpfeil.org wrote: > >If it supports rsync with ssh transport, you should at least be able to > >duplicate/push your local backup to their site, once the local backup is > >complete. I do the reverse with my webserver, pulling its backups to my > >local removable disk. But I guess you'd like to immediately store the backup > >to a remote site. > > > >I don't know if dirvish natively supports vaults on remote hosts. And I also > >don't know if this service can be mounted with 'sshfs' to a local > >directory (which could then be used with dirvish). But as SCP is supposed to > >work... maybe ;-) Yes it does. However I've done it only with hosts that I either controlled or had admin rights to. The issue I have run across is that very few places give you simple storage at a reasonable price that you can rsync to. I'd be really happy to hear from anyone who knows of such places. ___ Dirvish mailing list Dirvish@dirvish.org http://www.dirvish.org/mailman/listinfo/dirvish
Re: [Dirvish] [administrivia] taming web spiders, particularly Baidu?
On Mon, Jul 23, 2012 at 11:31:41AM -0700, Keith Lofstrom wrote: > Is there any way to tell the search spiders to visit once a day > or once a week, rather than four times per hour? Or send them > "recent changes" lists instead of them repeatedly downloading the > same files? Any other ideas for calming down the web crawlers? I don't know of a good answer but I could suggest a hack. Keep two copies of your robots.txt file and use a cron job to swap them out for an hour or two in the middle of the night or wherever your slowest traffic period falls. ___ Dirvish mailing list Dirvish@dirvish.org http://www.dirvish.org/mailman/listinfo/dirvish
Re: [Dirvish] Init fails on one partition, the largest
On Fri, Aug 17, 2012 at 01:39:31PM -0700, Rich Shepard wrote: > On Fri, 17 Aug 2012, Rich Shepard wrote: > salmo-home/. Unmounted the drive and ran fsck; it's clean and OK. > >Have no idea where the input/output error originated. I'll see what's on > the backup drive tomorrow morning after the cron job runs. I have seen overheated drives appear to be throwing errors all over the place, but after cooled down show no errors at all. ___ Dirvish mailing list Dirvish@dirvish.org http://www.dirvish.org/mailman/listinfo/dirvish
Re: [Dirvish] 1.3.2-r657 warnings
On Wed, Nov 07, 2012 at 08:51:26AM +0100, Rolf-Werner Eilert wrote: > And what do you mean "permission from buerocrats"? When it's your code, > you can do with it what you want, it's open source after all. Not sure what it means. I cannot he imagine there is an intent to modify the dirvish API in any substantial way since it is in mission critical applications all over the world. Anyone who broke the API would have Imperial Star Troopers (ones that went to a better firearms training school) out hunting them, backed by everything from small organizations to large labs to good sized corporations. So I doubt that is what is being discussed. I am guessing the talk is about some new and entirely different application which people would migrate to or not as they chose. ___ Dirvish mailing list Dirvish@dirvish.org http://www.dirvish.org/mailman/listinfo/dirvish
Re: [Dirvish] Migration/Mirror of Dirvish code to github?
Keith: Yep. Added capability good, changed capability evil. Anyone who works on mission critical software like dirvish is playing in a very conservative swimming pool because backup breakage can be invisible for weeks or months and when discovered can mean losses of thousands of manhours of labour, and in a corporate setting, millions of dollars. So my word of advice to everyone. Don't do that. ___ Dirvish mailing list Dirvish@dirvish.org http://www.dirvish.org/mailman/listinfo/dirvish
Re: [Dirvish] Any Burning Man participants here?
On Thu, Apr 10, 2014 at 08:38:48AM -0700, Keith Lofstrom wrote: > I have an idea for a < $100 build-it-yourself, fire-resistant, > theft-resistant backup enclosure. It would be fun to make the > beta test at Burning Man at the end of August after I work out > the details and make an alpha test. Are there any Burners > here who might enjoy testing this in an amusing public way? > > Discussion off the list until we think this through. I could connect you on Facebook with a friend who works at SpaceX who does things at Burning Man. ___ Dirvish mailing list Dirvish@dirvish.org http://www.dirvish.org/mailman/listinfo/dirvish
Re: [Dirvish] transferring dirvish.org/dirvish.com, RBL
On Tue, Mar 01, 2016 at 10:18:19PM +0100, Andreas Kotes wrote: > > My neurologist tells me that I may be in the first stages > > of cognitive decline, and that I should pass important > > responsibilities on to others. Shock. Pray for nanotech. Dale Amon ___ Dirvish mailing list Dirvish@dirvish.org http://www.dirvish.org/mailman/listinfo/dirvish
