[DNG] rdiff-backup compatibility

[Hendrik Boom]

rdiff-bckup has changed its communication protocol between versions 1 
and 2, so it is necessary to have compatible versions of rdiff-backup at 
both ends of an rdiff-backup over a network, and a version 2 
beowulf backport is available for rdiff for those who need to bridge 
rdiff-backup between older and newer systems.

I have used the backport to achieve this network-protocol compatibility.
What I would like to know:

Has the file format for the rdiff-backup metadata changed between 
versions 1 and 2 in any incompatible way?

I'm having trouble with backing up using version 2 to a version 1 backup 
directory.  I am wondering whether this is becuse of an incompatibility 
between the backup file formats, or because the backup has become 
corrupt by other means.

I would expect rdiff-backup to detect old backup formats and deal with 
them, but then, I would have expected rdiff-backup to deal with old 
networking protocols as well.

-- hendrik 

___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] rdiff-backup compatibility

[Mark Hindley]

This is documented in Debian's Bullseye Release Notes and referenced from
Devuan's Chimaera Release Notes.

Mark
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

[DNG] Long-term archiving versus medium fallibility

[Hendrik Boom]

These days an increasing amount of my personal information, bookd, 
mementos, family photos, and work data are being kept on digital media.  
And those are vulnerable.

It's well-known that to archive files long-term (say, ten years or more) 
it is necessary to keep multiple copies, preferably on different media.

So this is what I'd like to do with my critical files.

Yet these files are also working files, are kept online, and 
legitinately need to be modified from time to time.

So I keep backups.  Currently I use rdiff-backup, which does have the
ablity to keep older as well as newer versions of files on the same 
backup drive.  And I keep multple backups.

(This might even help somewhat against ransomware attacks)

--

Now storage media deteriorate over time.
It is necessary to read and transfer data from old media to new from 
time to time.  Yes, I know that.  My present method is to keep 
everything on my server, and make regular backups.

(OK, my backuos aren't all *that* regular, but I try)

Now the master copy is the working file system of my server.
And even in the absence of ransomware, there are occasional disk 
failures.

Yes, I use a RAID so any detected disk failures don't cause immediate 
data loss.  (it also has the side effect of letting me continue running 
apparently unaffected from the time a disk has failed until I manage to 
replace it.

And I also use the ext4 file system against unexpected shutdowns.  Yes, 
I journal everything, not just metadata.  So after a crash, or 
unexpected power outage, the file system is easy to restore to a 
consistent state.

Now for further protection against data failure, I'd like to introduce 
checksumming.  This is available with btrfs and zfs (or is it xfs?  I 
forget which is which).

But ... all of this relies on valid RAM.

Copying files to or from backup, updating files, all of it is done by 
copying into RAM and then copying it from RAM.  In the presence of 
faulty RAM, even a backup copy could be seriously damaged.

And this is worse with the newer b-tree file systems, which are 
constantly copying data. - even data which hasn't changed.  A single 
update will read a large block of data from disk, make the changes, and 
write it back.  The entire block is this written back, complete with 
changes, bit-failures from RAM problems, and a new check-sum to validate 
the bad bits.

I'm told the maintainers of thse file-systems laugh at you if you're not 
using ECC memory.

---

Now I'm wondering how to introduce chack-summing to protect against this 
kind of data loss despite occasionally (but rarely) filing memory.

* I could run memory checks frequently to catch failing memory.  But the 
circumstances in ordinary operation differ from the circumstances of the 
memory check program, and faulty memory might fail to be detected.

* I could install ECC memory.  But that has become difficlt to get, and 
some processors on the mass market won't even handle it properly.

* I could hope the ext4 developers will add checksums to the ext4 file 
system, possibly renaming it to ext5.

* Or I could try doing my own checksuming.  Perhaps checksumming 
everythin in the file system and catching files whose checksums have 
changed without a new modification date.  This could be done at backup 
time, flaggin such discrepancies for manual attention.  (note: need to 
check the checksum on the backup, too).

---

Anyone have other ideas?

-- hendrik 




___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] rdiff-backup compatibility

[Hendrik Boom]

On Mon, Nov 01, 2021 at 11:41:58AM +, Mark Hindley wrote:
> This is documented in Debian's Bullseye Release Notes and referenced from
> Devuan's Chimaera Release Notes.
> 
> Mark

The need to upgrade server and client in lockstep is indeed documented 
in the Bullseye release notes.  I ahve done that.

What is not mentioned is whether the backup archive format has changed 
incompatibly.
  
That's what I was asking.

-- hendrik
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] rdiff-backup compatibility

[Mark Hindley]

Hendrick,

On Mon, Nov 01, 2021 at 08:42:01AM -0400, Hendrik Boom wrote:
> What is not mentioned is whether the backup archive format has changed 
> incompatibly.
>   
> That's what I was asking.

Sorry. My misunderstanding.

Mark
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] mutt and html

[Jarkko Lavinen via Dng]

On Sat, Oct 30, 2021 at 04:13:31PM -0400, Hendrik Boom wrote:
> Until upgraded from ascii to beowulf, HTML messages were tolerable.

I am using Mutt with palemoon browser on Chimera. I chose separate browser to
avoid interference with firefox. The view command works just like you describe
it.

Jarkko Lavinen
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] snetaid debs...

[aitor]

Hi,

On 25/10/21 8:22, aitor wrote:


Yesterday night i pushed the new code to git. Packages of libnetaid, snetaid 
and simple-netaid-cdk
(still not updated in gitea.devuan.dev) for chimaera will be available in a 
couple of days.


The code of simple-netaid is ready to use:


- libnetaid: the shared library

https://gitea.devuan.dev/aitor_czr/libnetaid/src/branch/gbp-master  


- snetaid: the daemon

https://gitea.devuan.dev/aitor_czr/snetaid/src/branch/gbp-master  


- simple-netaid-cdk: the ncurses interface

https://gitea.devuan.dev/aitor_czr/simple-netaid-cdk/src/branch/gbp-master  


Packages will be available in a few hours...

Sorry for the delay, and thanks for your interest in the project :)

Cheers,

Aitor.


___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] Long-term archiving versus medium fallibility

[Steve Litt]

Hendrik Boom said on Mon, 1 Nov 2021 08:11:54 -0400


>Now storage media deteriorate over time.
>It is necessary to read and transfer data from old media to new from 
>time to time.  Yes, I know that.  My present method is to keep 
>everything on my server, and make regular backups.

Backup is, and always has been a problem. I've written several times on
the subject:

http://troubleshooters.com/tpromag/9807.htm

http://troubleshooters.com/lpm/200208/200208.htm

http://troubleshooters.com/lpm/200609/200609.htm

http://troubleshooters.com/lpm/201406/201406.htm

I've discovered that long term restorability depends on several things:

* How standard is the backup format? .tgz will be around for decades,
  Fastback format became almost impossible to read after a decade.

* How standard is the backup media? I have some QIC tapes from the mid
  90's that I don't even try to read. I have some CPM formatted
  floppies from 1984 that I'm still trying to find a way to read.
  Meanwhile, my ISO9660 CDs from the late 1990's, holding 8.3 filename
  format .tgz files, are trivial to read.

* How robust is the backup media? Consumer grade tapes are probably
  unreadable within a couple years. Some consumer grade tapes are
  unrestorable one second after making them. Even worse are floppy
  backups. My experience tells me that flash drives, even if formatted
  as ext4 instead of that guaranteed data loss Windows format, go bad.
  USB spinning rust hard drives have so far restored for me, but the
  whole concept is scary. SSD is too expensive per GB to use for
  backup. My restorability successes mostly revolve around CD and DVD
  ISO 9660 holding tarballs in 8.3 format. I keep them in a rolltop box
  to shelter them from light, and my house is air conditioned, and I see
  no degradation so far. My results using Blu-Ray are mixed. Some still
  restore, but some went bad, and I mean visibly bad where you could
  see discolorations, within a couple years.

Anybody whose backups would require less than 5 DVDs, I'd recommend
DVDs with ISO9660 8.3 filename tarballs, additionally storing an md5
checksum for each tarball. I've never had an md5 file read
differently than the md5 value of the tarball, even after a decade.

If more than 5 DVDs worth, it's tougher. I use a combination of
UDF formatted Blu-Ray media and USB spinning rust hard drives.

I keep most of my backups in a bank safety deposit vault. Try to get a
box as high off the ground as you can, to prevent flood damage.

HTH,

SteveT

Steve Litt 
Spring 2021 featured book: Troubleshooting Techniques of the Successful
Technologist http://www.troubleshooters.com/techniques
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

[DNG] November GoLUG meeting

[Steve Litt]

Hi all,

The November GoLUG meeting is Wednesday, 11/3/2021, at 7PM Orlando
time. I'll be presenting on the QOwnNotes authoring software. See
http://golug.info for details.

SteveT

Steve Litt 
Spring 2021 featured book: Troubleshooting Techniques of the Successful 
Technologist
http://www.troubleshooters.com/techniques
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng

Re: [DNG] Long-term archiving versus medium fallibility

[Syeed Ali]

On Mon, 1 Nov 2021 08:11:54 -0400
Hendrik Boom  wrote:

> Yet these files are also working files, are kept online, and
> legitinately need to be modified from time to time.

> My present method is to keep everything on my server, and make
> regular backups.

My first suggestion is to decide why you're keeping everything
together.  Perhaps there needs to be a separation between

  Convenience; everything in one place (and subject to the same
  procedures, benefits and flaws)

  Confidence; some things subject to enhanced archival techniques or
  media, such as using M-DISC [1]

Perhaps there does not need to be a separation.  Maybe the effort is
too high or the reward is not great enough.  Make this decision.

Maybe unchanging media, such as your media archives (photographs,
video, poetry) can be on M-DISC locally, with a second copy offsite.

Thinking about M-DISC and changes, it makes me wonder if multi-track / 
multi-session technology [2] is available to them.  This would allow
the appending of data to either add more files, or "modify" existing
files (versioning in a sense).


> Currently I use rdiff-backup, which does have the ablity to keep
> older as well as newer versions of files on the same backup drive.

Does rdiff-backup have the ability to keep older versions on different
storage?  If so, you could shunt those data to a more reliable medium.
It would also save you space for your "live" data, giving some
side-benefits like reduced cost.


> Now storage media deteriorate over time.
> 
> It is necessary to read and transfer data from old media to new from
> time to time.

If these things can be solved or made less difficult, it would help
alleviate a lot of pressure on the rest of your backups.  This is why I
suggested M-DISC [1]

-

As for your other thoughts, I have no experience.  Regarding
filesystems, I think it was ZFS that you mean. [3]





1.  https://en.wikipedia.org/wiki/M-DISC


2.  Regarding multi-session, I'm unfamiliar with it but see:

  "Red Book"  (1980), regarding CD-ROMs in general
  https://en.wikipedia.org/wiki/Rainbow_Books

  "Blue Book" (1995), regarding E-CD/CD+/CD Extra (Enhanced)
  https://en.wikipedia.org/wiki/Blue_Book_(CD_standard)
  https://en.wikipedia.org/wiki/Enhanced_CD

Is multi-session obscure knowledge?  Maybe USB storage rose to
prominence at about the same time.


3.  https://openzfs.github.io/openzfs-docs/Basic%20Concepts/Checksums.html
___
Dng mailing list
Dng@lists.dyne.org
https://mailinglists.dyne.org/cgi-bin/mailman/listinfo/dng