Re: HDD long-term data storage with ensured integrity

2024-05-04 Thread Marc SCHAEFER
On Fri, May 03, 2024 at 01:50:52PM -0700, David Christensen wrote: > Thank you for devising a benchmark and posting some data. :-) I did not do the comparison hosted on github. I just wrote the script which tests the dm-integrity on dm-raid error detection and error correction. > FreeBSD also o

Re: HDD long-term data storage with ensured integrity

2024-05-03 Thread David Christensen
On 5/3/24 04:26, Marc SCHAEFER wrote: On Mon, Apr 08, 2024 at 10:04:01PM +0200, Marc SCHAEFER wrote: For off-site long-term offline archiving, no, I am not using RAID. Now, as I had to think a bit about ONLINE integrity, I found this comparison: https://github.com/t13a/dm-integrity-benchmarks

Re: HDD long-term data storage with ensured integrity

2024-05-03 Thread Michael Kjörling
On 3 May 2024 13:26 +0200, from schae...@alphanet.ch (Marc SCHAEFER): > https://github.com/t13a/dm-integrity-benchmarks > > Contenders are btrfs, zfs, and notably ext4+dm-integrity+dm-raid ZFS' selling point is not performance, _especially_ on rotational drives. In fact, it's fairly widely accept

Re: HDD long-term data storage with ensured integrity

2024-05-03 Thread Marc SCHAEFER
On Mon, Apr 08, 2024 at 10:04:01PM +0200, Marc SCHAEFER wrote: > For off-site long-term offline archiving, no, I am not using RAID. Now, as I had to think a bit about ONLINE integrity, I found this comparison: https://github.com/t13a/dm-integrity-benchmarks Contenders are btrfs, zfs, and notably

Re: HDD long-term data storage with ensured integrity

2024-04-12 Thread David Christensen
On 4/12/24 08:14, piorunz wrote: On 10/04/2024 12:10, David Christensen wrote: Those sound like some compelling features. I believe the last time I tried Btrfs was Debian 9 (?).  I ran into problems because I did not do the required manual maintenance (rebalancing).  Does the Btrfs in Debian 1

Re: HDD long-term data storage with ensured integrity

2024-04-12 Thread piorunz
On 10/04/2024 12:10, David Christensen wrote: Those sound like some compelling features. I believe the last time I tried Btrfs was Debian 9 (?).  I ran into problems because I did not do the required manual maintenance (rebalancing).  Does the Btrfs in Debian 11 or Debian 12 still require manua

Re: HDD long-term data storage with ensured integrity

2024-04-10 Thread David Christensen
On 4/10/24 08:49, Paul Leiber wrote: Am 10.04.2024 um 13:10 schrieb David Christensen: Does the Btrfs in Debian 11 or Debian 12 still require manual maintenance?  If so, what and how often? Scrub and balance are actions which have been recommended. I am using btrfsmaintenance scripts [1][2] t

Re: HDD long-term data storage with ensured integrity

2024-04-10 Thread Paul Leiber
Am 10.04.2024 um 13:10 schrieb David Christensen: On 4/9/24 17:08, piorunz wrote: On 02/04/2024 13:53, David Christensen wrote: Does anyone have any comments or suggestions regarding how to use magnetic hard disk drives, commodity x86 computers, and Debian for long-term data storage with

Re: HDD long-term data storage with ensured integrity

2024-04-10 Thread Curt
On 2024-04-10, David Christensen wrote: >> >> I use Btrfs, on all my systems, including some servers, with soft Raid1 >> and Raid10 modes (because these modes are considered stable and >> production ready). I decided on Btrfs not ZFS, because Btrfs allows to >> migrate drives on the fly while par

Re: HDD long-term data storage with ensured integrity

2024-04-10 Thread David Christensen
On 4/9/24 17:08, piorunz wrote: On 02/04/2024 13:53, David Christensen wrote: Does anyone have any comments or suggestions regarding how to use magnetic hard disk drives, commodity x86 computers, and Debian for long-term data storage with ensured integrity? I use Btrfs, on all my systems

Re: HDD long-term data storage with ensured integrity

2024-04-09 Thread piorunz
On 02/04/2024 13:53, David Christensen wrote: Does anyone have any comments or suggestions regarding how to use magnetic hard disk drives, commodity x86 computers, and Debian for long-term data storage with ensured integrity? I use Btrfs, on all my systems, including some servers, with soft

Re: HDD long-term data storage with ensured integrity

2024-04-08 Thread David Christensen
On 4/8/24 13:04, Marc SCHAEFER wrote: Hello, On Mon, Apr 08, 2024 at 11:28:04AM -0700, David Christensen wrote: So, an ext4 file system on an LVM logical volume? Why LVM? Are you implementing redundancy (RAID)? Is your data larger than a single disk (concatenation/ JBOD)? Something else?

Re: Why LVM (was: HDD long-term data storage with ensured integrity)

2024-04-08 Thread DdB
Am 08.04.2024 um 23:08 schrieb Stefan Monnier: > David Christensen [2024-04-08 11:28:04] wrote: >> Why LVM? > > Personally, I've been using LVM everywhere I can (i.e. everywhere > except on my OpenWRT router, tho I've also used LVM there back when my > router had an HDD. I also use LVM on my 2GB

Why LVM (was: HDD long-term data storage with ensured integrity)

2024-04-08 Thread Stefan Monnier
David Christensen [2024-04-08 11:28:04] wrote: > Why LVM? Personally, I've been using LVM everywhere I can (i.e. everywhere except on my OpenWRT router, tho I've also used LVM there back when my router had an HDD. I also use LVM on my 2GB USB rescue image). To me the question is rather the rever

Re: HDD long-term data storage with ensured integrity

2024-04-08 Thread Marc SCHAEFER
Hello, On Mon, Apr 08, 2024 at 11:28:04AM -0700, David Christensen wrote: > So, an ext4 file system on an LVM logical volume? > > Why LVM? Are you implementing redundancy (RAID)? Is your data larger than > a single disk (concatenation/ JBOD)? Something else? For off-site long-term offline arc

Re: HDD long-term data storage with ensured integrity

2024-04-08 Thread David Christensen
On 4/8/24 02:38, Marc SCHAEFER wrote: For offline storage: On Tue, Apr 02, 2024 at 05:53:15AM -0700, David Christensen wrote: Does anyone have any comments or suggestions regarding how to use magnetic hard disk drives, commodity x86 computers, and Debian for long-term data storage with ensured

Re: HDD long-term data storage with ensured integrity

2024-04-08 Thread Marc SCHAEFER
For offline storage: On Tue, Apr 02, 2024 at 05:53:15AM -0700, David Christensen wrote: > Does anyone have any comments or suggestions regarding how to use magnetic > hard disk drives, commodity x86 computers, and Debian for long-term data > storage with ensured integrity? I use LVM on

Re: HDD long-term data storage with ensured integrity

2024-04-03 Thread Jonathan Dowland
On Tue Apr 2, 2024 at 10:57 PM BST, David Christensen wrote: > AIUI neither LVM nor ext4 have data and metadata checksum and correction > features. But, it should be possible to achieve such by including > dm-integrity (for checksumming) and some form of RAID (for correction) > in the storage s

Re: HDD long-term data storage with ensured integrity

2024-04-03 Thread David Christensen
On 4/2/24 14:57, David Christensen wrote: AIUI neither LVM nor ext4 have data and metadata checksum and correction features.  But, it should be possible to achieve such by including dm-integrity (for checksumming) and some form of RAID (for correction) in the storage stack.  I need to explore t

Re: HDD long-term data storage with ensured integrity

2024-04-02 Thread David Christensen
On 4/2/24 06:55, Stefan Monnier wrote: The most obvious alternative to ZFS on Debian would be Btrfs. Does anyone have any comments or suggestions regarding Btrfs and data corruption bugs, concurrency, CMM level, PSP, etc.? If you're worried about such things, I'd think "the most obvious altern

Re: HDD long-term data storage with ensured integrity

2024-04-02 Thread Stefan Monnier
> The most obvious alternative to ZFS on Debian would be Btrfs. Does anyone > have any comments or suggestions regarding Btrfs and data corruption bugs, > concurrency, CMM level, PSP, etc.? If you're worried about such things, I'd think "the most obvious alternative" is LVM+ext4. Both Btrfs and

HDD long-term data storage with ensured integrity

2024-04-02 Thread David Christensen
s alternative to ZFS on Debian would be Btrfs. Does anyone have any comments or suggestions regarding Btrfs and data corruption bugs, concurrency, CMM level, PSP, etc.? Does anyone have any comments or suggestions regarding how to use magnetic hard disk drives, commodity x86 computers, and Debi

large, shared data storage ???

2002-10-01 Thread Michael D. Schleif
Where can I find information to setup and run large disk arrays to be shared across multiple debian servers? We need to review all open source options; but, will also consider pointers to viable pay resources . . . -- Best Regards, mds mds resource 888.250.3987 Dare to fix things before the

Re: Data Storage

1999-02-25 Thread Peter Ludwig
On Thu, 25 Feb 1999, Stephen Lavelle wrote: > We are soon going to be installing a Linux Box on our Win98 network as a > file server - You'll notice a vast difference :) Plug here for Linux sponsored by no-one. > and i want to know of a good back up media supported by debian and easy to > conf

Re: Data Storage

1999-02-24 Thread Wojciech Zabolotny
On Thu, 25 Feb 1999, Stephen Lavelle wrote: > We are soon going to be installing a Linux Box on our Win98 network as a > file server - > and i want to know of a good back up media supported by debian and easy to > configure: > something like - zip or jazz drives. > Any suggestions? What about CD-r

Data Storage

1999-02-24 Thread Stephen Lavelle
We are soon going to be installing a Linux Box on our Win98 network as a file server - and i want to know of a good back up media supported by debian and easy to configure: something like - zip or jazz drives. Any suggestions? Regards, Stephen Lavelle Austanners Wet Blue Pty Ltd. ~ Australian Tanne