Making gmirror metadata cooperate with gpt metadata

[Abner Gershon]

Disclaimer, I am not a programmer but have tinkered with shell and perl
scripts. Having said that, I think many would appreciate the option of
using gmirror with disks larger than 2T.

The reason this is currently impossible is due to GPT and gmirror both
trying to store metadata in the last disk sector.

MBR does not allow for partitioning of disk greater 2T.

Workaround to gmirror partitions rather than whole disk is less than ideal.

Lines 254-293 of geom_mirror.c « mirror « geom « lib - src - FreeBSD source
tree

relate
to clearing and storing metadata on last sector of disk.

What would be the ramifications of altering gmirror to store metadata on
the 2nd to last sector?

Am I looking at the correct region to alter this code?

-Abner
___
freebsd-geom@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-geom
To unsubscribe, send any mail to "freebsd-geom-unsubscr...@freebsd.org"

Re: Making gmirror metadata cooperate with gpt metadata

[Paul Mather]

On Feb 7, 2021, at 8:50 AM, Abner Gershon <6731...@gmail.com> wrote:

> Disclaimer, I am not a programmer but have tinkered with shell and perl
> scripts. Having said that, I think many would appreciate the option of
> using gmirror with disks larger than 2T.
> 
> The reason this is currently impossible is due to GPT and gmirror both
> trying to store metadata in the last disk sector.
> 
> MBR does not allow for partitioning of disk greater 2T.
> 
> Workaround to gmirror partitions rather than whole disk is less than ideal.


Personally, I don't find mirroring individual partitions "less than ideal" or 
at least onerous enough to enforce "whole-disk" semantics.  For one, it lets 
you use different balance algorithms for different partitions, e.g., "prefer" 
on mirrored swap (so crash dumps work without extra effort) and other balance 
algorithms for other file systems.  For another, you might potentially have to 
rebuild less data, depending on the failure that leads to loss of mirror 
synchronisation.  In a whole-disk mirror, you are always going to have to 
resynchronise the whole disk, whereas with partition-level mirrors you might be 
lucky and have to resynchronise only a single partition.  Also, you could turn 
automatic resynchronisation off where it's not needed, e.g., for auto-encrypted 
swap partitions.

Gmirror's naive resilvering is also why I prefer to use ZFS for mirroring: it 
resilvers (resynchronises) only actual data, and so is much faster for pools 
that are emptier.

(I only use gmirror for swap nowadays and use ZFS for everything else.)


> Lines 254-293 of geom_mirror.c « mirror « geom « lib - src - FreeBSD source
> tree
> 
> relate
> to clearing and storing metadata on last sector of disk.
> 
> What would be the ramifications of altering gmirror to store metadata on
> the 2nd to last sector?


This is basically what it ends up doing already.  You make a GPT covering the 
whole disk and the GPT uses the last sector for the backup.  You then make a 
gmirror inside this GPT container and it uses the last sector of the container 
(second-to-last sector of the drive) for its gmirror stored metadata.

Also, couldn't you achieve your end using BSD partitions, or are they 
disallowed inside GPT partitions?  (I.e., create a GPT covering the whole disk; 
create a GPT partition covering the entire GPT; create a gmirror out of that; 
create a BSD partition on the gmirror for your file systems.)

Cheers,

Paul.
___
freebsd-geom@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-geom
To unsubscribe, send any mail to "freebsd-geom-unsubscr...@freebsd.org"

Problem reports for g...@freebsd.org that need special attention

[bugzilla-noreply]

To view an individual PR, use:
  https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id).

The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status  |Bug Id | Description
+---+---
In Progress |218679 | [geli] add a verify command   
Open|237269 | panic in glabel (g_label_destroy) stop after resi 
Open|238814 | geom: topology lock being dropped in dumpconf of  
Open|242747 | geli: AMD Epyc+GELI not using Hardware AES

4 problems total for which you should take action.
___
freebsd-geom@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-geom
To unsubscribe, send any mail to "freebsd-geom-unsubscr...@freebsd.org"

Re: Making gmirror metadata cooperate with gpt metadata

[Abner Gershon]

Wow, thanks for opening my eyes to this. I did not realize that BSD
partitions may be layered on top of a GPT partition.

If I understand this correctly, you are suggesting I use fdisk to partition
a GPT partition?

BSD partitions are still limited to 2TB but that would be 2TB per GPT
partition.

My concern with gmirror on partition level is that I have seen a comment
about how this may cause excessive stress on disk due to contention between
various sectors for writing data during initial mirroring. In other words
will gmirror update one partition at a time or will disk write head jitter
back and forth writing 4k to 1st partition, 4k to 2nd partition, 4k to 3rd
partition, etc.

I have been resisting ZFS because of inefficiencies of COW for updating
database files ( as opposed to updating one block of existing file ). I
plan to set up a server with some UFS gmirror and some ZFS storage and do
some comparisons. Will post back my results when I do. Most related posts I
have seen suggest ZFS is the way of the now/future but then again I am
driving a 1988 Ford ranger with manual transmission.

Regards,
Abner


On Sun, Feb 7, 2021 at 3:28 PM Paul Mather  wrote:

> On Feb 7, 2021, at 8:50 AM, Abner Gershon <6731...@gmail.com> wrote:
>
> > Disclaimer, I am not a programmer but have tinkered with shell and perl
> > scripts. Having said that, I think many would appreciate the option of
> > using gmirror with disks larger than 2T.
> >
> > The reason this is currently impossible is due to GPT and gmirror both
> > trying to store metadata in the last disk sector.
> >
> > MBR does not allow for partitioning of disk greater 2T.
> >
> > Workaround to gmirror partitions rather than whole disk is less than
> ideal.
>
>
> Personally, I don't find mirroring individual partitions "less than ideal"
> or at least onerous enough to enforce "whole-disk" semantics.  For one, it
> lets you use different balance algorithms for different partitions, e.g.,
> "prefer" on mirrored swap (so crash dumps work without extra effort) and
> other balance algorithms for other file systems.  For another, you might
> potentially have to rebuild less data, depending on the failure that leads
> to loss of mirror synchronisation.  In a whole-disk mirror, you are always
> going to have to resynchronise the whole disk, whereas with partition-level
> mirrors you might be lucky and have to resynchronise only a single
> partition.  Also, you could turn automatic resynchronisation off where it's
> not needed, e.g., for auto-encrypted swap partitions.
>
> Gmirror's naive resilvering is also why I prefer to use ZFS for mirroring:
> it resilvers (resynchronises) only actual data, and so is much faster for
> pools that are emptier.
>
> (I only use gmirror for swap nowadays and use ZFS for everything else.)
>
>
> > Lines 254-293 of geom_mirror.c « mirror « geom « lib - src - FreeBSD
> source
> > tree
> > <
> https://cgit.freebsd.org/src/tree/lib/geom/mirror/geom_mirror.c?h=releng/13.0
> >
> > relate
> > to clearing and storing metadata on last sector of disk.
> >
> > What would be the ramifications of altering gmirror to store metadata on
> > the 2nd to last sector?
>
>
> This is basically what it ends up doing already.  You make a GPT covering
> the whole disk and the GPT uses the last sector for the backup.  You then
> make a gmirror inside this GPT container and it uses the last sector of the
> container (second-to-last sector of the drive) for its gmirror stored
> metadata.
>
> Also, couldn't you achieve your end using BSD partitions, or are they
> disallowed inside GPT partitions?  (I.e., create a GPT covering the whole
> disk; create a GPT partition covering the entire GPT; create a gmirror out
> of that; create a BSD partition on the gmirror for your file systems.)
>
> Cheers,
>
> Paul.
___
freebsd-geom@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-geom
To unsubscribe, send any mail to "freebsd-geom-unsubscr...@freebsd.org"

Re: Making gmirror metadata cooperate with gpt metadata

[Paul Mather]

On Feb 7, 2021, at 4:23 PM, Abner Gershon <6731...@gmail.com> wrote:

> Wow, thanks for opening my eyes to this. I did not realize that BSD
> partitions may be layered on top of a GPT partition.


Hopefully it was clear from my original reply that I wasn't sure you could do 
this and you should try it out for yourself (e.g., in a VM or using an 
md-disk). :-)

It's not clear whether it is possible from the gpart man page.  For the BSD 
partitioning scheme it says, "Traditional BSD disklabel, usually used to 
subdivide MBR partitions.  (This scheme can also be used as the sole 
partitioning method, without an MBR. ..."  It's not clear to me whether you 
could create a partitioning scheme in this way and still have the resultant 
system boot via EFI or legacy BIOS---it's the latter "being able to boot" which 
is the most important, IMHO.


> If I understand this correctly, you are suggesting I use fdisk to partition
> a GPT partition?


No, my thought was just to add a partition of type "freebsd" inside the GPT.  I 
do note that the gpart man page discourages its use: "This is a legacy 
partition type and should not be used for the APM or GPT schemes."  Then, as I 
said above, there is the matter of whether a FreeBSD boot loader could 
successfully boot from such a layout. :-\


> My concern with gmirror on partition level is that I have seen a comment
> about how this may cause excessive stress on disk due to contention between
> various sectors for writing data during initial mirroring. In other words
> will gmirror update one partition at a time or will disk write head jitter
> back and forth writing 4k to 1st partition, 4k to 2nd partition, 4k to 3rd
> partition, etc.


To be honest, I don't remember what it does because I only use gmirror for swap 
nowadays, but I have a sneaking suspicion from memory that it was subject to 
the jitter you mention (at least years ago when I was still gmirroring UFS 
filesystems).  I ended up turning off autosynchronisation and doing it myself 
on the occasions when the mirror broke.

For initial mirroring you could make a special case for synchronising, if you 
were worried about disk stress.  People are increasingly using SSDs for OS 
drives nowadays, so stress from mechanical head movement becomes a moot point 
in that case.  (In fact, all those layout and rotational optimisations in UFS 
designed to minimise physical head movement and rotational latency become moot 
in that case.)


> I have been resisting ZFS because of inefficiencies of COW for updating
> database files ( as opposed to updating one block of existing file ).


Don't databases themselves use COW techniques to ensure data safety?


> I
> plan to set up a server with some UFS gmirror and some ZFS storage and do
> some comparisons. Will post back my results when I do. Most related posts I
> have seen suggest ZFS is the way of the now/future but then again I am
> driving a 1988 Ford ranger with manual transmission.


There's nothing wrong with sticking with what you know and what you feel 
comfortable with, and with what you believe best accommodates your use case.

Others in this thread have made some great points regarding not dismissing ZFS 
as an option.  I agree with what they said.  I've used FreeBSD since version 3 
and used ZFS from the moment it was available in FreeBSD (version 7).  Here's 
what I would add/echo to what has already been said as plus points for ZFS:

- Pooled storage: no more gnashing teeth over badly sizing your filesystems
- Snapshots: cheap and reliable; I never felt the same way about UFS snapshots
- Flexible filesets: tune the behaviour of "filesytems" according to use cases
- Integrity and durability: advanced "RAID" setups and data integrity 
protections throughout
- Administration: better control over administration and delegation of your 
file systems
- Efficiency: tiered storage model

As regards ZFS being "new," it would be fair to say that it has received more 
active development in the last few years than UFS.  I actually switched from 
UFS to ZFS on FreeBSD/arm64 (on a 1 GB Raspberry Pi) because I was tired of 
losing data due to UFS+SUJ crashing with non-fsck-able file systems.  Snapshots 
on UFS have also had a chequered history of working properly (e.g., for 
consistent dumps).  Data safety is the #1 thing that attracts me to ZFS.

Hopefully, data safety is important to you, too.  Also, one big concern I would 
have in changing the semantics of gmirror, as you propose, is the easy 
availability of rescue tools should something happen to my storage causing 
everything to go pear shaped. :-)  You'd have to factor it into your disaster 
recovery plan.

Cheers,

Paul.
___
freebsd-geom@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-geom
To unsubscribe, send any mail to "freebsd-geom-unsubscr...@freebsd.org"

Re: Making gmirror metadata cooperate with gpt metadata

[Abner Gershon]

Thanks for the clarifications and additional info.

-Ab

On Sun, Feb 7, 2021 at 5:56 PM Paul Mather  wrote:

> On Feb 7, 2021, at 4:23 PM, Abner Gershon <6731...@gmail.com> wrote:
>
> > Wow, thanks for opening my eyes to this. I did not realize that BSD
> > partitions may be layered on top of a GPT partition.
>
>
> Hopefully it was clear from my original reply that I wasn't sure you could
> do this and you should try it out for yourself (e.g., in a VM or using an
> md-disk). :-)
>
> It's not clear whether it is possible from the gpart man page.  For the
> BSD partitioning scheme it says, "Traditional BSD disklabel, usually used
> to subdivide MBR partitions.  (This scheme can also be used as the sole
> partitioning method, without an MBR. ..."  It's not clear to me whether you
> could create a partitioning scheme in this way and still have the resultant
> system boot via EFI or legacy BIOS---it's the latter "being able to boot"
> which is the most important, IMHO.
>
>
> > If I understand this correctly, you are suggesting I use fdisk to
> partition
> > a GPT partition?
>
>
> No, my thought was just to add a partition of type "freebsd" inside the
> GPT.  I do note that the gpart man page discourages its use: "This is a
> legacy partition type and should not be used for the APM or GPT schemes."
> Then, as I said above, there is the matter of whether a FreeBSD boot loader
> could successfully boot from such a layout. :-\
>
>
> > My concern with gmirror on partition level is that I have seen a comment
> > about how this may cause excessive stress on disk due to contention
> between
> > various sectors for writing data during initial mirroring. In other words
> > will gmirror update one partition at a time or will disk write head
> jitter
> > back and forth writing 4k to 1st partition, 4k to 2nd partition, 4k to
> 3rd
> > partition, etc.
>
>
> To be honest, I don't remember what it does because I only use gmirror for
> swap nowadays, but I have a sneaking suspicion from memory that it was
> subject to the jitter you mention (at least years ago when I was still
> gmirroring UFS filesystems).  I ended up turning off autosynchronisation
> and doing it myself on the occasions when the mirror broke.
>
> For initial mirroring you could make a special case for synchronising, if
> you were worried about disk stress.  People are increasingly using SSDs for
> OS drives nowadays, so stress from mechanical head movement becomes a moot
> point in that case.  (In fact, all those layout and rotational
> optimisations in UFS designed to minimise physical head movement and
> rotational latency become moot in that case.)
>
>
> > I have been resisting ZFS because of inefficiencies of COW for updating
> > database files ( as opposed to updating one block of existing file ).
>
>
> Don't databases themselves use COW techniques to ensure data safety?
>
>
> > I
> > plan to set up a server with some UFS gmirror and some ZFS storage and do
> > some comparisons. Will post back my results when I do. Most related
> posts I
> > have seen suggest ZFS is the way of the now/future but then again I am
> > driving a 1988 Ford ranger with manual transmission.
>
>
> There's nothing wrong with sticking with what you know and what you feel
> comfortable with, and with what you believe best accommodates your use case.
>
> Others in this thread have made some great points regarding not dismissing
> ZFS as an option.  I agree with what they said.  I've used FreeBSD since
> version 3 and used ZFS from the moment it was available in FreeBSD (version
> 7).  Here's what I would add/echo to what has already been said as plus
> points for ZFS:
>
> - Pooled storage: no more gnashing teeth over badly sizing your filesystems
> - Snapshots: cheap and reliable; I never felt the same way about UFS
> snapshots
> - Flexible filesets: tune the behaviour of "filesytems" according to use
> cases
> - Integrity and durability: advanced "RAID" setups and data integrity
> protections throughout
> - Administration: better control over administration and delegation of
> your file systems
> - Efficiency: tiered storage model
>
> As regards ZFS being "new," it would be fair to say that it has received
> more active development in the last few years than UFS.  I actually
> switched from UFS to ZFS on FreeBSD/arm64 (on a 1 GB Raspberry Pi) because
> I was tired of losing data due to UFS+SUJ crashing with non-fsck-able file
> systems.  Snapshots on UFS have also had a chequered history of working
> properly (e.g., for consistent dumps).  Data safety is the #1 thing that
> attracts me to ZFS.
>
> Hopefully, data safety is important to you, too.  Also, one big concern I
> would have in changing the semantics of gmirror, as you propose, is the
> easy availability of rescue tools should something happen to my storage
> causing everything to go pear shaped. :-)  You'd have to factor it into
> your disaster recovery plan.
>
> Cheers,
>
> Paul.
__