pvcreate can't open /dev/md1 exclusively

[Matthew Pounsett]

New problem getting FAI to run cleanly with LVM.  I'm sure I'm missing
something, but again I'm not sure what.  There was a similar question on
the list in Nov 2021, but I don't see an answer for it.



The tail end of my format.log looks like this:

Executing: yes | mdadm --create /dev/md0 --level=raid1 --force --run
 --raid-devices=2 /dev/sdb1 /dev/sda1
Executing: echo frozen | tee /sys/block/md0/md/sync_action
Executing: mkfs.ext4  /dev/md0
Executing: yes | mdadm --create /dev/md1 --level=raid1 --force --run
 --raid-devices=2 /dev/sda3 /dev/sdb3
Executing: echo frozen | tee /sys/block/md1/md/sync_action
Executing: pvcreate -ff -y  /dev/md1
pvcreate -ff -y  /dev/md1 had exit code 5
(STDERR)   Can't open /dev/md1 exclusively.  Mounted filesystem?
(STDERR)   Can't open /dev/md1 exclusively.  Mounted filesystem?
(STDERR)   Error opening device /dev/md1 for reading at 0 length 512.
(STDERR)   Can't open /dev/md1 exclusively.  Mounted filesystem?
(STDERR)   Error opening device /dev/md1 for reading at 0 length 4096.
(STDERR)   Cannot use /dev/md1: device has a signature
Command had non-zero exit code

The server has four drives:  two SSDs that I'm trying to mirror and put
volumes on, and two larger HDDs that FAI should be ignoring.   It does seem
to be ignoring the two HDDs now, thanks to help in the last thread.

$disklist shows up as expected:
root@srv03:/etc/lvm# echo $disklist
sda sdb sdc sdd

I found some suggestions in various places, that mostly seem to be echoing
this redhat article: https://access.redhat.com/solutions/110203 ... but so
far no solutions.  The md device is not mounted yet, and there should be no
reason for FAI holding the device open at this point in the install (that I
can think of, anyway).  `fuser -m -v /dev/md1` indicates no processes
holding the device open.

I have tried making sure any trace of previous data on the drives is gone,
just to make sure pvcreate isn't running up against its own old data, but I
get the same result after each run.  My cleanup log (run between FAI
attempts) is pasted below.  What else should I be trying?

root@srv03:/tmp/fai# pvscan
  PV /dev/md1   VG vg_system   lvm2 [<430.66 GiB / <4.18 GiB free]
  Total: 1 [<430.66 GiB] / in use: 1 [<430.66 GiB] / in no VG: 0 [0   ]

root@srv03:/tmp/fai# dmsetup table
vg_system-home: 0 632250368 linear 9:1 52430848
vg_system-root: 0 52428800 linear 9:1 2048
vg_system-usr: 0 209715200 linear 9:1 684681216

root@srv03:/tmp/fai# lsblk
NAME MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda8:00   7.3T  0 disk
├─sda1 8:10   350M  0 part
│ └─md09:00   349M  0 raid1
├─sda2 8:2016G  0 part
├─sda3 8:30   7.3T  0 part
│ └─md19:10   7.3T  0 raid1
│   ├─vg_system-root 254:0025G  0 lvm
│   ├─vg_system-home 254:10 301.5G  0 lvm
│   └─vg_system-usr  254:20   100G  0 lvm
└─sda4 8:40 1M  0 part
sdb8:16   0   7.3T  0 disk
├─sdb1 8:17   0   350M  0 part
│ └─md09:00   349M  0 raid1
├─sdb2 8:18   016G  0 part
├─sdb3 8:19   0   7.3T  0 part
│ └─md19:10   7.3T  0 raid1
│   ├─vg_system-root 254:0025G  0 lvm
│   ├─vg_system-home 254:10 301.5G  0 lvm
│   └─vg_system-usr  254:20   100G  0 lvm
└─sdb4 8:20   0 1M  0 part
sdc8:32   0 447.1G  0 disk
sdd8:48   0 447.1G  0 disk


root@srv03:/tmp/fai# for dev in vg_system-home vg_system-root
vg_system-usr; do dmsetup remove $dev; done

root@srv03:/tmp/fai# dmsetup table
No devices found

root@srv03:/tmp/fai# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
root@srv03:/tmp/fai# mdadm --stop /dev/md1
mdadm: stopped /dev/md1
root@srv03:/tmp/fai# for dev in sda1 sda3 sdb1 sdb3; do mdadm
--zero-superblock /dev/$dev; done

root@srv03:/tmp/fai# for dev in sda1 sda3 sdb1 sdb3; do mdadm
--zero-superblock /dev/$dev; done
root@srv03:/tmp/fai# for device in $disklist; do wipefs -a /dev/${device};
dd if=/dev/zero of=/dev/${device} bs=512 seek=$(( $(blockdev --getsz
/dev/${device}) - 1024 )) count=1024; done
/dev/sda: 8 bytes were erased at offset 0x0200 (gpt): 45 46 49 20 50 41
52 54
/dev/sda: 8 bytes were erased at offset 0x74702555e00 (gpt): 45 46 49 20 50
41 52 54
/dev/sda: 2 bytes were erased at offset 0x01fe (PMBR): 55 aa
/dev/sda: calling ioctl to re-read partition table: Success
1024+0 records in
1024+0 records out
524288 bytes (524 kB, 512 KiB) copied, 0.0108355 s, 48.4 MB/s
/dev/sdb: 8 bytes were erased at offset 0x0200 (gpt): 45 46 49 20 50 41
52 54
/dev/sdb: 8 bytes were erased at offset 0x74702555e00 (gpt): 45 46 49 20 50
41 52 54
/dev/sdb: 2 bytes were erased at offset 0x01fe (PMBR): 55 aa
/dev/sdb: calling ioctl to re-read partition table: Success
1024+

Re: pvcreate can't open /dev/md1 exclusively

[Christopher Huhn]

Hi Matthew,

Am 24.01.25 um 18:18 schrieb Matthew Pounsett:
The server has four drives:  two SSDs that I'm trying to mirror and put 
volumes on, and two larger HDDs that FAI should be ignoring.   It does 
seem to be ignoring the two HDDs now, thanks to help in the last thread.

[…]
$disklist shows up as expected:
root@srv03:/etc/lvm# echo $disklist
sda sdb sdc sdd

> […]

root@srv03:/tmp/fai# lsblk
NAME MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda    8:0    0   7.3T  0 disk
sdb    8:16   0   7.3T  0 disk
sdc    8:32   0 447.1G  0 disk
sdd    8:48   0 447.1G  0 disk
as you can see here the large disks are sda (disk1) and sdb (disk2) 
while the SSDs are sdc and sdd.
This mapping is not stable through reboots, so you have to make sure 
your disks are in the intended order in your disklist.


We use a very basic hook to sort our disks by size:

> # cat class/31-sortdisks.sh
> disklist=$(grep -H '^[1-9]' /sys/block/[hsv]d[a-z]/size 
/sys/block/nvme*n[0-9]/size | sort -n -t : -k 2 | cut -f4 -d / | xargs)

> echo "disklist=\"$disklist\"" >> "$LOGDIR/additional.var"

IIRC there were some discussions on this list on how to create a 
reliably sorted disklist but nothing seem to have made it to the example 
FAI config on github.


Best

Christopher


smime.p7s
Description: Kryptografische S/MIME-Signatur

Re: pvcreate can't open /dev/md1 exclusively

[Matthew Pounsett]

On Fri, Jan 24, 2025 at 1:07 PM Christopher Huhn  wrote:

> > sda8:00   7.3T  0 disk
> > sdb8:16   0   7.3T  0 disk
> > sdc8:32   0 447.1G  0 disk
> > sdd8:48   0 447.1G  0 disk
> as you can see here the large disks are sda (disk1) and sdb (disk2)
> while the SSDs are sdc and sdd.
> This mapping is not stable through reboots, so you have to make sure
> your disks are in the intended order in your disklist.
>

Ooof.  Embarrassing. I'm aware it's unstable, but it had been so stable in
my previous tests I stopped paying attention to that.  Thanks for the
second pair of eyes.

Unfortunately though, that doesn't fix my problem.  After correcting for
that, I'm still seeing the same LVM error.


> We use a very basic hook to sort our disks by size:
>
>  > # cat class/31-sortdisks.sh
>  > disklist=$(grep -H '^[1-9]' /sys/block/[hsv]d[a-z]/size
> /sys/block/nvme*n[0-9]/size | sort -n -t : -k 2 | cut -f4 -d / | xargs)
>  > echo "disklist=\"$disklist\"" >> "$LOGDIR/additional.var"
>

This looks REALLY helpful!  Thanks!

I'm making one small modification, using sort's -s modifier... in the
(likely futile) hope that the drives might still be in bus order. :)

Re: setup-storage pulling in disks it should ignore

[Matthew Pounsett]

On Fri, Jan 24, 2025 at 7:57 AM Thomas Lange  wrote:

> > On Thu, 23 Jan 2025 17:33:56 -0500, Matthew Pounsett <
> m...@conundrum.com> said:
>> I've used dd to wipe all four drives of data left behind by the
> previous hardware controller, and
> > when I started the install all four drives had no partitions, but
> somewhere along the way
> > setup-storage seems to have written partitions to all four drives,
> not just the first two.
> Did you used the command wipefs -a on all disks and partitions?
> Maybe also use dd and write some MB of zeros onto the disks. There may
> be signatures of soft raid and GPT tables at the end of the disk.


I'm already using dd to write zeros to the disks ("I've used dd to wipe all
four drives of data left behind by the previous hardware controller"
above), which has been sufficient in the past.  I guess I've never run into
a case where FAI assigned the same UUID to two different disks before ...
but wipefs seems to have worked around that.  Thanks for the suggestion!

Unfortunately, I'm now getting an entirely new error from LVM, for which
I'm running out of troubleshooting steps.  I'll start a new thread for that
if I can't solve it.

Thanks for the help!

Re: setup-storage pulling in disks it should ignore

[Thomas Lange]

> On Thu, 23 Jan 2025 17:33:56 -0500, Matthew Pounsett  
> said:

> Executing: mdadm --assemble --scan 
--config=/tmp/fai/mdadm-from-examine.conf
> mdadm --assemble --scan --config=/tmp/fai/mdadm-from-examine.conf had 
exit code 1
> (STDERR) mdadm: Devices UUID-01c77a40:7ab3107d:583f6322:117cbec4 and
> UUID-c5e32267:fd576f2b:ee8febe0:48c9d35a have the same name: /dev/md/0
> (STDERR) mdadm: Duplicate MD device names in conf file were found.
> Command had non-zero exit code
IMO this is the problem. For whatever reason there are still some UUID
on other disks.

> I've used dd to wipe all four drives of data left behind by the previous 
hardware controller, and
> when I started the install all four drives had no partitions, but 
somewhere along the way
> setup-storage seems to have written partitions to all four drives, not 
just the first two.
Did you used the command wipefs -a on all disks and partitions?
Maybe also use dd and write some MB of zeros onto the disks. There may
be signatures of soft raid and GPT tables at the end of the disk. 

In your format.log we only see sdb. So another FAI run must have
written to other disks.

Also check $disklist in /tmp/fai/variables.log

-- 
regards Thomas