Hi!
One of the disks of my softraid array of a four disk RAID5 has failed,
so I went on and replaced it with an identical make/model, and now I'm
trying to rebuild the RAID5 array:
So far I've replaced the HDD, fdisk/disklabeled it exactly the same as the
other drives:
# fdisk sd4 <--- one of the old disks
Disk: sd4 geometry: 267349/255/63 [4294961685 Sectors]
Offset: 0 Signature: 0xAA55
Starting Ending LBA Info:
#: id C H S - C H S [ start: size ]
-------------------------------------------------------------------------------
0: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
1: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
2: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
*3: A6 0 1 2 - 267348 254 63 [ 64: 4294961621 ] OpenBSD
# fdisk sd5 <--- new disk
Disk: sd5 geometry: 267349/255/63 [4294961685 Sectors]
Offset: 0 Signature: 0xAA55
Starting Ending LBA Info:
#: id C H S - C H S [ start: size ]
-------------------------------------------------------------------------------
0: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
1: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
2: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
*3: A6 0 1 2 - 267348 254 63 [ 64: 4294961621 ] OpenBSD
# disklabel sd4 <--- old disk
# /dev/rsd4c:
type: SCSI
disk: SCSI disk
label: WDC WD30EFRX-68E
duid: 2e2a1c57ee8b6dc3
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 364801
total sectors: 5860533168
boundstart: 64
boundend: 5860533168
drivedata: 0
16 partitions:
# size offset fstype [fsize bsize cpg]
a: 5860533104 64 RAID
c: 5860533168 0 unused
# disklabel sd5 <--- new disk
# /dev/rsd5c:
type: SCSI
disk: SCSI disk
label: WDC WD30EFRX-68E
duid: e095270eb1a82ff1
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 364801
total sectors: 5860533168
boundstart: 64
boundend: 5860533168
drivedata: 0
16 partitions:
# size offset fstype [fsize bsize cpg]
a: 5860533104 64 RAID
c: 5860533168 0 unused
Now when I boot the system, I get this in my dmesg (the full dmesg is
in-line at the end):
sd0 at scsibus1 targ 0 lun 0: <ATA, KINGSTON SV300S3, 603A> SCSI3 0/direct
fixed naa.50026b7759058c0d
sd0: 228936MB, 512 bytes/sector, 468862128 sectors, thin
sd1 at scsibus1 targ 1 lun 0: <ATA, KINGSTON SV300S3, 603A> SCSI3 0/direct
fixed naa.50026b7755033793
sd1: 228936MB, 512 bytes/sector, 468862128 sectors, thin
sd2 at scsibus1 targ 2 lun 0: <ATA, WDC WD30EFRX-68E, 82.0> SCSI3 0/direct
fixed naa.50014ee262ccb19e
sd2: 2861588MB, 512 bytes/sector, 5860533168 sectors
sd3 at scsibus1 targ 3 lun 0: <ATA, WDC WD30EFRX-68E, 82.0> SCSI3 0/direct
fixed naa.50014ee20d7712a7
sd3: 2861588MB, 512 bytes/sector, 5860533168 sectors
sd4 at scsibus1 targ 4 lun 0: <ATA, WDC WD30EFRX-68E, 82.0> SCSI3 0/direct
fixed naa.50014ee2b82277a5
sd4: 2861588MB, 512 bytes/sector, 5860533168 sectors
sd5 at scsibus1 targ 5 lun 0: <ATA, WDC WD30EFRX-68E, 82.0> SCSI3 0/direct
fixed naa.50014ee20e71f0c5
sd5: 2861588MB, 512 bytes/sector, 5860533168 sectors
sd6 at scsibus2 targ 1 lun 0: <StoreJet, TS512GESD400K, 0> SCSI4 0/direct fixed
serial.174c07d1B8502108567B
sd6: 488386MB, 512 bytes/sector, 1000215216 sectors
softraid0 at root
scsibus4 at softraid0: 256 targets
sd7 at scsibus4 targ 1 lun 0: <OPENBSD, SR RAID 1, 006> SCSI2 0/direct fixed
sd7: 228933MB, 512 bytes/sector, 468856433 sectors
softraid0: not all chunks were provided; attempting to bring volume 1 online
softraid0: trying to bring up sd7 degraded
softraid0: sd7 is offline, will not be brought online
root on sd7a (625bf83321ffc7ea.a) swap on sd7b dump on sd7b
sd8 at scsibus4 targ 2 lun 0: <OPENBSD, SR CRYPTO, 006> SCSI2 0/direct fixed
sd8: 155872MB, 512 bytes/sector, 319227056 sectors
But this is already suspicious. sd7 shouldn't have any problems, that's a two
disk RAID1 setup, the system being on it, and everything I do I'm doing on that
system right now. So maybe a bug in softraid logging?
Having said that, what I think softraid was trying to tell me is that I
have a missing disk from the RAID5 setup and I'd need to rebuild it with
the new disk as the new chunk -- is this correct?
Originally my softraid0 looked like this:
# bioctl softraid0
Volume Status Size Device
softraid0 0 Online 240054493696 sd7 RAID1
0 Online 240054493696 0:0.0 noencl <sd1a>
1 Online 240054493696 0:1.0 noencl <sd0a>
softraid0 1 Degraded 9001777889280 sd8 RAID5
0 Online 3000592678912 1:0.0 noencl <sd2a>
1 Online 3000592678912 1:1.0 noencl <sd3a>
2 Online 3000592678912 1:2.0 noencl <sd4a>
3 Offline 3000592678912 1:3.0 noencl <sd5a>
softraid0 2 Online 163444252672 sd9 CRYPTO
0 Online 163444252672 2:0.0 noencl <sd7l>
softraid0 3 Online 9001777487872 sd10 CRYPTO
0 Online 9001777487872 3:0.0 noencl <sd8a>
softraid0 4 Online 512105629696 sd11 CRYPTO
0 Online 512105629696 4:0.0 noencl <sd6a>
And now I have this:
Volume Status Size Device
softraid0 0 Online 240054493696 sd7 RAID1
0 Online 240054493696 0:0.0 noencl <sd1a>
1 Online 240054493696 0:1.0 noencl <sd0a>
softraid0 1 Online 163444252672 sd8 CRYPTO
0 Online 163444252672 1:0.0 noencl <sd7l>
So basically this is missing (but sd5a being Online :) ):
softraid0 1 Degraded 9001777889280 sd8 RAID5
0 Online 3000592678912 1:0.0 noencl <sd2a>
1 Online 3000592678912 1:1.0 noencl <sd3a>
2 Online 3000592678912 1:2.0 noencl <sd4a>
3 Offline 3000592678912 1:3.0 noencl <sd5a>
[...]
softraid0 3 Online 9001777487872 sd10 CRYPTO
0 Online 9001777487872 3:0.0 noencl <sd8a>
So the system disks (RAID1) are there, sd7l is decrypted as sd8 (so
that's also okay), and I'm missing the array of sd2,sd3,sd4,sd5. (no
worries about sd6 and sd11, that's an external drive, I've removed that
from the equation for now).
Every documentation I read tells me to rebuild the array like so:
# bioctl -R /dev/<new disk> <softraid disk>
So I guess I could run eg. `bioctl -R /dev/sd5a sd8`, if I had sd8 as a
degraded RAID5 array, but I don't have that -- softraid0 told me that it won't
bring it online, because of the missing chunk.
Is this kind of a catch 22, and somewehere along the road I messed up my array
completely and now I'm stuck here?
Could someone please hit me with a cluestick? :)
Thanks!
Daniel
full dmesg:
OpenBSD 6.1 (GENERIC.MP) #7: Mon Jun 12 20:41:01 CEST 2017
rob...@syspatch-61-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 17055162368 (16265MB)
avail mem = 16533594112 (15767MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xec250 (77 entries)
bios0: vendor American Megatrends Inc. version "F5" date 10/22/2014
bios0: Gigabyte Technology Co., Ltd. H97M-D3H
acpi0 at bios0: rev 2
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP APIC FPDT SSDT SSDT SSDT MCFG HPET SSDT SSDT DMAR
acpi0: wakeup devices PEGP(S4) PEG0(S4) PEGP(S4) PEG1(S4) PEGP(S4) PEG2(S4)
PXSX(S4) RP01(S4) PXSX(S4) PXSX(S4) RP03(S4) PXSX(S4) RP04(S4) PXSX(S4)
PXSX(S4) PXSX(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Core(TM) i5-4690S CPU @ 3.20GHz, 3200.38 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POP
CNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,SENSOR,ARAT
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: TSC frequency 3200379150 Hz
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 100MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Core(TM) i5-4690S CPU @ 3.20GHz, 3200.00 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POP
CNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,SENSOR,ARAT
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
cpu2: Intel(R) Core(TM) i5-4690S CPU @ 3.20GHz, 3200.00 MHz
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POP
CNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,SENSOR,ARAT
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) Core(TM) i5-4690S CPU @ 3.20GHz, 3200.00 MHz
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POP
CNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,SENSOR,ARAT
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 3, package 0
ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
acpimcfg0 at acpi0 addr 0xf8000000, bus 0-63
acpihpet0 at acpi0: 14318179 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus -1 (PEG0)
acpiprt2 at acpi0: bus -1 (PEG1)
acpiprt3 at acpi0: bus -1 (PEG2)
acpiprt4 at acpi0: bus 1 (RP01)
acpiprt5 at acpi0: bus 2 (RP03)
acpiprt6 at acpi0: bus 3 (RP04)
acpiec0 at acpi0: not present
acpicpu0 at acpi0: C2(200@117 mwait.1@0x30), C1(1000@1 mwait.1), PSS
acpicpu1 at acpi0: C2(200@117 mwait.1@0x30), C1(1000@1 mwait.1), PSS
acpicpu2 at acpi0: C2(200@117 mwait.1@0x30), C1(1000@1 mwait.1), PSS
acpicpu3 at acpi0: C2(200@117 mwait.1@0x30), C1(1000@1 mwait.1), PSS
acpipwrres0 at acpi0: PG00, resource for PEG0
acpipwrres1 at acpi0: PG01, resource for PEG1
acpipwrres2 at acpi0: PG02, resource for PEG2
acpipwrres3 at acpi0: FN00, resource for FAN0
acpipwrres4 at acpi0: FN01, resource for FAN1
acpipwrres5 at acpi0: FN02, resource for FAN2
acpipwrres6 at acpi0: FN03, resource for FAN3
acpipwrres7 at acpi0: FN04, resource for FAN4
acpitz0 at acpi0: critical temperature is 97 degC
acpitz1 at acpi0: critical temperature is 97 degC
"INT3F0D" at acpi0 not configured
"PNP0501" at acpi0 not configured
"INT33CA" at acpi0 not configured
acpibtn0 at acpi0: PWRB
acpibtn1 at acpi0: SLPB
"INT3394" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
acpivideo0 at acpi0: GFX0
acpivout0 at acpivideo0: DD1F
cpu0: Enhanced SpeedStep 3200 MHz: speeds: 3201, 3200, 3000, 2900, 2700, 2500,
2300, 2200, 2000, 1800, 1700, 1500, 1300, 1100, 1000, 800 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Core 4G Host" rev 0x06
inteldrm0 at pci0 dev 2 function 0 "Intel HD Graphics 4600" rev 0x06
drm0 at inteldrm0
inteldrm0: msi
inteldrm0: 1024x768, 32bpp
wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation)
wsdisplay0: screen 1-5 added (std, vt100 emulation)
azalia0 at pci0 dev 3 function 0 "Intel Core 4G HD Audio" rev 0x06: msi
azalia0: No codecs found
xhci0 at pci0 dev 20 function 0 "Intel 9 Series xHCI" rev 0x00: msi
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00
addr 1
"Intel 9 Series MEI" rev 0x00 at pci0 dev 22 function 0 not configured
ehci0 at pci0 dev 26 function 0 "Intel 9 Series USB" rev 0x00: apic 8 int 16
usb1 at ehci0: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00
addr 1
ppb0 at pci0 dev 28 function 0 "Intel 9 Series PCIE" rev 0xd0
pci1 at ppb0 bus 1
ppb1 at pci0 dev 28 function 2 "Intel 9 Series PCIE" rev 0xd0: msi
pci2 at ppb1 bus 2
ppb2 at pci0 dev 28 function 3 "Intel 9 Series PCIE" rev 0xd0: msi
pci3 at ppb2 bus 3
ppb3 at pci3 dev 0 function 0 "Intel 82801BA Hub-to-PCI" rev 0x41
pci4 at ppb3 bus 4
bge0 at pci4 dev 1 function 0 "Broadcom BCM5705" rev 0x01, BCM5705 A1 (0x3001):
apic 8 int 16, address 00:xx
brgphy0 at bge0 phy 1: BCM5705 10/100/1000baseT PHY, rev. 1
ehci1 at pci0 dev 29 function 0 "Intel 9 Series USB" rev 0x00: apic 8 int 23
usb2 at ehci1: USB revision 2.0
uhub2 at usb2 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00
addr 1
pcib0 at pci0 dev 31 function 0 "Intel H97 LPC" rev 0x00
ahci0 at pci0 dev 31 function 2 "Intel 9 Series AHCI" rev 0x00: msi, AHCI 1.3
ahci0: port 0: 6.0Gb/s
ahci0: port 1: 6.0Gb/s
ahci0: port 2: 6.0Gb/s
ahci0: port 3: 6.0Gb/s
ahci0: port 4: 6.0Gb/s
ahci0: port 5: 6.0Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, KINGSTON SV300S3, 603A> SCSI3 0/direct
fixed naa.50026b7759058c0d
sd0: 228936MB, 512 bytes/sector, 468862128 sectors, thin
sd1 at scsibus1 targ 1 lun 0: <ATA, KINGSTON SV300S3, 603A> SCSI3 0/direct
fixed naa.50026b7755033793
sd1: 228936MB, 512 bytes/sector, 468862128 sectors, thin
sd2 at scsibus1 targ 2 lun 0: <ATA, WDC WD30EFRX-68E, 82.0> SCSI3 0/direct
fixed naa.50014ee262ccb19e
sd2: 2861588MB, 512 bytes/sector, 5860533168 sectors
sd3 at scsibus1 targ 3 lun 0: <ATA, WDC WD30EFRX-68E, 82.0> SCSI3 0/direct
fixed naa.50014ee20d7712a7
sd3: 2861588MB, 512 bytes/sector, 5860533168 sectors
sd4 at scsibus1 targ 4 lun 0: <ATA, WDC WD30EFRX-68E, 82.0> SCSI3 0/direct
fixed naa.50014ee2b82277a5
sd4: 2861588MB, 512 bytes/sector, 5860533168 sectors
sd5 at scsibus1 targ 5 lun 0: <ATA, WDC WD30EFRX-68E, 82.0> SCSI3 0/direct
fixed naa.50014ee20e71f0c5
sd5: 2861588MB, 512 bytes/sector, 5860533168 sectors
ichiic0 at pci0 dev 31 function 3 "Intel 9 Series SMBus" rev 0x00: apic 8 int 18
iic0 at ichiic0
spdmem0 at iic0 addr 0x51: 8GB DDR3 SDRAM PC3-12800
spdmem1 at iic0 addr 0x53: 8GB DDR3 SDRAM PC3-12800
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vmm0 at mainbus0: VMX/EPT
umass0 at uhub0 port 18 configuration 1 interface 0 "StoreJet TS512GESD400K"
rev 3.00/10.00 addr 2
umass0: using SCSI over Bulk-Only
scsibus2 at umass0: 2 targets, initiator 0
sd6 at scsibus2 targ 1 lun 0: <StoreJet, TS512GESD400K, 0> SCSI4 0/direct fixed
serial.174c07d1B8502108567B
sd6: 488386MB, 512 bytes/sector, 1000215216 sectors
uhub3 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev
2.00/0.00 addr 2
uhub4 at uhub2 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev
2.00/0.00 addr 2
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
sd7 at scsibus4 targ 1 lun 0: <OPENBSD, SR RAID 1, 006> SCSI2 0/direct fixed
sd7: 228933MB, 512 bytes/sector, 468856433 sectors
softraid0: not all chunks were provided; attempting to bring volume 1 online
softraid0: trying to bring up sd7 degraded
softraid0: sd7 is offline, will not be brought online
root on sd7a (625bf83321ffc7ea.a) swap on sd7b dump on sd7b
sd8 at scsibus4 targ 2 lun 0: <OPENBSD, SR CRYPTO, 006> SCSI2 0/direct fixed
sd8: 155872MB, 512 bytes/sector, 319227056 sectors
--
LÉVAI Dániel
PGP key ID = 0x83B63A8F
Key fingerprint = DBEC C66B A47A DFA2 792D 650C C69B BE4C 83B6 3A8F
