> I have the 1GHz version with 4 cores (32 threads). Ok, so same per-core speed, so single-threaded performance should be the same. (Btw, you have 8 cores, not 4. 8 cores @ 4 threads each.)
> Otherwise it's probably similar to yours. > It's running 6.0 at the moment, yes. Some guests are running -current. Was the guest in which you ran softraid crypto -current or 6.0? > Here's a dmesg from a few years ago which I copied before LDOMs were configured. > With guests configured the host dmesg changes since it has fewer resources. > > real mem = 8455716864 (8064MB) > avail mem = 8304115712 (7919MB) > mainbus0 at root: SPARC Enterprise T1000 > cpu0 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu1 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu2 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu3 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu4 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu5 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu6 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu7 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu8 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu9 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu10 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu11 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu12 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu13 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu14 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu15 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu16 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu17 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu18 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu19 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu20 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu21 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu22 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu23 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu24 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu25 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu26 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu27 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu28 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu29 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu30 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > cpu31 at mainbus0: SUNW,UltraSPARC-T1 (rev 0.0) @ 1000 MHz > vbus0 at mainbus0 > "flashprom" at vbus0 not configured > cbus0 at vbus0 > vldc0 at cbus0 > vldcp0 at vldc0 chan 0x0: ivec 0x200, 0x201 channel "hvctl" > "ldom-primary" at vldc0 chan 0x1 not configured > "fmactl" at vldc0 chan 0x3 not configured > vldc1 at cbus0 > "ldmfma" at vldc1 chan 0x4 not configured > vldc2 at cbus0 > vldcp1 at vldc2 chan 0x14: ivec 0x228, 0x229 channel "spds" > "system-management" at vldc2 chan 0xd not configured > vcons0 at vbus0: ivec 0x111, console > vrtc0 at vbus0 > "fma" at vbus0 not configured > "sunvts" at vbus0 not configured > "sunmc" at vbus0 not configured > "explorer" at vbus0 not configured > "led" at vbus0 not configured > "flashupdate" at vbus0 not configured > "ncp" at vbus0 not configured > vpci0 at mainbus0: bus 2 to 2, dvma map 80000000-ffffffff > pci0 at vpci0 > ebus0 at mainbus0 > com0 at ebus0 addr c2c000-c2c007 ivec 0xa: st16650, 32 byte fifo > vpci1 at mainbus0: bus 2 to 4, dvma map 80000000-ffffffff > pci1 at vpci1 > ppb0 at pci1 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xb3 > pci2 at ppb0 bus 3 > bge0 at pci2 dev 4 function 0 "Broadcom BCM5714" rev 0xa2, BCM5715 A1 (0x9001): ivec 0x7d4, address 00:14:4f:ae:b5:28 > brgphy0 at bge0 phy 1: BCM5714 10/100/1000baseT/SX PHY, rev. 0 > bge1 at pci2 dev 4 function 1 "Broadcom BCM5714" rev 0xa2, BCM5715 A1 (0x9001): ivec 0x7d5, address 00:14:4f:ae:b5:29 > brgphy1 at bge1 phy 1: BCM5714 10/100/1000baseT/SX PHY, rev. 0 > ppb1 at pci2 dev 8 function 0 "ServerWorks HT-1000 PCIX" rev 0xb3 > pci3 at ppb1 bus 4 > bge2 at pci3 dev 1 function 0 "Broadcom BCM5704C" rev 0x10, BCM5704 B0 (0x2100): ivec 0x7c2, address 00:14:4f:ae:b5:2a > brgphy2 at bge2 phy 1: BCM5704 10/100/1000baseT PHY, rev. 0 > bge3 at pci3 dev 1 function 1 "Broadcom BCM5704C" rev 0x10, BCM5704 B0 (0x2100): ivec 0x7c1, address 00:14:4f:ae:b5:2b > brgphy3 at bge3 phy 1: BCM5704 10/100/1000baseT PHY, rev. 0 > mpi0 at pci3 dev 2 function 0 "Symbios Logic SAS1064" rev 0x02: msi > scsibus0 at mpi0: 63 targets > sd0 at scsibus0 targ 0 lun 0: <ATA, HITACHI HDS7250S, AJ0A> SCSI3 0/direct fixed naa.5000cca20ec9a366 > sd0: 476940MB, 512 bytes/sector, 976773168 sectors > vscsi0 at root > scsibus1 at vscsi0: 256 targets > softraid0 at root > scsibus2 at softraid0: 256 targets > bootpath: /pci@7c0,0/pci@0,0/pci@8,0/scsi@2,0/disk@0,0 > root on sd0a (c29f49f8ceac7c2e.a) swap on sd0b dump on sd0b Your dmesg looks similar to mine, except for a few differences: * Mine has mpath0 at root (missing in your output) * Mine has scsibus0 at mpath0: 256 targets (missing in your output) * Yours says SPARC Enterprise T1000, while mine says Sun Fire(TM) T1000 as mainbus0 * Mine has double the RAM * Yours shows 8 CPUs more * vldcp0 has different values: vldcp0 at vldc0 chan 0x0: ivec 0x0, 0x1 channel "hvctl" * vldcp1 has different values: vldcp1 at vldc2 chan 0x14: ivec 0x28, 0x29 channel "spds" * Different MAC addresses (obviously!) * Mine says mpi0: UNUSED, firmware 1.10.0.0 * sd0 at scsibus1 targ 0 lun 0: <LSILOGIC, Logical Volume, 3000> SCSI2 0/direct fixed naa.600508e0000000002ce78b8be2a9da06 * sd1 at scsibus3 targ 1 lun 0: <OPENBSD, SR CRYPTO, 006> SCSI2 0/direct fixed Since you said you have only 2 cores for the host, I decided to boot bsd.sp instead. This doubled the unencrypted disk performance to 80 MB/s, and more than doubled the encrypted disk performance to 4.5 MB/s. So there is definitely a problem with lock contention. In your output, it was 60 MB/s vs. 40 MB/s, which seems more reasonable, but almost too fast: 40 MB/s sounds like more than a single core should be able to do? Are you sure that LDOM was indeed using softraid crypto? I guess what I should also try is disabling the hardware RAID-1 again - though I guess that means I need to reinstall. -- Jonathan
