> Here are some more findings... > > The Nexenta box has 3 pools: > syspool: made of 2 mirrored (hardware RAID) local SAS > disks > pool_sas: made of 22 15K SAS disks in ZFS mirrors on > 2 JBODs on 2 controllers > pool_sata: made of 42 SATA disks in 6 RAIDZ2 vdevs on > a single controller > > When we copy data from any linux box to either the > pool_sas or pool_sata, it is painfully slow. > > When we copy data from any linux box directly to the > syspool, it is plenty fast > > When we copy data locally on the Nexenta box from the > syspool to either the pool_sas or pool_sata, it is > crazy fast. > > We also see the same pattern whether we use iSCSI or > NFS. We've also tested using different NICs (some at > 1GbE, some at 10GbE) and even tried bypassing the > switch by directly connecting the two boxes with a > cable- and it didn't made any difference. We've also > tried not using the SSD for the ZIL. > > So... > We've ruled out iSCSI, the networking, the ZIL > device, even the HBAs as it is fast when it is done > locally. > > Where should we look next? > > Thank you all for your help! > Ian
Looking at the list suggested earlier: 1. Linux network stack 2. Linux iSCSI issues 3. Network cabling/switch between the devices 4. Nexenta CPU constraints (unlikely, I know, but let's be thorough) 5. Nexenta network stack 6. COMSTAR problems It looks like you have ruled out everything. The only thing that still stands out is that network operations (iSCSI and NFS) to external drives are slow, correct? Just for completeness, what happens if you scp a file to the three different pools? If the results are the same as NFS and iSCSI, then I think the network can be ruled out. I would be leaning toward thinking there is some mismatch between the network protocols and the external controllers/cables/arrays. Are the controllers the same hardware/firmware/driver for the internal vs. external drives? Keep digging. I think you are getting close. Cheers, Marty -- This message posted from opensolaris.org _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
