Re: [zfs-discuss] Performance issues with iSCSI under Linux

Here's a long due update for you all... After updating countless drivers, BIOSes and Nexenta, it seems that our issue has disappeared. We're slowly moving our production to our three appliances and things are going well so far. Sadly we don't know exactly what update fixed our issue. I wish I

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> Then set the zfs_write_limit_override to a reasonable > value. Our first experiments are showing progress. We'll play with it some more and let you know. Thanks! Ian -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discu

Re: [zfs-discuss] Performance issues with iSCSI under Linux

Ross Walker wrote: On Nov 1, 2010, at 5:09 PM, Ian D wrote: Maybe you are experiencing this: http://opensolaris.org/jive/thread.jspa?threadID=11942 It does look like this... Is this really the expected behaviour? That's just unacceptable. It is so bad it sometimes drop connection

Re: [zfs-discuss] Performance issues with iSCSI under Linux

On Nov 1, 2010, at 5:09 PM, Ian D wrote: >> Maybe you are experiencing this: >> http://opensolaris.org/jive/thread.jspa?threadID=11942 > > It does look like this... Is this really the expected behaviour? That's just > unacceptable. It is so bad it sometimes drop connection and fail copies and

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> Maybe you are experiencing this: > http://opensolaris.org/jive/thread.jspa?threadID=11942 It does look like this... Is this really the expected behaviour? That's just unacceptable. It is so bad it sometimes drop connection and fail copies and SQL queries... Ian -- This message posted from

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> You "doubt" AMD or Intel cpu's suffer from bad cache > mgmt? In order to clear that out, we've tried using an older server (about 4 years old) as the head and we see the same pattern. It's actually more obvious that it consumes a whole lot of CPU cycles. Using the same box as a Linux-based N

Re: [zfs-discuss] Performance issues with iSCSI under Linux

Maybe you are experiencing this: http://opensolaris.org/jive/thread.jspa?threadID=119421 -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Performance issues with iSCSI under Linux

- Original Message - > > Likely you don't have enough ram or CPU in the box. > > The Nexenta box has 256G of RAM and the latest X7500 series CPUs. That > said, the load does get crazy high (like 35+) very quickly. We can't > figure out what's taking so much CPU. It happens even when > chec

Re: [zfs-discuss] Performance issues with iSCSI under Linux

Check your TXG settings, it could be a timing issue, nagles issue, also TCP buffer issue. Check setup system properties. On 1 Nov 2010 19:36, "SR" wrote: What if you connect locally via NFS or iscsi? SR -- This message posted from opensolaris.org __

Re: [zfs-discuss] Performance issues with iSCSI under Linux

What if you connect locally via NFS or iscsi? SR -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> If you do a dd to the storage from the heads do > you still get the same issues? no, local read/writes are great, they never choke. It's whenever NFS or iSCSI are involved and that the read/writes are done from a remote box that we experience the problem. Local operations barely affects the

Re: [zfs-discuss] Performance issues with iSCSI under Linux

If you do a dd to the storage from the heads do you still get the same issues? On 31 Oct 2010 12:40, "Ian D" wrote: I get that multi-cores doesn't necessarily better performances, but I doubt that both the latest AMD CPUs (the Magny-Cours) and the latest Intel CPUs (the Beckton) suffer from incr

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > boun...@opensolaris.org] On Behalf Of Ian D > > I get that multi-cores doesn't necessarily better performances, but I > doubt that both the latest AMD CPUs (the Magny-Cours) and the latest > Intel CPUs (the Beckton) suffer from inc

Re: [zfs-discuss] Performance issues with iSCSI under Linux

I get that multi-cores doesn't necessarily better performances, but I doubt that both the latest AMD CPUs (the Magny-Cours) and the latest Intel CPUs (the Beckton) suffer from incredibly bad cache management. Our two test system have 2 and 4 of each respectively. The thing is that the perform

Re: [zfs-discuss] Performance issues with iSCSI under Linux

Right, I realized it was Magny not Mangy, but I thought it was related to the race track or racing not a town. I completely agree with you on codenames, the Linux distro codename irk me - hey, guys it might be easy for you to keep track of which release is "Bushy Beaver" or "Itchy Ibis" or "Ma

Re: [zfs-discuss] Performance issues with iSCSI under Linux

On 10/30/2010 7:07 PM, zfs user wrote: I did it deliberately - how dumb are these product managers that they name products with weird names and not expect them to be abused? On the other hand, if you do a search for mangy cours you'll find a bunch of hits where it is clearly a misspelling on se

Re: [zfs-discuss] Performance issues with iSCSI under Linux

I did it deliberately - how dumb are these product managers that they name products with weird names and not expect them to be abused? On the other hand, if you do a search for mangy cours you'll find a bunch of hits where it is clearly a misspelling on serious tech articles, postings, etc. "

Re: [zfs-discuss] Performance issues with iSCSI under Linux

If you take a look at http://www.brendangregg.com/cachekit.html you will see some DTrace yummyness which should let you tell... --- W. A. Khushil Dep - khushil@gmail.com - 07905374843 Visit my blog at http://www.khushil.com/ On 30 October 2010 15:49, Eugen Leitl wrote: > On Sat, Oct

Re: [zfs-discuss] Performance issues with iSCSI under Linux

On Sat, Oct 30, 2010 at 02:10:49PM -0700, zfs user wrote: > 1 Mangy-Cours CPU ^ Dunno whether deliberate, or malapropism, but I love it. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-

Re: [zfs-discuss] Performance issues with iSCSI under Linux

We had the same issue with a 24 core box a while ago. Check your l2 cache hits and misses. Sometimes more cores does not mean more performance dtrace is your friend! On 30 Oct 2010 14:12, "zfs user" wrote: Here is a total guess - but what if it has to do with zfs processing running on one CPU ha

Re: [zfs-discuss] Performance issues with iSCSI under Linux

Here is a total guess - but what if it has to do with zfs processing running on one CPU having to talk to the memory "owned" by a different CPU? I don't know if many people are running fully populated boxes like you are, so maybe it is something people are not seeing due to not having huge amoun

Re: [zfs-discuss] Performance issues with iSCSI under Linux

So maybe a next step is to run zilstat, arcstat, iostat -xe?? (I forget what people like to use for these params), zpool iostat -v in 4 term windows while running the same test and try to see what is spiking when that high load period occurs. Not sure if there is a better version than this: h

Re: [zfs-discuss] Performance issues with iSCSI under Linux

I owe you all an update... We found out a clear pattern we can now recreate at will. Whenever we read/write the pool, it gives expected throughput and IOPS for a while, but at some point it slows down to a crawl, nothing is responding and pretty much "hang" for a few seconds and then things go

Re: [zfs-discuss] Performance issues with iSCSI under Linux

On Oct 23, 2010, at 4:31 AM, Ian D wrote: >> Likely you don't have enough ram or CPU in the box. > > The Nexenta box has 256G of RAM and the latest X7500 series CPUs. That said, > the load does get crazy high (like 35+) very quickly. We can't figure out > what's taking so much CPU. It happen

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> I don't think the switch model was ever identified...perhaps it is a 1 GbE > switch with a few 10 GbE ports?  (Drawing at straws.) It is a a Dell 8024F. It has 24 SPF+ 10GbE ports and every NICs we connect to it are Intel X520. One issue we do have with it is when we turn jumbo frames on,

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> Likely you don't have enough ram or CPU in the box. The Nexenta box has 256G of RAM and the latest X7500 series CPUs. That said, the load does get crazy high (like 35+) very quickly. We can't figure out what's taking so much CPU. It happens even when checksum/compression/unduping are off.

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> A network switch that is being maxed out? Some > switches cannot switch > at rated line speed on all their ports all at the > same time. Their > internal buses simply don't have the bandwidth needed > for that. Maybe > you are running into that limit? (I know you > mentioned bypassing the

Re: [zfs-discuss] Performance issues with iSCSI under Linux

Tim Cook wrote: On Fri, Oct 22, 2010 at 10:40 PM, Haudy Kazemi > wrote: One thing suspicious is that we notice a slow down of one pool when the other is under load. How can that be? Ian A network switch that is being maxed o

Re: [zfs-discuss] Performance issues with iSCSI under Linux

What more info could you provide? Quite a lot more, actually, like: how many streams of SQL and copy are you running? how are the filesystems/zvols configured (recordsize, etc)? some CPU, VM and network stats would also be nice. Based on the nexenta iostats you've provided (a tiny window on what

Re: [zfs-discuss] Performance issues with iSCSI under Linux

On Fri, Oct 22, 2010 at 10:40 PM, Haudy Kazemi wrote: > > >> One thing suspicious is that we notice a slow down of one pool when the >> other is under load. How can that be? >> >> Ian >> >> > A network switch that is being maxed out? Some switches cannot switch at > rated line speed on all thei

Re: [zfs-discuss] Performance issues with iSCSI under Linux

One thing suspicious is that we notice a slow down of one pool when the other is under load. How can that be? Ian A network switch that is being maxed out? Some switches cannot switch at rated line speed on all their ports all at the same time. Their internal buses simply don't have t

Re: [zfs-discuss] Performance issues with iSCSI under Linux

Some numbers... zpool status pool: Pool_sas state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM Pool_sas ONLINE 0 0 0 c4t5000C506A6D3d0 ONLINE 0 0 0 c4t5000C506

Re: [zfs-discuss] Performance issues with iSCSI under Linux

On Oct 15, 2010, at 5:34 PM, Ian D wrote: >> Has anyone suggested either removing L2ARC/SLOG >> entirely or relocating them so that all devices are >> coming off the same controller? You've swapped the >> external controller but the H700 with the internal >> drives could be the real culprit. Coul

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> Has anyone suggested either removing L2ARC/SLOG > entirely or relocating them so that all devices are > coming off the same controller? You've swapped the > external controller but the H700 with the internal > drives could be the real culprit. Could there be > issues with cross-controller IO in t

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> -Original Message- > From: zfs-discuss-boun...@opensolaris.org > [mailto:zfs-discuss-boun...@opensolaris.org] On Behalf Of Ian D > Sent: Friday, October 15, 2010 4:19 PM > To: zfs-discuss@opensolaris.org > Subject: Re: [zfs-discuss] Performance issues with iSCSI

Re: [zfs-discuss] Performance issues with iSCSI under Linux

On 15 oct. 2010, at 22:19, Ian D wrote: > A little setback We found out that we also have the issue with the Dell > H800 controllers, not just the LSI 9200-16e. With the Dell it's initially > faster as we benefit from the cache, but after a little while it goes sour- > from 350MB/sec down

Re: [zfs-discuss] Performance issues with iSCSI under Linux

A little setback We found out that we also have the issue with the Dell H800 controllers, not just the LSI 9200-16e. With the Dell it's initially faster as we benefit from the cache, but after a little while it goes sour- from 350MB/sec down to less than 40MB/sec. We've also tried with a

Re: [zfs-discuss] Performance issues with iSCSI under Linux

After contacting LSI they say that the 9200-16e HBA is not supported in OpenSolaris, just Solaris. Aren't Solaris drivers the same as OpenSolaris? Is there anyone here using 9200-16e HBAs? What about the 9200-8e? We have a couple lying around and we'll test one shortly. Ian -- This message

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> Does the Linux box have the same issue to any other > server ? > What if the client box isn't Linux but Solaris or > Windows or MacOS X ? That would be a good test. We'll try that. -- This message posted from opensolaris.org ___ zfs-discuss mailing l

Re: [zfs-discuss] Performance issues with iSCSI under Linux

On 15/10/2010 19:09, Ian D wrote: It's only when a Linux box SEND/RECEIVE data to the NFS/iSCSI shares that we have problems. But if the Linux box send/receive file through scp on the external disks mounted by the Nexenta box as a local filesystem then there is no problem. Does the Linux bo

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> As I have mentioned already, it would be useful to > know more about the > onfig, how the tests are being done, and to see some > basic system > performance stats. I will shortly. Thanks! -- This message posted from opensolaris.org ___ zfs-discuss

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> You mentioned a second Nexenta box earlier. To rule > out client-side issues, have you considered testing > with Nexenta as the iSCSI/NFS client? If you mean running the NFS client AND server on the same box then yes, and it doesn't show the same performance issues. It's only when a Linux box

Re: [zfs-discuss] Performance issues with iSCSI under Linux

As I have mentioned already, it would be useful to know more about the config, how the tests are being done, and to see some basic system performance stats. On 15/10/2010 15:58, Ian D wrote: As I have mentioned already, we have the same performance issues whether we READ or we WRITE to the a

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> I've had a few people sending emails directly > suggesting it might have something to do with the > ZIL/SLOG. I guess I should have said that the issue > happen both ways, whether we copy TO or FROM the > Nexenta box. You mentioned a second Nexenta box earlier. To rule out client-side issues,

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> He already said he has SSD's for dedicated log. This > means the best > solution is to disable WriteBack and just use > WriteThrough. Not only is it > more reliable than WriteBack, it's faster. > > And I know I've said this many times before, but I > don't mind repeating: If > you have slog d

Re: [zfs-discuss] Performance issues with iSCSI under Linux

As I have mentioned already, we have the same performance issues whether we READ or we WRITE to the array, shouldn't that rule out caching issues? Also we can get great performances with the LSI HBA if we use the JBODs as a local file system. The issues only arise when it is done through iSCSI

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > boun...@opensolaris.org] On Behalf Of Phil Harman > > I'm wondering whether your HBA has a write through or write back cache > enabled? The latter might make things very fast, but could put data at > risk if not sufficiently non-vo

Re: [zfs-discuss] Performance issues with iSCSI under Linux

Ian, It would help to have some config detail (e.g. what options are you using? zpool status output; property lists for specific filesystems and zvols; etc) Some basic Solaris stats can be very helpful too (e.g. peak flow samples of vmstat 1, mpstst 1, iostat -xnz 1, etc) It would also be grea

Re: [zfs-discuss] Performance issues with iSCSI under Linux [SEC=UNCLASSIFIED]

> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > boun...@opensolaris.org] On Behalf Of Wilkinson, Alex > > can you paste them anyway ? Note: If you have more than one adapter, I believe you can specify -aALL in the commands below, instead of -a0 I have 2 disks (slots 4 & 5) th

Re: [zfs-discuss] Performance issues with iSCSI under Linux [SEC=UNCLASSIFIED]

0n Thu, Oct 14, 2010 at 09:54:09PM -0400, Edward Ned Harvey wrote: >If you happen to find that MegaCLI is the right tool for your hardware, let >me know, and I'll paste a few commands here, which will simplify your life. >When I first started using it, I found it terribly cumbers

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > boun...@opensolaris.org] On Behalf Of Ian D > > ok... we're making progress. After swapping the LSI HBA for a Dell > H800 the issue disappeared. Now, I'd rather not use those controllers > because they don't have a JBOD mode. We

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> Earlier you said you had eliminated the ZIL as an > issue, but one difference > between the Dell H800 and the LSI HBA is that the > H800 has an NV cache (if > you have the battery backup present). > > A very simple test would be when things are running > slow, try disabling > the ZIL temporarily

Re: [zfs-discuss] Performance issues with iSCSI under Linux

rewar...@hotmail.com said: > ok... we're making progress. After swapping the LSI HBA for a Dell H800 the > issue disappeared. Now, I'd rather not use those controllers because they > don't have a JBOD mode. We have no choice but to make individual RAID0 > volumes for each disks which means we nee

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> Our next test is to try with a different kind of HBA, > we have a Dell H800 lying around. ok... we're making progress. After swapping the LSI HBA for a Dell H800 the issue disappeared. Now, I'd rather not use those controllers because they don't have a JBOD mode. We have no choice but to mak

Re: [zfs-discuss] Performance issues with iSCSI under Linux

I've had a few people sending emails directly suggesting it might have something to do with the ZIL/SLOG. I guess I should have said that the issue happen both ways, whether we copy TO or FROM the Nexenta box. -- This message posted from opensolaris.org

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> Sounding more and more like a networking issue - are > the network cards set up in an aggregate? I had some > similar issues on GbE where there was a mismatch > between the aggregate settings on the switches and > the LACP settings on the server. Basically the > network was wasting a ton of time

Re: [zfs-discuss] Performance issues with iSCSI under Linux

On 13 oct. 2010, at 18:37, Marty Scholes wrote: > The only thing that still stands out is that network operations (iSCSI and > NFS) to external drives are slow, correct? > > Just for completeness, what happens if you scp a file to the three different > pools? If the results are the same as NF

Re: [zfs-discuss] Performance issues with iSCSI under Linux

More stuff... We ran the same tests on another Nexenta box with fairly similar hardware and had the exact same issues. The two boxes have the same models of HBAs, NICs and JBODs but different CPUs and motherboards. Our next test is to try with a different kind of HBA, we have a Dell H800 lying

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> Would it be possible to install OpenSolaris to an USB > disk and boot from it and try? That would take 1-2h > and could maybe help you narrow things down further? I'm a little afraid to lose my data, i wouldnt be the end of the world, but I'd rather avoid that. I'll do it in last resort. Ian

Re: [zfs-discuss] Performance issues with iSCSI under Linux

Would it be possible to install OpenSolaris to an USB disk and boot from it and try? That would take 1-2h and could maybe help you narrow things down further? -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolar

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> The only thing that still stands out is that network > operations (iSCSI and NFS) to external drives are > slow, correct? Yes, that pretty much resume it. > Just for completeness, what happens if you scp a file > to the three different pools? If the results are the > same as NFS and iSCSI, th

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> Here are some more findings... > > The Nexenta box has 3 pools: > syspool: made of 2 mirrored (hardware RAID) local SAS > disks > pool_sas: made of 22 15K SAS disks in ZFS mirrors on > 2 JBODs on 2 controllers > pool_sata: made of 42 SATA disks in 6 RAIDZ2 vdevs on > a single controller > > Whe

Re: [zfs-discuss] Performance issues with iSCSI under Linux

Here are some more findings... The Nexenta box has 3 pools: syspool: made of 2 mirrored (hardware RAID) local SAS disks pool_sas: made of 22 15K SAS disks in ZFS mirrors on 2 JBODs on 2 controllers pool_sata: made of 42 SATA disks in 6 RAIDZ2 vdevs on a single controller When we copy data from an

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> From the Linux side, it appears the drive in question > is either sdb or dm-3, and both appear to be the same > drive. Since switching to zfs, my Linux-disk-fu has > become a bit rusty. Is one an alias for the other? Yes, dm-3 is the alias created by LVM while sdb is the "physical" (or raw) d

Re: [zfs-discuss] Performance issues with iSCSI under Linux

Ok, Let's think about this for a minute. The log drive is c1t11d0 and it appears to be almost completely unused, so we probably can rule out a ZIL bottleneck. I run Ubuntu booting iSCSI against OSol 128a and the writes do not appear to be synchronous. So, writes aren't the issue. >From the

Re: [zfs-discuss] Performance issues with iSCSI under Linux

\> We're aware of that. The original plan was to use > mirrored DDRDrive X1s but we're experiencing > stability issues. Chris George is being very > responsible and we'll help us out investigate that > once we figure out our most pressing performance > problems. I feel I need to add to my commen

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> If you have a single SSD for dedicated log, that will > surely be a bottleneck > for you. We're aware of that. The original plan was to use mirrored DDRDrive X1s but we're experiencing stability issues. Chris George is being very responsible and we'll help us out investigate that once we f

Re: [zfs-discuss] Performance issues with iSCSI under Linux

>> What sort of drives are these? It looks like iSCSI or FC device names, and >> not local drives > > The "Pool_sas" is made of 15K SAS drives on external JBOD arrays (Dell > MD1000) connected on mirrored LSI 9200-8e SAS HBAs. > > The "Pool_sata" is made of SATA drives on other JBODs. > > Th

Re: [zfs-discuss] Performance issues with iSCSI under Linux

>A couple of notes: we know the "Pool_sata" is resilvering, but we're >concerned about the performances of the other pool ("Pool_sas"). We also know >that we're >not using jumbo frames as for some reason it makes the linux box >crash. Could that explain it all? >What sort of drives are thes

Re: [zfs-discuss] Performance issues with iSCSI under Linux

A couple of notes: we know the "Pool_sata" is resilvering, but we're concerned about the performances of the other pool ("Pool_sas"). We also know that we're not using jumbo frames as for some reason it makes the linux box crash. Could that explain it all? What sort of drives are these? It l

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> I'll suggest trying something completely different, like, dd if=/dev/zero > bs=1024k | pv | ssh othermachine 'cat > /dev/null' ... Just to verify there > isn't something horribly wrong with your hardware (network). > > In linux, run "ifconfig" ... You should see "errors:0" > > Make sure each

Re: [zfs-discuss] Performance issues with iSCSI under Linux

To see if it is iscsi related or zfs, have you tried to test performance over nfs to a zfs filesystem instead of a zvol? SR -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.o

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > boun...@opensolaris.org] On Behalf Of Ian D > > the help to community can provide.  We're running the latest version of > Nexenta on a pretty powerful machine (4x Xeon 7550, 256GB RAM, 12x > 100GB Samsung SSDs for the cache, 50GB S

Re: [zfs-discuss] Performance issues with iSCSI under Linux

> Where should we look at? What more information should I provide? Start Start with 'iostat -xdn 1'. That'll provide info about the actual device I/O. Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 97542685 r...@karlsbakk.net http://blogg.karlsbakk.net/ -- I all peda

[zfs-discuss] Performance issues with iSCSI under Linux

Hi!We're trying to pinpoint our performance issues and we could use all the help to community can provide. We're running the latest version of Nexenta on a pretty powerful machine (4x Xeon 7550, 256GB RAM, 12x 100GB Samsung SSDs for the cache, 50GB Samsung SSD for the ZIL, 10GbE on a dedicated