Marion Hakanson wrote:
[EMAIL PROTECTED] said:
...
I know, I know, I should have gone with a JBOD setup, but it's too late for
that in this iteration of this server. We we set this up, I had the gear
already, and it's not in my budget to get new stuff right now.
What kind of array are you seeing this problem with? It sounds very much
like our experience here with a 3-yr-old HDS ATA array.
It's not that old. It's a Supermicro system with a 3ware 9650SE-8LP.
Open-E iSCSI-R3 DOM module. The system is plenty fast. I can pretty
handily pull 120MB/sec from it, and write at over 100MB/sec. It falls
apart more on random I/O. The server/initiator side is a T2000 with
Solaris 10u4. It never sees over 25% CPU, ever. Oh yeah, and two 1GB
network links to the SAN
When the crunch
came here, I didn't know enough dtrace to help, but I threw the following
into crontab to run every five minutes (24x7), and it at least collected
the info I needed to see what LUN/filesystem was busying things out.
Way crude, but effective enough:
/bin/ksh -c "date && mpstat 2 20 && iostat -xn 2 20 \
&& fsstat $(zfs list -H -o mountpoint -t filesystem | egrep '^/') 2 20 \
&& vmstat 2 20" >> /var/tmp/iostats.log 2>&1 </dev/null
A quick scan using "egrep" could pull out trouble spots; E.g. the following
would identify "iostat" lines that showed 90-100% busy:
egrep '^Sun |^Mon |^Tue |^Wed |^Thu |^Fri |^Sat | 1[0-9][0-9] c6| 9[0-9]
c6'\
/var/tmp/iostats.log
yeah, I have some running traditional *stat utilities running. If I
capture more than a second at a time, things look good. I was hoping to
get a real distribution of service times, to catch the outliers, that
don't get absorbed into the average. Hence why I wanted to use dtrace.
My opinion is, if when the array got really loaded up, everything slowed
down evenly, users wouldn't mind or notice much. But when every 20 or
so reads/writes gets delayed my 10s of seconds, the users start to line
up at my door.
Thanks for the tips.
Jon
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
