I recently tweaked Oracle (8K blocks, log_buffer gt 2M) on a Solaris
AMD64 system for max performance on a Sun 6140 with one tray of 73 GB
15K RPM drives. Definitely needed to place the datafiles and redo logs
on isolated RAID groups. Wasn't sure how many blocks Oracle batches for
IO. Used dtrace's bitesize script to generate the distributions below.
Based on the dtrace output, and after testing multiple segment sizes,
finally settled on Segment Size (stripe size) 256K for both datafiles
and redo logs.
Also observed performance boost by using forcedirectio and noatime on
the 6140 mount points and observed smoother performance by using 2M
pagesize (MPSS) by adding the line below to Oracle's .profile (and
verified with pmap -s [ORACLE PID]|grep 2M).
Oracle MPSS .profile
LD_PRELOAD=$LD_PRELOAD:mpss.so.1
MPSSHEAP=2M
MPSSSTACK=2M
export LD_PRELOAD MPSSHEAP MPSSSTACK
MPSSERRFILE=~/mpsserr
export MPSSERRFILE
Here's the final 6140 config:
Oracle datafiles => 12 drives RAID 10 Sement Size 256
Oracle redo log A => 2 drives RAID 0 Sement Size 256
Oracle redo log B => 2 drives RAID 0 Sement Size 256
./bitesize.d
1452 ora_dbw2_prf02\0
value ------------- Distribution ------------- count
16384 | 0
32768 |@@@@@@@@@@@@@@@@@@@@ 1
65536 | 0
131072 |@@@@@@@@@@@@@@@@@@@@ 1
262144 | 0
1454 ora_dbw3_prf02\0
value ------------- Distribution ------------- count
4096 | 0
8192 |@@@@@@@@@@@@@@@@@@@@@@@ 4
16384 |@@@@@@ 1
32768 |@@@@@@ 1
65536 | 0
131072 |@@@@@@ 1
262144 | 0
1448 ora_dbw0_prf02\0
value ------------- Distribution ------------- count
4096 | 0
8192 |@@@@@@@@@@@@@@@@@@@@@@ 5
16384 |@@@@@@@@@@@@@ 3
32768 | 0
65536 | 0
131072 |@@@@ 1
262144 | 0
1450 ora_dbw1_prf02\0
value ------------- Distribution ------------- count
65536 | 0
131072 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 2
262144 | 0
1458 ora_ckpt_prf02\0
value ------------- Distribution ------------- count
8192 | 0
16384 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 43
32768 | 0
1456 ora_lgwr_prf02\0
value ------------- Distribution ------------- count
256 | 0
512 |@@@@@@@@ 24
1024 |@@@@ 12
2048 |@@@@@ 15
4096 |@@@@@ 14
8192 | 0
16384 | 1
32768 |@ 4
65536 | 0
131072 |@ 4
262144 |@@ 6
524288 |@@@@@@@@@@@@@@ 42
1048576 | 0
This email message is for the sole use of the intended recipient(s) and
may contain GDC4S confidential or privileged information. Any
unauthorized review, use, disclosure or distribution is prohibited. If
you are not an intended recipient, please contact the sender by reply
email and destroy all copies of the original message.
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Asif Iqbal
Sent: Tuesday, November 20, 2007 3:08 PM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]; perf-discuss@opensolaris.org;
[EMAIL PROTECTED]
Subject: Re: [perf-discuss] [storage-discuss] zpool io to 6140 is really
slow
On Nov 20, 2007 10:40 AM, Andrew Wilson <[EMAIL PROTECTED]> wrote:
>
> What kind of workload are you running. If you are you doing these
> measurements with some sort of "write as fast as possible"
> microbenchmark,
Oracle database with blocksize 16K .. populating the database as fast I
can
> once the 4 GB of nvram is full, you will be limited by backend
> performance (FC disks and their interconnect) rather than the host /
controller bus.
>
> Since, best case, 4 gbit FC can transfer 4 GBytes of data in about 10
> seconds, you will fill it up, even with the backend writing out data
> as fast as it can, in about 20 seconds. Once the nvram is full, you
> will only see the backend (e.g. 2 Gbit) rate.
>
> The reason these controller buffers are useful with real applications
> is that they smooth the bursts of writes that real applications tend
> to generate, thus reducing the latency of those writes and improving
> performance. They will then "catch up" during periods when few writes
> are being issued. But a typical microbenchmark that pumps out a steady
> stream of writes won't see this benefit.
>
> Drew Wilson
>
>
>
> Asif Iqbal wrote:
> On Nov 20, 2007 7:01 AM, Chad Mynhier <[EMAIL PROTECTED]> wrote:
>
>
> On 11/20/07, Asif Iqbal <[EMAIL PROTECTED]> wrote:
>
>
> On Nov 19, 2007 1:43 AM, Louwtjie Burger <[EMAIL PROTECTED]>
wrote:
>
>
> On Nov 17, 2007 9:40 PM, Asif Iqbal <[EMAIL PROTECTED]> wrote:
>
>
> (Including storage-discuss)
>
> I have 6 6140s with 96 disks. Out of which 64 of them are Seagate
> ST3300007FC (300GB - 10000 RPM FC-AL)
>
> Those disks are 2Gb disks, so the tray will operate at 2Gb.
>
>
> That is still 256MB/s . I am getting about 194MB/s
>
> 2Gb fibre channel is going to max out at a data transmission rate
>
> But I am running 4GB fiber channels with 4GB NVRAM on a 6 tray of
> 300GB FC 10K rpm (2Gb/s) disks
>
> So I should get "a lot" more than ~ 200MB/s. Shouldn't I?
>
>
>
>
> around 200MB/s rather than the 256MB/s that you'd expect. Fibre
> channel uses an 8-bit/10-bit encoding, so it transmits 8-bits of data
> in 10 bits on the wire. So while 256MB/s is being transmitted on the
> connection itself, only 200MB/s of that is the data that you're
> transmitting.
>
> Chad Mynhier
>
>
>
>
>
>
>
--
Asif Iqbal
PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu
_______________________________________________
perf-discuss mailing list
perf-discuss@opensolaris.org
_______________________________________________
perf-discuss mailing list
perf-discuss@opensolaris.org
