On Dec 27, 2009, at 10:21 PM, Joe Little wrote:
I've had this happen to me too. I found some dtrace scripts at the
time that showed that the file system was spending too much time
finding available 128k blocks or the like as I was near full per each
disk, even though combined I still had 140GB left of my 3TB pool. The
SPA code I believe it was was spending too much time walking the
available pool for continguous space for new writes, and this
affecting both read and write performance dramatically (measured in
kb/sec).
I was able to alleviate the pressure so to speak by adjusting the
recordsize for the pool down to 8k (32k is likely more recommended)
and from there I could then start to clear out space. Anything below
10% available space seems to cause ZFS to start behaving poorly, and
getting down lower increases the problems. But the root cause was
metadata management on pools w/ less than 5-10% disk space left.
The better solution is to use b129 or later, where CR 6869229,
zfs should switch to shiny new metaslabs more frequently, was
integrated.
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6869229
-- richard
In my case, I had lots of symlinks, lots of small files, and also
dozens of snapshots. My pool was a RAID10 (aka, 3 mirror sets
striped).
On Sun, Dec 27, 2009 at 4:52 PM, Morten-Christian Bernson
<m...@uib.no> wrote:
Lately my zfs pool in my home server has degraded to a state where
it can be said it doesn't work at all. Read spead is slower than I
can read from the internet on my slow dsl-line... This is compared
to just a short while ago, where I could read from it with over
50mb/sec over the network.
My setup:
Running latest Solaris 10: # uname -a
SunOS solssd01 5.10 Generic_142901-02 i86pc i386 i86pc
# zpool status DATA
pool: DATA
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
DATA ONLINE 0 0 0
raidz1 ONLINE 0 0 0
c2t5d0 ONLINE 0 0 0
c2t4d0 ONLINE 0 0 0
c2t3d0 ONLINE 0 0 0
c2t2d0 ONLINE 0 0 0
spares
c0t2d0 AVAIL
errors: No known data errors
# zfs list -r DATA
NAME USED AVAIL REFER MOUNTPOINT
DATA 3,78T 229G 3,78T /DATA
All of the drives in this pool are 1.5tb western digital green
drives. I am not seeing any error messages in /var/adm/messages,
and "fmdump -eV" shows no errors... However, I am seeing some
soft faults in "iostat -eEn":
---- errors ---
s/w h/w trn tot device
2 0 0 2 c0t0d0
1 0 0 1 c1t0d0
2 0 0 2 c2t1d0
151 0 0 151 c2t2d0
151 0 0 151 c2t3d0
153 0 0 153 c2t4d0
153 0 0 153 c2t5d0
2 0 0 2 c0t1d0
3 0 0 3 c0t2d0
0 0 0 0 solssd01:vold(pid531)
c0t0d0 Soft Errors: 2 Hard Errors: 0 Transport Errors: 0
Vendor: Sun Product: STK RAID INT Revision: V1.0 Serial No:
Size: 31.87GB <31866224128 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 2 Predictive Failure Analysis: 0
c1t0d0 Soft Errors: 1 Hard Errors: 0 Transport Errors: 0
Vendor: _NEC Product: DVD_RW ND-3500AG Revision: 2.16 Serial No:
Size: 0.00GB <0 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 1 Predictive Failure Analysis: 0
c2t1d0 Soft Errors: 2 Hard Errors: 0 Transport Errors: 0
Vendor: ATA Product: SAMSUNG HD753LJ Revision: 1113 Serial No:
Size: 750.16GB <750156373504 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 2 Predictive Failure Analysis: 0
c2t2d0 Soft Errors: 151 Hard Errors: 0 Transport Errors: 0
Vendor: ATA Product: WDC WD15EADS-00R Revision: 0A01 Serial No:
Size: 1500.30GB <1500301909504 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 151 Predictive Failure Analysis: 0
c2t3d0 Soft Errors: 151 Hard Errors: 0 Transport Errors: 0
Vendor: ATA Product: WDC WD15EADS-00R Revision: 0A01 Serial No:
Size: 1500.30GB <1500301909504 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 151 Predictive Failure Analysis: 0
c2t4d0 Soft Errors: 153 Hard Errors: 0 Transport Errors: 0
Vendor: ATA Product: WDC WD15EADS-00R Revision: 0A01 Serial No:
Size: 1500.30GB <1500301909504 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 153 Predictive Failure Analysis: 0
c2t5d0 Soft Errors: 153 Hard Errors: 0 Transport Errors: 0
Vendor: ATA Product: WDC WD15EADS-00R Revision: 0A01 Serial No:
Size: 1500.30GB <1500301909504 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 153 Predictive Failure Analysis: 0
c0t1d0 Soft Errors: 2 Hard Errors: 0 Transport Errors: 0
Vendor: Sun Product: STK RAID INT Revision: V1.0 Serial No:
Size: 31.87GB <31866224128 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 2 Predictive Failure Analysis: 0
c0t2d0 Soft Errors: 3 Hard Errors: 0 Transport Errors: 0
Vendor: Sun Product: STK RAID INT Revision: V1.0 Serial No:
Size: 1497.86GB <1497859358208 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 3 Predictive Failure Analysis: 0
I am curious as to why the counter for "Illegal request" goes up
all the time. The machine was rebooted ~11 hours ago, and it goes
up all the time when I try to use the pool...
The machine is a quite powerful one, and top shows no cpu load, no
iowait and plenty of available memory. The machine basicly doesn't
do anything at the moment, still it can take several minutes to
copy a 300mb file from somewhere in the pool to /tmp/...
# top
last pid: 1383; load avg: 0.01, 0.00, 0.00; up
0
+
10
:
47
:
57
01:39:17
55 processes: 54 sleeping, 1 on cpu
CPU states: 99.0% idle, 0.0% user, 1.0% kernel, 0.0% iowait,
0.0% swap
Kernel: 193 ctxsw, 3 trap, 439 intr, 298 syscall, 3 flt
Memory: 8186M phys mem, 4699M free mem, 2048M total swap, 2048M
free swap
I thought I might have run into problems described here on the
forums with the ARC and fragmentation, but it doesn't seem so:
# echo "::arc"|mdb -k
hits = 490044
misses = 37004
demand_data_hits = 282392
demand_data_misses = 2113
demand_metadata_hits = 191757
demand_metadata_misses = 21034
prefetch_data_hits = 851
prefetch_data_misses = 10265
prefetch_metadata_hits = 15044
prefetch_metadata_misses = 3592
mru_hits = 73416
mru_ghost_hits = 16
mfu_hits = 401500
mfu_ghost_hits = 24
deleted = 1555
recycle_miss = 0
mutex_miss = 0
evict_skip = 1487
hash_elements = 37032
hash_elements_max = 37045
hash_collisions = 10094
hash_chains = 4365
hash_chain_max = 4
p = 3576 MB
c = 7154 MB
c_min = 894 MB
c_max = 7154 MB
size = 1797 MB
hdr_size = 8002680
data_size = 1866272256
other_size = 10519712
l2_hits = 0
l2_misses = 0
l2_feeds = 0
l2_rw_clash = 0
l2_read_bytes = 0
l2_write_bytes = 0
l2_writes_sent = 0
l2_writes_done = 0
l2_writes_error = 0
l2_writes_hdr_miss = 0
l2_evict_lock_retry = 0
l2_evict_reading = 0
l2_free_on_write = 0
l2_abort_lowmem = 0
l2_cksum_bad = 0
l2_io_error = 0
l2_size = 0
l2_hdr_size = 0
memory_throttle_count = 0
arc_no_grow = 0
arc_tempreserve = 0 MB
arc_meta_used = 372 MB
arc_meta_limit = 1788 MB
arc_meta_max = 372 MB
I then tried to start a scrub, and it seems like it will take
forever... It used to take a few hours, now it says it will be done
in almost 700 hours:
scrub: scrub in progress for 4h43m, 0,68% done, 685h2m to go
Does anyone have any clue as to what is happening, and what I can
do? If a disk is failing without the OS noticing, it would be nice
to find a way to know which drive it is, and get it exchanged
before it's too late.
All help is apreciated...
Yours sincerly,
Morten-Christian Bernson
--
This message posted from opensolaris.org
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
