Re: Help! two machines ran out of swap and corrupted their zpools!

2016-11-24 Thread Pete French 
> Instapanic, huh...
>
> Ok, let's put documentation aside and focus on unsupported development 
> features.

Hi, sorry for not replying untiltoday, but basivally we got to the point
where getting the machine sp again was more importnat than debugging so
unfortunately I had to clone the drives off other leaves, so the opportunity
to debug this was lost :-( Sorry...

however, the info below is *very* useful! Thankyou! Will file taht way for
the next time we get any similar issues.

much appreciated.

-pete.

>
> You'll need a list of available transactions:
>
> zdb -ul 
>
> This one when given a vdev lists all uberblocks with their respective 
> transaction ids. You can take the highest one (it's not the last one) 
> and try to mount pool with:
>
> zpool import -N -O readonly=on -f -R /mnt/somezpool -F -T 
>  
>
> Then check available filesystems. If it breaks again - retry with older 
> transaction id.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

10-STABLE zfs: strange memory stats

2016-11-24 Thread Peter 

I observe a strange reading of the ZFS memory stats:

Mem: 298M Active, 207M Inact, 446M Wired, 10M Cache, 91M Buf, 29M Free
ARC: 339M Total, 8758K MFU, 43M MRU, 52K Anon, 35M Header, 40M Other
Swap: 2441M Total, 402M Used, 2040M Free, 16% Inuse

Usually I perceived the "Total" value being approx. the sum of the other 
values. Now this is still the case after system start, but after a day 
the significant difference appears like shown above.

(40+35+43+9 = 127 << 339)
Also it seems the ARC is reluctant to grow when free mem is avail nor 
does it shrink much while paging out. The build is r309023M. Definitely 
the behaviour is different than what I tried before (r306589:306943M), 
but that one was probably unstable, and I see a bunch of ZFS related 
commits interim.


Also I now see some counts on "l2_cksum_bad" which weren't there before.

BTW: is there some specific mailing-list where ZFS changes are pronounced?

Machine is i386 with 1GB mem. Probably hardware is somehow crappy,
but at least the mem readings are difficult to explain by hardware weakness.

Config (in case it matters):
vm.kmem_size="576M"
vm.kmem_size_max="576M"
vfs.zfs.arc_max="320M"
vfs.zfs.arc_min="120M"
vfs.zfs.vdev.cache.size="5M"
vfs.zfs.prefetch_disable="0"
vfs.zfs.l2arc_norw="0"
vfs.zfs.l2arc_noprefetch="0"

kstat.zfs.misc.arcstats.demand_hit_predictive_prefetch: 1016019
kstat.zfs.misc.arcstats.sync_wait_for_async: 1157
kstat.zfs.misc.arcstats.arc_meta_min: 62914560
kstat.zfs.misc.arcstats.arc_meta_max: 242711832
kstat.zfs.misc.arcstats.arc_meta_limit: 83886080
kstat.zfs.misc.arcstats.arc_meta_used: 133996612
kstat.zfs.misc.arcstats.memory_throttle_count: 0
kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 272242
kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 489828
kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 3372460809216
kstat.zfs.misc.arcstats.l2_write_pios: 14313
kstat.zfs.misc.arcstats.l2_write_buffer_iter: 122496
kstat.zfs.misc.arcstats.l2_write_full: 177
kstat.zfs.misc.arcstats.l2_write_not_cacheable: 4673385
kstat.zfs.misc.arcstats.l2_write_io_in_progress: 925
kstat.zfs.misc.arcstats.l2_write_in_l2: 93122523
kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 196362282
kstat.zfs.misc.arcstats.l2_write_passed_headroom: 57198
kstat.zfs.misc.arcstats.l2_write_trylock_fail: 20575
kstat.zfs.misc.arcstats.l2_padding_needed: 0
kstat.zfs.misc.arcstats.l2_hdr_size: 33567112
kstat.zfs.misc.arcstats.l2_asize: 4040757248
kstat.zfs.misc.arcstats.l2_size: 4472570880
kstat.zfs.misc.arcstats.l2_io_error: 0
kstat.zfs.misc.arcstats.l2_cksum_bad: 61
kstat.zfs.misc.arcstats.l2_abort_lowmem: 15
kstat.zfs.misc.arcstats.l2_free_on_write: 26703
kstat.zfs.misc.arcstats.l2_evict_l1cached: 0
kstat.zfs.misc.arcstats.l2_evict_reading: 0
kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0
kstat.zfs.misc.arcstats.l2_writes_lock_retry: 173
kstat.zfs.misc.arcstats.l2_writes_error: 0
kstat.zfs.misc.arcstats.l2_writes_done: 14313
kstat.zfs.misc.arcstats.l2_writes_sent: 14313
kstat.zfs.misc.arcstats.l2_write_bytes: 6030606336
kstat.zfs.misc.arcstats.l2_read_bytes: 11140009984
kstat.zfs.misc.arcstats.l2_rw_clash: 0
kstat.zfs.misc.arcstats.l2_feeds: 122496
kstat.zfs.misc.arcstats.l2_misses: 4370503
kstat.zfs.misc.arcstats.l2_hits: 2932017
kstat.zfs.misc.arcstats.mfu_ghost_evictable_metadata: 46062080
kstat.zfs.misc.arcstats.mfu_ghost_evictable_data: 1047040
kstat.zfs.misc.arcstats.mfu_ghost_size: 47109120
kstat.zfs.misc.arcstats.mfu_evictable_metadata: 0
kstat.zfs.misc.arcstats.mfu_evictable_data: 114688
kstat.zfs.misc.arcstats.mfu_size: 9073664
kstat.zfs.misc.arcstats.mru_ghost_evictable_metadata: 178836480
kstat.zfs.misc.arcstats.mru_ghost_evictable_data: 86231040
kstat.zfs.misc.arcstats.mru_ghost_size: 265067520
kstat.zfs.misc.arcstats.mru_evictable_metadata: 5632
kstat.zfs.misc.arcstats.mru_evictable_data: 1155072
kstat.zfs.misc.arcstats.mru_size: 49945088
kstat.zfs.misc.arcstats.anon_evictable_metadata: 0
kstat.zfs.misc.arcstats.anon_evictable_data: 0
kstat.zfs.misc.arcstats.anon_size: 53248
kstat.zfs.misc.arcstats.other_size: 44759120
kstat.zfs.misc.arcstats.metadata_size: 50840064
kstat.zfs.misc.arcstats.data_size: 231464448
kstat.zfs.misc.arcstats.hdr_size: 4830316
kstat.zfs.misc.arcstats.overhead_size: 41351168
kstat.zfs.misc.arcstats.uncompressed_size: 52131328
kstat.zfs.misc.arcstats.compressed_size: 17729024
kstat.zfs.misc.arcstats.size: 365461060
kstat.zfs.misc.arcstats.c_max: 335544320
kstat.zfs.misc.arcstats.c_min: 125829120
kstat.zfs.misc.arcstats.c: 315017029
kstat.zfs.misc.arcstats.p: 145334923
kstat.zfs.misc.arcstats.hash_chain_max: 17
kstat.zfs.misc.arcstats.hash_chains: 119135
kstat.zfs.misc.arcstats.hash_collisions: 6453863
kstat.zfs.misc.arcstats.hash_elements_max: 538227
kstat.zfs.misc.arcstats.hash_elements: 525460
kstat.zfs.misc.arcstats.evict_l2_skip: 4277
kstat.zfs.misc.arcstats.evict_l2_ineligible: 7410790400
kstat.zfs.misc.arcstats.evict_l2_eligible: 14946466816
kstat.zfs.misc.arcstats.evict_l2_cached: 26608123904
kstat.