Hello ZFS,
System was rebooted and after reboot server again
System is snv_39, SPARC, T2000
bash-3.00# ptree
7 /lib/svc/bin/svc.startd -s
163 /sbin/sh /lib/svc/method/fs-local
254 /usr/sbin/zfs mount -a
[...]
bash-3.00# zfs list|wc -l
46
Using df I can see most file systems are already mounted.
> ::ps!grep zfs
R 254 163 7 7 0 0x4a004000 00000600219e1800 zfs
> 00000600219e1800::walk thread|::findstack -v
stack pointer for thread 300013026a0: 2a10069ebb1
[ 000002a10069ebb1 cv_wait+0x40() ]
000002a10069ec61 txg_wait_synced+0x54(3000052f0d0, 2f2216d, 3000107fb90,
3000052f110, 3000052f112, 3000052f0c8)
000002a10069ed11 zil_replay+0xbc(60022609c08, 600226a1840, 600226a1870,
700d13d8, 7ba2d000, 60006821200)
000002a10069ee01 zfs_domount+0x1f0(7ba35000, 700d1000, 60021a8bae0, 1, 0, 0)
000002a10069eed1 zfs_mount+0x160(600225f6080, 1868800, 2a10069f9d8,
60021a8bae0, 2f, 0)
000002a10069efb1 domount+0x9b0(2a10069f8b0, 1, 600225f6080, 0, 0, 0)
000002a10069f121 mount+0x110(600013fbcf8, 2a10069fad8, 0, 0, ff38e40c, 100)
000002a10069f221 syscall_ap+0x44(2a0, ffbfeca0, 1118bf4, 600013fbc40, 15, 0)
000002a10069f2e1 syscall_trap32+0xcc(52b3c8, ffbfeca0, 100, ff38e40c, 0, 0)
>
# iostat -xnz 1
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
3.0 513.0 192.0 2822.1 0.0 2.1 0.0 4.0 0 95
c4t600C0FF00000000009258F3E4C4C5601d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
7.0 598.1 388.6 1832.9 0.0 2.0 0.0 3.4 0 93
c4t600C0FF00000000009258F3E4C4C5601d0
^C
bash-3.00#
However for many seconds no IOs at all are issued.
# mpstat 1
[...]
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 0 0 74 209 109 0 0 0 0 0 0 0 1 0 99
1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
2 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
3 0 0 0 1 0 0 0 0 0 0 0 0 100 0 0
4 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
5 0 0 0 6 1 8 0 0 0 0 0 0 0 0 100
6 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
7 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
8 3 0 1 7 0 12 0 0 2 0 245 1 1 0 98
9 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
10 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
11 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
12 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
13 0 0 0 6 0 10 0 0 0 0 0 0 0 0 100
14 0 0 0 3 0 2 1 0 0 0 188 0 1 0 99
15 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
16 0 0 0 8 1 12 0 0 0 0 0 0 0 0 100
17 0 0 0 7 0 12 0 0 0 0 0 0 0 0 100
18 0 0 0 23 0 43 0 0 1 0 0 0 0 0 100
19 0 0 0 9 0 16 0 0 0 0 0 0 0 0 100
20 0 0 0 8 0 14 0 0 0 0 0 0 0 0 100
21 0 0 0 7 2 8 0 1 0 0 1 0 1 0 99
22 0 0 17 38 34 7 0 1 1 0 5 0 0 0 100
23 0 0 0 7 0 12 0 0 0 0 0 0 0 0 100
24 0 0 0 4 0 6 0 1 0 0 0 0 0 0 100
25 0 0 1 5 0 8 0 1 0 0 0 0 0 0 100
26 0 0 0 2 0 2 0 0 0 0 0 0 0 0 100
27 0 0 0 2 0 2 0 0 0 0 0 0 0 0 100
28 0 0 0 2 0 2 0 0 1 0 0 0 0 0 100
29 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
30 0 0 0 1 0 0 0 0 0 0 0 0 0 0 100
31 0 0 0 6 0 8 0 0 0 0 0 0 0 0 100
^C
bash-3.00#
# dtrace -n fbt:::entry'{self->t=timestamp;}' -n fbt:::return'/self->t/[EMAIL
PROTECTED](timestamp-self->t);self->t=0;}' -n tick-5s'{printa(@);exit(0);}'
[...]
syscall_mstate 12636328
callout_schedule_1 14428108
hwblkclr 16656308
avl_rotation 17196252
page_pptonum 19457456
sfmmu_mlist_enter 20078508
sfmmu_mlist_exit 20804176
page_numtomemseg_nolock 23377972
avl_insert 23727512
avl_insert_here 25281752
sfmmu_mlspl_enter 32277524
kmem_cache_alloc 38497240
xc_serv 94880184
disp_getwork 118366960
disp_anywork 191004096
space_map_seg_compare 573117700
resume 7059674296
#
bash-3.00# dtrace -n fbt::space_map_seg_compare:entry'[EMAIL
PROTECTED]()]=count();}'
dtrace: description 'fbt::space_map_seg_compare:entry' matched 1 probe
^C
genunix`avl_find+0x38
zfs`space_map_add+0x12c
zfs`space_map_load+0x214
zfs`metaslab_activate+0x3c
zfs`metaslab_group_alloc+0x1b0
zfs`metaslab_alloc_dva+0x10c
zfs`metaslab_alloc+0x2c
zfs`zio_dva_allocate+0x50
zfs`zio_write_compress+0x1e4
zfs`arc_write+0xbc
zfs`dbuf_sync+0x6b0
zfs`dnode_sync+0x300
zfs`dmu_objset_sync_dnodes+0x68
zfs`dmu_objset_sync+0x50
zfs`dsl_dataset_sync+0xc
zfs`dsl_pool_sync+0x60
zfs`spa_sync+0xe0
zfs`txg_sync_thread+0x130
unix`thread_start+0x4
316769
genunix`avl_find+0x38
zfs`space_map_remove+0x98
zfs`space_map_load+0x214
zfs`metaslab_activate+0x3c
zfs`metaslab_group_alloc+0x1b0
zfs`metaslab_alloc_dva+0x10c
zfs`metaslab_alloc+0x2c
zfs`zio_dva_allocate+0x50
zfs`zio_write_compress+0x1e4
zfs`arc_write+0xbc
zfs`dbuf_sync+0x6b0
zfs`dnode_sync+0x300
zfs`dmu_objset_sync_dnodes+0x68
zfs`dmu_objset_sync+0x50
zfs`dsl_dataset_sync+0xc
zfs`dsl_pool_sync+0x60
zfs`spa_sync+0xe0
zfs`txg_sync_thread+0x130
unix`thread_start+0x4
404210
bash-3.00#
Above was running for 3-4seconds
I can see some progress - I mean it mounts another filesystem in every
3-10 minutes.
However overall it could take even over an hour to boot server before
zfs mount -a mounts actually all file systems. Also with default
Solaris configs during this stage you can't login either from console
or ssh and no messages are issued. That is bad also as you don't know
what is going on. We had to reboot again, boot to single user, run
sshd, login via ssh and let boot process to proceed in order to see
why server won't boot.
Using zpool status all pools are ok and no errors at all are reported.
Summary:
1. zfs mount -a shouldn't "sleep" 1h!
2. one virtual CPU core of T2000 1GH is consumed for
most of the time during zfs mount -a - why is it so CPU
intensive? On that server CPU became limiting in that way
that usually no IOs were issued to disks during mount.
When it happened on faster CPU disks were doing 100MB/s,
but still it took 10-20 minutes to zfs mount disks.
3. I thought zfs mount problem so it exits earlier was already
addressed. So this is completely new or still something wrong is
going on. I'm thinking about
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6377670
4. Would be better if zfs mount -a would actually do it in parallel
as it looks like it mounts in serial fashion right now
Above behavior is really BAD and it makes ZFS looks like fsck is
actually build-in - I know it isn't.
--
Best regards,
Robert mailto:[EMAIL PROTECTED]
http://milek.blogspot.com
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
