On 06/09/2011 10:14 AM, Ding Honghui wrote:
On 06/09/2011 12:23 AM, Donald Stahl wrote:
Another (less satisfying) workaround is to increase the amount of
free space
in the pool, either by reducing usage or adding more storage. Observed
behavior is that allocation is fast until usage crosses a
threshhold, then
performance hits a wall.
We actually tried this solution. We were at 70% usage and performance
hit a wall. We figured it was because of the change of fit algorithm
so we added 16 2TB disks in mirrors. (Added 16TB to an 18TB pool). It
made almost no difference in our pool performance. It wasn't until we
told the metaslab allocator to stop looking for such large chunks that
the problem went away.
The original poster's pool is about 78% full. If possible, try freeing
stuff until usage goes back under 75% or 70% and see if your
performance
returns.
Freeing stuff did fix the problem for us (temporarily) but only in an
indirect way. When we freed up a bunch of space, the metaslab
allocator was able to find large enough blocks to write to without
searching all over the place. This would fix the performance problem
until those large free blocks got used up. Then- even though we were
below the usage problem threshold from earlier- we would still have
the performance problem.
-Don
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Don,
From your words, my symptom is almost same with yours.
We have examine the metaslab layout, when metaslab_df_free_pct is 35,
there are 65 free metaslab(64G),
The write performance is very low and the rough test shows no new free
metaslab will be loaded and activated.
Then we tune the metaslab_df_free_pct to 4, the performance keep good
for 1 week and the free metaslab reduce to 51.
But now, the write bandwidth is poor again ( maybe I'd better trace
the free space of each metaslab? )
Maybe there are some problem in metaslab rating score(weight) for
select the metaslab and block allocator algorithm?
There is snapshot of metaslab layout, the last 51 metaslabs have 64G
free space.
vdev offset spacemap free
---------- ------------------- ---------------
-------------
... snip
vdev 3 offset 27000000000 spacemap 440 free
21.0G
vdev 3 offset 28000000000 spacemap 31 free
7.36G
vdev 3 offset 29000000000 spacemap 32 free
2.44G
vdev 3 offset 2a000000000 spacemap 33 free
2.91G
vdev 3 offset 2b000000000 spacemap 34 free
3.25G
vdev 3 offset 2c000000000 spacemap 35 free
3.03G
vdev 3 offset 2d000000000 spacemap 36 free
3.20G
vdev 3 offset 2e000000000 spacemap 90 free
3.28G
vdev 3 offset 2f000000000 spacemap 91 free
2.46G
vdev 3 offset 30000000000 spacemap 92 free
2.98G
vdev 3 offset 31000000000 spacemap 93 free
2.19G
vdev 3 offset 32000000000 spacemap 94 free
2.42G
vdev 3 offset 33000000000 spacemap 95 free
2.83G
vdev 3 offset 34000000000 spacemap 252 free
41.6G
vdev 3 offset 35000000000 spacemap 0 free
64G
vdev 3 offset 36000000000 spacemap 0 free
64G
vdev 3 offset 37000000000 spacemap 0 free
64G
vdev 3 offset 38000000000 spacemap 0 free
64G
vdev 3 offset 39000000000 spacemap 0 free
64G
vdev 3 offset 3a000000000 spacemap 0 free
64G
vdev 3 offset 3b000000000 spacemap 0 free
64G
vdev 3 offset 3c000000000 spacemap 0 free
64G
vdev 3 offset 3d000000000 spacemap 0 free
64G
vdev 3 offset 3e000000000 spacemap 0 free
64G
...snip
I free up some disk space(about 300GB), the performance is back again.
I'm sure the performance will degrade again soon.
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
