G'Day Ben,
ARC visibility is important; did you see Neel's arcstat?:
http://www.solarisinternals.com/wiki/index.php/Arcstat
Try -x for various sizes, and -v for definitions.
On Thu, Aug 21, 2008 at 10:23:24AM -0700, Ben Rockwood wrote:
> Its a starting point anyway. The key is to try and draw useful conclusions
> from the info to answer the torrent of "why is my ARC 30GB???"
>
> There are several things I'm unclear on whether or not I'm properly
> interpreting such as:
>
> * As you state, the anon pages. Even the comment in code is, to me anyway, a
> little vague. I include them because otherwise you look at the hit counters
> and wonder where a large chunk of them went.
Yes, anon hits doesn't make sense - they are dirty pages and won't have a DVA,
and so won't be findable by other threads in arc_read(). I can see why
arc_summary.pl thinks they exist - accounting for the discrepancy between
arcstats:hits and the sum of the hits from the four ARC lists. Ghost list
hits aren't part of arcstats:hits - arcstats:hits are real hits, the ghost
hits are an artifact of the ARC algorithm. If you do want to break down
arcstats:hits into it's components, use:
zfs:0:arcstats:demand_data_hits
zfs:0:arcstats:demand_metadata_hits
zfs:0:arcstats:prefetch_data_hits
zfs:0:arcstats:prefetch_metadata_hits
And for a different perspective on the demand hits:
zfs:0:arcstats:mru_hits
zfs:0:arcstats:mfu_hits
Also, arc_summary.pl's reported MRU and MFU sizes aren't actual, these
are target sizes. The ARC will try to steer itself towards them, but in at
least one case (where the ARC has yet to fill) they can be very different
from actual (until arc_adjust() is called to whip them back to size.)
> * Prefetch... I want to use the Prefetch Data hit ratio as a judgment call on
> the efficiency of prefetch. If the value is very low it might be best to
> turn it off..... but I'd like to hear that from someone else before I go
> saying that.
Sounds good to me.
> In high latency environments, such as ZFS on iSCSI, prefetch can either
> significantly help or hurt, determining which is difficult without some type
> of metric as as above.
>
> * There are several instances (based on dtracing) in which the ARC is
> bypassed... for ZIL I understand, in some other cases I need to spend more
> time analyzing the DMU (dbuf_*) for why.
>
> * In answering the "Is having a 30GB ARC good?" question, I want to say that
> if MFU is >60% of ARC, and if the hits are mostly MFU that you are deriving
> significant benefit from your large ARC.... but on a system with a 2GB ARC or
> a 30GB ARC the overall hit ratio tends to be 99%. Which is nuts, and tends
> to reinforce a misinterpretation of anon hits.
I wouldn't read *too* much into MRU vs MFU hits. MFU means 2 hits, MRU
means 1.
> The only way I'm seeing to _really_ understand ARC's efficiency is to look at
> the overall number of reads and then how many are intercepted by ARC and how
> many actually made it to disk... and why (prefetch or demand). This is
> tricky to implement via kstats because you have to pick out and monitor the
> zpool disks themselves.
This would usually have more to do with the workload than the ARC's efficiency.
> I've spent a lot of time in this code (arc.c) and still have a lot of
> questions. I really wish there was an "Advanced ZFS Internals" talk coming
> up; I simply can't keep spending so much time on this.
Maybe you could try forgetting about the kstats for a moment and draw a
fantasy arc_summary.pl output. Then we can look at adding kstats to make
writing that script possible/easy (Mark and I could add the kstats, and
Neel could provide the script, for example). Of course, if we do add more
kstats, it's not going to help on older rev kernels out there...
cheers,
Brendan
--
Brendan
[CA, USA]
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
