Hi everyone, I'd like to come back to a discussion from 2012 (thread at http://marc.info/?l=ceph-devel&m=134808745719233) to estimate the expected MDS memory consumption from file metadata caching. I am certain the following is full of untested assumptions, some of which are probably inaccurate, so please shoot those down as needed.
I did an entirely unscientific study of a real data set (my laptop, in case you care to know) which currently holds about 70G worth of data in a huge variety of file sizes and several file systems, and currently lists about 944,000 inodes as being in use. So going purely by order of magnitude and doing a wild approximation, I'll assume a ratio of 1 million files in 100G, or 10,000 files per gigabyte, which means an average file size of about 100KB -- again, approximating and forgetting about the difference between 10^3 and 2^10, and using a stupid arithmetic mean rather than a median which would probably be much more useful. If I were to assume that all those files were in CephFS, and they were all somehow regularly in use (or at least one file in each directory), then the Ceph MDS would have to keep the metadata of all those files in cache. Suppose further that the stat struct for all those files is anywhere between 1 and 2KB, and we go by an average of 1.5KB metadata per file including some overhead, then that would mean the average metadata per file is about 1.5% of the average file size. So for my 100G of data, the MDS would use about 1.5G of RAM for caching. If you scale that up for a filestore of say a petabyte, that means all your Ceph MDSs would consume a relatively whopping 15TB in total RAM for metadata caching, again assuming that *all* the data is actually used by clients. Now of course it's entirely unrealistic that in a production system data is actually ever used across the board, but are the above considerations "close enough" for a rule-of-thumb approximation of MDS memory footprint? As in, Total MDS RAM = (Total used storage) * (fraction of data in regular use) * 0.015 If CephFS users could use a rule of thumb like that, it would help them answer questions like "given a filesystem of size X, will a single MDS be enough to hold my metadata caches if Y is the maximum amount of memory I can afford for budget Z". All thoughts and comments much appreciated. Thank you! Cheers, Florian _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
