I have been running some speed tests in POSIX file operations and I noticed
even just listing files can take a while compared to an attached HDD. I am
wondering is there a reason it takes so long to even just list files.
Here is the test I ran
time for i in {1..100000}; do touch $i; done
Internal HDD:
real 4m37.492s
user 0m18.125s
sys 1m5.040s
Ceph Dir
real 12m30.059s
user 0m16.749s
sys 0m53.451s
~300% faster on HDD
*I am actually ok with this but nice to be quicker.
When I am listing the directory it is taking a lot longer compared to an
attached HDD
time ls -1
Internal HDD
real 0m2.112s
user 0m0.560s
sys 0m0.440s
Ceph Dir
real 3m35.982s
user 0m2.788s
sys 0m4.580s
~1000% faster on HDD
*I understand there is some time in the display so what is really making it
odd is the following test.
time ls -1 > /dev/null
Internal HDD
real 0m0.367s
user 0m0.324s
sys 0m0.040s
Ceph Dir
real 0m2.807s
user 0m0.128s
sys 0m0.052s
~700% faster on HDD
My guess the performance issue is with the batch requests as you stated. So
I am wondering if the file deletion of the 40M files is not just deleting
the files but even just traversing that many files takes a while.
I am running this on 0.94.6 with Ceph Fuse Client
And config
fuse multithreaded = false
Since multithreaded crashes in hammer.
It would be interesting to see the performance on newer versions.
Any thoughts or comments would be good.
On Tue, Apr 5, 2016 at 9:22 AM Gregory Farnum <gfar...@redhat.com> wrote:
> On Mon, Apr 4, 2016 at 9:55 AM, Gregory Farnum <gfar...@redhat.com> wrote:
> > Deletes are just slow right now. You can look at the ops in flight on you
> > client or MDS admin socket to see how far along it is and watch them to
> see
> > how long stuff is taking -- I think it's a sync disk commit for each
> unlink
> > though so at 40M it's going to be a good looong while. :/
> > -Greg
>
> Oh good, I misremembered — it's a synchronous request to the MDS, but
> it's not a synchronous disk commit. They get batched up normally in
> the metadata log. :)
> Still, a sync MDS request can take a little bit of time. Someday we
> will make the client able to respond to these more quickly locally and
> batch up MDS requests or something, but it'll be tricky. Faster file
> creates will probably come first. (If we're lucky they can use some of
> the same client-side machinery.)
> -Greg
>
> >
> >
> > On Monday, April 4, 2016, Kenneth Waegeman <kenneth.waege...@ugent.be>
> > wrote:
> >>
> >> Hi all,
> >>
> >> I want to remove a large directory containing +- 40M files /160TB of
> data
> >> in CephFS by running rm -rf on the directory via the ceph kernel client.
> >> After 7h , the rm command is still running. I checked the rados df
> output,
> >> and saw that only about 2TB and 2M files are gone.
> >> I know this output of rados df can be confusing because ceph should
> delete
> >> objects asyncroniously, but then I don't know why the rm command still
> >> hangs.
> >> Is there some way to speed this up? And is there a way to check how far
> >> the marked for deletion has progressed ?
> >>
> >> Thank you very much!
> >>
> >> Kenneth
> >>
> >> _______________________________________________
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
