Thanks for the thought. It’s mounted with this entry in fstab (one line, if email wraps it):
cephmon-s01,cephmon-s02,cephmon-s03:/ /loam ceph noauto,name=clientname,secretfile=/etc/ceph/secret,noatime,_netdev 0 2 Pretty plain, but I'm open to tweaking! peter Peter Eisch virginpulse.com |globalchallenge.virginpulse.com Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore | Switzerland | United Kingdom | USA Confidentiality Notice: The information contained in this e-mail, including any attachment(s), is intended solely for use by the designated recipient(s). Unauthorized use, dissemination, distribution, or reproduction of this message by anyone other than the intended recipient(s), or a person designated as responsible for delivering such messages to the intended recipient, is strictly prohibited and may be unlawful. This e-mail may contain proprietary, confidential or privileged information. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Virgin Pulse, Inc. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender and delete this e-mail message. v2.10 From: Gregory Farnum <gfar...@redhat.com> Date: Thursday, August 30, 2018 at 11:47 AM To: Peter Eisch <peter.ei...@virginpulse.com> Cc: "ceph-users@lists.ceph.com" <ceph-users@lists.ceph.com> Subject: Re: [ceph-users] cephfs speed How are you mounting CephFS? It may be that the cache settings are just set very badly for a 10G pipe. Plus rados bench is a very parallel large-IO benchmark and many benchmarks you might dump into a filesystem are definitely not. -Greg On Thu, Aug 30, 2018 at 7:54 AM Peter Eisch <mailto:peter.ei...@virginpulse.com> wrote: Hi, I have a cluster serving cephfs and it works. It’s just slow. Client is using the kernel driver. I can ‘rados bench’ writes to the cephfs_data pool at wire speeds (9580Mb/s on a 10G link) but when I copy data into cephfs it is rare to get above 100Mb/s. Large file writes may start fast (2Gb/s) but within a minute slows. In the dashboard at the OSDs I get lots of triangles (it doesn't stream) which seems to be lots of starts and stops. By contrast the graphs show constant flow when using 'rados bench.' I feel like I'm missing something obvious. What can I do to help diagnose this better or resolve the issue? Errata: Version: 12.2.7 (on everything) mon: 3 daemons, quorum cephmon-s01,cephmon-s03,cephmon-s02 mgr: cephmon-s02(active), standbys: cephmon-s01, cephmon-s03 mds: cephfs1-1/1/1 up {0=cephmon-s02=up:active}, 2 up:standby osd: 70 osds: 70 up, 70 in rgw: 3 daemons active rados bench summary: Total time run: 600.043733 Total writes made: 167725 Write size: 4194304 Object size: 4194304 Bandwidth (MB/sec): 1118.09 Stddev Bandwidth: 7.23868 Max bandwidth (MB/sec): 1140 Min bandwidth (MB/sec): 1084 Average IOPS: 279 Stddev IOPS: 1 Max IOPS: 285 Min IOPS: 271 Average Latency(s): 0.057239 Stddev Latency(s): 0.0354817 Max latency(s): 0.367037 Min latency(s): 0.0120791 peter Peter Eisch https://www.virginpulse.com/ | https://globalchallenge.virginpulse.com/ Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore | Switzerland | United Kingdom | USA Confidentiality Notice: The information contained in this e-mail, including any attachment(s), is intended solely for use by the designated recipient(s). Unauthorized use, dissemination, distribution, or reproduction of this message by anyone other than the intended recipient(s), or a person designated as responsible for delivering such messages to the intended recipient, is strictly prohibited and may be unlawful. This e-mail may contain proprietary, confidential or privileged information. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Virgin Pulse, Inc. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender and delete this e-mail message. v2.10 _______________________________________________ ceph-users mailing list mailto:ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
