I have installed diamond(built by ksingh found at
https://github.com/ksingh7/ceph-calamari-packages) on the MDS node and I am
not seeing the mds_server.handle_client_request OR objecter.op_active
metrics being sent to graphite. Mind you, this is not the graphite that is
part of the calamari install but our own internal graphite cluster.
Perhaps that is the reason? I could not get calamari working correctly on
hammerhead/centos7.1 so I put it on pause for now to concentrate on the
cluster itself.
Ultimately, I need to find a way to get a hold of these metrics to
determine the health of my MDS so I can justify moving forward on a SSD
based cephfs metadata pool.
On Wed, Aug 5, 2015 at 4:05 PM, Bob Ababurko <b...@ababurko.net> wrote:
> Hi John,
>
> You are correct in that my expectations may be incongruent with what is
> possible with ceph(fs). I'm currently copying many small files(images)
> from a netapp to the cluster...~35k sized files to be exact and the number
> of objects/files copied thus far is fairly significant(below in bold):
>
> [bababurko@cephmon01 ceph]$ sudo rados df
> pool name KB objects clones degraded
> unfound rd rd KB wr wr KB
> cephfs_data 3289284749 *163993660* 0 0
> 0 0 0 328097038 3369847354
> cephfs_metadata 133364 524363 0 0
> 0 3600023 5264453980 95600004 1361554516
> rbd 0 0 0 0
> 0 0 0 0 0
> total used 9297615196 164518023
> total avail 19990923044
> total space 29288538240
>
> Yes, that looks like ~164 million objects copied to the cluster. I would
> assume this will potentially be a burden to the MDS but I have yet to
> confirm with the ceph daemontool mds.<id>. I cannot seem to run it on the
> mds host as it doesn't seem to know about that command:
>
> [bababurko@cephmds01]$ sudo ceph daemonperf mds.cephmds01
> no valid command found; 10 closest matches:
> osd lost <int[0-]> {--yes-i-really-mean-it}
> osd create {<uuid>}
> osd primary-temp <pgid> <id>
> osd primary-affinity <osdname (id|osd.id)> <float[0.0-1.0]>
> osd reweight <int[0-]> <float[0.0-1.0]>
> osd pg-temp <pgid> {<id> [<id>...]}
> osd in <ids> [<ids>...]
> osd rm <ids> [<ids>...]
> osd down <ids> [<ids>...]
> osd out <ids> [<ids>...]
> Error EINVAL: invalid command
>
> This fails in a similar manner on all the hosts in the cluster. I'm very
> green w/ ceph and i'm probably missing something obvious. Is there
> something I need to install to get access to the 'ceph daemonperf' command
> in hammerhead?
>
> thanks,
> Bob
>
> On Wed, Aug 5, 2015 at 2:43 AM, John Spray <jsp...@redhat.com> wrote:
>
>> On Tue, Aug 4, 2015 at 10:36 PM, Bob Ababurko <b...@ababurko.net> wrote:
>> > My writes are not going as I would expect wrt to IOPS(50-1000 IOPs) &
>> write
>> > throughput( ~25MB/s max). I'm interested in understanding what it
>> takes to
>> > create a SSD pool that I can then migrate the current Cephfs_metadata
>> pool
>> > to. I suspect that the spinning disk metadata pool is a bottleneck and
>> I
>> > want to try to get the max performance out of this cluster to prove
>> that we
>> > would build out a larger version. One caveat is that I have copied
>> about 4
>> > TB of data to the cluster via cephfs and dont want to lose the data so I
>> > obviously need to keep the metadata intact.
>>
>> I'm a bit suspicious of this: your IOPS expectations sort of imply
>> doing big files, but you're then suggesting that metadata is the
>> bottleneck (i.e. small file workload).
>>
>> There are lots of statistics that come out of the MDS, you may be
>> particular interested in mds_server.handle_client_request,
>> objecter.op_active, to work out if there really are lots of RADOS
>> operations getting backed up on the MDS (which would be the symptom of
>> a too-slow metadata pool). "ceph daemonperf mds.<id>" may be some
>> help if you don't already have graphite or similar set up.
>>
>> > If anyone has done this OR understands how this can be done, I would
>> > appreciate the advice.
>>
>> You could potentially do this in a two-phase process where you
>> initially set a crush rule that includes both SSDs and spinners, and
>> then finally set a crush rule that just points to SSDs. Obviously
>> that'll do lots of data movement, but your metadata is probably a fair
>> bit smaller than your data so that might be acceptable.
>>
>> John
>>
>
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
