Hello Chris, We don't use SSD as journal. each host has one intel E5-2620 CPU which is 6 cores. the networking (both cluster and data networks) is 10Gbps.
My further questions include, (1) osd_mkfs_type = xfs osd_mkfs_options_xfs = -f filestore_xattr_use_omap = true for XFS filesystem, we should not enable filestore_xattr_use_omap = true, is it? (2) filestore_queue_max_ops = 25000 filestore_queue_max_bytes = 10485760 filestore_queue_committing_max_ops = 5000 filestore_queue_committing_max_bytes = 10485760000 journal_max_write_bytes = 1073714824 journal_max_write_entries = 10000 journal_queue_max_ops = 50000 journal_queue_max_bytes = 10485760000 Since we don't have SSD as journals, all these setup are too large? what are the better values? (3) osd_mount_options_xfs = "rw,noexec,nodev,noatime,nodiratime,nobarrier" What's your suggested options here? Thanks a lot. 2016-05-10 15:31 GMT+08:00 Christian Balzer <ch...@gol.com>: > On Tue, 10 May 2016 11:48:07 +0800 Geocast wrote: > > Hello, > > > We have 21 hosts for ceph OSD servers, each host has 12 SATA disks (4TB > > each), 64GB memory. > No journal SSDs? > What CPU(s) and network? > > > ceph version 10.2.0, Ubuntu 16.04 LTS > > The whole cluster is new installed. > > > > Can you help check what the arguments we put in ceph.conf is reasonable > > or not? > > thanks. > > > > [osd] > > osd_data = /var/lib/ceph/osd/ceph-$id > > osd_journal_size = 20000 > Overkill most likely, but not an issue. > > > osd_mkfs_type = xfs > > osd_mkfs_options_xfs = -f > > filestore_xattr_use_omap = true > > filestore_min_sync_interval = 10 > Are you aware what this does and have you actually tested this (IOPS AND > throughput) with various other setting on your hardware to arrive at this > number? > > > filestore_max_sync_interval = 15 > That's fine in and by itself, unlikely to ever be reached anyway. > > > filestore_queue_max_ops = 25000 > > filestore_queue_max_bytes = 10485760 > > filestore_queue_committing_max_ops = 5000 > > filestore_queue_committing_max_bytes = 10485760000 > > journal_max_write_bytes = 1073714824 > > journal_max_write_entries = 10000 > > journal_queue_max_ops = 50000 > > journal_queue_max_bytes = 10485760000 > Same as above, have you tested these setting (from filestore_queue_max_ops > onward) compared to the defaults? > > With HDDs only I'd expect any benefits to be small and/or things to become > very uneven once the HDDs are saturated. > > > osd_max_write_size = 512 > > osd_client_message_size_cap = 2147483648 > > osd_deep_scrub_stride = 131072 > > osd_op_threads = 8 > > osd_disk_threads = 4 > > osd_map_cache_size = 1024 > > osd_map_cache_bl_size = 128 > > osd_mount_options_xfs = "rw,noexec,nodev,noatime,nodiratime,nobarrier" > The nobarrier part is a a potential recipe for disaster unless you have all > on-disk caches disabled and every other cache battery backed. > > The only devices I trust to mount nobarrier are SSDs with powercaps that > have been proven to do the right thing (Intel DC S amongst them). > > > osd_recovery_op_priority = 4 > > osd_recovery_max_active = 10 > > osd_max_backfills = 4 > > > That's sane enough. > > > [client] > > rbd_cache = true > AFAIK that's the case with recent Ceph versions anyway. > > > rbd_cache_size = 268435456 > > Are you sure that you have 256MB per client to waste on RBD cache? > If so, bully for you, but you might find that depending on your use case a > smaller RBD cache but more VM memory (for pagecache, SLAB, etc) could be > more beneficial. > > > rbd_cache_max_dirty = 134217728 > > rbd_cache_max_dirty_age = 5 > > Christian > -- > Christian Balzer Network/Systems Engineer > ch...@gol.com Global OnLine Japan/Rakuten Communications > http://www.gol.com/ >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
