This patch adds more information about hardware preconditions and practices.
Signed-off-by: Alwin Antreich <a.antre...@proxmox.com> --- pveceph.adoc | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 51 insertions(+), 6 deletions(-) diff --git a/pveceph.adoc b/pveceph.adoc index f5ccdd1..b7378d5 100644 --- a/pveceph.adoc +++ b/pveceph.adoc @@ -72,16 +72,56 @@ footnote:[Ceph glossary http://docs.ceph.com/docs/luminous/glossary]. Precondition ------------ -To build a Proxmox Ceph Cluster there should be at least three (preferably) -identical servers for the setup. - -A 10Gb network, exclusively used for Ceph, is recommended. A meshed network -setup is also an option if there are no 10Gb switches available, see our wiki -article footnote:[Full Mesh Network for Ceph {webwiki-url}Full_Mesh_Network_for_Ceph_Server] . +To build a hyper-converged Proxmox + Ceph Cluster there should be at least +three (preferably) identical servers for the setup. Check also the recommendations from http://docs.ceph.com/docs/luminous/start/hardware-recommendations/[Ceph's website]. +.CPU +As higher the core frequency the better, this will reduce latency. This will +amongst others benefit Ceph's services, as they can process data quicker. As a +simple measure to ease planning dedicate a CPU core (or thread) for each Ceph +service to provide enough resources for a stable and enduring Ceph performance. + +.Memory +Especially in a hyper-converged setup, the memory consumption needs to be +carefully monitored. In addition to the intended workload (VM / Container), +Ceph needs enough memory to provide good and stable performance. As a rule of +thumb, for roughly 1TiB of data, 1 GiB of memory will be used by an OSD. With +additionally needed memory for the OSD caching. + +.Network +A 10 Gb or higher bandwidth network, exclusively used for Ceph, is recommended. +A meshed network setup is also an option if there are no 10 Gb switches +available, see our wiki article footnote:[Full Mesh Network for Ceph +{webwiki-url}Full_Mesh_Network_for_Ceph_Server] . + +To be insistently about networking, as Ceph is a network distributed storage, +its traffic needs to be separated onto its own physical network. The volume of +traffic especially during recovery will interfere with other services on the +same network. + +Further, estimate your bandwidth needs. While one HDD might not saturate a 1 Gb +link, a SSD or a NVMe SSD certainly can. Modern NVMe SSDs will even saturate 10 +Gb of bandwidth. You also should consider higher bandwidths, as these tend to +come with lower latency. + +.Disks +When planning the storage size of your Ceph cluster, it is important to take +the recovery time into consideration. Especially with small clusters, the +recovery might take a long time. It is advised to use SSDs or NVMe SSDs in +small setups to decrease the recovery time and therefore minimise the +probability of a subsequent failure event during recovery. + +In general SSDs or NVMe SSDs will provide more IOPs then spinning disks. This +fact and the higher cost may make a xref:pve_ceph_device_classes[class based] +separation of pools appealing. Another possibility to speedup OSDs is to use a +faster disk as journal or DB/WAL device. See below on how to create these. If a +faster disk is used for multiple OSDs an adequate ratio between OSD and WAL/DB +(or journal) device needs to be picked, as otherwise the faster disk will +become the bottleneck for all connected OSDs. + .Avoid RAID As Ceph handles data object redundancy and multiple parallel writes to disks (OSDs) on its own, using a RAID controller normally doesn’t improve @@ -93,6 +133,10 @@ the ones from Ceph. WARNING: Avoid RAID controller, use host bus adapter (HBA) instead. +NOTE: Above recommendations should be seen as a rough guidance for choosing +hardware. Hence it is still indispensable to test your setup and employ health +& performance monitoring. + [[pve_ceph_install]] Installation of Ceph Packages @@ -316,6 +360,7 @@ operation footnote:[Ceph pool operation http://docs.ceph.com/docs/luminous/rados/operations/pools/] manual. +[[pve_ceph_device_classes]] Ceph CRUSH & device classes --------------------------- The foundation of Ceph is its algorithm, **C**ontrolled **R**eplication -- 2.11.0 _______________________________________________ pve-devel mailing list pve-devel@pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
