Hi Soeren. First, I would like to clarify something. There are two options:
*stretched cluster* and *stretch mode.* Sometimes this cannot be relied upon. If you have a “stretched-cluster” deployment in which much of your cluster is behind a single network component, you might need to use stretch mode to ensure data integrity. source: https://docs.ceph.com/en/latest/rados/operations/stretch-mode/#id1 The focus in this sentence is on ‘single network component’. I hope you don't have a single point of failure in your setup. Which option is best for your requirements? Regards, Joachim joachim.kraftma...@clyso.com www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE275430677 Am Mo., 28. Apr. 2025 um 02:50 Uhr schrieb Anthony D'Atri < anthony.da...@gmail.com>: > > Please make list posts in plain text. > > > > i am working on the plan for a 3 datacenter setup using ceph (in proxmox > nodes). > > > > Each datacenter has 3 physical nodes to start with and 100Gbit switches. > I will also have 2 x 100 Gbit/s connectivity between the datacenters (each > datacenter to each other). > > the physical nodes have 2 x 100Gbit/s for the public network and 2 x > 100Gbit/s for the cluster network. > > You almost certainly don’t need a cluster / replication network unless > these are exceptionally large nodes. > > > > > About this setup i have 2 questions. > > > > is it even necessary to evaluate a stretched cluster since the WAN > connections are as fast as the local ones (including the latency, since it > is only 25km) ? > > There’s more to latency than just distance. What is the measured > latency? A:B, B:C, C:A? > > > > > > > If using a stretched pool across all 3 datacenters, what happens if one > datacenter fails ? I did read the documentation and the question came up, > because it do not understand the sentence "Individual Stretch Pools do not > support I/O operations during a netsplit scenario between two or more > zones" completely, does it mean there is no IO already if one datacenter > fails ? > > That sentence refers to a non-stretch cluster. > > Tell us why you’re spreading across three DCs, what you’re trying to > accomplish, and what your performance requirements are. > > AIUI a stretch 3-site cluster requires all pools to be replicated, > size=6. > > Explicit stretch mode treats the mon quorum in a different way. With two > OSD sites you deploy a tiebreaker at a third site, which is possibly just a > cloud VM. With three OSD sites, I might speculate that one would deploy 7 > mons, 2 At each OSD site + tiebreaker. > > Operations on a stretch cluster can be slow. Sometimes separate clusters > with asynchronous replication make more sense. > > > > > > > If i am on the wrong path, maybe someone has a link for me, where is can > find information on this setup ? > > > > Cheers > > Soeren > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
