Ok, thank you guys The version is 10.2.10
Matteo > Il giorno 20 nov 2017, alle ore 23:15, Christian Balzer <ch...@gol.com> ha > scritto: > > On Mon, 20 Nov 2017 10:35:36 -0800 Chris Taylor wrote: > >> On 2017-11-20 3:39 am, Matteo Dacrema wrote: >>> Yes I mean the existing Cluster. >>> SSDs are on a fully separate pool. >>> Cluster is not busy during recovery and deep scrubs but I think it’s >>> better to limit replication in some way when switching to replica 3. >>> >>> My question is to understand if I need to set some options parameters >>> to limit the impact of the creation of new objects.I’m also concerned >>> about disk filling up during recovery because of inefficient data >>> balancing. >> >> You can try using osd_recovery_sleep to slow down the backfilling so it >> does not cause the client io to hang. >> >> ceph tell osd.* injectargs "--osd_recovery_sleep 0.1" >> > > Which is one of the things that is version specific and we don't know the > version yet. > > The above will work with Hammer and should again with Luminous, but not so > much with the unified queue bits inbetween. > > Christian > >> >>> >>> Here osd tree >>> >>> ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY >>> -10 19.69994 root ssd >>> -11 5.06998 host ceph101 >>> 166 0.98999 osd.166 up 1.00000 1.00000 >>> 167 1.00000 osd.167 up 1.00000 1.00000 >>> 168 1.00000 osd.168 up 1.00000 1.00000 >>> 169 1.07999 osd.169 up 1.00000 1.00000 >>> 170 1.00000 osd.170 up 1.00000 1.00000 >>> -12 4.92998 host ceph102 >>> 171 0.98000 osd.171 up 1.00000 1.00000 >>> 172 0.92999 osd.172 up 1.00000 1.00000 >>> 173 0.98000 osd.173 up 1.00000 1.00000 >>> 174 1.00000 osd.174 up 1.00000 1.00000 >>> 175 1.03999 osd.175 up 1.00000 1.00000 >>> -13 4.69998 host ceph103 >>> 176 0.84999 osd.176 up 1.00000 1.00000 >>> 177 0.84999 osd.177 up 1.00000 1.00000 >>> 178 1.00000 osd.178 up 1.00000 1.00000 >>> 179 1.00000 osd.179 up 1.00000 1.00000 >>> 180 1.00000 osd.180 up 1.00000 1.00000 >>> -14 5.00000 host ceph104 >>> 181 1.00000 osd.181 up 1.00000 1.00000 >>> 182 1.00000 osd.182 up 1.00000 1.00000 >>> 183 1.00000 osd.183 up 1.00000 1.00000 >>> 184 1.00000 osd.184 up 1.00000 1.00000 >>> 185 1.00000 osd.185 up 1.00000 1.00000 >>> -1 185.19835 root default >>> -2 18.39980 host ceph001 >>> 63 0.79999 osd.63 up 1.00000 1.00000 >>> 64 0.79999 osd.64 up 1.00000 1.00000 >>> 65 0.79999 osd.65 up 1.00000 1.00000 >>> 146 0.70000 osd.146 up 1.00000 1.00000 >>> 147 0.70000 osd.147 up 1.00000 1.00000 >>> 148 0.90999 osd.148 up 1.00000 1.00000 >>> 149 0.79999 osd.149 up 1.00000 1.00000 >>> 150 0.79999 osd.150 up 1.00000 1.00000 >>> 151 0.79999 osd.151 up 1.00000 1.00000 >>> 152 0.79999 osd.152 up 1.00000 1.00000 >>> 153 0.70000 osd.153 up 1.00000 1.00000 >>> 154 0.70000 osd.154 up 1.00000 1.00000 >>> 155 0.89999 osd.155 up 1.00000 1.00000 >>> 156 0.84999 osd.156 up 1.00000 1.00000 >>> 157 0.70000 osd.157 up 1.00000 1.00000 >>> 158 0.70000 osd.158 up 1.00000 1.00000 >>> 159 0.84999 osd.159 up 1.00000 1.00000 >>> 160 0.90999 osd.160 up 1.00000 1.00000 >>> 161 0.90999 osd.161 up 1.00000 1.00000 >>> 162 0.90999 osd.162 up 1.00000 1.00000 >>> 163 0.79999 osd.163 up 1.00000 1.00000 >>> 164 0.90999 osd.164 up 1.00000 1.00000 >>> 165 0.64999 osd.165 up 1.00000 1.00000 >>> -3 19.41982 host ceph002 >>> 23 0.79999 osd.23 up 1.00000 1.00000 >>> 24 0.70000 osd.24 up 1.00000 1.00000 >>> 25 0.90999 osd.25 up 1.00000 1.00000 >>> 26 0.59999 osd.26 up 1.00000 1.00000 >>> 27 0.95000 osd.27 up 1.00000 1.00000 >>> 28 0.64999 osd.28 up 1.00000 1.00000 >>> 29 0.75000 osd.29 up 1.00000 1.00000 >>> 30 0.89999 osd.30 up 1.00000 1.00000 >>> 31 0.90999 osd.31 up 1.00000 1.00000 >>> 32 0.90999 osd.32 up 1.00000 1.00000 >>> 33 0.89999 osd.33 up 1.00000 1.00000 >>> 34 0.90999 osd.34 up 1.00000 1.00000 >>> 35 0.90999 osd.35 up 1.00000 1.00000 >>> 36 0.84999 osd.36 up 1.00000 1.00000 >>> 37 0.89999 osd.37 up 1.00000 1.00000 >>> 38 1.00000 osd.38 up 1.00000 1.00000 >>> 39 0.79999 osd.39 up 1.00000 1.00000 >>> 40 0.90999 osd.40 up 1.00000 1.00000 >>> 41 0.84999 osd.41 up 1.00000 1.00000 >>> 42 0.84999 osd.42 up 1.00000 1.00000 >>> 43 0.90999 osd.43 up 1.00000 1.00000 >>> 44 0.75000 osd.44 up 1.00000 1.00000 >>> 45 0.79999 osd.45 up 1.00000 1.00000 >>> -4 16.78984 host ceph003 >>> 20 1.00000 osd.20 up 1.00000 1.00000 >>> 21 0.90999 osd.21 up 1.00000 1.00000 >>> 22 0.90999 osd.22 up 1.00000 1.00000 >>> 46 0.79999 osd.46 up 1.00000 1.00000 >>> 47 0.84999 osd.47 up 1.00000 1.00000 >>> 48 0.75000 osd.48 up 1.00000 1.00000 >>> 49 0.90999 osd.49 up 1.00000 1.00000 >>> 50 0.90999 osd.50 up 1.00000 1.00000 >>> 51 0.90999 osd.51 up 1.00000 1.00000 >>> 52 0.90999 osd.52 up 1.00000 1.00000 >>> 53 0.70000 osd.53 up 1.00000 1.00000 >>> 54 0.70000 osd.54 up 1.00000 1.00000 >>> 55 0.90999 osd.55 up 1.00000 1.00000 >>> 56 0.90999 osd.56 up 1.00000 1.00000 >>> 57 0.75000 osd.57 up 1.00000 1.00000 >>> 58 0.79999 osd.58 up 1.00000 1.00000 >>> 59 0.70000 osd.59 up 1.00000 1.00000 >>> 60 0.90999 osd.60 up 1.00000 1.00000 >>> 61 0.75000 osd.61 up 1.00000 1.00000 >>> 62 0.79999 osd.62 up 1.00000 1.00000 >>> -5 16.30986 host ceph004 >>> 66 0.95000 osd.66 up 1.00000 1.00000 >>> 67 0.79999 osd.67 up 1.00000 1.00000 >>> 68 0.84999 osd.68 up 1.00000 1.00000 >>> 69 0.84999 osd.69 up 1.00000 1.00000 >>> 70 0.70000 osd.70 up 1.00000 1.00000 >>> 71 0.75000 osd.71 up 1.00000 1.00000 >>> 72 0.64999 osd.72 up 1.00000 1.00000 >>> 73 0.90999 osd.73 up 1.00000 1.00000 >>> 74 0.75000 osd.74 up 1.00000 1.00000 >>> 75 0.81999 osd.75 up 1.00000 1.00000 >>> 76 0.90999 osd.76 up 1.00000 1.00000 >>> 77 0.79999 osd.77 up 1.00000 1.00000 >>> 78 0.59999 osd.78 up 1.00000 1.00000 >>> 79 1.00000 osd.79 up 1.00000 1.00000 >>> 80 0.90999 osd.80 up 1.00000 1.00000 >>> 81 0.79999 osd.81 up 1.00000 1.00000 >>> 82 0.89999 osd.82 up 1.00000 1.00000 >>> 83 0.70000 osd.83 up 1.00000 1.00000 >>> 84 0.75000 osd.84 up 1.00000 1.00000 >>> 85 0.90999 osd.85 up 1.00000 1.00000 >>> -6 16.71983 host ceph005 >>> 86 0.75000 osd.86 up 1.00000 1.00000 >>> 87 0.90999 osd.87 up 1.00000 1.00000 >>> 88 0.89999 osd.88 up 1.00000 1.00000 >>> 89 0.90999 osd.89 up 1.00000 1.00000 >>> 90 0.90999 osd.90 up 1.00000 1.00000 >>> 91 0.89999 osd.91 up 1.00000 1.00000 >>> 92 0.89999 osd.92 up 1.00000 1.00000 >>> 93 0.90999 osd.93 up 1.00000 1.00000 >>> 94 0.84999 osd.94 up 1.00000 1.00000 >>> 95 0.90999 osd.95 up 1.00000 1.00000 >>> 96 0.50000 osd.96 up 1.00000 1.00000 >>> 97 0.79999 osd.97 up 1.00000 1.00000 >>> 98 0.59999 osd.98 up 1.00000 1.00000 >>> 99 0.75000 osd.99 up 1.00000 1.00000 >>> 100 1.00000 osd.100 up 1.00000 1.00000 >>> 101 0.79999 osd.101 up 1.00000 1.00000 >>> 102 0.90999 osd.102 up 1.00000 1.00000 >>> 103 0.79999 osd.103 up 1.00000 1.00000 >>> 104 0.90999 osd.104 up 1.00000 1.00000 >>> 105 0.79999 osd.105 up 1.00000 1.00000 >>> -7 17.19980 host ceph006 >>> 106 0.90999 osd.106 up 1.00000 1.00000 >>> 107 0.89999 osd.107 up 1.00000 1.00000 >>> 108 0.70000 osd.108 up 1.00000 1.00000 >>> 109 0.84999 osd.109 up 1.00000 1.00000 >>> 110 0.79999 osd.110 up 1.00000 1.00000 >>> 111 0.79999 osd.111 up 1.00000 1.00000 >>> 112 0.79999 osd.112 up 1.00000 1.00000 >>> 113 0.90999 osd.113 up 1.00000 1.00000 >>> 115 0.90999 osd.115 up 1.00000 1.00000 >>> 116 0.90999 osd.116 up 1.00000 1.00000 >>> 117 0.90999 osd.117 up 1.00000 1.00000 >>> 118 0.90999 osd.118 up 1.00000 1.00000 >>> 119 0.59999 osd.119 up 1.00000 1.00000 >>> 120 0.90999 osd.120 up 1.00000 1.00000 >>> 121 0.90999 osd.121 up 1.00000 1.00000 >>> 122 0.90999 osd.122 up 1.00000 1.00000 >>> 123 0.79999 osd.123 up 1.00000 1.00000 >>> 124 0.75000 osd.124 up 1.00000 1.00000 >>> 125 1.09999 osd.125 up 1.00000 1.00000 >>> 114 0.90999 osd.114 up 1.00000 1.00000 >>> -8 16.61984 host ceph007 >>> 126 0.90999 osd.126 up 1.00000 1.00000 >>> 127 0.75000 osd.127 up 1.00000 1.00000 >>> 128 0.84999 osd.128 up 1.00000 1.00000 >>> 129 0.90999 osd.129 up 1.00000 1.00000 >>> 130 0.70000 osd.130 up 1.00000 1.00000 >>> 131 0.78000 osd.131 up 1.00000 1.00000 >>> 132 0.79999 osd.132 up 1.00000 1.00000 >>> 133 0.54999 osd.133 up 1.00000 1.00000 >>> 134 0.79999 osd.134 up 1.00000 1.00000 >>> 135 0.87000 osd.135 up 1.00000 1.00000 >>> 136 0.90999 osd.136 up 1.00000 1.00000 >>> 137 0.90999 osd.137 up 1.00000 1.00000 >>> 138 0.90999 osd.138 up 1.00000 1.00000 >>> 139 0.89999 osd.139 up 1.00000 1.00000 >>> 140 0.75000 osd.140 up 1.00000 1.00000 >>> 141 0.75000 osd.141 up 1.00000 1.00000 >>> 142 0.90999 osd.142 up 1.00000 1.00000 >>> 143 0.84999 osd.143 up 1.00000 1.00000 >>> 145 0.90999 osd.145 up 1.00000 1.00000 >>> 144 0.89999 osd.144 up 1.00000 1.00000 >>> -9 16.93982 host ceph008 >>> 0 0.90999 osd.0 up 1.00000 1.00000 >>> 1 0.70000 osd.1 up 1.00000 1.00000 >>> 2 0.90999 osd.2 up 1.00000 1.00000 >>> 3 0.90999 osd.3 up 1.00000 1.00000 >>> 4 0.79999 osd.4 up 1.00000 1.00000 >>> 5 0.64999 osd.5 up 1.00000 1.00000 >>> 6 0.64999 osd.6 up 1.00000 1.00000 >>> 7 0.90999 osd.7 up 1.00000 1.00000 >>> 8 0.79999 osd.8 up 1.00000 1.00000 >>> 9 0.90999 osd.9 up 1.00000 1.00000 >>> 10 0.75000 osd.10 up 1.00000 1.00000 >>> 11 0.90999 osd.11 up 1.00000 1.00000 >>> 12 0.89999 osd.12 up 1.00000 1.00000 >>> 13 0.75000 osd.13 up 1.00000 1.00000 >>> 14 1.09999 osd.14 up 1.00000 1.00000 >>> 15 0.84999 osd.15 up 1.00000 1.00000 >>> 16 0.90999 osd.16 up 1.00000 1.00000 >>> 17 0.90999 osd.17 up 1.00000 1.00000 >>> 18 0.79999 osd.18 up 1.00000 1.00000 >>> 19 0.90999 osd.19 up 1.00000 1.00000 >>> -15 28.79987 host ceph009 >>> 186 1.00000 osd.186 up 1.00000 1.00000 >>> 187 1.00000 osd.187 up 1.00000 1.00000 >>> 188 1.00000 osd.188 up 1.00000 1.00000 >>> 189 1.00000 osd.189 up 1.00000 1.00000 >>> 190 1.00000 osd.190 up 1.00000 1.00000 >>> 191 1.00000 osd.191 up 1.00000 1.00000 >>> 192 1.00000 osd.192 up 1.00000 1.00000 >>> 193 1.00000 osd.193 up 1.00000 1.00000 >>> 194 1.00000 osd.194 up 1.00000 1.00000 >>> 195 1.79999 osd.195 up 1.00000 1.00000 >>> 196 1.79999 osd.196 up 1.00000 1.00000 >>> 197 1.79999 osd.197 up 1.00000 1.00000 >>> 198 1.79999 osd.198 up 1.00000 1.00000 >>> 199 1.79999 osd.199 up 1.00000 1.00000 >>> 200 1.79999 osd.200 up 1.00000 1.00000 >>> 201 1.79999 osd.201 up 1.00000 1.00000 >>> 202 1.79999 osd.202 up 1.00000 1.00000 >>> 203 1.79999 osd.203 up 1.00000 1.00000 >>> 204 1.79999 osd.204 up 1.00000 1.00000 >>> 205 1.79999 osd.205 up 1.00000 1.00000 >>> -16 17.99988 host ceph010 >>> 206 1.79999 osd.206 up 1.00000 1.00000 >>> 207 1.79999 osd.207 up 1.00000 1.00000 >>> 208 1.79999 osd.208 up 1.00000 1.00000 >>> 209 1.79999 osd.209 up 1.00000 1.00000 >>> 210 1.79999 osd.210 up 1.00000 1.00000 >>> 211 1.79999 osd.211 up 1.00000 1.00000 >>> 212 1.79999 osd.212 up 1.00000 1.00000 >>> 213 1.79999 osd.213 up 1.00000 1.00000 >>> 214 1.79999 osd.214 up 1.00000 1.00000 >>> 215 1.79999 osd.215 up 1.00000 1.00000 >>> >>> >>> >>>> Il giorno 20 nov 2017, alle ore 12:17, Christian Balzer >>>> <ch...@gol.com> ha scritto: >>>> >>>> >>>> Hello, >>>> >>>> On Mon, 20 Nov 2017 11:56:31 +0100 Matteo Dacrema wrote: >>>> >>>>> Hi, >>>>> >>>>> I need to switch a cluster of over 200 OSDs from replica 2 to replica >>>>> 3 >>>> I presume this means the existing cluster and not adding 100 OSDs... >>>> >>>>> There are two different crush maps for HDD and SSDs also mapped to >>>>> two different pools. >>>>> >>>>> Is there a best practice to use? Can this provoke troubles? >>>>> >>>> Are your SSDs a cache-tier or are they a fully separate pool? >>>> >>>> As for troubles, how busy is your cluster during the recovery of >>>> failed >>>> OSDs or deep scrubs? >>>> >>>> There are 2 things to consider here: >>>> >>>> 1. The re-balancing and additional replication of all the data, which >>>> you >>>> can control/ease by the various knobs present. Ceph version matters to >>>> which are relevant/useful. It shouldn't impact things too much, unless >>>> your cluster was at the very edge of it's capacity anyway. >>>> >>>> 2. The little detail that after 1) is done, your cluster will be >>>> noticeably slower than before, especially in the latency department. >>>> In short, you don't just need to have the disk space to go 3x, but >>>> also >>>> enough IOPS/bandwidth reserves. >>>> >>>> Christian >>>> >>>>> Thank you >>>>> Matteo >>>>> _______________________________________________ >>>>> ceph-users mailing list >>>>> ceph-users@lists.ceph.com >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>>> >>>> >>>> >>>> -- >>>> Christian Balzer Network/Systems Engineer >>>> ch...@gol.com Rakuten Communications >>>> >>>> -- >>>> Questo messaggio e' stato analizzato con Libra ESVA ed e' risultato >>>> non infetto. >>>> Seguire il link qui sotto per segnalarlo come spam: >>>> http://mx01.enter.it/cgi-bin/learn-msg.cgi?id=524464756E.A33EC >>>> >>>> >>> >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > > -- > Christian Balzer Network/Systems Engineer > ch...@gol.com <mailto:ch...@gol.com> Rakuten Communications > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com> > > -- > Questo messaggio e' stato analizzato con Libra ESVA ed e' risultato non > infetto. > Seguire il link qui sotto per segnalarlo come spam: > http://mx01.enter.it/cgi-bin/learn-msg.cgi?id=78F40403AD.A839B > <http://mx01.enter.it/cgi-bin/learn-msg.cgi?id=78F40403AD.A839B>
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
