[ceph-users] Ceph Usage web and terminal.

[Сергей Цаболов]

Hello,

My instalation of ceph is:

6 Node of Proxmox with 2 disk (8 TB) on the every node.

I make 12 OSD from all 8TB disk.

Ceph installed is - ceph version 15.2.14  octopus (stable)

I installed 6 monitor (all runnig) and 6 Manager 1 of them runnig 
(*active*) all others is *standby*.


In ceph I have 4 pools

device_health_metrics Size/min 3/2, Crush Rule: replicated_rule, of PGs 
1 PG Autoscale Mode : on , Min. # of PGs 1


cephfs_data Size/min 2/2, Crush Rule: replicated_rule,  of PGs 32 PG 
Autoscale Mode : on ,Min. # of PGs


cephfs_metadata Size/min 2/2, Crush Rule: replicated_rule,  of PGs 32 PG 
Autoscale Mode : on,Target Size: 500GB,  Min. # of PGs 16


pool_vm Size/min 2/2, Crush Rule: replicated_rule,  of PGs 512, PG 
Autoscale Mode : on,Target Ratio: 1


And now it confuses me  usage pools on web and on terminal

Storage cephfs : on web I see 42.80 TB,  on terminal with ceph df : 39 TiB

Storage pool_vm: on web I see 45.27 TB, on terminal with   ceph df : 39 TiB

All pools usage in terminal I see with ceph df

--- RAW STORAGE ---
CLASS  SIZE    AVAIL   USED RAW USED  %RAW USED
hdd    87 TiB  83 TiB  4.6 TiB   4.6 TiB   5.27
TOTAL  87 TiB  83 TiB  4.6 TiB   4.6 TiB   5.27

--- POOLS ---
POOL   ID  PGS  STORED   OBJECTS  USED %USED  MAX AVAIL
device_health_metrics   1    1  1.2 MiB   12  3.6 MiB 0 26 TiB
pool_vm 2  512  2.3 TiB  732.57k  4.6 TiB 5.55 39 TiB
cephfs_data 3   32  0 B    0  0 B 0 39 TiB
cephfs_metadata 4   32  9.8 MiB   24   21 MiB 0 39 TiB

I don't quite understand the discrepancy in TB usage on web and on terminal.
Maybe I misunderstood something.

P.S. And the question is which of usage disk I can use for stored data: 
the usage what I see on web or what I see on terminal?



--
-
Sergey TS

With Best Regard

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Disable PG Autoscale - globally

[Nagaraj Akkina]

We are using ceph-ansible to deploy our ceph clusters, Octopus version and
passing required parameters to ceph.conf . We are unable  send parameter
to  disable pg auto scaling for existing default pools and also new pools.

 global osd pool default pg autoscale mode: "off"

osd pool default pg autoscale mode: "off"

is not working, How can I disable PG auto scaling while using ceph-ansible?

Regards,
Akkina
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Usage web and terminal.

[Eugen Block]

Also note that you need an odd number of MONs to be able to form a  
quorum. So I would recommend to remove one MON to have 5.



Zitat von Eneko Lacunza :


Hi,

El 27/10/21 a las 9:55, Сергей Цаболов escribió:

My instalation of ceph is:

6 Node of Proxmox with 2 disk (8 TB) on the every node.

I make 12 OSD from all 8TB disk.

Ceph installed is - ceph version 15.2.14  octopus (stable)

I installed 6 monitor (all runnig) and 6 Manager 1 of them runnig  
(*active*) all others is *standby*.


In ceph I have 4 pools

device_health_metrics Size/min 3/2, Crush Rule: replicated_rule, of  
PGs 1 PG Autoscale Mode : on , Min. # of PGs 1


cephfs_data Size/min 2/2, Crush Rule: replicated_rule,  of PGs 32  
PG Autoscale Mode : on ,Min. # of PGs


cephfs_metadata Size/min 2/2, Crush Rule: replicated_rule,  of PGs  
32 PG Autoscale Mode : on,Target Size: 500GB,  Min. # of PGs 16


pool_vm Size/min 2/2, Crush Rule: replicated_rule,  of PGs 512, PG  
Autoscale Mode : on,Target Ratio: 1


You're aware that size 2/2 makes it very likely you will have disk  
write problems, right (an OSD issue will prevent writes)?




And now it confuses me  usage pools on web and on terminal

Storage cephfs : on web I see 42.80 TB,  on terminal with ceph df : 39 TiB

Storage pool_vm: on web I see 45.27 TB, on terminal with   ceph df : 39 TiB


This is TB->TiB conversion, 42.80TB = 428000 bytes/1024⁴ ~= 39TiB

Also, it can't reallistically be usage, must be total available  
space (roughly half the raw space due to your pools being replicated  
size=2)




All pools usage in terminal I see with ceph df

--- RAW STORAGE ---
CLASS  SIZE    AVAIL   USED RAW USED  %RAW USED
hdd    87 TiB  83 TiB  4.6 TiB   4.6 TiB   5.27
TOTAL  87 TiB  83 TiB  4.6 TiB   4.6 TiB   5.27

--- POOLS ---
POOL   ID  PGS  STORED   OBJECTS  USED %USED  MAX AVAIL
device_health_metrics   1    1  1.2 MiB   12  3.6 MiB 0 26 TiB
pool_vm 2  512  2.3 TiB  732.57k  4.6 TiB 5.55 39 TiB
cephfs_data 3   32  0 B    0  0 B 0 39 TiB
cephfs_metadata 4   32  9.8 MiB   24   21 MiB 0 39 TiB

I don't quite understand the discrepancy in TB usage on web and on terminal.
Maybe I misunderstood something.

P.S. And the question is which of usage disk I can use for stored  
data: the usage what I see on web or what I see on terminal?




Hope this helps ;)

Cheers

Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project

Tel. +34 943 569 206 | https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun

https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Usage web and terminal.

[Janne Johansson]

Den ons 27 okt. 2021 kl 11:10 skrev Eugen Block :
>
> Also note that you need an odd number of MONs to be able to form a
> quorum. So I would recommend to remove one MON to have 5.

Well, you need to have a distinct majority, so using 6 is not better
than 5, but not worse either.
Both will let the cluster run as long as no more than 2 mons are currently out.


-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Usage web and terminal.

[Сергей Цаболов]

Hi,

27.10.2021 12:03, Eneko Lacunza пишет:

Hi,

El 27/10/21 a las 9:55, Сергей Цаболов escribió:

My instalation of ceph is:

6 Node of Proxmox with 2 disk (8 TB) on the every node.

I make 12 OSD from all 8TB disk.

Ceph installed is - ceph version 15.2.14  octopus (stable)

I installed 6 monitor (all runnig) and 6 Manager 1 of them runnig 
(*active*) all others is *standby*.


In ceph I have 4 pools

device_health_metrics Size/min 3/2, Crush Rule: replicated_rule, of 
PGs 1 PG Autoscale Mode : on , Min. # of PGs 1


cephfs_data Size/min 2/2, Crush Rule: replicated_rule,  of PGs 32 PG 
Autoscale Mode : on ,Min. # of PGs


cephfs_metadata Size/min 2/2, Crush Rule: replicated_rule,  of PGs 32 
PG Autoscale Mode : on,Target Size: 500GB,  Min. # of PGs 16


pool_vm Size/min 2/2, Crush Rule: replicated_rule,  of PGs 512, PG 
Autoscale Mode : on,Target Ratio: 1


You're aware that size 2/2 makes it very likely you will have disk 
write problems, right (an OSD issue will prevent writes)?


No, I not have disk problem, I looking for when I lost more TB, because 
before of change Size & min Size  to 2/2 I have the 3/2 and the space is 
28-29 TB.





And now it confuses me  usage pools on web and on terminal

Storage cephfs : on web I see 42.80 TB,  on terminal with ceph df : 
39 TiB


Storage pool_vm: on web I see 45.27 TB, on terminal with   ceph df : 
39 TiB


This is TB->TiB conversion, 42.80TB = 428000 bytes/1024⁴ ~= 39TiB


Ox, I not the count right space, is my mistake !!! 😉



Also, it can't reallistically be usage, must be total available space 
(roughly half the raw space due to your pools being replicated size=2)




All pools usage in terminal I see with ceph df

--- RAW STORAGE ---
CLASS  SIZE    AVAIL   USED RAW USED  %RAW USED
hdd    87 TiB  83 TiB  4.6 TiB   4.6 TiB   5.27
TOTAL  87 TiB  83 TiB  4.6 TiB   4.6 TiB   5.27

--- POOLS ---
POOL   ID  PGS  STORED   OBJECTS  USED %USED MAX AVAIL
device_health_metrics   1    1  1.2 MiB   12  3.6 MiB 0 26 TiB
pool_vm 2  512  2.3 TiB  732.57k  4.6 TiB 5.55 39 TiB
cephfs_data 3   32  0 B    0  0 B 0 39 TiB
cephfs_metadata 4   32  9.8 MiB   24   21 MiB 0 39 TiB

I don't quite understand the discrepancy in TB usage on web and on 
terminal.

Maybe I misunderstood something.

P.S. And the question is which of usage disk I can use for stored 
data: the usage what I see on web or what I see on terminal?




Hope this helps ;)

Cheers

Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project

Tel. +34 943 569 206 | https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun

https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


--
-
С уважением
Сергей Цаболов,
Системный администратор
ООО "Т8"
Тел.: +74992716161,
Моб: +79850334875
tsabo...@t8.ru
ООО «Т8», 107076, г. Москва, Краснобогатырская ул., д. 44, стр.1
www.t8.ru

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Usage web and terminal.

[Сергей Цаболов]

I need to destroy it or just stopped ?

27.10.2021 12:09, Eugen Block пишет:
Also note that you need an odd number of MONs to be able to form a 
quorum. So I would recommend to remove one MON to have 5.



Zitat von Eneko Lacunza :


Hi,

El 27/10/21 a las 9:55, Сергей Цаболов escribió:

My instalation of ceph is:

6 Node of Proxmox with 2 disk (8 TB) on the every node.

I make 12 OSD from all 8TB disk.

Ceph installed is - ceph version 15.2.14  octopus (stable)

I installed 6 monitor (all runnig) and 6 Manager 1 of them runnig 
(*active*) all others is *standby*.


In ceph I have 4 pools

device_health_metrics Size/min 3/2, Crush Rule: replicated_rule, of 
PGs 1 PG Autoscale Mode : on , Min. # of PGs 1


cephfs_data Size/min 2/2, Crush Rule: replicated_rule,  of PGs 32 PG 
Autoscale Mode : on ,Min. # of PGs


cephfs_metadata Size/min 2/2, Crush Rule: replicated_rule,  of PGs 
32 PG Autoscale Mode : on,Target Size: 500GB,  Min. # of PGs 16


pool_vm Size/min 2/2, Crush Rule: replicated_rule,  of PGs 512, PG 
Autoscale Mode : on,Target Ratio: 1


You're aware that size 2/2 makes it very likely you will have disk 
write problems, right (an OSD issue will prevent writes)?




And now it confuses me  usage pools on web and on terminal

Storage cephfs : on web I see 42.80 TB,  on terminal with ceph df : 
39 TiB


Storage pool_vm: on web I see 45.27 TB, on terminal with ceph df : 
39 TiB


This is TB->TiB conversion, 42.80TB = 428000 bytes/1024⁴ ~= 
39TiB


Also, it can't reallistically be usage, must be total available space 
(roughly half the raw space due to your pools being replicated size=2)




All pools usage in terminal I see with ceph df

--- RAW STORAGE ---
CLASS  SIZE    AVAIL   USED RAW USED  %RAW USED
hdd    87 TiB  83 TiB  4.6 TiB   4.6 TiB   5.27
TOTAL  87 TiB  83 TiB  4.6 TiB   4.6 TiB   5.27

--- POOLS ---
POOL   ID  PGS  STORED   OBJECTS  USED %USED MAX AVAIL
device_health_metrics   1    1  1.2 MiB   12  3.6 MiB 0 26 TiB
pool_vm 2  512  2.3 TiB  732.57k  4.6 TiB 5.55 39 TiB
cephfs_data 3   32  0 B    0  0 B 0 39 TiB
cephfs_metadata 4   32  9.8 MiB   24   21 MiB 0 39 TiB

I don't quite understand the discrepancy in TB usage on web and on 
terminal.

Maybe I misunderstood something.

P.S. And the question is which of usage disk I can use for stored 
data: the usage what I see on web or what I see on terminal?




Hope this helps ;)

Cheers

Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project

Tel. +34 943 569 206 | https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun

https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


--
-
С уважением
Сергей Цаболов,
Системный администратор
ООО "Т8"
Тел.: +74992716161,
Моб: +79850334875
tsabo...@t8.ru
ООО «Т8», 107076, г. Москва, Краснобогатырская ул., д. 44, стр.1
www.t8.ru

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] After nautilus to pacific update rbd_header omapkey broken

[Lanore Ronan]

Hi list,

We have a lot of failed instances when reboot host  after upgrade cpeh client 
to pacific.
On  160 instances failed represent 15-20%

We have been impacted by this treads too when upgrade cluster: 
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/C7FIBFYP32KSWJXL2XSJJKXNMASKMAQ4/

I'm working with beard lionel and for now we have stopped to do 
bluestore_fsck_quick_fix_on_mount on OSD with

  osd.95 legacy (not per-pool) omap detected, suggest to run store repair to 
benefit from per-pool omap usage statistics

message. Because the process failed 50% of time and we can't start ever OSD

Exemple all keys are prefixed by IO. with other rbd volume header keys are 
prefixed by 'X.'

root@ceph-mon4:/usr/local/sbin# rados -p openstack2_vms listomapvals 
rbd_header.db979141b71efb
key (25 bytes):
  00 00 00 00 00 0a 49 4f  2e 63 72 65 61 74 65 5f  |..IO.create_|
0010  74 69 6d 65 73 74 61 6d  70   |timestamp|
0019

value (8 bytes) :
  5c 22 87 5e 2d de 61 2c   |\".^-.a,|
0008

key (17 bytes):
  00 00 00 00 00 0a 49 4f  2e 66 65 61 74 75 72 65  |..IO.feature|
0010  73|s|
0011

To repair our VM I writed a little script it delete rbd_header obekct and 
re-create one with correct keys name and content from backuped header.

Any advise are welcome
thanks


Ronan Lanore



Ce message et toutes les pièces jointes (ci-après le "message") sont établis à 
l'intention exclusive de ses destinataires et sont confidentiels. Si vous 
recevez ce message par erreur ou s'il ne vous est pas destiné, merci de le 
détruire ainsi que toute copie de votre système et d'en avertir immédiatement 
l'expéditeur. Toute lecture non autorisée, toute utilisation de ce message qui 
n'est pas conforme à sa destination, toute diffusion ou toute publication, 
totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer 
l'intégrité de ce message électronique susceptible d'altération, l'expéditeur 
(et ses filiales) décline(nt) toute responsabilité au titre de ce message dans 
l'hypothèse où il aurait été modifié ou falsifié.

This message and any attachments (the "message") is intended solely for the 
intended recipient(s) and is confidential. If you receive this message in 
error, or are not the intended recipient(s), please delete it and any copies 
from your systems and immediately notify the sender. Any unauthorized view, use 
that does not comply with its purpose, dissemination or disclosure, either 
whole or partial, is prohibited. Since the internet cannot guarantee the 
integrity of this message which may not be reliable, the sender (and its 
subsidiaries) shall not be liable for the message if modified or falsified.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Usage web and terminal.

[Eugen Block]

I need to destroy it or just stopped ?


Destroy it so you only have 5 existing MONs in the cluster.


Zitat von Сергей Цаболов :


I need to destroy it or just stopped ?

27.10.2021 12:09, Eugen Block пишет:
Also note that you need an odd number of MONs to be able to form a  
quorum. So I would recommend to remove one MON to have 5.



Zitat von Eneko Lacunza :


Hi,

El 27/10/21 a las 9:55, Сергей Цаболов escribió:

My instalation of ceph is:

6 Node of Proxmox with 2 disk (8 TB) on the every node.

I make 12 OSD from all 8TB disk.

Ceph installed is - ceph version 15.2.14  octopus (stable)

I installed 6 monitor (all runnig) and 6 Manager 1 of them runnig  
(*active*) all others is *standby*.


In ceph I have 4 pools

device_health_metrics Size/min 3/2, Crush Rule: replicated_rule,  
of PGs 1 PG Autoscale Mode : on , Min. # of PGs 1


cephfs_data Size/min 2/2, Crush Rule: replicated_rule,  of PGs 32  
PG Autoscale Mode : on ,Min. # of PGs


cephfs_metadata Size/min 2/2, Crush Rule: replicated_rule,  of  
PGs 32 PG Autoscale Mode : on,Target Size: 500GB,  Min. # of PGs 16


pool_vm Size/min 2/2, Crush Rule: replicated_rule,  of PGs 512,  
PG Autoscale Mode : on,Target Ratio: 1


You're aware that size 2/2 makes it very likely you will have disk  
write problems, right (an OSD issue will prevent writes)?




And now it confuses me  usage pools on web and on terminal

Storage cephfs : on web I see 42.80 TB,  on terminal with ceph df : 39 TiB

Storage pool_vm: on web I see 45.27 TB, on terminal with ceph df : 39 TiB


This is TB->TiB conversion, 42.80TB = 428000 bytes/1024⁴ ~= 39TiB

Also, it can't reallistically be usage, must be total available  
space (roughly half the raw space due to your pools being  
replicated size=2)




All pools usage in terminal I see with ceph df

--- RAW STORAGE ---
CLASS  SIZE    AVAIL   USED RAW USED  %RAW USED
hdd    87 TiB  83 TiB  4.6 TiB   4.6 TiB   5.27
TOTAL  87 TiB  83 TiB  4.6 TiB   4.6 TiB   5.27

--- POOLS ---
POOL   ID  PGS  STORED   OBJECTS  USED %USED MAX AVAIL
device_health_metrics   1    1  1.2 MiB   12  3.6 MiB 0 26 TiB
pool_vm 2  512  2.3 TiB  732.57k  4.6 TiB 5.55 39 TiB
cephfs_data 3   32  0 B    0  0 B 0 39 TiB
cephfs_metadata 4   32  9.8 MiB   24   21 MiB 0 39 TiB

I don't quite understand the discrepancy in TB usage on web and  
on terminal.

Maybe I misunderstood something.

P.S. And the question is which of usage disk I can use for stored  
data: the usage what I see on web or what I see on terminal?




Hope this helps ;)

Cheers

Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project

Tel. +34 943 569 206 | https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun

https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


--
-
С уважением
Сергей Цаболов,
Системный администратор
ООО "Т8"
Тел.: +74992716161,
Моб: +79850334875
tsabo...@t8.ru
ООО «Т8», 107076, г. Москва, Краснобогатырская ул., д. 44, стр.1
www.t8.ru




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Usage web and terminal.

[Сергей Цаболов]

Thank you!

I destroy it.

27.10.2021 13:18, Eugen Block пишет:

I need to destroy it or just stopped ?


Destroy it so you only have 5 existing MONs in the cluster.


Zitat von Сергей Цаболов :


I need to destroy it or just stopped ?

27.10.2021 12:09, Eugen Block пишет:
Also note that you need an odd number of MONs to be able to form a 
quorum. So I would recommend to remove one MON to have 5.



Zitat von Eneko Lacunza :


Hi,

El 27/10/21 a las 9:55, Сергей Цаболов escribió:

My instalation of ceph is:

6 Node of Proxmox with 2 disk (8 TB) on the every node.

I make 12 OSD from all 8TB disk.

Ceph installed is - ceph version 15.2.14  octopus (stable)

I installed 6 monitor (all runnig) and 6 Manager 1 of them runnig 
(*active*) all others is *standby*.


In ceph I have 4 pools

device_health_metrics Size/min 3/2, Crush Rule: replicated_rule, 
of PGs 1 PG Autoscale Mode : on , Min. # of PGs 1


cephfs_data Size/min 2/2, Crush Rule: replicated_rule,  of PGs 32 
PG Autoscale Mode : on ,Min. # of PGs


cephfs_metadata Size/min 2/2, Crush Rule: replicated_rule,  of PGs 
32 PG Autoscale Mode : on,Target Size: 500GB,  Min. # of PGs 16


pool_vm Size/min 2/2, Crush Rule: replicated_rule,  of PGs 512, PG 
Autoscale Mode : on,Target Ratio: 1


You're aware that size 2/2 makes it very likely you will have disk 
write problems, right (an OSD issue will prevent writes)?




And now it confuses me  usage pools on web and on terminal

Storage cephfs : on web I see 42.80 TB,  on terminal with ceph df 
: 39 TiB


Storage pool_vm: on web I see 45.27 TB, on terminal with ceph df : 
39 TiB


This is TB->TiB conversion, 42.80TB = 428000 bytes/1024⁴ ~= 
39TiB


Also, it can't reallistically be usage, must be total available 
space (roughly half the raw space due to your pools being 
replicated size=2)




All pools usage in terminal I see with ceph df

--- RAW STORAGE ---
CLASS  SIZE    AVAIL   USED RAW USED  %RAW USED
hdd    87 TiB  83 TiB  4.6 TiB   4.6 TiB   5.27
TOTAL  87 TiB  83 TiB  4.6 TiB   4.6 TiB   5.27

--- POOLS ---
POOL   ID  PGS  STORED   OBJECTS  USED %USED MAX 
AVAIL
device_health_metrics   1    1  1.2 MiB   12  3.6 MiB 0 26 
TiB

pool_vm 2  512  2.3 TiB  732.57k  4.6 TiB 5.55 39 TiB
cephfs_data 3   32  0 B    0  0 B 0 39 
TiB
cephfs_metadata 4   32  9.8 MiB   24   21 MiB 0 39 
TiB


I don't quite understand the discrepancy in TB usage on web and on 
terminal.

Maybe I misunderstood something.

P.S. And the question is which of usage disk I can use for stored 
data: the usage what I see on web or what I see on terminal?




Hope this helps ;)

Cheers

Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project

Tel. +34 943 569 206 | https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun

https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


--
-
С уважением
Сергей Цаболов,
Системный администратор
ООО "Т8"
Тел.: +74992716161,
Моб: +79850334875
tsabo...@t8.ru
ООО «Т8», 107076, г. Москва, Краснобогатырская ул., д. 44, стр.1
www.t8.ru






--
-
С уважением
Сергей Цаболов,
Системный администратор
ООО "Т8"
Тел.: +74992716161,
Моб: +79850334875
tsabo...@t8.ru
ООО «Т8», 107076, г. Москва, Краснобогатырская ул., д. 44, стр.1
www.t8.ru

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: After nautilus to pacific update rbd_header omapkey broken

[Igor Fedotov]

Hi Lanore,

as I've already mentioned at my last reply to 
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/XDMISQC74Z67RXP2PJHERARJ7KT2ADW4/ 
there is a bug in BlueStore's quick-fix/repair in OSDs upgraded to 
Pacific. Looks like omaps' records are getting improper keys...


I'm working on the fsck's fix at the moment. . Is it correct that you 
managed to recover your OSDs after the incident so the need for 
additional means to recover  broken omaps isn't that urgent at the moment?



Thanks,

Igor


On 10/27/2021 1:11 PM, Lanore Ronan wrote:

pool omap us


--
Igor Fedotov
Ceph Lead Developer

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Doing SAML2 Auth With Containerized mgrs

[Ernesto Puerta]

Hi Edward,

You don't need to worry about keeping those certs persistently, the Ceph
Dashboard does that for you (they're persisted inside the ceph-mon KV
store). You just need to ensure that the paths you provide are reachable to
the ceph-mgr daemon. And I agree: that's a bit tricky with containerized
deployments (it was originally designed for bare-metal ones in mind). We
could improve that by making the Ceph CLI tool read CephFile-type args and
send the contents of these files, instead of sending the paths to the
daemon (I don't know what was the original intent of this behaviour, maybe
avoid sending sensitive data unencrypted or avoiding the multiple loggers
that might dump what's sent from the Ceph CLI).

Kind Regards,
Ernesto


On Mon, Oct 25, 2021 at 6:21 PM Edward R Huyer  wrote:

> No worries.  It's a pretty specific problem, and the documentation could
> be better.
>
> -Original Message-
> From: Yury Kirsanov 
> Sent: Monday, October 25, 2021 12:17 PM
> To: Edward R Huyer 
> Cc: ceph-users@ceph.io
> Subject: [ceph-users] Re: Doing SAML2 Auth With Containerized mgrs
>
> Hi Edward,
> Yes, you probably are right, I thought about dashboard SSL certificate,
> not the SAML2, sorry for that.
>
> Regards,
> Yury.
>
> On Tue, Oct 26, 2021 at 3:10 AM Edward R Huyer  wrote:
>
> > I don’t think that’s correct?  I already have a certificate set up for
> > HTTPS, and it doesn’t show up in the SAML2 configuration.  Maybe I’m
> > mistaken, but I think the SAML2 cert is separate from the regular
> > HTTPS cert?
> >
> >
> >
> > *From:* Yury Kirsanov 
> > *Sent:* Monday, October 25, 2021 11:52 AM
> > *To:* Edward R Huyer 
> > *Cc:* ceph-users@ceph.io
> > *Subject:* Re: [ceph-users] Doing SAML2 Auth With Containerized mgrs
> >
> >
> >
> > *CAUTION: This message came from outside RIT. If you are unsure about
> > the source or content of this message, please contact the RIT Service
> > Center at
> > 585-475-5000 or help.rit.edu  before clicking
> > links, opening attachments or responding.*
> >
> > Hi Edward,
> >
> > You need to set configuration like this, assuming that certificate and
> > key are on your local disk:
> >
> > ceph mgr module disable dashboard
> > ceph dashboard set-ssl-certificate -i .crt ceph
> > dashboard set-ssl-certificate-key -i .key ceph
> > config-key set mgr/cephadm/grafana_crt -i .crt ceph
> > config-key set mgr/cephadm/grafana_key -i .key
> > ceph orch reconfig grafana ceph mgr module enable dashboard
> >
> > Hope this helps!
> >
> > Regards,
> > Yury.
> >
> >
> >
> > On Tue, Oct 26, 2021 at 2:45 AM Edward R Huyer  wrote:
> >
> > Continuing my containerized Ceph adventures
> >
> > I'm trying to set up SAML2 auth for the dashboard (specifically
> > pointing at the institute Shibboleth service).  The service requires
> > the use of the
> > x509 certificates.  Following the instructions in the documentation (
> > https://docs.ceph.com/en/latest/mgr/dashboard/#dashboard-sso-support )
> > leads to an error about the certificate file not existing.
> >
> > Some digging suggests that's because the daemon is looking in the
> > container's filesystem rather than the physical host's filesystem.
> > That makes some sense, but it annoying.
> >
> > So my question is:  How do I get the cert and key file into the
> > container's filesystem in a persistent way?  cephadm enter --name
> > "mgr.hostname" results in a "no such container" error.  cephadm shell
> > --name "mgr.hostname" works, but changes don't persist.
> >
> > Any suggestions about this problem specifically, authing the dashboard
> > against Shibboleth, or SAML2 in general?
> >
> > -
> > Edward Huyer
> > Golisano College of Computing and Information Sciences Rochester
> > Institute of Technology Golisano 70-2373
> > 152 Lomb Memorial Drive
> > Rochester, NY 14623
> > 585-475-6651
> > erh...@rit.edu
> >
> > Obligatory Legalese:
> > The information transmitted, including attachments, is intended only
> > for the person(s) or entity to which it is addressed and may contain
> > confidential and/or privileged material. Any review, retransmission,
> > dissemination or other use of, or taking of any action in reliance
> > upon this information by persons or entities other than the intended
> > recipient is prohibited. If you received this in error, please contact
> > the sender and destroy any copies of this information.
> >
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
> > email to ceph-users-le...@ceph.io
> >
> >
> ___
> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
> email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io

[ceph-users] After nautilus to pacific update rbd_header omapkey broken

[Lanore Ronan]

Hi, thanks for reply. I haven't received your answer because of my subscription 
delivered set to None: Fixed

You are wright. We have provisionned other OSD to fix our cluster at the moment.
waiting for fix
thanks

Ronan Lanore



Ce message et toutes les pièces jointes (ci-après le "message") sont établis à 
l'intention exclusive de ses destinataires et sont confidentiels. Si vous 
recevez ce message par erreur ou s'il ne vous est pas destiné, merci de le 
détruire ainsi que toute copie de votre système et d'en avertir immédiatement 
l'expéditeur. Toute lecture non autorisée, toute utilisation de ce message qui 
n'est pas conforme à sa destination, toute diffusion ou toute publication, 
totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer 
l'intégrité de ce message électronique susceptible d'altération, l'expéditeur 
(et ses filiales) décline(nt) toute responsabilité au titre de ce message dans 
l'hypothèse où il aurait été modifié ou falsifié.

This message and any attachments (the "message") is intended solely for the 
intended recipient(s) and is confidential. If you receive this message in 
error, or are not the intended recipient(s), please delete it and any copies 
from your systems and immediately notify the sender. Any unauthorized view, use 
that does not comply with its purpose, dissemination or disclosure, either 
whole or partial, is prohibited. Since the internet cannot guarantee the 
integrity of this message which may not be reliable, the sender (and its 
subsidiaries) shall not be liable for the message if modified or falsified.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Usage web and terminal.

[Eneko Lacunza]

Hi,

El 27/10/21 a las 9:55, Сергей Цаболов escribió:

My instalation of ceph is:

6 Node of Proxmox with 2 disk (8 TB) on the every node.

I make 12 OSD from all 8TB disk.

Ceph installed is - ceph version 15.2.14  octopus (stable)

I installed 6 monitor (all runnig) and 6 Manager 1 of them runnig 
(*active*) all others is *standby*.


In ceph I have 4 pools

device_health_metrics Size/min 3/2, Crush Rule: replicated_rule, of 
PGs 1 PG Autoscale Mode : on , Min. # of PGs 1


cephfs_data Size/min 2/2, Crush Rule: replicated_rule,  of PGs 32 PG 
Autoscale Mode : on ,Min. # of PGs


cephfs_metadata Size/min 2/2, Crush Rule: replicated_rule,  of PGs 32 
PG Autoscale Mode : on,Target Size: 500GB,  Min. # of PGs 16


pool_vm Size/min 2/2, Crush Rule: replicated_rule,  of PGs 512, PG 
Autoscale Mode : on,Target Ratio: 1


You're aware that size 2/2 makes it very likely you will have disk write 
problems, right (an OSD issue will prevent writes)?




And now it confuses me  usage pools on web and on terminal

Storage cephfs : on web I see 42.80 TB,  on terminal with ceph df : 39 
TiB


Storage pool_vm: on web I see 45.27 TB, on terminal with   ceph df : 
39 TiB


This is TB->TiB conversion, 42.80TB = 428000 bytes/1024⁴ ~= 39TiB

Also, it can't reallistically be usage, must be total available space 
(roughly half the raw space due to your pools being replicated size=2)




All pools usage in terminal I see with ceph df

--- RAW STORAGE ---
CLASS  SIZE    AVAIL   USED RAW USED  %RAW USED
hdd    87 TiB  83 TiB  4.6 TiB   4.6 TiB   5.27
TOTAL  87 TiB  83 TiB  4.6 TiB   4.6 TiB   5.27

--- POOLS ---
POOL   ID  PGS  STORED   OBJECTS  USED %USED  MAX AVAIL
device_health_metrics   1    1  1.2 MiB   12  3.6 MiB 0 26 TiB
pool_vm 2  512  2.3 TiB  732.57k  4.6 TiB 5.55 39 TiB
cephfs_data 3   32  0 B    0  0 B 0 39 TiB
cephfs_metadata 4   32  9.8 MiB   24   21 MiB 0 39 TiB

I don't quite understand the discrepancy in TB usage on web and on 
terminal.

Maybe I misunderstood something.

P.S. And the question is which of usage disk I can use for stored 
data: the usage what I see on web or what I see on terminal?




Hope this helps ;)

Cheers

Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project

Tel. +34 943 569 206 | https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun

https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Usage web and terminal.

[Eugen Block]

Sorry, I saw your answer too late, you're right of course. I  
internalised the odd number of MONs for quorum, sometimes it's  
difficult to get these things out. ;-)



Zitat von Janne Johansson :


Den ons 27 okt. 2021 kl 11:10 skrev Eugen Block :


Also note that you need an odd number of MONs to be able to form a
quorum. So I would recommend to remove one MON to have 5.


Well, you need to have a distinct majority, so using 6 is not better
than 5, but not worse either.
Both will let the cluster run as long as no more than 2 mons are  
currently out.



--
May the most significant bit of your life be positive.




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Doing SAML2 Auth With Containerized mgrs

[Edward R Huyer]

Thank you for the reply.  Even if there’s a good reason for the CLI tool to not 
send the contents of the files, I feel like the docs should at least have “this 
is how we recommend you handle this if you’re using a containerized (e.g. 
cephadm) deployment”.

Speaking of which, do you have any specific suggestions as to how to approach 
this?  I’m not familiar enough the details of the cephadm-deploy containers 
specifically or containers in general to know where to start.

From: Ernesto Puerta 
Sent: Wednesday, October 27, 2021 6:53 AM
To: Edward R Huyer 
Cc: Yury Kirsanov ; ceph-users@ceph.io
Subject: Re: [ceph-users] Re: Doing SAML2 Auth With Containerized mgrs

Hi Edward,

You don't need to worry about keeping those certs persistently, the Ceph 
Dashboard does that for you (they're persisted inside the ceph-mon KV store). 
You just need to ensure that the paths you provide are reachable to the 
ceph-mgr daemon. And I agree: that's a bit tricky with containerized 
deployments (it was originally designed for bare-metal ones in mind). We could 
improve that by making the Ceph CLI tool read CephFile-type args and send the 
contents of these files, instead of sending the paths to the daemon (I don't 
know what was the original intent of this behaviour, maybe avoid sending 
sensitive data unencrypted or avoiding the multiple loggers that might dump 
what's sent from the Ceph CLI).

Kind Regards,
Ernesto


On Mon, Oct 25, 2021 at 6:21 PM Edward R Huyer 
mailto:erh...@rit.edu>> wrote:
No worries.  It's a pretty specific problem, and the documentation could be 
better.

-Original Message-
From: Yury Kirsanov mailto:y.kirsa...@gmail.com>>
Sent: Monday, October 25, 2021 12:17 PM
To: Edward R Huyer mailto:erh...@rit.edu>>
Cc: ceph-users@ceph.io
Subject: [ceph-users] Re: Doing SAML2 Auth With Containerized mgrs

Hi Edward,
Yes, you probably are right, I thought about dashboard SSL certificate, not the 
SAML2, sorry for that.

Regards,
Yury.

On Tue, Oct 26, 2021 at 3:10 AM Edward R Huyer 
mailto:erh...@rit.edu>> wrote:

> I don’t think that’s correct?  I already have a certificate set up for
> HTTPS, and it doesn’t show up in the SAML2 configuration.  Maybe I’m
> mistaken, but I think the SAML2 cert is separate from the regular
> HTTPS cert?
>
>
>
> *From:* Yury Kirsanov mailto:y.kirsa...@gmail.com>>
> *Sent:* Monday, October 25, 2021 11:52 AM
> *To:* Edward R Huyer mailto:erh...@rit.edu>>
> *Cc:* ceph-users@ceph.io
> *Subject:* Re: [ceph-users] Doing SAML2 Auth With Containerized mgrs
>
>
>
> *CAUTION: This message came from outside RIT. If you are unsure about
> the source or content of this message, please contact the RIT Service
> Center at
> 585-475-5000 or help.rit.edu  
> before clicking
> links, opening attachments or responding.*
>
> Hi Edward,
>
> You need to set configuration like this, assuming that certificate and
> key are on your local disk:
>
> ceph mgr module disable dashboard
> ceph dashboard set-ssl-certificate -i .crt ceph
> dashboard set-ssl-certificate-key -i .key ceph
> config-key set mgr/cephadm/grafana_crt -i .crt ceph
> config-key set mgr/cephadm/grafana_key -i .key
> ceph orch reconfig grafana ceph mgr module enable dashboard
>
> Hope this helps!
>
> Regards,
> Yury.
>
>
>
> On Tue, Oct 26, 2021 at 2:45 AM Edward R Huyer 
> mailto:erh...@rit.edu>> wrote:
>
> Continuing my containerized Ceph adventures
>
> I'm trying to set up SAML2 auth for the dashboard (specifically
> pointing at the institute Shibboleth service).  The service requires
> the use of the
> x509 certificates.  Following the instructions in the documentation (
> https://docs.ceph.com/en/latest/mgr/dashboard/#dashboard-sso-support )
> leads to an error about the certificate file not existing.
>
> Some digging suggests that's because the daemon is looking in the
> container's filesystem rather than the physical host's filesystem.
> That makes some sense, but it annoying.
>
> So my question is:  How do I get the cert and key file into the
> container's filesystem in a persistent way?  cephadm enter --name
> "mgr.hostname" results in a "no such container" error.  cephadm shell
> --name "mgr.hostname" works, but changes don't persist.
>
> Any suggestions about this problem specifically, authing the dashboard
> against Shibboleth, or SAML2 in general?
>
> -
> Edward Huyer
> Golisano College of Computing and Information Sciences Rochester
> Institute of Technology Golisano 70-2373
> 152 Lomb Memorial Drive
> Rochester, NY 14623
> 585-475-6651
> erh...@rit.edu>
>
> Obligatory Legalese:
> The information transmitted, including attachments, is intended only
> for the person(s) or entity to which it is addressed and may contain
> confidential and/or privileged material. Any review, retransmission,
> dissemination or other use of, or taking of any action in r

[ceph-users] DocuBetter meetings cancelled in perpetuity

[John Zachary Dover]

DocuBetter meetings are now cancelled in perpetuity.

These meetings are cancelled because they are sparsely attended, and the
few people who do attend them are in more frequent contact with Zac through
channels that are not the DocuBetter meeting.

Zac is available to field documentation-related requests at his email
address (zac.do...@gmail.com). Write to him anytime with any complaint or
request.

Currently Zac is working on rewriting the RADOS documentation and writing
the Ceph-for-Hobbyists/Ceph-at-Home/MicroCeph documentation project (a
project proposed by Patrick Donnelly with the aim of explaining how to
build a Ceph cluster at home for testing or experimentation).

Zac Dover
Documentation
Ceph Foundation
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: v15.2.15 Octopus released

[David Galloway]

On 10/25/21 4:45 AM, Stefan Kooman wrote:
> On 10/20/21 21:57, David Galloway wrote:
>> We're happy to announce the 15th backport release in the Octopus series.
>> We recommend users to update to this release. 
> 
> ...
> 
>> Getting Ceph
>> 
>> * Git at git://github.com/ceph/ceph.git
>> * Tarball at https://download.ceph.com/tarballs/ceph-15.2.15.tar.gz
>> * Containers at https://quay.io/repository/ceph/ceph
> 
> No containers tagged with version 15.2.15 as of yet. How long does it
> normally take to build containers after a new version gets released?
> 

Sorry about that.  The x86_64 container was built and available as
ceph-amd64/v15.2.15-20211020.  The arm64 container build was failing,
however, so the ceph/v15.2.15 manifest list tag was missing.

Fixed now though.

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 1 MDS report slow metadata IOs

[Abdelillah Asraoui]

OSD flapping was due to ports blocked by firewall.
while mounting the file system, the directory structure shows csi volumes
subfolder folders
as in /tmp/cephFS/csi/csi-vol-/container-name-log
is there a way to not show the csi volumes in the path to the container log
as an example:
/tmp/cephFS/container-name-log
this would make it easier to update the containers configuration without
dealing the volumes sub folders in the path ..

Thanks!

On Wed, Oct 6, 2021 at 2:58 AM Eugen Block  wrote:

> > what is causing the slow MDS metadata IOs ?
>
> Your flapping OSDs.
>
> > currently, there are 2 mds and 3 monitors deployed ..
> > would it help to just one mds and one monitor ?
>
> No, you need to figure out why your OSDs crash. More details about
> your setup (ceph version, deployment method, hardware resources) and
> the logs from a crashing OSD could help identify the issue.
>
>
> Zitat von Abdelillah Asraoui :
>
> > The osds are continuously flapping up/down due to the slow MDS metadata
> IOs
> > ..
> > what is causing the slow MDS metadata IOs ?
> > currently, there are 2 mds and 3 monitors deployed ..
> > would it help to just one mds and one monitor ?
> >
> > thanks!
> >
> > On Tue, Oct 5, 2021 at 1:42 PM Eugen Block  wrote:
> >
> >> All your PGs are inactive, if two of four OSDs are down and you
> >> probably have a pool size of 3 then no IO can be served. You’d need at
> >> least three up ODSs to resolve that.
> >>
> >>
> >> Zitat von Abdelillah Asraoui :
> >>
> >> > Ceph is reporting warning on slow metdataIOs on one of the MDS server,
> >> > this is
> >> >
> >> > a new cluster with no upgrade..
> >> >
> >> > Anyone has encountered this and is there a workaround ..
> >> >
> >> > ceph -s
> >> >
> >> >   cluster:
> >> >
> >> > id: 801691e6xx-x-xx-xx-xx
> >> >
> >> > health: HEALTH_WARN
> >> >
> >> > 1 MDSs report slow metadata IOs
> >> >
> >> > noscrub,nodeep-scrub flag(s) set
> >> >
> >> > 2 osds down
> >> >
> >> > 2 hosts (2 osds) down
> >> >
> >> > Reduced data availability: 97 pgs inactive, 66 pgs
> peering,
> >> 53
> >> > pgs stale
> >> >
> >> > Degraded data redundancy: 31 pgs undersized
> >> >
> >> > 2 slow ops, oldest one blocked for 30 sec, osd.0 has slow
> ops
> >> >
> >> >
> >> >
> >> >   services:
> >> >
> >> > mon: 3 daemons, quorum a,c,f (age 15h)
> >> >
> >> > mgr: a(active, since 17h)
> >> >
> >> > mds: myfs:1 {0=myfs-a=up:creating} 1 up:standby
> >> >
> >> > osd: 4 osds: 2 up (since 36s), 4 in (since 10h)
> >> >
> >> >  flags noscrub,nodeep-scrub
> >> >
> >> >
> >> >
> >> >   data:
> >> >
> >> > pools:   4 pools, 97 pgs
> >> >
> >> > objects: 0 objects, 0 B
> >> >
> >> > usage:   1.0 GiB used, 1.8 TiB / 1.8 TiB avail
> >> >
> >> > pgs: 100.000% pgs not active
> >> >
> >> >  44 creating+peering
> >> >
> >> >  31 stale+undersized+peered
> >> >
> >> >  22 stale+creating+peering
> >> >
> >> >
> >> >
> >> >   progress:
> >> >
> >> > Rebalancing after osd.2 marked in (10h)
> >> >
> >> >   []
> >> >
> >> > Rebalancing after osd.3 marked in (10h)
> >> >
> >> >   []
> >> >
> >> >
> >> > Thanks!
> >> > ___
> >> > ceph-users mailing list -- ceph-users@ceph.io
> >> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >>
> >>
> >>
> >>
>
>
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Fwd: radosgw bucket stats "ver" and "master_ver"

[Trey Palmer]

Hi,

We have a couple of buckets used for media transcoding scratch space that
have huge lists under "ver" and "master_ver" when doing a "radosgw-admin
bucket stats".

Can anyone tell me what these version lists are for?  I saw that Casey said
they're not related to bucket versioning, but I'm just wondering what they
*are* used for, and if they're causing any inefficiencies and if so if
there's any way to get rid of tracking them?

These happen to be on Luminous (I know) but I checked some Nautilus
clusters and they have similar version entries, though not as big I'm
assuming because the buckets are less transient.

Thanks,

Trey Palmer
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 16.2.6 OSD down, out but container running....

[Marco Pizzolo]

Is there any command or log I can provide a sample from that would help to
pinpoint the issue?  The 119 of 120 OSDs are working correctly by all
accounts, but I am just unable to have the bring the last one fully online.

Thank you,

On Tue, Oct 26, 2021 at 3:59 PM Marco Pizzolo 
wrote:

> Thanks for the responses so far.
>
> Putting the OSD in is not a problem, but it will not report in the mgr as
> being online.  Podman ps does show the container for osd.13 as online
> though.  I've tried killing it and it relaunches, but the OSD does not
> report as up in mgr.  ceph orch restart osd.all-available-devices likewise
> restarts all OSDs (120 so far) but this one never comes up.  Same with
> rebooting.
>
> Here is a snippet of the log requested.
>
>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb:Options.write_buffer_si>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb:  Options.max_write_buffer_numb>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb:  Options.compression: >
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb:  Options.botto>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb:   Options.prefix_extractor>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb:   Options.memtable_insert_with>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb: Options.num_levels>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb:Options.min_write_buffe>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb: Options.max_write_buffer_n>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb: Options.max_write_buffer_s>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb:Options.bottommost_>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb:  Options.botto>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb:   Options.bottommo>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb: Options.bottommost_com>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb: Options.bottommost_com>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb:  Options.botto>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb:Options.compression>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb:  Options.compr>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb:   Options.compress>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb: Options.compression_op>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb: Options.compression_op>
> Oct 26 19:31:59  conmon[780974]: rain_bytes: 0
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb:  Options.compr>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb:  Options.level0_file_num_c>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb:  Options.level0_slowdo>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb:  Options.level0_st>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb:   Options.targ>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb: Options.target_fil>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb:Options.max_byt>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb: Options.level_compaction_dynam>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb:  Options.max_bytes_for>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb: Options.max_bytes_for_level_mu>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827+
> 7f08e65a4080  4 rocksdb: Options.max_bytes_for_level_mu>
> Oct 26 19:31:59  conmon[780974]: debug 2021-10-26T19:31:59.827

[ceph-users] Re: Fwd: radosgw bucket stats "ver" and "master_ver"

[Casey Bodley]

(oops, forgot to reply-all)

On Wed, Oct 27, 2021 at 12:58 PM Trey Palmer  wrote:
>
> Hi,
>
> We have a couple of buckets used for media transcoding scratch space that
> have huge lists under "ver" and "master_ver" when doing a "radosgw-admin
> bucket stats".
>
> Can anyone tell me what these version lists are for?  I saw that Casey said
> they're not related to bucket versioning, but I'm just wondering what they
> *are* used for, and if they're causing any inefficiencies and if so if
> there's any way to get rid of tracking them?

it looks like these are coming from
librados::IoCtx::get_last_version(), which is a per-object version
number tracked by rados. cls_rgw seems to be storing the latest
version number for each bucket index shard object, but i can't tell
what exactly it's using them for - i just assume it's somehow related
to ordering of operations

> These happen to be on Luminous (I know) but I checked some Nautilus
> clusters and they have similar version entries, though not as big I'm
> assuming because the buckets are less transient.
>
> Thanks,
>
> Trey Palmer
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] 回复: 16.2.6 OSD down, out but container running....

[胡 玮文]

Hi Marco, the log lines are truncated. I recommend you to send the logs to a 
file rather than copying from terminal:

cephadm logs --name osd.13 > osd.13.log

I see “read stalled” in the log. Just a guess, can you check the kernel logs 
and the SMART info to see if there is something wrong with this disk? Maybe 
also do a self-test.

从 Windows 版邮件发送

发件人: Marco Pizzolo
发送时间: 2021年10月28日 1:17
收件人: 胡 玮文
抄送: ceph-users
主题: Re: [ceph-users] 16.2.6 OSD down, out but container running

Is there any command or log I can provide a sample from that would help to 
pinpoint the issue?  The 119 of 120 OSDs are working correctly by all accounts, 
but I am just unable to have the bring the last one fully online.

Thank you,
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 16.2.6 OSD down, out but container running....

[Marco Pizzolo]

Thanks Hu Weiwen,

These hosts and drives are perhaps 2 months old or so, and this is the
first cluster we build on them so I was not anticipating a drive issue
already.

The smartmontools show:

root@:~# smartctl -H /dev/sdag
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.11.0-38-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Health Status: FIRMWARE IMPENDING FAILURE TOO MANY BLOCK REASSIGNS
[asc=5d, ascq=64]

Grown defects during certification 
Total blocks reassigned during format 
Total new blocks reassigned 
Power on minutes since format 
root@:~# smartctl -H /dev/sdah
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.11.0-38-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org



On Wed, Oct 27, 2021 at 1:26 PM 胡 玮文  wrote:

> Hi Marco, the log lines are truncated. I recommend you to send the logs to
> a file rather than copying from terminal:
>
>
>
> cephadm logs --name osd.13 > osd.13.log
>
>
>
> I see “read stalled” in the log. Just a guess, can you check the kernel
> logs and the SMART info to see if there is something wrong with this disk?
> Maybe also do a self-test.
>
>
>
> 从 Windows 版邮件 发送
>
>
>
> *发件人: *Marco Pizzolo 
> *发送时间: *2021年10月28日 1:17
> *收件人: *胡 玮文 
> *抄送: *ceph-users 
> *主题: *Re: [ceph-users] 16.2.6 OSD down, out but container running
>
>
>
> Is there any command or log I can provide a sample from that would help to
> pinpoint the issue?  The 119 of 120 OSDs are working correctly by all
> accounts, but I am just unable to have the bring the last one fully online.
>
>
>
> Thank you,
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: jj's "improved" ceph balancer

[Neha Ojha]

Jonas, would you be interested in joining one of our performance
meetings and presenting some of your work there? Seems like we can
have a good discussion about further improvements to the balancer.

Thanks,
Neha

On Mon, Oct 25, 2021 at 11:39 AM Josh Salomon  wrote:
>
> Hi Jonas,
>
> I have some comments -
> IMHO you should replace 3 & 4 - if the PGs are not split optimally between 
> the OSDs per pool, the primary balancing will not help, so I believe 4 is 
> more important than 3.
> There is also a practical reason for this, after we have 1,2,3 constraints 
> fulfilled, we can implement 4 by just changing the order of OSDs inside the 
> PGs (at least for replica) which is a cheap operation since it is only upmap 
> operation and does not require any data movement.
> Regards,
>
> Josh
>
>
> On Mon, Oct 25, 2021 at 9:01 PM Jonas Jelten  wrote:
>>
>> Hi Josh,
>>
>> yes, there's many factors to optimize... which makes it kinda hard to 
>> achieve an optimal solution.
>>
>> I think we have to consider all these things, in ascending priority:
>>
>> * 1: Minimize distance to CRUSH (prefer fewest upmaps, and remove upmap 
>> items if balance is better)
>> * 2: Relocation of PGs in remapped state (since they are not fully moved 
>> yet, hence 'easier' to relocate)
>> * 3: Per-Pool PG distribution, respecting OSD device size -> ideal_pg_count 
>> = osd_size * (pg_num / sum(possible_osd_sizes)
>> * 4: Primary/EC-N distribution (all osds have equal primary/EC-N counts, for 
>> workload balancing, not respecting device size (for hdd at least?), else 
>> this is just 3)
>> * 5: Capacity balancing (all osds equally full)
>> * 6: And of course CRUSH constraints
>>
>> Beautiful optimization problem, which could be fed into a solver :)
>> My approach currently optimizes for 3, 5, 6, iteratively...
>>
>> > My only comment about what you did is that it should somehow work pool by 
>> > pool and manage the +-1 globally.
>>
>> I think this is already implemented!
>> Since each iteration I pick the "fullest" device first, it has to have more 
>> pools (or data) than other OSDs (e.g. through +1), and we try to migrate a 
>> PG off it.
>> And we only migrate a particular PG of a pool from such a source OSD if it 
>> has >ideal_amount_for_pool (float, hence we allow moving +1s or worse).
>> Same for a destination OSD, it's only selected if has other PGs of that pool 
>> > more).
>> So we eliminate global imbalance, and respect equal PG distribution per pool.
>>
>> I can try to hack in (optional) constraints so it also supports optimization 
>> 4, but this works very much against the CRUSH placement (because we'd have 
>> to ignore OSD size).
>> But since this is basically bypassing CRUSH-weights, it could also be done 
>> by placing all desired devices in a custom crush hierarchy with identical 
>> weighted buckets (even though "wasting" storage).
>> Then we don't have to fight CRUSH and it's a 'simple' optimization 3 again.
>>
>> To achieve 2 and 1 it's just an re-ordering of candidate PGs.
>> So in theory it should be doable™.
>>
>> -- Jonas
>>
>>
>> On 25/10/2021 11.12, Josh Salomon wrote:
>> > Hi Jonas,
>> >
>> > I want to clarify a bit my thoughts (it may be long) regarding balancing 
>> > in general.
>> >
>> > 1 - Balancing the capacity correctly is of top priority, this is because 
>> > we all know that the system is as full as the fullest device and as a 
>> > storage system we can't allow large capacity which is wasted and can't be 
>> > used. This is a top functional requirement.
>> > 2 - Workload balancing is a performance requirement, and an important one, 
>> > but we should not optimize workload on behalf of capacity so the challenge 
>> > is how to do both simultaneously. (hint: it is not always possible, but 
>> > when this is not possible the system performs less than the aggregated 
>> > performance of the devices)
>> >
>> > Assumption 1: Per pool the workload on a PG is linear with the capacity, 
>> > which means either all PGs have the same workload (#PGs is a power of 2) 
>> > or some PGs has exactly twice the load as the others. From now on I will 
>> > assume the number of PGs is a power of 2, since the adjustments to the 
>> > other case are pretty simple.
>> >
>> > Conclusion 1: Balancing capacity based on all the PGs is the system may 
>> > cause workload imbalance - balancing capacity should be done on a pool by 
>> > pool basis. (assume 2 pools H(ot) and C(old) with exactly the same 
>> > settings (#PGs, capacity and protection scheme). If you balance per PG 
>> > capacity only you can have a device with all the PGs from C pool and a 
>> > device with all the PGs from the H pool -
>> > This will cause the second device to be fully loaded while the first 
>> > device is idle).
>> >
>> > On the other hand, your point about the +-1 PGs when working on a pool by 
>> > pool basis is correct and should be fixed (when working on pool by pool 
>> > basis)
>> >
>> > When all the devices are identical, the othe