date:20240131

[ceph-users] Re: NFS HA - "virtual_ip": null after upgrade to reef

2024-01-31 Thread Eugen Block


Hi,

if I understand this correctly, with the "keepalive-only" option only  
one ganesha instance is supposed to be deployed:


If a user additionally supplies --ingress-mode keepalive-only a  
partial ingress service will be deployed that still provides a  
virtual IP, but has nfs directly binding to that virtual IP and  
leaves out any sort of load balancing or traffic redirection. This  
setup will restrict users to deploying only 1 nfs daemon as multiple  
cannot bind to the same port on the virtual IP.


Maybe that's why it disappeared as you have 3 hosts in the placement  
parameter? Is the ingress service still present in 'ceph orch ls'?


Regards,
Eugen

Zitat von Torkil Svensgaard :


Hi

Last week we created an NFS service like this:

"
ceph nfs cluster create jumbo "ceph-flash1,ceph-flash2,ceph-flash3"  
--ingress --virtual_ip 172.21.15.74/22 --ingress-mode keepalive-only

"

Worked like a charm. Yesterday we upgraded from 17.2.7 to 18.20.0  
and the NFS virtual IP seems to have gone missing in the process:


"
# ceph nfs cluster info jumbo
{
  "jumbo": {
"backend": [
  {
"hostname": "ceph-flash1",
"ip": "172.21.15.148",
"port": 2049
  }
],
"virtual_ip": null
  }
}
"

Service spec:

"
service_type: nfs
service_id: jumbo
service_name: nfs.jumbo
placement:
  count: 1
  hosts:
  - ceph-flash1
  - ceph-flash2
  - ceph-flash3
spec:
  port: 2049
  virtual_ip: 172.21.15.74
"

I've tried restarting the nfs.jumbo service which didn't help. Suggestions?

Mvh.

Torkil

--
Torkil Svensgaard
Sysadmin
MR-Forskningssektionen, afs. 714
DRCMR, Danish Research Centre for Magnetic Resonance
Hvidovre Hospital
Kettegård Allé 30
DK-2650 Hvidovre
Denmark
Tel: +45 386 22828
E-mail: tor...@drcmr.dk
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] how can install latest dev release?

2024-01-31 Thread garcetto

good morning,
 how can i install latest dev release using cephadm?
thank you.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: NFS HA - "virtual_ip": null after upgrade to reef

2024-01-31 Thread Torkil Svensgaard




On 31/01/2024 08:38, Torkil Svensgaard wrote:

Hi

Last week we created an NFS service like this:

"
ceph nfs cluster create jumbo "ceph-flash1,ceph-flash2,ceph-flash3" 
--ingress --virtual_ip 172.21.15.74/22 --ingress-mode keepalive-only

"

Worked like a charm. Yesterday we upgraded from 17.2.7 to 18.20.0 and 
the NFS virtual IP seems to have gone missing in the process:


"
# ceph nfs cluster info jumbo
{
   "jumbo": {
     "backend": [
   {
     "hostname": "ceph-flash1",
     "ip": "172.21.15.148",
     "port": 2049
   }
     ],
     "virtual_ip": null
   }
}
"

Service spec:

"
service_type: nfs
service_id: jumbo
service_name: nfs.jumbo
placement:
   count: 1
   hosts:
   - ceph-flash1
   - ceph-flash2
   - ceph-flash3
spec:
   port: 2049
   virtual_ip: 172.21.15.74
"

I've tried restarting the nfs.jumbo service which didn't help. Suggestions?


Should I have an ingress service with "--ingress-mode keepalive-only"?

We created the cluster with the nfs module[1] but looking at the cephadm 
documentation[2] it looks like that should also have created an ingress 
service, which is currently missing.


[1] https://docs.ceph.com/en/latest/mgr/nfs/#create-nfs-ganesha-cluster
[2] 
https://docs.ceph.com/en/latest/cephadm/services/nfs/#nfs-with-virtual-ip-but-no-haproxy


Mvh.

Torkil




Mvh.

Torkil



--
Torkil Svensgaard
Sysadmin
MR-Forskningssektionen, afs. 714
DRCMR, Danish Research Centre for Magnetic Resonance
Hvidovre Hospital
Kettegård Allé 30
DK-2650 Hvidovre
Denmark
Tel: +45 386 22828
E-mail: tor...@drcmr.dk
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Help on rgw metrics (was rgw_user_counters_cache)

2024-01-31 Thread garcetto

good morning,
  i was struggling trying to understand why i cannot find this setting on
my reef version, is it because is only on latest dev ceph version and not
before?

https://docs.ceph.com/en/*latest*
/radosgw/metrics/#user-bucket-counter-caches

Reef gives 404
https://docs.ceph.com/en/reef/radosgw/metrics/

thank you!
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: NFS HA - "virtual_ip": null after upgrade to reef

2024-01-31 Thread Torkil Svensgaard




On 31/01/2024 09:36, Eugen Block wrote:

Hi,

if I understand this correctly, with the "keepalive-only" option only 
one ganesha instance is supposed to be deployed:


If a user additionally supplies --ingress-mode keepalive-only a 
partial ingress service will be deployed that still provides a virtual 
IP, but has nfs directly binding to that virtual IP and leaves out any 
sort of load balancing or traffic redirection. This setup will 
restrict users to deploying only 1 nfs daemon as multiple cannot bind 
to the same port on the virtual IP.


Maybe that's why it disappeared as you have 3 hosts in the placement 
parameter? Is the ingress service still present in 'ceph orch ls'?


As I read the documentation[1] the "count: 1" handles that so what I 
have is a placement pool from which only one is selected for deployment?


The absence of the ingress service is puzzling me, as it worked just 
fine prior to the upgrade and the upgrade shouldn't have touched the 
service spec in any way?


Mvh.

Torkil

[1] 
https://docs.ceph.com/en/latest/cephadm/services/nfs/#nfs-with-virtual-ip-but-no-haproxy




Regards,
Eugen

Zitat von Torkil Svensgaard :


Hi

Last week we created an NFS service like this:

"
ceph nfs cluster create jumbo "ceph-flash1,ceph-flash2,ceph-flash3" 
--ingress --virtual_ip 172.21.15.74/22 --ingress-mode keepalive-only

"

Worked like a charm. Yesterday we upgraded from 17.2.7 to 18.20.0 and 
the NFS virtual IP seems to have gone missing in the process:


"
# ceph nfs cluster info jumbo
{
  "jumbo": {
    "backend": [
  {
    "hostname": "ceph-flash1",
    "ip": "172.21.15.148",
    "port": 2049
  }
    ],
    "virtual_ip": null
  }
}
"

Service spec:

"
service_type: nfs
service_id: jumbo
service_name: nfs.jumbo
placement:
  count: 1
  hosts:
  - ceph-flash1
  - ceph-flash2
  - ceph-flash3
spec:
  port: 2049
  virtual_ip: 172.21.15.74
"

I've tried restarting the nfs.jumbo service which didn't help. 
Suggestions?


Mvh.

Torkil

--
Torkil Svensgaard
Sysadmin
MR-Forskningssektionen, afs. 714
DRCMR, Danish Research Centre for Magnetic Resonance
Hvidovre Hospital
Kettegård Allé 30
DK-2650 Hvidovre
Denmark
Tel: +45 386 22828
E-mail: tor...@drcmr.dk
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


--
Torkil Svensgaard
Sysadmin
MR-Forskningssektionen, afs. 714
DRCMR, Danish Research Centre for Magnetic Resonance
Hvidovre Hospital
Kettegård Allé 30
DK-2650 Hvidovre
Denmark
Tel: +45 386 22828
E-mail: tor...@drcmr.dk
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: NFS HA - "virtual_ip": null after upgrade to reef

2024-01-31 Thread Eugen Block

I believe so, I just deployed it with the same command as you did, but  
on a single-node cluster (hence only one host):


ceph:~ # ceph orch ls
NAME  PORTS RUNNING  REFRESHED  AGE  PLACEMENT
crash   1/1  3m ago 44m  *
ingress.nfs.test  192.168.124.187:9049  1/1  3m ago 36m  ceph;count:1
mds.cephfs  2/2  3m ago 31m  count:2
mgr 2/2  3m ago 44m  count:2
mon 1/5  3m ago 44m  count:5
nfs.test  ?:20491/1  3m ago 36m  ceph;count:1
osd.osd-hdd-ssd   3  3m ago 41m  ceph

The ingress.nfs.test has the virtual IP a provided.


Zitat von Torkil Svensgaard :


On 31/01/2024 08:38, Torkil Svensgaard wrote:

Hi

Last week we created an NFS service like this:

"
ceph nfs cluster create jumbo "ceph-flash1,ceph-flash2,ceph-flash3"  
--ingress --virtual_ip 172.21.15.74/22 --ingress-mode keepalive-only

"

Worked like a charm. Yesterday we upgraded from 17.2.7 to 18.20.0  
and the NFS virtual IP seems to have gone missing in the process:


"
# ceph nfs cluster info jumbo
{
  "jumbo": {
    "backend": [
  {
    "hostname": "ceph-flash1",
    "ip": "172.21.15.148",
    "port": 2049
  }
    ],
    "virtual_ip": null
  }
}
"

Service spec:

"
service_type: nfs
service_id: jumbo
service_name: nfs.jumbo
placement:
  count: 1
  hosts:
  - ceph-flash1
  - ceph-flash2
  - ceph-flash3
spec:
  port: 2049
  virtual_ip: 172.21.15.74
"

I've tried restarting the nfs.jumbo service which didn't help. Suggestions?


Should I have an ingress service with "--ingress-mode keepalive-only"?

We created the cluster with the nfs module[1] but looking at the  
cephadm documentation[2] it looks like that should also have created  
an ingress service, which is currently missing.


[1] https://docs.ceph.com/en/latest/mgr/nfs/#create-nfs-ganesha-cluster
[2]  
https://docs.ceph.com/en/latest/cephadm/services/nfs/#nfs-with-virtual-ip-but-no-haproxy


Mvh.

Torkil




Mvh.

Torkil



--
Torkil Svensgaard
Sysadmin
MR-Forskningssektionen, afs. 714
DRCMR, Danish Research Centre for Magnetic Resonance
Hvidovre Hospital
Kettegård Allé 30
DK-2650 Hvidovre
Denmark
Tel: +45 386 22828
E-mail: tor...@drcmr.dk
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: NFS HA - "virtual_ip": null after upgrade to reef

2024-01-31 Thread Eugen Block

I deployed the nfs with ceph version 17.2.7 and then upgraded to  
18.2.1 successfully, the ingress service is still present. Can you  
tell if it was there while you were on quincy? To fix it I would just  
apply the nfs.yaml again and see if the ingress service is deployed.  
To know what happened during (or after) the upgrade you'd probably  
have to look through the mgr logs...



Zitat von Torkil Svensgaard :


On 31/01/2024 09:36, Eugen Block wrote:

Hi,

if I understand this correctly, with the "keepalive-only" option  
only one ganesha instance is supposed to be deployed:


If a user additionally supplies --ingress-mode keepalive-only a  
partial ingress service will be deployed that still provides a  
virtual IP, but has nfs directly binding to that virtual IP and  
leaves out any sort of load balancing or traffic redirection. This  
setup will restrict users to deploying only 1 nfs daemon as  
multiple cannot bind to the same port on the virtual IP.


Maybe that's why it disappeared as you have 3 hosts in the  
placement parameter? Is the ingress service still present in 'ceph  
orch ls'?


As I read the documentation[1] the "count: 1" handles that so what I  
have is a placement pool from which only one is selected for  
deployment?


The absence of the ingress service is puzzling me, as it worked just  
fine prior to the upgrade and the upgrade shouldn't have touched the  
service spec in any way?


Mvh.

Torkil

[1]  
https://docs.ceph.com/en/latest/cephadm/services/nfs/#nfs-with-virtual-ip-but-no-haproxy




Regards,
Eugen

Zitat von Torkil Svensgaard :


Hi

Last week we created an NFS service like this:

"
ceph nfs cluster create jumbo  
"ceph-flash1,ceph-flash2,ceph-flash3" --ingress --virtual_ip  
172.21.15.74/22 --ingress-mode keepalive-only

"

Worked like a charm. Yesterday we upgraded from 17.2.7 to 18.20.0  
and the NFS virtual IP seems to have gone missing in the process:


"
# ceph nfs cluster info jumbo
{
  "jumbo": {
    "backend": [
  {
    "hostname": "ceph-flash1",
    "ip": "172.21.15.148",
    "port": 2049
  }
    ],
    "virtual_ip": null
  }
}
"

Service spec:

"
service_type: nfs
service_id: jumbo
service_name: nfs.jumbo
placement:
  count: 1
  hosts:
  - ceph-flash1
  - ceph-flash2
  - ceph-flash3
spec:
  port: 2049
  virtual_ip: 172.21.15.74
"

I've tried restarting the nfs.jumbo service which didn't help. Suggestions?

Mvh.

Torkil

--
Torkil Svensgaard
Sysadmin
MR-Forskningssektionen, afs. 714
DRCMR, Danish Research Centre for Magnetic Resonance
Hvidovre Hospital
Kettegård Allé 30
DK-2650 Hvidovre
Denmark
Tel: +45 386 22828
E-mail: tor...@drcmr.dk
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


--
Torkil Svensgaard
Sysadmin
MR-Forskningssektionen, afs. 714
DRCMR, Danish Research Centre for Magnetic Resonance
Hvidovre Hospital
Kettegård Allé 30
DK-2650 Hvidovre
Denmark
Tel: +45 386 22828
E-mail: tor...@drcmr.dk
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: NFS HA - "virtual_ip": null after upgrade to reef

2024-01-31 Thread Torkil Svensgaard




On 31/01/2024 09:52, Eugen Block wrote:
I deployed the nfs with ceph version 17.2.7 and then upgraded to 18.2.1 
successfully, the ingress service is still present. Can you tell if it 
was there while you were on quincy? To fix it I would just apply the 
nfs.yaml again and see if the ingress service is deployed. To know what 
happened during (or after) the upgrade you'd probably have to look 
through the mgr logs...


We just discussed that internally. We don't know for sure if the ingress 
service was in the service spec, we just know that we could use the 
virtual IP for mounting.


We did quite a bit of mucking around getting it to work so I guess it is 
possible the backend host just happened to have a leftover ingress 
container running and it got lost in the upgrade, or something like that.


I got it running for now by crafting a ingress.yml manually. I'll go dig 
in the logs. Thanks!


Mvh.

Torkil



Zitat von Torkil Svensgaard :


On 31/01/2024 09:36, Eugen Block wrote:

Hi,

if I understand this correctly, with the "keepalive-only" option only 
one ganesha instance is supposed to be deployed:


If a user additionally supplies --ingress-mode keepalive-only a 
partial ingress service will be deployed that still provides a 
virtual IP, but has nfs directly binding to that virtual IP and 
leaves out any sort of load balancing or traffic redirection. This 
setup will restrict users to deploying only 1 nfs daemon as multiple 
cannot bind to the same port on the virtual IP.


Maybe that's why it disappeared as you have 3 hosts in the placement 
parameter? Is the ingress service still present in 'ceph orch ls'?


As I read the documentation[1] the "count: 1" handles that so what I 
have is a placement pool from which only one is selected for deployment?


The absence of the ingress service is puzzling me, as it worked just 
fine prior to the upgrade and the upgrade shouldn't have touched the 
service spec in any way?


Mvh.

Torkil

[1] 
https://docs.ceph.com/en/latest/cephadm/services/nfs/#nfs-with-virtual-ip-but-no-haproxy




Regards,
Eugen

Zitat von Torkil Svensgaard :


Hi

Last week we created an NFS service like this:

"
ceph nfs cluster create jumbo "ceph-flash1,ceph-flash2,ceph-flash3" 
--ingress --virtual_ip 172.21.15.74/22 --ingress-mode keepalive-only

"

Worked like a charm. Yesterday we upgraded from 17.2.7 to 18.20.0 
and the NFS virtual IP seems to have gone missing in the process:


"
# ceph nfs cluster info jumbo
{
  "jumbo": {
    "backend": [
  {
    "hostname": "ceph-flash1",
    "ip": "172.21.15.148",
    "port": 2049
  }
    ],
    "virtual_ip": null
  }
}
"

Service spec:

"
service_type: nfs
service_id: jumbo
service_name: nfs.jumbo
placement:
  count: 1
  hosts:
  - ceph-flash1
  - ceph-flash2
  - ceph-flash3
spec:
  port: 2049
  virtual_ip: 172.21.15.74
"

I've tried restarting the nfs.jumbo service which didn't help. 
Suggestions?


Mvh.

Torkil

--
Torkil Svensgaard
Sysadmin
MR-Forskningssektionen, afs. 714
DRCMR, Danish Research Centre for Magnetic Resonance
Hvidovre Hospital
Kettegård Allé 30
DK-2650 Hvidovre
Denmark
Tel: +45 386 22828
E-mail: tor...@drcmr.dk
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


--
Torkil Svensgaard
Sysadmin
MR-Forskningssektionen, afs. 714
DRCMR, Danish Research Centre for Magnetic Resonance
Hvidovre Hospital
Kettegård Allé 30
DK-2650 Hvidovre
Denmark
Tel: +45 386 22828
E-mail: tor...@drcmr.dk
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


--
Torkil Svensgaard
Sysadmin
MR-Forskningssektionen, afs. 714
DRCMR, Danish Research Centre for Magnetic Resonance
Hvidovre Hospital
Kettegård Allé 30
DK-2650 Hvidovre
Denmark
Tel: +45 386 22828
E-mail: tor...@drcmr.dk
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Cannot recreate monitor in upgrade from pacific to quincy (leveldb -> rocksdb)

2024-01-31 Thread Mark Schouten


Hi,

During an upgrade from pacific to quincy, we needed to recreate the mons 
because the mons were pretty old and still using leveldb.


So step one was to destroy one of the mons. After that we recreated the 
monitor, and although it starts, it remains in state ‘probing’, as you 
can see below.


No matter what I tried, it won’t come up. I’ve seen quite some messages 
that the MTU might be an issue, but that seems to be ok:

root@proxmox03:/var/log/ceph# fping -b 1472 10.10.10.{1..3} -M
10.10.10.1 is alive
10.10.10.2 is alive
10.10.10.3 is alive


Does anyone have an idea how to fix this? I’ve tried destroying and 
recreating the mon a few times now. Could it be that the leveldb mons 
only support mon.$id notation for the monitors?


root@proxmox03:/var/log/ceph# ceph daemon mon.proxmox03 mon_status
{
"name": “proxmox03”,
"rank": 2,
"state": “probing”,
"election_epoch": 0,
"quorum": [],
"features": {
"required_con": “2449958197560098820”,
"required_mon": [
“kraken”,
“luminous”,
“mimic”,
"osdmap-prune”,
“nautilus”,
“octopus”,
“pacific”,
"elector-pinging”
],
"quorum_con": “0”,
"quorum_mon": []
},
"outside_quorum": [
“proxmox03”
],
"extra_probe_peers": [],
"sync_provider": [],
"monmap": {
"epoch": 0,
"fsid": "39b1e85c-7b47-4262-9f0a-47ae91042bac”,
"modified": "2024-01-23T21:02:12.631320Z”,
"created": "2017-03-15T14:54:55.743017Z”,
"min_mon_release": 16,
"min_mon_release_name": “pacific”,
"election_strategy": 1,
"disallowed_leaders: ": “”,
"stretch_mode": false,
"tiebreaker_mon": “”,
"removed_ranks: ": “2”,
"features": {
"persistent": [
“kraken”,
“luminous”,
“mimic”,
"osdmap-prune”,
“nautilus”,
“octopus”,
“pacific”,
"elector-pinging”
],
"optional": []
},
"mons": [
{
"rank": 0,
"name": “0”,
"public_addrs": {
"addrvec": [
{
"type": “v2”,
"addr": "10.10.10.1:3300”,
"nonce": 0
},
{
"type": “v1”,
"addr": "10.10.10.1:6789”,
"nonce": 0
}
]
},
"addr": "10.10.10.1:6789/0”,
"public_addr": "10.10.10.1:6789/0”,
"priority": 0,
"weight": 0,
"crush_location": “{}”
},
{
"rank": 1,
"name": “1”,
"public_addrs": {
"addrvec": [
{
"type": “v2”,
"addr": "10.10.10.2:3300”,
"nonce": 0
},
{
"type": “v1”,
"addr": "10.10.10.2:6789”,
"nonce": 0
}
]
},
"addr": "10.10.10.2:6789/0”,
"public_addr": "10.10.10.2:6789/0”,
"priority": 0,
"weight": 0,
"crush_location": “{}”
},
{
"rank": 2,
"name": “proxmox03”,
"public_addrs": {
"addrvec": [
{
"type": “v2”,
"addr": "10.10.10.3:3300”,
"nonce": 0
},
{
"type": “v1”,
"addr": "10.10.10.3:6789”,
"nonce": 0
}
]
},
"addr": "10.10.10.3:6789/0”,
"public_addr": "10.10.10.3:6789/0”,
"priority": 0,
"weight": 0,
"crush_location": “{}”
}
]
},
"feature_map": {
"mon": [
{
"features": “0x3f01cfbdfffd”,
"release": “luminous”,
"num": 1
}
]
},
"stretch_mode": false
}

—
Mark Schouten
CTO, Tuxis B.V.
+31 318 200208 / m...@tuxis.nl
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: how can install latest dev release?

2024-01-31 Thread Christian Rohmann


On 31.01.24 09:38, garcetto wrote:

  how can i install latest dev release using cephadm?
I suppose you found 
https://docs.ceph.com/en/quincy/install/get-packages/#ceph-development-packages, 
but yes, that only seems to target a package installation.
Would be nice if there were also dev containers being built somewhere to 
use with cephadm.




Regards

Christian



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: how can install latest dev release?

2024-01-31 Thread garcetto

thank you, but seems related to quincy, there is nothing on latest vesions
in the doc...maybe the doc is not updated?

On Wed, Jan 31, 2024 at 10:46 AM Christian Rohmann <
christian.rohm...@inovex.de> wrote:

> On 31.01.24 09:38, garcetto wrote:
> >   how can i install latest dev release using cephadm?
> I suppose you found
>
> https://docs.ceph.com/en/quincy/install/get-packages/#ceph-development-packages,
>
> but yes, that only seems to target a package installation.
> Would be nice if there were also dev containers being built somewhere to
> use with cephadm.
>
>
>
> Regards
>
> Christian
>
>
>
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cannot recreate monitor in upgrade from pacific to quincy (leveldb -> rocksdb)

2024-01-31 Thread Eugen Block


Hi Mark,

as I'm not familiar with proxmox I'm not sure what happens under the  
hood. There are a couple of things I would try, not necessarily in  
this order:


- Check the troubleshooting guide [1], for example a clock skew could  
be one reason, have you verified ntp/chronyd functionality?
- Inspect debug log output, maybe first on the probing mon and if  
those don't reveal the reason, enable debug logs for the other MONs as  
well:

ceph config set mon.proxmox03 debug_mon 20
ceph config set mon.proxmox03 debug_paxos 20

or for all MONs:
ceph config set mon debug_mon 20
ceph config set mon debug_paxos 20

- Try to deploy an additional MON on a different server (if you have  
more available) and see if that works.

- Does proxmox log anything?
- Maybe last resort, try to start a MON manually after adding it to  
the monmap with the monmaptool, but only if you know what you're  
doing. I wonder if the monmap doesn't get updated...


Regards,
Eugen

[1] https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/

Zitat von Mark Schouten :


Hi,

During an upgrade from pacific to quincy, we needed to recreate the  
mons because the mons were pretty old and still using leveldb.


So step one was to destroy one of the mons. After that we recreated  
the monitor, and although it starts, it remains in state ‘probing’,  
as you can see below.


No matter what I tried, it won’t come up. I’ve seen quite some  
messages that the MTU might be an issue, but that seems to be ok:

root@proxmox03:/var/log/ceph# fping -b 1472 10.10.10.{1..3} -M
10.10.10.1 is alive
10.10.10.2 is alive
10.10.10.3 is alive


Does anyone have an idea how to fix this? I’ve tried destroying and  
recreating the mon a few times now. Could it be that the leveldb  
mons only support mon.$id notation for the monitors?


root@proxmox03:/var/log/ceph# ceph daemon mon.proxmox03 mon_status
{
"name": “proxmox03”,
"rank": 2,
"state": “probing”,
"election_epoch": 0,
"quorum": [],
"features": {
"required_con": “2449958197560098820”,
"required_mon": [
“kraken”,
“luminous”,
“mimic”,
"osdmap-prune”,
“nautilus”,
“octopus”,
“pacific”,
"elector-pinging”
],
"quorum_con": “0”,
"quorum_mon": []
},
"outside_quorum": [
“proxmox03”
],
"extra_probe_peers": [],
"sync_provider": [],
"monmap": {
"epoch": 0,
"fsid": "39b1e85c-7b47-4262-9f0a-47ae91042bac”,
"modified": "2024-01-23T21:02:12.631320Z”,
"created": "2017-03-15T14:54:55.743017Z”,
"min_mon_release": 16,
"min_mon_release_name": “pacific”,
"election_strategy": 1,
"disallowed_leaders: ": “”,
"stretch_mode": false,
"tiebreaker_mon": “”,
"removed_ranks: ": “2”,
"features": {
"persistent": [
“kraken”,
“luminous”,
“mimic”,
"osdmap-prune”,
“nautilus”,
“octopus”,
“pacific”,
"elector-pinging”
],
"optional": []
},
"mons": [
{
"rank": 0,
"name": “0”,
"public_addrs": {
"addrvec": [
{
"type": “v2”,
"addr": "10.10.10.1:3300”,
"nonce": 0
},
{
"type": “v1”,
"addr": "10.10.10.1:6789”,
"nonce": 0
}
]
},
"addr": "10.10.10.1:6789/0”,
"public_addr": "10.10.10.1:6789/0”,
"priority": 0,
"weight": 0,
"crush_location": “{}”
},
{
"rank": 1,
"name": “1”,
"public_addrs": {
"addrvec": [
{
"type": “v2”,
"addr": "10.10.10.2:3300”,
"nonce": 0
},
{
"type": “v1”,
"addr": "10.10.10.2:6789”,
"nonce": 0
}
]
},
"addr": "10.10.10.2:6789/0”,
"public_addr": "10.10.10.2:6789/0”,
"priority": 0,
"weight": 0,
"crush_location": “{}”
},
{
"rank": 2,
"name": “proxmox03”,
"public_addrs": {
"addrvec": [
{
"type": “v2”,

[ceph-users] Re: NFS HA - "virtual_ip": null after upgrade to reef

2024-01-31 Thread Eugen Block


Hi,

As I read the documentation[1] the "count: 1" handles that so what I  
have is a placement pool from which only one is selected for  
deployment?


you're probably right, when using your example command with multiple  
hosts it automatically sets "count:1" (don't mind the hostnames, it's  
an upgraded cluster currently running 18.2.1):


# ceph orch ls nfs
NAME  PORTS   RUNNING  REFRESHED  AGE  PLACEMENT
nfs.nfs-reef  ?:2049  1/1  114s ago   3m
nautilus;nautilus2;nautilus3;count:1


# ceph orch ls ingress
NAME  PORTS RUNNING  REFRESHED  AGE  PLACEMENT
ingress.nfs.nfs-reef  192.168.168.114:9049  1/1  2m ago 4m
nautilus;nautilus2;nautilus3;count:1


So it's not really clear what happened to your ingress service. :-)  
But at least it works now, so that's good.


Zitat von Torkil Svensgaard :


On 31/01/2024 09:36, Eugen Block wrote:

Hi,

if I understand this correctly, with the "keepalive-only" option  
only one ganesha instance is supposed to be deployed:


If a user additionally supplies --ingress-mode keepalive-only a  
partial ingress service will be deployed that still provides a  
virtual IP, but has nfs directly binding to that virtual IP and  
leaves out any sort of load balancing or traffic redirection. This  
setup will restrict users to deploying only 1 nfs daemon as  
multiple cannot bind to the same port on the virtual IP.


Maybe that's why it disappeared as you have 3 hosts in the  
placement parameter? Is the ingress service still present in 'ceph  
orch ls'?


As I read the documentation[1] the "count: 1" handles that so what I  
have is a placement pool from which only one is selected for  
deployment?


The absence of the ingress service is puzzling me, as it worked just  
fine prior to the upgrade and the upgrade shouldn't have touched the  
service spec in any way?


Mvh.

Torkil

[1]  
https://docs.ceph.com/en/latest/cephadm/services/nfs/#nfs-with-virtual-ip-but-no-haproxy




Regards,
Eugen

Zitat von Torkil Svensgaard :


Hi

Last week we created an NFS service like this:

"
ceph nfs cluster create jumbo  
"ceph-flash1,ceph-flash2,ceph-flash3" --ingress --virtual_ip  
172.21.15.74/22 --ingress-mode keepalive-only

"

Worked like a charm. Yesterday we upgraded from 17.2.7 to 18.20.0  
and the NFS virtual IP seems to have gone missing in the process:


"
# ceph nfs cluster info jumbo
{
  "jumbo": {
    "backend": [
  {
    "hostname": "ceph-flash1",
    "ip": "172.21.15.148",
    "port": 2049
  }
    ],
    "virtual_ip": null
  }
}
"

Service spec:

"
service_type: nfs
service_id: jumbo
service_name: nfs.jumbo
placement:
  count: 1
  hosts:
  - ceph-flash1
  - ceph-flash2
  - ceph-flash3
spec:
  port: 2049
  virtual_ip: 172.21.15.74
"

I've tried restarting the nfs.jumbo service which didn't help. Suggestions?

Mvh.

Torkil

--
Torkil Svensgaard
Sysadmin
MR-Forskningssektionen, afs. 714
DRCMR, Danish Research Centre for Magnetic Resonance
Hvidovre Hospital
Kettegård Allé 30
DK-2650 Hvidovre
Denmark
Tel: +45 386 22828
E-mail: tor...@drcmr.dk
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


--
Torkil Svensgaard
Sysadmin
MR-Forskningssektionen, afs. 714
DRCMR, Danish Research Centre for Magnetic Resonance
Hvidovre Hospital
Kettegård Allé 30
DK-2650 Hvidovre
Denmark
Tel: +45 386 22828
E-mail: tor...@drcmr.dk
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cannot recreate monitor in upgrade from pacific to quincy (leveldb -> rocksdb)

2024-01-31 Thread Mark Schouten


Hi,

I don’t have a fourth machine available, so that’s not an option 
unfortunatly.


I did enable a lot of debugging earlier, but that shows no information 
as to why stuff is not working as to be expected.


Proxmox just deploys the mons, nothing fancy there, no special cases.

Can anyone confirm that ancient (2017) leveldb database mons should just 
accept ‘mon.$hostname’ names for mons, a well as ‘mon.$id’ ?


—
Mark Schouten
CTO, Tuxis B.V.
+31 318 200208 / m...@tuxis.nl


-- Original Message --
From "Eugen Block" 
To ceph-users@ceph.io
Date 31/01/2024, 13:02:04
Subject [ceph-users] Re: Cannot recreate monitor in upgrade from pacific 
to quincy (leveldb -> rocksdb)



Hi Mark,

as I'm not familiar with proxmox I'm not sure what happens under the  hood. 
There are a couple of things I would try, not necessarily in  this order:

- Check the troubleshooting guide [1], for example a clock skew could  be one 
reason, have you verified ntp/chronyd functionality?
- Inspect debug log output, maybe first on the probing mon and if  those don't 
reveal the reason, enable debug logs for the other MONs as  well:
ceph config set mon.proxmox03 debug_mon 20
ceph config set mon.proxmox03 debug_paxos 20

or for all MONs:
ceph config set mon debug_mon 20
ceph config set mon debug_paxos 20

- Try to deploy an additional MON on a different server (if you have  more 
available) and see if that works.
- Does proxmox log anything?
- Maybe last resort, try to start a MON manually after adding it to  the monmap 
with the monmaptool, but only if you know what you're  doing. I wonder if the 
monmap doesn't get updated...

Regards,
Eugen

[1] https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/

Zitat von Mark Schouten :


Hi,

During an upgrade from pacific to quincy, we needed to recreate the  mons 
because the mons were pretty old and still using leveldb.

So step one was to destroy one of the mons. After that we recreated  the 
monitor, and although it starts, it remains in state ‘probing’,  as you can see 
below.

No matter what I tried, it won’t come up. I’ve seen quite some  messages that 
the MTU might be an issue, but that seems to be ok:
root@proxmox03:/var/log/ceph# fping -b 1472 10.10.10.{1..3} -M
10.10.10.1 is alive
10.10.10.2 is alive
10.10.10.3 is alive


Does anyone have an idea how to fix this? I’ve tried destroying and  recreating 
the mon a few times now. Could it be that the leveldb  mons only support 
mon.$id notation for the monitors?

root@proxmox03:/var/log/ceph# ceph daemon mon.proxmox03 mon_status
{
"name": “proxmox03”,
"rank": 2,
"state": “probing”,
"election_epoch": 0,
"quorum": [],
"features": {
"required_con": “2449958197560098820”,
"required_mon": [
“kraken”,
“luminous”,
“mimic”,
"osdmap-prune”,
“nautilus”,
“octopus”,
“pacific”,
"elector-pinging”
],
"quorum_con": “0”,
"quorum_mon": []
},
"outside_quorum": [
“proxmox03”
],
"extra_probe_peers": [],
"sync_provider": [],
"monmap": {
"epoch": 0,
"fsid": "39b1e85c-7b47-4262-9f0a-47ae91042bac”,
"modified": "2024-01-23T21:02:12.631320Z”,
"created": "2017-03-15T14:54:55.743017Z”,
"min_mon_release": 16,
"min_mon_release_name": “pacific”,
"election_strategy": 1,
"disallowed_leaders: ": “”,
"stretch_mode": false,
"tiebreaker_mon": “”,
"removed_ranks: ": “2”,
"features": {
"persistent": [
“kraken”,
“luminous”,
“mimic”,
"osdmap-prune”,
“nautilus”,
“octopus”,
“pacific”,
"elector-pinging”
],
"optional": []
},
"mons": [
{
"rank": 0,
"name": “0”,
"public_addrs": {
"addrvec": [
{
"type": “v2”,
"addr": "10.10.10.1:3300”,
"nonce": 0
},
{
"type": “v1”,
"addr": "10.10.10.1:6789”,
"nonce": 0
}
]
},
"addr": "10.10.10.1:6789/0”,
"public_addr": "10.10.10.1:6789/0”,
"priority": 0,
"weight": 0,
"crush_location": “{}”
},
{
"rank": 1,
"name": “1”,
"public_addrs": {
"addrvec": [
{
"type": “v2”,
"addr": "10.10.10.2:3300”,
"nonce": 0

[ceph-users] Re: pacific 16.2.15 QE validation status

2024-01-31 Thread Venky Shankar

On Tue, Jan 30, 2024 at 3:08 AM Yuri Weinstein  wrote:
>
> Details of this release are summarized here:
>
> https://tracker.ceph.com/issues/64151#note-1
>
> Seeking approvals/reviews for:
>
> rados - Radek, Laura, Travis, Ernesto, Adam King
> rgw - Casey
> fs - Venky

fs approved. Failures are -
https://tracker.ceph.com/projects/cephfs/wiki/Pacific#31-Jan-2024

> rbd - Ilya
> krbd - in progress
>
> upgrade/nautilus-x (pacific) - Casey PTL (regweed tests failed)
> upgrade/octopus-x (pacific) - Casey PTL (regweed tests failed)
>
> upgrade/pacific-x (quincy) - in progress
> upgrade/pacific-p2p - Ilya PTL (maybe rbd related?)
>
> ceph-volume - Guillaume
>
> TIA
> YuriW
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>


-- 
Cheers,
Venky
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Help on rgw metrics (was rgw_user_counters_cache)

2024-01-31 Thread Casey Bodley

On Wed, Jan 31, 2024 at 3:43 AM garcetto  wrote:
>
> good morning,
>   i was struggling trying to understand why i cannot find this setting on
> my reef version, is it because is only on latest dev ceph version and not
> before?

that's right, this new feature will be part of the squid release. we
don't plan to backport it to reef

>
> https://docs.ceph.com/en/*latest*
> /radosgw/metrics/#user-bucket-counter-caches
>
> Reef gives 404
> https://docs.ceph.com/en/reef/radosgw/metrics/
>
> thank you!
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: pacific 16.2.15 QE validation status

2024-01-31 Thread Ilya Dryomov

On Tue, Jan 30, 2024 at 9:24 PM Yuri Weinstein  wrote:
>
> Update.
> Seeking approvals/reviews for:
>
> rados - Radek, Laura, Travis, Ernesto, Adam King
> rgw - Casey
> fs - Venky
> rbd - Ilya

Hi Yuri,

rbd looks good overall but we are missing iSCSI coverage due to
https://tracker.ceph.com/issues/64126:

https://pulpito.ceph.com/yuriw-2024-01-25_17:27:10-rbd-pacific-release-distro-default-smithi/7532167
https://pulpito.ceph.com/yuriw-2024-01-26_17:05:20-rbd-pacific-release-distro-default-smithi/7535193

Adam, did you get a chance to look into it?

> krbd - Ilya

Please do another rerun for krbd -- I want to see one of those
remaining jobs pass.

Thanks,

Ilya
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Pacific: Drain hosts does not remove mgr daemon

2024-01-31 Thread Mevludin Blazevic


Hi all,

after performing "ceph orch host drain" on one of our host with only the 
mgr container left, I encounter that another mgr daemon is indeed 
deployed on another host, but the "old" does not get removed from the 
drain command. The same happens if I edit the mgr service via UI to 
define different hosts for the daemon and again the old mgr daemons are 
not getting removed. Any recommendations? I am using a setup with podman 
and RHEL.


Best,
Mevludin
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Pacific: Drain hosts does not remove mgr daemon

2024-01-31 Thread Adam King

If you just manually run `ceph orch daemon rm
` does it get removed? I know there's
some logic in host drain that does some ok-to-stop checks that can cause
things to be delayed or stuck if it doesn't think it's safe to remove the
daemon for some reason. I wonder if it's being overly cautious here. The
manual command to remove the mgr doesn't do all those checks I believe.

On Wed, Jan 31, 2024 at 10:16 AM Mevludin Blazevic 
wrote:

> Hi all,
>
> after performing "ceph orch host drain" on one of our host with only the
> mgr container left, I encounter that another mgr daemon is indeed
> deployed on another host, but the "old" does not get removed from the
> drain command. The same happens if I edit the mgr service via UI to
> define different hosts for the daemon and again the old mgr daemons are
> not getting removed. Any recommendations? I am using a setup with podman
> and RHEL.
>
> Best,
> Mevludin
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] S3 object appears in ListObject but 404 when issuing a GET

2024-01-31 Thread Mathias Chapelain

Hello,

Recently we got a problem from an internal customer on our S3. Our setup consist
of roughly 10 servers with 140 OSDs. Our 3 RGWs are collocated with monitors on
dedicated servers in a HA setup with HAProxy in front. We are running 16.2.14
on Podman with Cephadm.

Our S3 is constantly having a traffic of 500 req/s average per RGW instance.

The problem is described in this issue: https://tracker.ceph.com/issues/63935.

Basically this customer is having a Grafana Mimir instance pushing to our S3 and
during a compaction process it does a special pattern like this:

```
29/Dec/2023:17:13:28.961 rgw-frontend~ rgw-backend/server-mon-01-rgw0 
0/0/0/127/127 200 228 - -  132/132/70/67/0 0/0 "PUT /1234/object HTTP/1.1" 
29/Dec/2023:17:13:29.101 rgw-frontend~ rgw-backend/server-mon-01-rgw0 0/0/0/1/1 
200 381 - -  132/132/76/71/0 0/0 "GET /1234/object HTTP/1.1" 
29/Dec/2023:17:13:29.121 rgw-frontend~ rgw-backend/server-mon-01-rgw0 0/0/0/1/1 
200 381 - -  132/132/71/59/0 0/0 "GET /1234/object HTTP/1.1" 
29/Dec/2023:17:13:29.137 rgw-frontend~ rgw-backend/server-mon-03-rgw0 0/0/0/4/4 
204 153 - -  132/132/71/6/0 0/0 "DELETE /1234/object HTTP/1.1" 
29/Dec/2023:19:03:21.671 rgw-frontend~ rgw-backend/server-mon-03-rgw0 0/0/0/1/1 
404 472 - -  55/55/26/0/0 0/0 "GET /1234/object HTTP/1.1" 
```

It is doing PUT, GET and DELETE in the same second. Afterwards the customer can
see the deleted object when doing a ListObjects in the bucket but if he tries 
to access it then RGW
returns a 404.

After looking in Ceph, it appears the object has a bucket index entry but the
associated RADOS object does not exist anymore. The bucket does not have
versioning or object locking.

Did someone encounter something similar? Thank you!

Regards,


--
Mathias Chapelain
Storage Engineer
Proton AG
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: how can install latest dev release?

2024-01-31 Thread Kai Stian Olstad


On 31.01.2024 09:38, garcetto wrote:

good morning,
 how can i install latest dev release using cephadm?


Have you looked at this page?
https://docs.ceph.com/en/latest/install/containers/#development-builds

--
Kai Stian Olstad
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Successfully using dm-cache

2024-01-31 Thread Michael Lipp

Just in case anybody is interested: Using dm-cache works and boosts 
performance -- at least for my use case.


The "challenge" was to get 100 (identical) Linux-VMs started on a three 
node hyperconverged cluster. The hardware is nothing special, each node 
has a Supermicro server board with a single CPU with 24 cores and 4 x 4 
TB hard disks. And there's that extra 1 TB NVMe...


I know that the general recommendation is to use the NVMe for WAL and 
metadata, but this didn't seem appropriate for my use case and I'm still 
not quite sure about failure scenarios with this configuration. So 
instead I made each drive a logical volume (managed by an OSD) and added 
85 GiB NVMe to each LV as read-only cache.


Each VM uses as system disk an RBD based on a snapshot from the master 
image. The idea was that with this configuration, all VMs should share 
most (actually almost all) of the data on their system disk and this 
data should be available from the cache.


Well, it works. When booting the 100 VMs, almost all read operations are 
satisfied from the cache. So I get close to NVMe speed but have payed 
for conventional hard drives only (well, SSDs aren't that much more 
expensive nowadays, but the hardware is 4 years old).


So, nothing sophisticated, but as I couldn't find anything about this 
kind of setup, it might be of interest nevertheless.


 - Michael
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Performance improvement suggestion

2024-01-31 Thread quag...@bol.com.br

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephfs inode backtrace information

2024-01-31 Thread Gregory Farnum

The docs recommend a fast SSD pool for the CephFS *metadata*, but the
default data pool can be more flexible. The backtraces are relatively
small — it's an encoded version of the path an inode is located at,
plus the RADOS hobject, which is probably more of the space usage. So
it should fit fine in your SSD pool, but if all the cephfs file data
is living in the hard drive pool I'd just set it up there.
-Greg

On Tue, Jan 30, 2024 at 2:03 AM Dietmar Rieder
 wrote:
>
> Hello,
>
> I have a question regarding the default pool of a cephfs.
>
> According to the docs it is recommended to use a fast ssd replicated
> pool as default pool for cephfs. I'm asking what are the space
> requirements for storing the inode backtrace information?
>
> Let's say I have a 85 TiB replicated ssd pool (hot data) and as 3 PiB EC
> data pool (cold data).
>
> Does it make sense to create a third pool as default pool which only
> holds the inode backtrace information (what would be a good size), or is
> it OK to use the ssd pool as default pool?
>
> Thanks
> Dietmar
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephfs inode backtrace information

2024-01-31 Thread Patrick Donnelly

On Tue, Jan 30, 2024 at 5:03 AM Dietmar Rieder
 wrote:
>
> Hello,
>
> I have a question regarding the default pool of a cephfs.
>
> According to the docs it is recommended to use a fast ssd replicated
> pool as default pool for cephfs. I'm asking what are the space
> requirements for storing the inode backtrace information?

The actual recommendation is to use a replicated pool for the default
data pool. Regular hard drives are fine for the storage device.

> Let's say I have a 85 TiB replicated ssd pool (hot data) and as 3 PiB EC
> data pool (cold data).
>
> Does it make sense to create a third pool as default pool which only
> holds the inode backtrace information (what would be a good size), or is
> it OK to use the ssd pool as default pool?

Assuming your 85 TiB rep ssd pool is the default data pool already, use that.

(I am curious why this question is asked now when the file system
already has a significant amount of data? Are you thinking about
recreating the fs?)

-- 
Patrick Donnelly, Ph.D.
He / Him / His
Red Hat Partner Engineer
IBM, Inc.
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: pacific 16.2.15 QE validation status

2024-01-31 Thread Casey Bodley

On Mon, Jan 29, 2024 at 4:39 PM Yuri Weinstein  wrote:
>
> Details of this release are summarized here:
>
> https://tracker.ceph.com/issues/64151#note-1
>
> Seeking approvals/reviews for:
>
> rados - Radek, Laura, Travis, Ernesto, Adam King
> rgw - Casey

rgw approved, thanks

> fs - Venky
> rbd - Ilya
> krbd - in progress
>
> upgrade/nautilus-x (pacific) - Casey PTL (regweed tests failed)
> upgrade/octopus-x (pacific) - Casey PTL (regweed tests failed)
>
> upgrade/pacific-x (quincy) - in progress
> upgrade/pacific-p2p - Ilya PTL (maybe rbd related?)
>
> ceph-volume - Guillaume
>
> TIA
> YuriW
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Performance improvement suggestion

2024-01-31 Thread quag...@bol.com.br

Hello everybody,
 I would like to make a suggestion for improving performance in Ceph architecture.
 I don't know if this group would be the best place or if my proposal is correct.

 My suggestion would be in the item https://docs.ceph.com/en/latest/architecture/, at the end of the topic "Smart Daemons Enable Hyperscale".

 The Client needs to "wait" for the configured amount of replicas to be written (so that the client receives an ok and continues). This way, if there is slowness on any of the disks on which the PG will be updated, the client is left waiting.
     
 It would be possible:
     
 1-) Only record on the primary OSD
 2-) Write other replicas in background (like the same way as when an OSD fails: "degraded" ).

 This way, client has a faster response when writing to storage: improving latency and performance (throughput and IOPS).
     
 I would find it plausible to accept a period of time (seconds) until all replicas are ok (written asynchronously) at the expense of improving performance.
     
 Could you evaluate this scenario?


Rafael.

 ___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Performance improvement suggestion

2024-01-31 Thread Can Özyurt

I never tried this myself but "min_size = 1" should do what you want to achieve.

On Wed, 31 Jan 2024 at 22:48, quag...@bol.com.br  wrote:
>
> Hello everybody,
>  I would like to make a suggestion for improving performance in Ceph 
> architecture.
>  I don't know if this group would be the best place or if my proposal is 
> correct.
>
>  My suggestion would be in the item 
> https://docs.ceph.com/en/latest/architecture/, at the end of the topic "Smart 
> Daemons Enable Hyperscale".
>
>  The Client needs to "wait" for the configured amount of replicas to be 
> written (so that the client receives an ok and continues). This way, if there 
> is slowness on any of the disks on which the PG will be updated, the client 
> is left waiting.
>
>  It would be possible:
>
>  1-) Only record on the primary OSD
>  2-) Write other replicas in background (like the same way as when an OSD 
> fails: "degraded" ).
>
>  This way, client has a faster response when writing to storage: 
> improving latency and performance (throughput and IOPS).
>
>  I would find it plausible to accept a period of time (seconds) until all 
> replicas are ok (written asynchronously) at the expense of improving 
> performance.
>
>  Could you evaluate this scenario?
>
>
> Rafael.
>
>  ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Performance improvement suggestion

2024-01-31 Thread Anthony D'Atri

Would you be willing to accept the risk of data loss?

> On Jan 31, 2024, at 2:48 PM, quag...@bol.com.br wrote:
> 
> Hello everybody,
>  I would like to make a suggestion for improving performance in Ceph 
> architecture.
>  I don't know if this group would be the best place or if my proposal is 
> correct.
> 
>  My suggestion would be in the item 
> https://docs.ceph.com/en/latest/architecture/, at the end of the topic "Smart 
> Daemons Enable Hyperscale".
> 
>  The Client needs to "wait" for the configured amount of replicas to be 
> written (so that the client receives an ok and continues). This way, if there 
> is slowness on any of the disks on which the PG will be updated, the client 
> is left waiting.
>  
>  It would be possible:
>  
>  1-) Only record on the primary OSD
>  2-) Write other replicas in background (like the same way as when an OSD 
> fails: "degraded" ).
> 
>  This way, client has a faster response when writing to storage: 
> improving latency and performance (throughput and IOPS).
>  
>  I would find it plausible to accept a period of time (seconds) until all 
> replicas are ok (written asynchronously) at the expense of improving 
> performance.
>  
>  Could you evaluate this scenario?
> 
> 
> Rafael.
> 
>  ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Performance improvement suggestion

2024-01-31 Thread Anthony D'Atri

I’ve heard conflicting asserts on whether the write returns with min_size 
shards have been persisted, or all of them.

> On Jan 31, 2024, at 2:58 PM, Can Özyurt  wrote:
> 
> I never tried this myself but "min_size = 1" should do what you want to 
> achieve.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Understanding subvolumes

2024-01-31 Thread Matthew Melendy


In our department we're getting starting with Ceph 'reef', using Ceph FUSE 
client for our Ubuntu workstations.

So far so good, except I can't quite figure out one aspect of subvolumes.

When I do the commands:

[root@ceph1 ~]# ceph fs subvolumegroup create cephfs csvg
[root@ceph1 ~]# ceph fs subvolume create cephfs staff csvg --size 2
 
I get these results:


- A subvolume group csvg is created on volume cephfs
- A subvolume called staff is created in csvg subvolume (like 
/volumes/csvg/staff ) however there is no size limit set at this folder in the 
Ceph dashboard view
- A folder with an random UUID name is created inside the subvolume staff ( 
like /volumes/csvg/staff/6a1b3de5-f6ab-4878-aea3-3c3c6f96ffcf ); this folder 
does have a size set on it of 2TB

My questions are:
- what's the purpose of this UUID, and is it a requirement?
- which directory should be mounted for my clients, staff/ or staff/{UUID}, for 
the size limit to take effect?
- is there any way to hide or disable this UUID for client mounts? (eg in 
/etc/fstab)

[root@ceph1 ~]# ceph fs subvolume getpath cephfs staff csvg
/volumes/csvg/staff/6a1b3de5-f6ab-4878-aea3-3c3c6f96ffcf

[root@ceph1 ~]# ceph fs subvolume ls cephfs csvg
[
{
"name": "staff"
}
]



--
Sincerely,


Matthew Melendy

IT Services Specialist
CS System Services Group
FEC 3550, University of New Mexico
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: pacific 16.2.15 QE validation status

2024-01-31 Thread Laura Flores

I reviewed the rados suite. @Adam King , @Nizamudeen A
 would appreciate a look from you, as there are some
orchestrator and dashboard trackers that came up.

pacific-release, 16.2.15

Failures:
1. https://tracker.ceph.com/issues/62225
2. https://tracker.ceph.com/issues/64278
3. https://tracker.ceph.com/issues/58659
4. https://tracker.ceph.com/issues/58658
5. https://tracker.ceph.com/issues/64280 -- new tracker, worth a look
from Orch
6. https://tracker.ceph.com/issues/63577
7. https://tracker.ceph.com/issues/63894
8. https://tracker.ceph.com/issues/64126
9. https://tracker.ceph.com/issues/63887
10. https://tracker.ceph.com/issues/61602
11. https://tracker.ceph.com/issues/54071
12. https://tracker.ceph.com/issues/57386
13. https://tracker.ceph.com/issues/64281
14. https://tracker.ceph.com/issues/49287

Details:
1. pacific upgrade test fails on 'ceph versions | jq -e' command - Ceph
- RADOS
2. Unable to update caps for client.iscsi.iscsi.a - Ceph - Orchestrator
3. mds_upgrade_sequence: failure when deploying node-exporter - Ceph -
Orchestrator
4. mds_upgrade_sequence: Error: initializing source
docker://prom/alertmanager:v0.20.0 - Ceph - Orchestrator
5. mgr-nfs-upgrade test times out from failed cephadm daemons - Ceph -
Orchestrator
6. cephadm: docker.io/library/haproxy: toomanyrequests: You have
reached your pull rate limit. You may increase the limit by authenticating
and upgrading: https://www.docker.com/increase-rate-limit - Ceph -
Orchestrator
7. qa: cephadm failed with an error code 1, alertmanager container not
found. - Ceph - Orchestrator
8. ceph-iscsi build was retriggered and now missing
package_manager_version attribute - Ceph
9. Starting alertmanager fails from missing container - Ceph -
Orchestrator
10. pacific: cls/test_cls_sdk.sh: Health check failed: 1 pool(s) do not
have an application enabled (POOL_APP_NOT_ENABLED) - Ceph - RADOS
11. rados/cephadm/osds: Invalid command: missing required parameter
hostname() - Ceph - Orchestrator
12. cephadm/test_dashboard_e2e.sh: Expected to find content: '/^foo$/'
within the selector: 'cd-modal .badge' but never did - Ceph - Mgr -
Dashboard
13. Failed to download key at
http://download.ceph.com/keys/autobuild.asc: Request failed:  - Infrastructure
14. podman: setting cgroup config for procHooks process caused: Unit
libpod-$hash.scope not found - Ceph - Orchestrator

On Wed, Jan 31, 2024 at 1:41 PM Casey Bodley  wrote:

> On Mon, Jan 29, 2024 at 4:39 PM Yuri Weinstein 
> wrote:
> >
> > Details of this release are summarized here:
> >
> > https://tracker.ceph.com/issues/64151#note-1
> >
> > Seeking approvals/reviews for:
> >
> > rados - Radek, Laura, Travis, Ernesto, Adam King
> > rgw - Casey
>
> rgw approved, thanks
>
> > fs - Venky
> > rbd - Ilya
> > krbd - in progress
> >
> > upgrade/nautilus-x (pacific) - Casey PTL (regweed tests failed)
> > upgrade/octopus-x (pacific) - Casey PTL (regweed tests failed)
> >
> > upgrade/pacific-x (quincy) - in progress
> > upgrade/pacific-p2p - Ilya PTL (maybe rbd related?)
> >
> > ceph-volume - Guillaume
> >
> > TIA
> > YuriW
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email to dev-le...@ceph.io
>


-- 

Laura Flores

She/Her/Hers

Software Engineer, Ceph Storage 

Chicago, IL

lflo...@ibm.com | lflo...@redhat.com 
M: +17087388804
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Performance improvement suggestion

2024-01-31 Thread Janne Johansson

> I’ve heard conflicting asserts on whether the write returns with min_size 
> shards have been persisted, or all of them.

I think it waits until all replicas have written the data, but from
simplistic tests with fast network and slow drives, the extra time
taken to write many copies is not linear to what it takes to write the
first, so unless you do go min_size=1 (not recommended at all), the
extra copies are not slowing you down as much as you'd expect. At
least not if the other drives are not 100% busy.

I get that this thread started on having one bad drive, and that is
another scenario of course, but having repl=2 or repl=3 is not about
writes taking 100% - 200% more time than the single write, it is less.

-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [EXTERN] Re: cephfs inode backtrace information

2024-01-31 Thread Dietmar Rieder


On 1/31/24 20:13, Patrick Donnelly wrote:

On Tue, Jan 30, 2024 at 5:03 AM Dietmar Rieder
 wrote:


Hello,

I have a question regarding the default pool of a cephfs.

According to the docs it is recommended to use a fast ssd replicated
pool as default pool for cephfs. I'm asking what are the space
requirements for storing the inode backtrace information?


The actual recommendation is to use a replicated pool for the default
data pool. Regular hard drives are fine for the storage device.


yes true that, I was saying ssd because my replicated pool is on ssds.


Let's say I have a 85 TiB replicated ssd pool (hot data) and as 3 PiB EC
data pool (cold data).

Does it make sense to create a third pool as default pool which only
holds the inode backtrace information (what would be a good size), or is
it OK to use the ssd pool as default pool?


Assuming your 85 TiB rep ssd pool is the default data pool already, use that.


yes, I was planning to uses the 85 TiB rep ssd pool as the default, I 
was just not sure if it might get substantially filled by  inode 
backtrace information from files that will get stored in the 3 PiB EC pool.



(I am curious why this question is asked now when the file system
already has a significant amount of data? Are you thinking about
recreating the fs?)


the system is being set up, so no data there yet, but we plan to use the 
85 TiB rep pool for user homes and the 3 PiB EC pool for data. We use 
ceph.dir.layout.pool settings to separate the storage devices/pool for 
/home and /data.


Dietmar




OpenPGP_signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Syslog server log naming

2024-01-31 Thread Torkil Svensgaard

We have ceph (currently 18.2.0) log to an rsyslog server with the 
following file name format:


template (name="DynaFile" type="string" 
string="/tank/syslog/%fromhost-ip%/%hostname%/%programname%.log")



Around May 25th this year something changed so instead of getting the 
usual program log names we are also getting a lot of diffent logs with 
weird names. Here's an ls excerpt:


"
...
-rw---. 1 root root  517K Feb  1 05:42 hardcore_raman.log
-rw---. 1 root root  198K Feb  1 05:42 sleepy_moser.log
-rw---. 1 root root  203K Feb  1 05:42 friendly_gagarin.log
-rw---. 1 root root  1.1K Feb  1 05:42 goofy_hypatia.log
-rw---. 1 root root  164K Feb  1 05:42 kind_chebyshev.log
-rw---. 1 root root   11K Feb  1 05:42 magical_archimedes.log
-rw---. 1 root root  373K Feb  1 05:42 busy_bardeen.log
-rw---. 1 root root  178K Feb  1 05:42 trusting_euler.log
-rw---. 1 root root  526K Feb  1 05:42 inspiring_golick.log
-rw---. 1 root root  369K Feb  1 06:12 condescending_ganguly.log
-rw---. 1 root root  191K Feb  1 06:12 mystifying_torvalds.log
-rw---. 1 root root  475K Feb  1 06:12 charming_nash.log
-rw---. 1 root root  168K Feb  1 06:12 zealous_sinoussi.log
-rw---. 1 root root  325K Feb  1 06:12 amazing_booth.log
-rw---. 1 root root  516K Feb  1 06:12 great_ardinghelli.log
-rw---. 1 root root  313K Feb  1 06:12 magical_bell.log
-rw---. 1 root root   22K Feb  1 06:12 nifty_swartz.log
-rw---. 1 root root   426 Feb  1 06:12 upbeat_beaver.log
-rw---. 1 root root  166K Feb  1 06:13 funny_lederberg.log
-rw---. 1 root root  164K Feb  1 06:13 frosty_murdock.log
-rw---. 1 root root  374K Feb  1 06:13 elastic_banach.log
-rw---. 1 root root  308K Feb  1 06:13 inspiring_cohen.log
-rw---. 1 root root  176K Feb  1 06:13 angry_wu.log
-rw---. 1 root root   662 Feb  1 06:42 admiring_kalam.log
-rw---. 1 root root  3.1K Feb  1 06:43 thirsty_colden.log
-rw---. 1 root root  4.5M Feb  1 07:01 run-parts.log
-rw---. 1 root root   16M Feb  1 07:01 CROND.log
-rw---. 1 root root  109M Feb  1 07:06 python3.log
-rw---. 1 root root  3.4M Feb  1 07:29 systemd-journald.log
-rw---. 1 root root  596M Feb  1 07:34 sudo.log
-rw---. 1 root root   549 Feb  1 07:44 interesting_rosalind.log
-rw---. 1 root root  342K Feb  1 07:45 beautiful_hamilton.log
-rw---. 1 root root  348K Feb  1 07:45 cool_ride.log
-rw---. 1 root root   15G Feb  1 07:45 conmon.log
-rw---. 1 root root  395K Feb  1 07:45 compassionate_satoshi.log
-rw---. 1 root root   11K Feb  1 07:45 hardcore_noether.log
-rw---. 1 root root  223K Feb  1 07:45 wizardly_johnson.log
-rw---. 1 root root  270M Feb  1 07:49 sshd.log
-rw---. 1 root root  111M Feb  1 07:49 systemd-logind.log
-rw---. 1 root root  1.6G Feb  1 07:50 systemd.log
-rw---. 1 root root  119M Feb  1 07:54 rsyslogd.log
-rw---. 1 root root   94G Feb  1 07:55 
ceph-8ee2d228-ed21-4580-8bbf-064.log

-rw---. 1 root root  1.1G Feb  1 07:56 podman.log
-rw---. 1 root root  1.8G Feb  1 07:58 ceph-mgr.log
-rw---. 1 root root  213G Feb  1 07:58 ceph-osd.log
-rw---. 1 root root   48G Feb  1 07:58 ceph-mon.log

"

Those are container names or something like that? The file content seems 
to be assorted bits from the ceph disk tool:


"
# cat goofy_hypatia.log
Jun  7 04:12:43 dopey goofy_hypatia[3224681]: 167 167
Jun 24 09:00:08 dopey goofy_hypatia[2319188]: --> passed data devices: 
22 physical, 0 LVM

Jun 24 09:00:08 dopey goofy_hypatia[2319188]: --> relative data size: 1.0
Jun 24 09:00:08 dopey goofy_hypatia[2319188]: --> All data devices are 
unavailable

Jun 24 09:00:08 dopey goofy_hypatia[2319188]: []
Sep 13 14:22:10 dopey goofy_hypatia[2027428]: --> passed data devices: 
22 physical, 0 LVM

Sep 13 14:22:10 dopey goofy_hypatia[2027428]: --> relative data size: 1.0
Sep 13 14:22:10 dopey goofy_hypatia[2027428]: --> All data devices are 
unavailable
2023-11-11T03:36:33+01:00 dopey goofy_hypatia[411330]: --> passed data 
devices: 2 physical, 0 LVM
2023-11-11T03:36:33+01:00 dopey goofy_hypatia[411330]: --> relative data 
size: 1.0
2023-11-11T03:36:33+01:00 dopey goofy_hypatia[411330]: --> All data 
devices are unavailable

2023-12-26T09:38:59+01:00 dopey goofy_hypatia[893177]: 167 167
2023-12-26T10:41:14+01:00 dopey goofy_hypatia[1057467]: 167 167
2024-01-05T21:00:37+01:00 dopey goofy_hypatia[3395314]: {}
2024-02-01T05:42:17+01:00 dopey goofy_hypatia[845150]: 167 167
"

Anyone else had this issue? Suggestions on how to get a real program 
name instead?


Mvh.

Torkil

--
Torkil Svensgaard
Sysadmin
MR-Forskningssektionen, afs. 714
DRCMR, Danish Research Centre for Magnetic Resonance
Hvidovre Hospital
Kettegård Allé 30
DK-2650 Hvidovre
Denmark
Tel: +45 386 22828
E-mail: tor...@drcmr.dk
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [EXTERN] Re: cephfs inode backtrace information

2024-01-31 Thread Dietmar Rieder


On 1/31/24 18:53, Gregory Farnum wrote:

The docs recommend a fast SSD pool for the CephFS *metadata*, but the
default data pool can be more flexible. The backtraces are relatively
small — it's an encoded version of the path an inode is located at,
plus the RADOS hobject, which is probably more of the space usage. So
it should fit fine in your SSD pool, but if all the cephfs file data
is living in the hard drive pool I'd just set it up there.


Right, I wrote *ssd* because our replicated pool is on SSD disks and the 
docs say:


 "If erasure-coded pools are planned for file system data, it is 
best to configure the default as a *replicated pool* to improve 
small-object write and read performance when updating backtraces..."


so of course replicated HDD pool would also fit there, I assume.

Well we plan to use the 85 TiB rep ssd pool for user homes (default 
pool) and the 3 PiB EC HDD pool for data, so most of the data will 
eventually be stored on the EC pool, so I was thinking if the need for 
storing all inode backtrace information on the default pool will 
substantially fill up the pool where also the user homes are going to end.


Thanks
  Dietmar



On Tue, Jan 30, 2024 at 2:03 AM Dietmar Rieder
 wrote:


Hello,

I have a question regarding the default pool of a cephfs.

According to the docs it is recommended to use a fast ssd replicated
pool as default pool for cephfs. I'm asking what are the space
requirements for storing the inode backtrace information?

Let's say I have a 85 TiB replicated ssd pool (hot data) and as 3 PiB EC
data pool (cold data).

Does it make sense to create a third pool as default pool which only
holds the inode backtrace information (what would be a good size), or is
it OK to use the ssd pool as default pool?

Thanks
 Dietmar

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io






OpenPGP_signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Changing A Ceph Cluster's Front- And/Or Back-End Networks IP Address(es)

2024-01-31 Thread Alexander E. Patrakov

Hi.

On Tue, Jan 30, 2024 at 5:24 PM duluxoz  wrote:
>
> Hi All,
>
> Quick Q: How easy/hard is it to change the IP networks of:
>
> 1) A Ceph Cluster's "Front-End" Network?

This is hard, but doable. The problem is that the MON database
contains the expected addresses of all MONs, and therefore, you cannot
just change them. What you can do is:

* Make sure that routing between the old network and the new one is functional
* Set at least "noout"
* Change the cluster config so that it lists two public networks (old and new)
* Remove one MON
* Adjust ceph.conf on all MONs to point it to the new MON address
* Add a new MON on the new network, wait for it to join the quorum
* Repeat the process for other MONs, one by one
* Do another rolling restart of all MONs with the correct ceph.conf,
just for good measure

Then change the network config and ceph.conf on OSD nodes and clients.

> 2) A Ceph Cluster's "Back-End" Network?

I have never tried this. In fact, any cluster that has a back-end
network not over the same physical interfaces as the front-end network
will not pass my audit. VLANs are OK. A failure of a back-end network
card on a single OSD would put the whole cluster to a halt.

The problem is described at
https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd/#flapping-osds

>
> Is it a "simply" matter of:
>
> a) Placing the Nodes in maintenance mode
>
> b) Changing a config file (I assume it's /etc/ceph/ceph.conf) on each Node
>
> c) Rebooting the Nodes
>
> d) Taking each Node out of Maintenance Mode
>
> Thanks in advance
>
> Cheers
>
> Dulux-Oz
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io



-- 
Alexander E. Patrakov
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

37 matches

Mail list logo