[ceph-users] Re: The snaptrim queue of PGs has not decreased for several days.
Quick update: we decided to switch to wpq to see if that would confirm our suspicion, and it did. After a few hours all PGs in the snaptrim queue had been processed. We haven't looked into the average object sizes yet, maybe we'll try that approach next week or so. If you have any other ideas, let us know. Zitat von Eugen Block : Hi, as expected the issue is not resolved and turned up again a couple of hours later. Here's the tracker issue: https://tracker.ceph.com/issues/67702 I also attached a log snippet from one osd with debug_osd 10 to the tracker. Let me know if you need anything else, I'll stay in touch with Giovanna. Thanks! Eugen Zitat von Sridhar Seshasayee : Hi Eugen, On Fri, Aug 23, 2024 at 1:37 PM Eugen Block wrote: Hi again, I have a couple of questions about this. What exactly happened to the PGs? They were queued for snaptrimming, but we didn't see any progress. Let's assume the average object size in that pool was around 2 MB (I don't have the actual numbers). Does that mean if osd_snap_trim_cost (1M default) was too low, those too large objects weren't trimmed? And then we split the PGs, reducing the average object size to 1 MB, these objects could be trimmed then, obviously. Does this explanation make sense? If you have the OSD logs, I can take a look and see why the snaptrim ops did not make progress. The cost is one contributing factor on the position of the op in the queue. Therefore, even though the cost incorrectly represents the actual average size of the objects in the PG, the op should be scheduled based on the set cost and the profile allocations. The OSDs appear to be NVMe based is what I understand from the thread. Based on the actions taken to resolve the situation (increased pg_num to 64), I think something else was up on the cluster. For NVMe based cluster, the current cost shouldn't cause stalling of the snaptrim ops. I'd suggest raising an upstream tracker with your observation and OSD logs to investigate this further. I just browsed through the changes, if I understand the fix correctly, the average object size is now calculated automatically, right? Which makes a lot of sense to me, as an operator I don't want to care too much about the average object sizes since ceph should know them better than me. ;-) Yes, that's correct. This fix was part of the effort to incrementally include background OSD operations to be scheduled by mClock. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: ceph orch host drain daemon type
Hi, What would be nice is if the drain command could take care of OSDs by default and drain all services only when called with a --remove-all-services flag or something similar. but that would mean that you wouldn't be able to drain only specific services, and OSDs would be drained either way. From my perspective, it would suffice to have the --daemon-type flag which is already present in 'ceph orch ps --daemon-type' command. You could either drain a specific daemon-type or drain the entire host (can be the default with the same behaviour as it currently works). That would allow more control about non-osd daemons. Zitat von Frédéric Nass : Hi Robert, Thanks for the suggestion. Unfortunately, removing the 'osds' label from a host does not remove the OSDs, unlike with other labels (mons, mgrs, mdss, nfss, crash, _admin, etc.). This is because this kind of service is tightly bound to the host and less 'portable' than other services, I think. This is likely the purpose of the drain command. What would be nice is if the drain command could take care of OSDs by default and drain all services only when called with a --remove-all-services flag or something similar. Frédéric - Le 30 Aoû 24, à 1:07, Robert W. Eckert r...@rob.eckert.name a écrit : If you are using cephadm, couldn't the host be removed from placing osds? On my cluster, I labeled the hosts for each service (OSD/MON/MGR/...) and have the services deployed by label. I believe that if you had that, then when a label is removed from the host the services eventually drain. -Original Message- From: Frédéric Nass Sent: Thursday, August 29, 2024 11:30 AM To: Eugen Block Cc: ceph-users ; dev Subject: [ceph-users] Re: ceph orch host drain daemon type Hello Eugen, A month back, while playing with a lab cluster, I drained a multi-service host (OSDs, MGR, MON, etc.) in order to recreate all of its OSDs. During this operation, all cephadm containers were removed as expected, including the MGR. As a result, I got into a situation where the orchestrator backend 'cephadm' was missing and wouldn't load anymore. I didn't have much time to investigate this, so I decided to recreate the lab cluster. But I think this is due to a bug. I would probably have avoided this situation if I had been able to ask the orchestrator to only drain services of type 'osd'. Makes sense. Cheers, Frédéric. - Le 27 Aoû 24, à 15:19, Eugen Block ebl...@nde.ag a écrit : Hi, is there anything on the road map to be able to choose a specific daemon type to be entirely removed from a host instead of all cephadm managed daemons? I just did a quick search in tracker and github, but it may be "hidden" somewhere else. I was thinking about colocated daemons on a host, for example MON, MGR, OSDs, node-exporter, crash. That's quite common, but if I just wanted to drain all OSDs (maybe mark them as "destroyed" in order to replace the drives), the usual 'ceph orch host drain ' command would remove all daemons. That seems unnecessary if I'm going to add the OSDs back. Since there are a couple of other daemon types which can be deployed multiple times per host, e. g. MDS, RGW, it doesn't only make sense for OSDs but for other daemons as well. And those other daemons usually have some cryptic suffix, we wouldn't need that in order to get rid of them, it doesn't save that much time, but it could be a nice enhancement. Should I create a tracker issue in the enhancement section for that? Thanks! Eugen ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: ceph orch host drain daemon type
Hi Eugen, I know, but removing other services is generally done by removing labels on hosts, isn't it? Whatever the way, another concern would be how to deal with _no_schedule and _no_conf_keyring labels when not draining all services on the host. Would it require per service type _no_schedule_labels? I don't know. That's where devs usually chime in. :-) Cheers, Frédéric - Le 30 Aoû 24, à 12:33, Eugen Block ebl...@nde.ag a écrit : > Hi, > >> What would be nice is if the drain command could take care of OSDs >> by default and drain all services only when called with a >> --remove-all-services flag or something similar. > > but that would mean that you wouldn't be able to drain only specific > services, and OSDs would be drained either way. From my perspective, > it would suffice to have the --daemon-type flag which is already > present in 'ceph orch ps --daemon-type' command. You could either > drain a specific daemon-type or drain the entire host (can be the > default with the same behaviour as it currently works). That would > allow more control about non-osd daemons. > > Zitat von Frédéric Nass : > >> Hi Robert, >> >> Thanks for the suggestion. Unfortunately, removing the 'osds' label >> from a host does not remove the OSDs, unlike with other labels >> (mons, mgrs, mdss, nfss, crash, _admin, etc.). This is because this >> kind of service is tightly bound to the host and less 'portable' >> than other services, I think. This is likely the purpose of the >> drain command. >> >> What would be nice is if the drain command could take care of OSDs >> by default and drain all services only when called with a >> --remove-all-services flag or something similar. >> >> Frédéric >> >> - Le 30 Aoû 24, à 1:07, Robert W. Eckert r...@rob.eckert.name a écrit : >> >>> If you are using cephadm, couldn't the host be removed from placing >>> osds? On my >>> cluster, I labeled the hosts for each service (OSD/MON/MGR/...) and have the >>> services deployed by label. I believe that if you had that, then >>> when a label >>> is removed from the host the services eventually drain. >>> >>> >>> >>> -Original Message- >>> From: Frédéric Nass >>> Sent: Thursday, August 29, 2024 11:30 AM >>> To: Eugen Block >>> Cc: ceph-users ; dev >>> Subject: [ceph-users] Re: ceph orch host drain daemon type >>> >>> Hello Eugen, >>> >>> A month back, while playing with a lab cluster, I drained a >>> multi-service host >>> (OSDs, MGR, MON, etc.) in order to recreate all of its OSDs. During this >>> operation, all cephadm containers were removed as expected, >>> including the MGR. >>> As a result, I got into a situation where the orchestrator backend 'cephadm' >>> was missing and wouldn't load anymore. I didn't have much time to >>> investigate >>> this, so I decided to recreate the lab cluster. But I think this is due to a >>> bug. >>> >>> I would probably have avoided this situation if I had been able to ask the >>> orchestrator to only drain services of type 'osd'. Makes sense. >>> >>> Cheers, >>> Frédéric. >>> >>> - Le 27 Aoû 24, à 15:19, Eugen Block ebl...@nde.ag a écrit : >>> Hi, is there anything on the road map to be able to choose a specific daemon type to be entirely removed from a host instead of all cephadm managed daemons? I just did a quick search in tracker and github, but it may be "hidden" somewhere else. I was thinking about colocated daemons on a host, for example MON, MGR, OSDs, node-exporter, crash. That's quite common, but if I just wanted to drain all OSDs (maybe mark them as "destroyed" in order to replace the drives), the usual 'ceph orch host drain ' command would remove all daemons. That seems unnecessary if I'm going to add the OSDs back. Since there are a couple of other daemon types which can be deployed multiple times per host, e. g. MDS, RGW, it doesn't only make sense for OSDs but for other daemons as well. And those other daemons usually have some cryptic suffix, we wouldn't need that in order to get rid of them, it doesn't save that much time, but it could be a nice enhancement. Should I create a tracker issue in the enhancement section for that? Thanks! Eugen ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io >>> ___ >>> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send >>> an email to > >> ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: ceph orch host drain daemon type
The label removal approach is great, but it still doesn't allow you to drain only OSDs, no other daemons. I didn't think about the other labels, that's a good point. Let's see what the devs have to say. :-) Zitat von Frédéric Nass : Hi Eugen, I know, but removing other services is generally done by removing labels on hosts, isn't it? Whatever the way, another concern would be how to deal with _no_schedule and _no_conf_keyring labels when not draining all services on the host. Would it require per service type _no_schedule_labels? I don't know. That's where devs usually chime in. :-) Cheers, Frédéric - Le 30 Aoû 24, à 12:33, Eugen Block ebl...@nde.ag a écrit : Hi, What would be nice is if the drain command could take care of OSDs by default and drain all services only when called with a --remove-all-services flag or something similar. but that would mean that you wouldn't be able to drain only specific services, and OSDs would be drained either way. From my perspective, it would suffice to have the --daemon-type flag which is already present in 'ceph orch ps --daemon-type' command. You could either drain a specific daemon-type or drain the entire host (can be the default with the same behaviour as it currently works). That would allow more control about non-osd daemons. Zitat von Frédéric Nass : Hi Robert, Thanks for the suggestion. Unfortunately, removing the 'osds' label from a host does not remove the OSDs, unlike with other labels (mons, mgrs, mdss, nfss, crash, _admin, etc.). This is because this kind of service is tightly bound to the host and less 'portable' than other services, I think. This is likely the purpose of the drain command. What would be nice is if the drain command could take care of OSDs by default and drain all services only when called with a --remove-all-services flag or something similar. Frédéric - Le 30 Aoû 24, à 1:07, Robert W. Eckert r...@rob.eckert.name a écrit : If you are using cephadm, couldn't the host be removed from placing osds? On my cluster, I labeled the hosts for each service (OSD/MON/MGR/...) and have the services deployed by label. I believe that if you had that, then when a label is removed from the host the services eventually drain. -Original Message- From: Frédéric Nass Sent: Thursday, August 29, 2024 11:30 AM To: Eugen Block Cc: ceph-users ; dev Subject: [ceph-users] Re: ceph orch host drain daemon type Hello Eugen, A month back, while playing with a lab cluster, I drained a multi-service host (OSDs, MGR, MON, etc.) in order to recreate all of its OSDs. During this operation, all cephadm containers were removed as expected, including the MGR. As a result, I got into a situation where the orchestrator backend 'cephadm' was missing and wouldn't load anymore. I didn't have much time to investigate this, so I decided to recreate the lab cluster. But I think this is due to a bug. I would probably have avoided this situation if I had been able to ask the orchestrator to only drain services of type 'osd'. Makes sense. Cheers, Frédéric. - Le 27 Aoû 24, à 15:19, Eugen Block ebl...@nde.ag a écrit : Hi, is there anything on the road map to be able to choose a specific daemon type to be entirely removed from a host instead of all cephadm managed daemons? I just did a quick search in tracker and github, but it may be "hidden" somewhere else. I was thinking about colocated daemons on a host, for example MON, MGR, OSDs, node-exporter, crash. That's quite common, but if I just wanted to drain all OSDs (maybe mark them as "destroyed" in order to replace the drives), the usual 'ceph orch host drain ' command would remove all daemons. That seems unnecessary if I'm going to add the OSDs back. Since there are a couple of other daemon types which can be deployed multiple times per host, e. g. MDS, RGW, it doesn't only make sense for OSDs but for other daemons as well. And those other daemons usually have some cryptic suffix, we wouldn't need that in order to get rid of them, it doesn't save that much time, but it could be a nice enhancement. Should I create a tracker issue in the enhancement section for that? Thanks! Eugen ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to >> ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] MDS cache always increasing
I hope someone can help us with a MDS caching problem. Ceph version 18.2.4 with cephadm container deployment. Question 1: For me it's not clear how much cache/memory you should allocate for the MDS. Is this based on the number of open files, caps or something else? Question 2/Problem: At the moment we have MDS nodes with 32 GB of memory and a configured cache limit of 20 GB. There are 4 MDS nodes: 2 active and 2 in standby-replay mode (with max_mds set at 2 of course). We pinned top directories to specific ranks, so the balancer isn't used. The memory usage is for the most part increasing, sometimes a little dip with couple hundred MB's freed. After all the memory is consumed, SWAP gets used. This results in a couple of hundred MB's of freed memory, but not much. When eventually the SWAP runs out and the memory is full, the MDS service stops and the cluster logs show: 1. no beacon from mds 2. marking mds up:active laggy 3. replacing mds 4. MDS daemon is removed because it is dead or otherwise unavailable For example: we have the top folder app2 and app4 which is pinnend to rank 1. Folder app2 is always accessed by 4 clients (application servers), the same happens with folder app4. Folder app2 is 3 times larger than folder app4 (last time I checked, don't wanna do a du at the moment). After a couple of hours the memory usage of the MDS server stays around 18% (Grafana shows a flatline for 7 hours). At night the 9the client connects and makes first a backup with rsync of the latest snapshot folder of app2 and afterwards the same happens for folder app4 with a pause for 5 minutes. When the backup starts, the memory increases to 70% and stays at 70% after the backup of app2 is completed. 5 minutes later the memory starts increases again with the start of the backup of folder app4. When the backup is done, it's at 78% and stays there for the rest of the day. Why isn't the memory usage decreasing after the rsync is completed? Is there a memory leak with the MDS service? Ps. I have some small log files/Grafana screenshots, not sure how to share. Kind regards, Sake ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: How to know is ceph is ready?
steps 2 to 4 are exactly my idea. In step 1 All I will check is all active (o non inactive or unknown). Clean is not necessary, I can boot VMs with degraded.as long they are active. El jue, 29 ago 2024 a las 8:03, Bogdan Adrian Velica () escribió: > Hi, > > It's a hacky idea but create a script that checks if the Ceph RBD pool is > fully "active+clean" to ensure it's ready before starting Proxmox VMs. > Something like this... > > 1. Bash Script: > - Write a script that checks if the Ceph RBD pool is in the "active+clean" > state using ceph pg stat or ceph -s. > - The script should run until the pool is ready before exiting. A loop or > something... > > 2. Systemd Service for Ceph Readiness: > - Create a systemd service unit file (something like: ceph-ready.service) > that runs the script at startup. > - Ensure the service waits until the Ceph cluster is ready before > proceeding. > > 3. Systemd Service for Proxmox VMs: > - Modify an existing Proxmox service or create a new one to depend on the > ceph-ready.service. > - Use After=ceph-ready.service in the unit file to ensure Proxmox VMs > start only after Ceph is ready. > > 4. Enable and Start the Services: > - Enable both systemd services to start at boot with systemctl enable. > - Reboot to ensure that Proxmox waits for Ceph before starting VMs. > > Just an idea... > > Thank you, > Bogdan Velica > croit.io > > On Thu, Aug 29, 2024 at 1:53 PM Alfredo Rezinovsky > wrote: > >> I have a proxmox cluster using an external CEPH cluster. >> >> Sometimes due to blackouts the servers need to restart. If proxmox starts >> before CEPH is ready the VMs fail to boot. >> >> I want to add a dependency in proxmox to wait for ceph to be ready. >> >> I can work with a HEALTH_WARN as long the RBD pool is usable. >> >> ceph status exit status doesn´t helps >> >> Should I grep for "pgs not active" in ceph status or for "inactive" pgs in >> ceph health or is there something more direct to know if everything is >> alright? >> >> >> >> -- >> Alfrenovsky >> ___ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-le...@ceph.io >> > -- Alfrenovsky ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] squid 19.2.0 QE validation status
Details of this release are summarized here: https://tracker.ceph.com/issues/67779#note-1 Release Notes - TBD Gibba upgrade -TBD LRC upgrade - TBD It was decided and agreed upon that there would be limited testing for this release, given it is based on 19.1.1 rather than a full rebase. Seeking approvals/reviews for: (some reruns are still in progress) rgw - Eric, Adam E fs - Venky orch - Adam King quincy-x - Laura, Neha reef-x - Laura, Neha Pls let me know if any tests were missed from this list. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] ceph-ansible installation error
Dear team , I configuring ceph cluster using ceph-ansible , ubuntu OS 20.04 . my previous production cluster have been configured using the same configuration and it is working perfectly , now I am trying to build the new cluster using the same configurations , but I am facing the following errors: ansible-playbook site.yml Traceback (most recent call last): File "/usr/bin/ansible-playbook", line 66, in from ansible.utils.display import Display, initialize_locale ImportError: cannot import name 'initialize_locale' from 'ansible.utils.display' (/usr/local/lib/python3.8/dist-packages/ansible/utils/display.py) Can someone help to solve the issue? Best regards Michel ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: ceph-ansible installation error
I believe that the original Ansible installation process is deprecated. It was pretty messy, anyway, since it had to do a lot of grunt work. Likewise the ceph-install program, which is in the Octopus docs, but wasn't actually available in the release of Octopus I installed on my servers. The Ansible playbooks also assumed things about the environment that didn't fit my setup, so I never used them. The good news is that unless you're determined to install a pre-Octopus release of Ceph or want the legacy setup, the cephadm application makes life considerably simpler. In fact, I rolled my own playbooks and all it contains are the following steps: 1. Install the ceph repository that corresponds to the release of Ceph you want. 2. Install the cephadm package from that repository. Optionally you can install ceph-common if you don't want to be restricted to doing all your work via "cephadm shell". 3. Copy your ceph config files to the targer machine. Now when you first run cephadm it will download and install the docker/podman container image that is shared between most of the Ceph daemons and after that, you're pretty much good to go. Image installation might not get done until you attempt something such as creating an OSD on that machine, but it will be done when needed. As a final note, even in the latest Ceph releases prefer you to the old (legecy) "Manual procedure". IGNORE THAT Lock for the cephadm equivalents. They're shy, but a little searching will show them. In other words, you don't really need a ready-made playbook now. You can roll your own like I did, but that's just a convenience. Tim On Fri, 2024-08-30 at 17:53 +0300, Michel Niyoyita wrote: > Dear team , > > > I configuring ceph cluster using ceph-ansible , ubuntu OS 20.04 . my > previous production cluster have been configured using the same > configuration and it is working perfectly , now I am trying to build > the > new cluster using the same configurations , but I am facing the > following errors: > > ansible-playbook site.yml > Traceback (most recent call last): > File "/usr/bin/ansible-playbook", line 66, in > from ansible.utils.display import Display, initialize_locale > ImportError: cannot import name 'initialize_locale' from > 'ansible.utils.display' > (/usr/local/lib/python3.8/dist-packages/ansible/utils/display.py) > > Can someone help to solve the issue? > > Best regards > > Michel > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: How to know is ceph is ready?
"Ceph is ready" covers a lot of territory. It's more like, "How can I delay util Ceph is available for the particular service I need? I've been taking a systemd-bsaed approach. Since I don't actually care bout Ceph in the abstract, but I'm actually looking for the Ceph or Ceph NFS shares, I create dependencies in my systemd setup such that the dependent VM or container (quadlet) requires its ceph mounts to be online before that service can start. I'm still evaluating how well that works. I'll let you know after the next hurricane hits. But so far, so good. Tim On Fri, 2024-08-30 at 11:02 -0300, Alfredo Rezinovsky wrote: > steps 2 to 4 are exactly my idea. > > In step 1 All I will check is all active (o non inactive or unknown). > Clean > is not necessary, I can boot VMs with degraded.as long they are > active. > > El jue, 29 ago 2024 a las 8:03, Bogdan Adrian Velica > () > escribió: > > > Hi, > > > > It's a hacky idea but create a script that checks if the Ceph RBD > > pool is > > fully "active+clean" to ensure it's ready before starting Proxmox > > VMs. > > Something like this... > > > > 1. Bash Script: > > - Write a script that checks if the Ceph RBD pool is in the > > "active+clean" > > state using ceph pg stat or ceph -s. > > - The script should run until the pool is ready before exiting. A > > loop or > > something... > > > > 2. Systemd Service for Ceph Readiness: > > - Create a systemd service unit file (something like: ceph- > > ready.service) > > that runs the script at startup. > > - Ensure the service waits until the Ceph cluster is ready before > > proceeding. > > > > 3. Systemd Service for Proxmox VMs: > > - Modify an existing Proxmox service or create a new one to depend > > on the > > ceph-ready.service. > > - Use After=ceph-ready.service in the unit file to ensure Proxmox > > VMs > > start only after Ceph is ready. > > > > 4. Enable and Start the Services: > > - Enable both systemd services to start at boot with systemctl > > enable. > > - Reboot to ensure that Proxmox waits for Ceph before starting VMs. > > > > Just an idea... > > > > Thank you, > > Bogdan Velica > > croit.io > > > > On Thu, Aug 29, 2024 at 1:53 PM Alfredo Rezinovsky > > > > wrote: > > > > > I have a proxmox cluster using an external CEPH cluster. > > > > > > Sometimes due to blackouts the servers need to restart. If > > > proxmox starts > > > before CEPH is ready the VMs fail to boot. > > > > > > I want to add a dependency in proxmox to wait for ceph to be > > > ready. > > > > > > I can work with a HEALTH_WARN as long the RBD pool is usable. > > > > > > ceph status exit status doesn´t helps > > > > > > Should I grep for "pgs not active" in ceph status or for > > > "inactive" pgs in > > > ceph health or is there something more direct to know if > > > everything is > > > alright? > > > > > > > > > > > > -- > > > Alfrenovsky > > > ___ > > > ceph-users mailing list -- ceph-users@ceph.io > > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > > > > > -- > Alfrenovsky > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: ceph-ansible installation error
Hi Michel, This is likely related to your ansible installation or system locale configuration. Try to gather more info. E.g.: which ansible ansible --version locale locale -a Also check if you run matching versions of ansible, ceph-ansible and ceph as is listed in Releases section at https://docs.ceph.com/projects/ceph-ansible/en/latest/ Milan -- Milan Kupcevic Research Computing Lead Storage Engineer Harvard University HUIT, University Research Computing On 8/30/24 10:53, Michel Niyoyita wrote: Dear team , I configuring ceph cluster using ceph-ansible , ubuntu OS 20.04 . my previous production cluster have been configured using the same configuration and it is working perfectly , now I am trying to build the new cluster using the same configurations , but I am facing the following errors: ansible-playbook site.yml Traceback (most recent call last): File "/usr/bin/ansible-playbook", line 66, in from ansible.utils.display import Display, initialize_locale ImportError: cannot import name 'initialize_locale' from 'ansible.utils.display' (/usr/local/lib/python3.8/dist-packages/ansible/utils/display.py) Can someone help to solve the issue? Best regards Michel ___ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: ceph-ansible installation error
On 8/30/24 12:38, Tim Holloway wrote: I believe that the original Ansible installation process is deprecated. This would be a bad news as I repeatedly hear from admins running large storage deployments that they prefer to stay away from containers. Milan -- Milan Kupcevic Research Computing Lead Storage Engineer Harvard University HUIT, University Research Computing ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: MDS cache always increasing
On Fri, Aug 30, 2024 at 9:22 PM Sake Ceph wrote: > > I hope someone can help us with a MDS caching problem. > > Ceph version 18.2.4 with cephadm container deployment. > > Question 1: > For me it's not clear how much cache/memory you should allocate for the MDS. > Is this based on the number of open files, caps or something else? > > Question 2/Problem: > At the moment we have MDS nodes with 32 GB of memory and a configured cache > limit of 20 GB. There are 4 MDS nodes: 2 active and 2 in standby-replay mode > (with max_mds set at 2 of course). We pinned top directories to specific > ranks, so the balancer isn't used. > The memory usage is for the most part increasing, sometimes a little dip with > couple hundred MB's freed. After all the memory is consumed, SWAP gets used. > This results in a couple of hundred MB's of freed memory, but not much. When > eventually the SWAP runs out and the memory is full, the MDS service stops > and the cluster logs show: > 1. no beacon from mds > 2. marking mds up:active laggy > 3. replacing mds > 4. MDS daemon is removed because it is dead or otherwise unavailable > > For example: we have the top folder app2 and app4 which is pinnend to rank 1. > Folder app2 is always accessed by 4 clients (application servers), the same > happens with folder app4. Folder app2 is 3 times larger than folder app4 > (last time I checked, don't wanna do a du at the moment). > After a couple of hours the memory usage of the MDS server stays around 18% > (Grafana shows a flatline for 7 hours). > At night the 9the client connects and makes first a backup with rsync of the > latest snapshot folder of app2 and afterwards the same happens for folder > app4 with a pause for 5 minutes. > When the backup starts, the memory increases to 70% and stays at 70% after > the backup of app2 is completed. 5 minutes later the memory starts increases > again with the start of the backup of folder app4. When the backup is done, > it's at 78% and stays there for the rest of the day. > Why isn't the memory usage decreasing after the rsync is completed? > > Is there a memory leak with the MDS service? > > Ps. I have some small log files/Grafana screenshots, not sure how to share. > > Kind regards, > Sake > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io Hello Sake, There was a known memory leak in MDS standby-replay: https://tracker.ceph.com/issues/48673. However, it is supposedly fixed in version 18.2.4, which you are running. I don't have any non-test cluster on Reef yet, and so can't tell for sure if this is true. Therefore, I guess what you see might be a different issue. Could you please disable standby-replay and retest whether the memory leak still exists? If so, it would be sufficient proof that it is not the same issue. ceph fs set cephfs allow_standby_replay 0 -- Alexander Patrakov ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: MDS cache always increasing
@Anthony: it's a small virtualized cluster and indeed SWAP shouldn't be used, but this doesn't change the problem. @Alexander: the problem is in the active nodes, the standby replay don't have issues anymore. Last night's backup run increased the memory usage to 86% when rsync was running for app2. It dropped to 77,8% when it was done. When the rsync for app4 was running it increased to 84% and dropping to 80%. After a few hours it's now settled on 82%. It looks to me the MDS server is caching something forever while it isn't being used.. The underlying host is running on RHEL 8. Upgrade to RHEL 9 is planned, but hit some issues with automatically upgrading hosts. Kind regards, Sake ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: MDS cache always increasing
Hello Sake, The combination of two active MDSs and RHEL8 does ring a bell, and I have seen this with Quincy, too. However, what's relevant is the kernel version on the clients. If they run the default 4.18.x kernel from RHEL8, please either upgrade to the mainline kernel or decrease max_mds to 1. If they run a modern kernel, then it is something I do not know about. On Sat, Aug 31, 2024 at 1:21 PM Sake Ceph wrote: > > @Anthony: it's a small virtualized cluster and indeed SWAP shouldn't be used, > but this doesn't change the problem. > > @Alexander: the problem is in the active nodes, the standby replay don't have > issues anymore. > > Last night's backup run increased the memory usage to 86% when rsync was > running for app2. It dropped to 77,8% when it was done. When the rsync for > app4 was running it increased to 84% and dropping to 80%. After a few hours > it's now settled on 82%. > It looks to me the MDS server is caching something forever while it isn't > being used.. > > The underlying host is running on RHEL 8. Upgrade to RHEL 9 is planned, but > hit some issues with automatically upgrading hosts. > > Kind regards, > Sake > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io -- Alexander Patrakov ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: MDS cache always increasing
I was talking about the hosts where the MDS containers are running on. The clients are all RHEL 9. Kind regards, Sake > Op 31-08-2024 08:34 CEST schreef Alexander Patrakov : > > > Hello Sake, > > The combination of two active MDSs and RHEL8 does ring a bell, and I > have seen this with Quincy, too. However, what's relevant is the > kernel version on the clients. If they run the default 4.18.x kernel > from RHEL8, please either upgrade to the mainline kernel or decrease > max_mds to 1. If they run a modern kernel, then it is something I do > not know about. > > On Sat, Aug 31, 2024 at 1:21 PM Sake Ceph wrote: > > > > @Anthony: it's a small virtualized cluster and indeed SWAP shouldn't be > > used, but this doesn't change the problem. > > > > @Alexander: the problem is in the active nodes, the standby replay don't > > have issues anymore. > > > > Last night's backup run increased the memory usage to 86% when rsync was > > running for app2. It dropped to 77,8% when it was done. When the rsync for > > app4 was running it increased to 84% and dropping to 80%. After a few hours > > it's now settled on 82%. > > It looks to me the MDS server is caching something forever while it isn't > > being used.. > > > > The underlying host is running on RHEL 8. Upgrade to RHEL 9 is planned, but > > hit some issues with automatically upgrading hosts. > > > > Kind regards, > > Sake > > ___ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > > > > -- > Alexander Patrakov ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
