[ceph-users] Re: upgrade problem nautilus 14.2.15 -> 14.2.18? (Broken ceph!)
On 25/03/2021 21:08, Simon Oosthoek wrote: > > I'll wait a bit before upgrading the remaining nodes. I hope 14.2.19 > will be available quickly. > Hi Dan, Just FYI, I upgraded the cluster this week to 14.2.19 and all systems are good now. I've removed the workaround configuration in the /etc/ceph/ceph.conf again. Thanks for the quick help at the time! Cheers /Simon ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Version of podman for Ceph 15.2.10
Thank you for confirming that podman 3.0.1 is fine.
I have now bootstrapped my first node with cephadm bootstrap command on my
RasPi 4 (8GB RAM) with Ubuntu 20.04 LTS that worked well but in the logs I can
see that it fails to deploy the graphana container as you can see from the log
below:
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/module.py", line 1021, in _remote_connection
yield (conn, connr)
File "/usr/share/ceph/mgr/cephadm/module.py", line 1168, in _run_cephadm
code, '\n'.join(err)))
orchestrator._interface.OrchestratorError: cephadm exited with an error code:
1, stderr:Deploy daemon grafana.ceph1a ...
Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --net=host
--entrypoint stat -e CONTAINER_IMAGE=docker.io/ceph/ceph-grafana:6.7.4 -e
NODE_NAME=ceph1a docker.io/ceph/ceph-grafana:6.7.4 -c %u %g /var/lib/grafana
stat: stderr {"msg":"exec container process `/usr/bin/stat`: Exec format
error","level":"error","time":"2021-04-09T06:17:54.000910863Z"}
Traceback (most recent call last):
File "", line 6153, in
File "", line 1412, in _default_image
File "", line 3431, in command_deploy
File "", line 3362, in extract_uid_gid_monitoring
File "", line 2099, in extract_uid_gid
RuntimeError: uid/gid not found
Does anyone have a clue what could be going wrong here?? and how to fix that?
Right now my bootstraped node has the following containers running:
$ sudo podman ps
CONTAINER ID IMAGE COMMAND
CREATED STATUS PORTS NAMES
5985c38ef718 docker.io/ceph/ceph:v15 -n mon.ceph1a -f ... 15
hours ago Up 15 hours ago
ceph-8d47792c-987d-11eb-9bb6-a5302e00e1fa-mon.ceph1a
0773ea8c6951 docker.io/ceph/ceph:v15 -n mgr.ceph1a.rzc... 15
hours ago Up 15 hours ago
ceph-8d47792c-987d-11eb-9bb6-a5302e00e1fa-mgr.ceph1a.rzcwjd
941db20cdc2e docker.io/ceph/ceph:v15 -n client.crash.c... 15
hours ago Up 15 hours ago
ceph-8d47792c-987d-11eb-9bb6-a5302e00e1fa-crash.ceph1a
897286d3c80f docker.io/prom/node-exporter:v0.18.1 --no-collector.ti... 15
hours ago Up 15 hours ago
ceph-8d47792c-987d-11eb-9bb6-a5302e00e1fa-node-exporter.ceph1a
08c4e95c0c03 docker.io/prom/prometheus:v2.18.1 --config.file=/et... 15
hours ago Up 15 hours ago
ceph-8d47792c-987d-11eb-9bb6-a5302e00e1fa-prometheus.ceph1a
19944dbf7a63 docker.io/prom/alertmanager:v0.20.0 --web.listen-addr... 15
hours ago Up 15 hours ago
ceph-8d47792c-987d-11eb-9bb6-a5302e00e1fa-alertmanager.ceph1a
‐‐‐ Original Message ‐‐‐
On Friday, April 9, 2021 3:37 AM, David Orman wrote:
> The latest podman 3.0.1 release is fine (we have many production clusters
> running this). We have not tested 3.1 yet, however, but will soon.
>
> > On Apr 8, 2021, at 10:32, mabi m...@protonmail.ch wrote:
> > Hello,
> > I would like to install Ceph 15.2.10 using cephadm and just found the
> > following table by checking the requirements on the host:
> > https://docs.ceph.com/en/latest/cephadm/compatibility/#compatibility-with-podman-versions
> > Do I understand this table correctly that I should be using podman version
> > 2.1?
> > and what happens if I use the latest podman version 3.0
> > Best regards,
> > Mabi
> >
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Version of podman for Ceph 15.2.10
That container (ceph-grafana) is not built for ARM based processors,
only AMD64:
https://hub.docker.com/r/ceph/ceph-grafana/tags?page=1&ordering=last_updated
. You'll probably need to disable that (I think it's part of the
dashboard module - I don't know - we run our own Prometheus/Grafana
infrastructure outside of Ceph).
On Fri, Apr 9, 2021 at 1:32 AM mabi wrote:
>
> Thank you for confirming that podman 3.0.1 is fine.
>
> I have now bootstrapped my first node with cephadm bootstrap command on my
> RasPi 4 (8GB RAM) with Ubuntu 20.04 LTS that worked well but in the logs I
> can see that it fails to deploy the graphana container as you can see from
> the log below:
>
> Traceback (most recent call last):
> File "/usr/share/ceph/mgr/cephadm/module.py", line 1021, in
> _remote_connection
> yield (conn, connr)
> File "/usr/share/ceph/mgr/cephadm/module.py", line 1168, in _run_cephadm
> code, '\n'.join(err)))
> orchestrator._interface.OrchestratorError: cephadm exited with an error code:
> 1, stderr:Deploy daemon grafana.ceph1a ...
> Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --net=host
> --entrypoint stat -e CONTAINER_IMAGE=docker.io/ceph/ceph-grafana:6.7.4 -e
> NODE_NAME=ceph1a docker.io/ceph/ceph-grafana:6.7.4 -c %u %g /var/lib/grafana
> stat: stderr {"msg":"exec container process `/usr/bin/stat`: Exec format
> error","level":"error","time":"2021-04-09T06:17:54.000910863Z"}
> Traceback (most recent call last):
> File "", line 6153, in
> File "", line 1412, in _default_image
> File "", line 3431, in command_deploy
> File "", line 3362, in extract_uid_gid_monitoring
> File "", line 2099, in extract_uid_gid
> RuntimeError: uid/gid not found
>
> Does anyone have a clue what could be going wrong here?? and how to fix that?
>
> Right now my bootstraped node has the following containers running:
>
> $ sudo podman ps
> CONTAINER ID IMAGE COMMAND
> CREATED STATUS PORTS NAMES
> 5985c38ef718 docker.io/ceph/ceph:v15 -n mon.ceph1a -f ... 15
> hours ago Up 15 hours ago
> ceph-8d47792c-987d-11eb-9bb6-a5302e00e1fa-mon.ceph1a
> 0773ea8c6951 docker.io/ceph/ceph:v15 -n mgr.ceph1a.rzc... 15
> hours ago Up 15 hours ago
> ceph-8d47792c-987d-11eb-9bb6-a5302e00e1fa-mgr.ceph1a.rzcwjd
> 941db20cdc2e docker.io/ceph/ceph:v15 -n client.crash.c... 15
> hours ago Up 15 hours ago
> ceph-8d47792c-987d-11eb-9bb6-a5302e00e1fa-crash.ceph1a
> 897286d3c80f docker.io/prom/node-exporter:v0.18.1 --no-collector.ti... 15
> hours ago Up 15 hours ago
> ceph-8d47792c-987d-11eb-9bb6-a5302e00e1fa-node-exporter.ceph1a
> 08c4e95c0c03 docker.io/prom/prometheus:v2.18.1 --config.file=/et... 15
> hours ago Up 15 hours ago
> ceph-8d47792c-987d-11eb-9bb6-a5302e00e1fa-prometheus.ceph1a
> 19944dbf7a63 docker.io/prom/alertmanager:v0.20.0 --web.listen-addr... 15
> hours ago Up 15 hours ago
> ceph-8d47792c-987d-11eb-9bb6-a5302e00e1fa-alertmanager.ceph1a
>
>
>
> ‐‐‐ Original Message ‐‐‐
> On Friday, April 9, 2021 3:37 AM, David Orman wrote:
>
> > The latest podman 3.0.1 release is fine (we have many production clusters
> > running this). We have not tested 3.1 yet, however, but will soon.
> >
> > > On Apr 8, 2021, at 10:32, mabi m...@protonmail.ch wrote:
> > > Hello,
> > > I would like to install Ceph 15.2.10 using cephadm and just found the
> > > following table by checking the requirements on the host:
> > > https://docs.ceph.com/en/latest/cephadm/compatibility/#compatibility-with-podman-versions
> > > Do I understand this table correctly that I should be using podman
> > > version 2.1?
> > > and what happens if I use the latest podman version 3.0
> > > Best regards,
> > > Mabi
> > >
> > > ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Abandon incomplete (damaged EC) pgs - How to manage the impact on cephfs?
Hi Joshua,
I'll dig into this output a bit more later, but here are my thoughts
right now. I'll preface this by saying that I've never had to clean up
from unrecoverable incomplete PGs, so some of what I suggest may not
work/apply or be the ideal fix in your case.
Correct me if I'm wrong, but you are willing to throw away all of the
data on this pool? This should make it easier because we don't have to
worry about recovering any lost data.
If this is the case, then I think the general strategy would be:
1) Identify and remove any files/directories in cephfs that are located
on this pool (based on ceph.file.layout.pool=claypool and
ceph.dir.layout.pool=claypool). Use 'unlink' instead of 'rm' to remove
the files; it should be less prone to hanging.
2) Wait a bit for ceph to clean up any unreferenced objects. Watch the
output of 'ceph df' to see how many objects are listed for the pool.
3) Use 'rados -p claypool ls' to identify the remaining objects. Use
the OID identifier to calculate the inode number of each file, then
search cephfs to identify which files these belong to. I would expect
it would be none, as you already deleted the files in step 1.
4) With nothing in the cephfs metadata referring to the objects anymore,
it should be safe to remove them with 'rados -p rm'.
5) Remove the now-empty pool from cephfs
6) Remove the now-empty pool from ceph
Can you also include the output of 'ceph df'?
--Mike
On 4/9/21 7:31 AM, Joshua West wrote:
Thank you Mike!
This is honestly a way more detailed reply than I was expecting.
You've equipped me with new tools to work with. Thank you!
I don't actually have any unfound pgs... only "incomplete" ones, which
limits the usefulness of:
`grep recovery_unfound`
`ceph pg $pg list_unfound`
`ceph pg $pg mark_unfound_lost delete`
I don't seem to see equivalent commands for incomplete pgs, save for
grep of course.
This does make me slightly more hopeful that recovery might be
possible if the pgs are incomplete and stuck, but not unfound..? Not
going to get my hopes too high.
Going to attach a few items just to keep from bugging me, if anyone
can take a glance, it would be appreciated.
In the meantime, in the absence of the above commands, what's the best
way to clean this up under the assumption that the data is lost?
~Joshua
Joshua West
President
403-456-0072
CAYK.ca
On Thu, Apr 8, 2021 at 6:15 PM Michael Thomas wrote:
Hi Joshua,
I have had a similar issue three different times on one of my cephfs
pools (15.2.10). The first time this happened I had lost some OSDs. In
all cases I ended up with degraded PGs with unfound objects that could
not be recovered.
Here's how I recovered from the situation. Note that this will
permanently remove the affected files from ceph. Restoring them from
backup is an excercise left to the reader.
* Make a list of the affected PGs:
ceph pg dump_stuck | grep recovery_unfound > pg.txt
* Make a list of the affected objects (OIDs):
cat pg.txt | awk '{print $1}' | while read pg ; do echo $pg ; ceph pg
$pg list_unfound | jq '.objects[].oid.oid' ; done | sed -e 's/"//g' >
oid.txt
* Convert the OID numbers to inodes using 'printf "%d\n" 0x${oid}' and
put the results in a file called 'inum.txt'
* On a ceph client, find the files that correspond to the affected inodes:
cat inum.txt | while read inum ; do echo -n "${inum} " ; find
/ceph/frames/O3/raw -inum ${inum} ; done > files.txt
* It may be helpful to put this table of PG, OID, inum, and files into a
spreadsheet to keep track of what's been done.
* On the ceph client, use 'unlink' to remove the files from the
filesystem. Do not use 'rm', as it will hang while calling 'stat()' on
each file. Even unlink may hang when you first try it. If it does
hang, do the following to get it unstuck:
- Reboot the client
- Restart each mon and the mgr. I rebooted each mon/mgr, but it may
be sufficient to restart the services without a reboot.
- Try using 'unlink' again
* After all of the affected files have been removed, go through the list
of PGs and remove the unfound OIDs:
ceph pg $pgid mark_unfound_lost delete
...or if you're feeling brave, delete them all at once:
cat pg.txt | awk '{print $1}' | while read pg ; do echo $pg ; ceph pg
$pg mark_unfound_lost delete ; done
* Watch the output of 'ceph -s' to see the health of the pools/pgs recover.
* Restore the deleted files from backup, or decide that you don't care
about them and don't do anything
This procedure lets you fix the problem without deleting the affected
pool. To be honest, the first time it happened, my solution was to
first copy all of the data off of the affected pool and onto a new pool.
I later found this to be unnecessary. But if you want to pursue this,
here's what I suggest:
* Follow the steps above to get rid of the affected files. I feel this
should still be done even though you don't care about saving the data,
to prevent corruption in
[ceph-users] Re: Nautilus 14.2.19 mon 100% CPU
I'm attempting to deep scrub all the PGs to see if that helps clear up some accounting issues, but that's going to take a really long time on 2PB of data. Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Thu, Apr 8, 2021 at 9:48 PM Robert LeBlanc wrote: > > Good thought. The storage for the monitor data is a RAID-0 over three > NVMe devices. Watching iostat, they are completely idle, maybe 0.8% to > 1.4% for a second every minute or so. > > Robert LeBlanc > PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 > > On Thu, Apr 8, 2021 at 7:48 PM Zizon Qiu wrote: > > > > Will it be related to some kind of disk issue of that mon located in,which > > may casually > > slow down IO and further the rocksdb? > > > > > > On Fri, Apr 9, 2021 at 4:29 AM Robert LeBlanc wrote: > >> > >> I found this thread that matches a lot of what I'm seeing. I see the > >> ms_dispatch thread going to 100%, but I'm at a single MON, the > >> recovery is done and the rocksdb MON database is ~300MB. I've tried > >> all the settings mentioned in that thread with no noticeable > >> improvement. I was hoping that once the recovery was done (backfills > >> to reformatted OSDs) that it would clear up, but not yet. So any other > >> ideas would be really helpful. Our MDS is functioning, but stalls a > >> lot because the mons miss heartbeats. > >> > >> mon_compact_on_start = true > >> rocksdb_cache_size = 1342177280 > >> mon_lease = 30 > >> mon_osd_cache_size = 20 > >> mon_sync_max_payload_size = 4096 > >> > >> > >> Robert LeBlanc > >> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 > >> > >> On Thu, Apr 8, 2021 at 1:11 PM Stefan Kooman wrote: > >> > > >> > On 4/8/21 6:22 PM, Robert LeBlanc wrote: > >> > > I upgraded our Luminous cluster to Nautilus a couple of weeks ago and > >> > > converted the last batch of FileStore OSDs to BlueStore about 36 hours > >> > > ago. Yesterday our monitor cluster went nuts and started constantly > >> > > calling elections because monitor nodes were at 100% and wouldn't > >> > > respond to heartbeats. I reduced the monitor cluster to one to prevent > >> > > the constant elections and that let the system limp along until the > >> > > backfills finished. There are large amounts of time where ceph commands > >> > > hang with the CPU is at 100%, when the CPU drops I see a lot of work > >> > > getting done in the monitor logs which stops as soon as the CPU is at > >> > > 100% again. > >> > > >> > > >> > Try reducing mon_sync_max_payload_size=4096. I have seen Frank Schilder > >> > advise this several times because of monitor issues. Also recently for a > >> > cluster that got upgraded from Luminous -> Mimic -> Nautilus. > >> > > >> > Worth a shot. > >> > > >> > Otherwise I'll try to look in depth and see if I can come up with > >> > something smart (for now I need to go catch some sleep). > >> > > >> > Gr. Stefan ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Nautilus 14.2.19 mon 100% CPU
On Fri, Apr 9, 2021 at 9:25 AM Stefan Kooman wrote: > Are you running with 1 mon now? Have you tried adding mons from scratch? > So with a fresh database? And then maybe after they have joined, kill > the donor mon and start from scratch. > > You have for sure not missed a step during the upgrade (just checking > mode), i.e. ceph osd require-osd-release nautilus. I have tried adding one of the other monitors by removing the data directory and starting from scratch, but it would go back to the monitor elections and I didn't feel comfortable that it's up to sync to fail over to it so I took it back out. I have run `ceph osd-require-osd-release nautilus` after the upgrade of all the OSDs. I'll go back and double check all the steps, but I think I got them all. Thank you, Robert LeBlanc ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Nautilus 14.2.19 mon 100% CPU
The only step not yet taken was to move to straw2. That was the last step we were going to do next. Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Fri, Apr 9, 2021 at 10:41 AM Robert LeBlanc wrote: > > On Fri, Apr 9, 2021 at 9:25 AM Stefan Kooman wrote: > > Are you running with 1 mon now? Have you tried adding mons from scratch? > > So with a fresh database? And then maybe after they have joined, kill > > the donor mon and start from scratch. > > > > You have for sure not missed a step during the upgrade (just checking > > mode), i.e. ceph osd require-osd-release nautilus. > > I have tried adding one of the other monitors by removing the data > directory and starting from scratch, but it would go back to the > monitor elections and I didn't feel comfortable that it's up to sync > to fail over to it so I took it back out. I have run `ceph > osd-require-osd-release nautilus` after the upgrade of all the OSDs. > I'll go back and double check all the steps, but I think I got them > all. > > Thank you, > Robert LeBlanc ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Nautilus 14.2.19 mon 100% CPU
Hi Robert, Have you checked a log with debug_mon=20 yet to try to see what it's doing? .. Dan On Fri, Apr 9, 2021, 7:02 PM Robert LeBlanc wrote: > The only step not yet taken was to move to straw2. That was the last > step we were going to do next. > > Robert LeBlanc > PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 > > On Fri, Apr 9, 2021 at 10:41 AM Robert LeBlanc > wrote: > > > > On Fri, Apr 9, 2021 at 9:25 AM Stefan Kooman wrote: > > > Are you running with 1 mon now? Have you tried adding mons from > scratch? > > > So with a fresh database? And then maybe after they have joined, kill > > > the donor mon and start from scratch. > > > > > > You have for sure not missed a step during the upgrade (just checking > > > mode), i.e. ceph osd require-osd-release nautilus. > > > > I have tried adding one of the other monitors by removing the data > > directory and starting from scratch, but it would go back to the > > monitor elections and I didn't feel comfortable that it's up to sync > > to fail over to it so I took it back out. I have run `ceph > > osd-require-osd-release nautilus` after the upgrade of all the OSDs. > > I'll go back and double check all the steps, but I think I got them > > all. > > > > Thank you, > > Robert LeBlanc > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Nautilus 14.2.19 mon 100% CPU
On Fri, Apr 9, 2021 at 11:05 AM Dan van der Ster wrote: > > Hi Robert, > > Have you checked a log with debug_mon=20 yet to try to see what it's doing? > I've posted the logs with debug_mon=20 for a period during high CPU here https://owncloud.leblancnet.us/owncloud/index.php/s/OtHsBAYN9r5eSbU You can look near the end of the log for the verbose logging. I'm not sure what to look for in there, nothing sticks out to me. I did disable cephx in the config file to see if that would help, but we still have the 100% CPU. Thank you, Robert LeBlanc ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Nautilus 14.2.19 mon 100% CPU
On Fri, Apr 9, 2021 at 7:24 PM Robert LeBlanc wrote: > > On Fri, Apr 9, 2021 at 11:05 AM Dan van der Ster wrote: > > > > Hi Robert, > > > > Have you checked a log with debug_mon=20 yet to try to see what it's doing? > > > I've posted the logs with debug_mon=20 for a period during high CPU > here https://owncloud.leblancnet.us/owncloud/index.php/s/OtHsBAYN9r5eSbU > > You can look near the end of the log for the verbose logging. I'm not > sure what to look for in there, nothing sticks out to me. I did > disable cephx in the config file to see if that would help, but we > still have the 100% CPU. > Thanks. I didn't see anything ultra obvious to me. But I did notice the nearfull warnings so I wonder if this cluster is churning through osdmaps? Did you see a large increase in inbound or outbound network traffic on this mon following the upgrade? Totally speculating here, but maybe there is an issue where you have some old clients, which can't decode an incremental osdmap from a nautilus mon, so the single mon is busy serving up these maps to the clients. Does the mon load decrease if you stop the osdmap churn?, e.g. by setting norebalance if that is indeed ongoing. Could you also share debug_ms = 1 for a minute of busy cpu mon? -- dan > Thank you, > Robert LeBlanc ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Version of podman for Ceph 15.2.10
I see, I guess I can live without grafana anyway.
What would be the "cephadm bootstrap" command or parameter in order to skip
installing grafana?
And now that it is already installed which command can I use to disable it? It
is trying to get deployed every 10 minutes...
‐‐‐ Original Message ‐‐‐
On Friday, April 9, 2021 2:32 PM, David Orman wrote:
> That container (ceph-grafana) is not built for ARM based processors,
> only AMD64:
> https://hub.docker.com/r/ceph/ceph-grafana/tags?page=1&ordering=last_updated
> . You'll probably need to disable that (I think it's part of the
> dashboard module - I don't know - we run our own Prometheus/Grafana
> infrastructure outside of Ceph).
>
> On Fri, Apr 9, 2021 at 1:32 AM mabi m...@protonmail.ch wrote:
>
> > Thank you for confirming that podman 3.0.1 is fine.
> > I have now bootstrapped my first node with cephadm bootstrap command on my
> > RasPi 4 (8GB RAM) with Ubuntu 20.04 LTS that worked well but in the logs I
> > can see that it fails to deploy the graphana container as you can see from
> > the log below:
> > Traceback (most recent call last):
> > File "/usr/share/ceph/mgr/cephadm/module.py", line 1021, in
> > _remote_connection
> > yield (conn, connr)
> > File "/usr/share/ceph/mgr/cephadm/module.py", line 1168, in _run_cephadm
> > code, '\n'.join(err)))
> > orchestrator._interface.OrchestratorError: cephadm exited with an error
> > code: 1, stderr:Deploy daemon grafana.ceph1a ...
> > Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --net=host
> > --entrypoint stat -e CONTAINER_IMAGE=docker.io/ceph/ceph-grafana:6.7.4 -e
> > NODE_NAME=ceph1a docker.io/ceph/ceph-grafana:6.7.4 -c %u %g /var/lib/grafana
> > stat: stderr {"msg":"exec container process `/usr/bin/stat`: Exec format
> > error","level":"error","time":"2021-04-09T06:17:54.000910863Z"}
> > Traceback (most recent call last):
> > File "", line 6153, in
> > File "", line 1412, in _default_image
> > File "", line 3431, in command_deploy
> > File "", line 3362, in extract_uid_gid_monitoring
> > File "", line 2099, in extract_uid_gid
> > RuntimeError: uid/gid not found
> > Does anyone have a clue what could be going wrong here?? and how to fix
> > that?
> > Right now my bootstraped node has the following containers running:
> > $ sudo podman ps
> > CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
> > 5985c38ef718 docker.io/ceph/ceph:v15 -n mon.ceph1a -f ... 15 hours ago Up
> > 15 hours ago ceph-8d47792c-987d-11eb-9bb6-a5302e00e1fa-mon.ceph1a
> > 0773ea8c6951 docker.io/ceph/ceph:v15 -n mgr.ceph1a.rzc... 15 hours ago Up
> > 15 hours ago ceph-8d47792c-987d-11eb-9bb6-a5302e00e1fa-mgr.ceph1a.rzcwjd
> > 941db20cdc2e docker.io/ceph/ceph:v15 -n client.crash.c... 15 hours ago Up
> > 15 hours ago ceph-8d47792c-987d-11eb-9bb6-a5302e00e1fa-crash.ceph1a
> > 897286d3c80f docker.io/prom/node-exporter:v0.18.1 --no-collector.ti... 15
> > hours ago Up 15 hours ago
> > ceph-8d47792c-987d-11eb-9bb6-a5302e00e1fa-node-exporter.ceph1a
> > 08c4e95c0c03 docker.io/prom/prometheus:v2.18.1 --config.file=/et... 15
> > hours ago Up 15 hours ago
> > ceph-8d47792c-987d-11eb-9bb6-a5302e00e1fa-prometheus.ceph1a
> > 19944dbf7a63 docker.io/prom/alertmanager:v0.20.0 --web.listen-addr... 15
> > hours ago Up 15 hours ago
> > ceph-8d47792c-987d-11eb-9bb6-a5302e00e1fa-alertmanager.ceph1a
> > ‐‐‐ Original Message ‐‐‐
> > On Friday, April 9, 2021 3:37 AM, David Orman orma...@corenode.com wrote:
> >
> > > The latest podman 3.0.1 release is fine (we have many production clusters
> > > running this). We have not tested 3.1 yet, however, but will soon.
> > >
> > > > On Apr 8, 2021, at 10:32, mabi m...@protonmail.ch wrote:
> > > > Hello,
> > > > I would like to install Ceph 15.2.10 using cephadm and just found the
> > > > following table by checking the requirements on the host:
> > > > https://docs.ceph.com/en/latest/cephadm/compatibility/#compatibility-with-podman-versions
> > > > Do I understand this table correctly that I should be using podman
> > > > version 2.1?
> > > > and what happens if I use the latest podman version 3.0
> > > > Best regards,
> > > > Mabi
> > > > ceph-users mailing list -- ceph-users@ceph.io
> > > > To unsubscribe send an email to ceph-users-le...@ceph.io
> > >
> > > ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to ceph-users-le...@ceph.io
>
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Nautilus 14.2.19 mon 100% CPU
On Fri, Apr 9, 2021 at 11:49 AM Dan van der Ster wrote: > > Thanks. I didn't see anything ultra obvious to me. > > But I did notice the nearfull warnings so I wonder if this cluster is > churning through osdmaps? Did you see a large increase in inbound or > outbound network traffic on this mon following the upgrade? > Totally speculating here, but maybe there is an issue where you have > some old clients, which can't decode an incremental osdmap from a > nautilus mon, so the single mon is busy serving up these maps to the > clients. > > Does the mon load decrease if you stop the osdmap churn?, e.g. by > setting norebalance if that is indeed ongoing. > > Could you also share debug_ms = 1 for a minute of busy cpu mon? Here are the new logs with the debug_ms=1 for a bit. https://owncloud.leblancnet.us/owncloud/index.php/s/1hvtJo3s2oLPpWn We do have nearfill, but there are no backfills going on (we don't have any auto balancing, we only use a tool that we wrote and only do it periodically). All PGs are currently active+clean. It appears that some osdmaps are still being trimmed on OSDs, but I'm not sure how to validate that. Our system is always under heavy load so there is always a lot of new data and deletions. We did have our monitor cluster freeze up after the upgrade. It was completely locked up when `ceph -s` command would run. It would not recover on it's own. We reduced the mon cluster to a single node then and we had to take the monitor node off the network. Then we started and stopped the monitor a few times and could run `ceph -s` commands without the 100% CPU. We then put it back on the network and it was fine. We added the remaining monitor nodes after deleting the data directory (fresh install) and it was fine until the last batch of OSDs were converted to BlueStore. The CPU for the entire upgrade is here https://owncloud.leblancnet.us/owncloud/index.php/s/Jku5z575PdE3AOr and the upgrade started just before 3/25. Some of that is due to the mgr. We do have some very old clients (Ubuntu 14.04) that we can't easily upgrade that use CephFS (we can probably upgrade to FUSE on these nodes, but it will take a long time). A good portion of our clients are Ubuntu 18.04 with a 5.3 kernel. I did set up a for loop to deep scrub all the PGs in case it needed to update some internal data structures as indicated from the `ceph pg ls` command. ``` * NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilisation. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. ``` Which I thought may help get `ceph df` working properly so we know how much storage is actually being used. We do have crons running to get metrics into graphite and some of them do ceph commands, others are creating/writing/reading/deleting files to get some file system performance metrics but none doing scrubs, rebalance, etc. Thank you, Robert LeBlanc ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Nautilus 14.2.19 mon 100% CPU
On 4/9/21 3:40 PM, Robert LeBlanc wrote: I'm attempting to deep scrub all the PGs to see if that helps clear up some accounting issues, but that's going to take a really long time on 2PB of data. Are you running with 1 mon now? Have you tried adding mons from scratch? So with a fresh database? And then maybe after they have joined, kill the donor mon and start from scratch. You have for sure not missed a step during the upgrade (just checking mode), i.e. ceph osd require-osd-release nautilus. Gr. Stefan ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Nautilus 14.2.19 mon 100% CPU
On Fri, Apr 9, 2021 at 8:39 PM Robert LeBlanc wrote:
>
> On Fri, Apr 9, 2021 at 11:49 AM Dan van der Ster wrote:
> >
> > Thanks. I didn't see anything ultra obvious to me.
> >
> > But I did notice the nearfull warnings so I wonder if this cluster is
> > churning through osdmaps? Did you see a large increase in inbound or
> > outbound network traffic on this mon following the upgrade?
> > Totally speculating here, but maybe there is an issue where you have
> > some old clients, which can't decode an incremental osdmap from a
> > nautilus mon, so the single mon is busy serving up these maps to the
> > clients.
> >
> > Does the mon load decrease if you stop the osdmap churn?, e.g. by
> > setting norebalance if that is indeed ongoing.
> >
> > Could you also share debug_ms = 1 for a minute of busy cpu mon?
>
> Here are the new logs with the debug_ms=1 for a bit.
> https://owncloud.leblancnet.us/owncloud/index.php/s/1hvtJo3s2oLPpWn
Something strange in this is there is one hammer client that is asking
for nearly a million incremental osdmaps, seemingly every 30s:
client.131831153 at 172.16.212.55 is asking for incrementals from
1170448..1987355 (see [1])
Can you try to evict/kill/block that client and see if your mon load drops?
-- dan
[1]
-43> 2021-04-09 13:12:37.032 7f50de246700 5
mon.sun-storemon01@0(leader).osd e1987341 send_incremental
[1170448..1987341] to client.131831153
2021-04-09 17:07:27.238 7f9fc83e4700 10 mon.sun-storemon01@0(leader)
e45 handle_subscribe
mon_subscribe({mdsmap=3914079+,monmap=0+,osdmap=1170448})
2021-04-09 17:07:27.238 7f9fc83e4700 10
mon.sun-storemon01@0(leader).osd e1987355 check_osdmap_sub
0x55e2e2133de0 next 1170448 (onetime)
2021-04-09 17:07:27.238 7f9fc83e4700 5
mon.sun-storemon01@0(leader).osd e1987355 send_incremental
[1170448..1987355] to client.131831153
2021-04-09 17:07:50.910 7f9fc83e4700 5 mon.sun-storemon01@0(leader)
e45 dispatch_op client.131831153 v1:172.16.212.55:0/527701465 is not
authenticated, dropping
mon_subscribe({mdsmap=3914079+,monmap=0+,osdmap=1170448})
2021-04-09 18:14:47.295 7f9fc83e4700 1 --
[v2:10.65.7.203:3300/0,v1:10.65.7.203:6789/0] <== client.131831153
v1:172.16.212.55:0/527701465 3
mon_subscribe({mdsmap=3914127+,monmap=0+,osdmap=1170448}) 85+0+0
(unknown 1413914345 0 0) 0x55e2dbc52c00 con 0x55e2e1cf5680
2021-04-09 18:15:17.006 7f9fc83e4700 1 --
[v2:10.65.7.203:3300/0,v1:10.65.7.203:6789/0] <== client.131831153
v1:172.16.212.55:0/527701465 2
mon_subscribe({mdsmap=3914127+,monmap=0+,osdmap=1170448}) 85+0+0
(unknown 1413914345 0 0) 0x55e2da565200 con 0x55e2df00a880
2021-04-09 18:15:17.278 7f9fc83e4700 1 --
[v2:10.65.7.203:3300/0,v1:10.65.7.203:6789/0] <== client.131831153
v1:172.16.212.55:0/527701465 3
mon_subscribe({mdsmap=3914127+,monmap=0+,osdmap=1170448}) 85+0+0
(unknown 1413914345 0 0) 0x55e2de443000 con 0x55e2ee3d8400
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Nautilus 14.2.19 mon 100% CPU
On Fri, Apr 9, 2021 at 9:37 PM Dan van der Ster wrote: > > On Fri, Apr 9, 2021 at 8:39 PM Robert LeBlanc wrote: > > > > On Fri, Apr 9, 2021 at 11:49 AM Dan van der Ster > > wrote: > > > > > > Thanks. I didn't see anything ultra obvious to me. > > > > > > But I did notice the nearfull warnings so I wonder if this cluster is > > > churning through osdmaps? Did you see a large increase in inbound or > > > outbound network traffic on this mon following the upgrade? > > > Totally speculating here, but maybe there is an issue where you have > > > some old clients, which can't decode an incremental osdmap from a > > > nautilus mon, so the single mon is busy serving up these maps to the > > > clients. > > > > > > Does the mon load decrease if you stop the osdmap churn?, e.g. by > > > setting norebalance if that is indeed ongoing. > > > > > > Could you also share debug_ms = 1 for a minute of busy cpu mon? > > > > Here are the new logs with the debug_ms=1 for a bit. > > https://owncloud.leblancnet.us/owncloud/index.php/s/1hvtJo3s2oLPpWn > > Something strange in this is there is one hammer client that is asking > for nearly a million incremental osdmaps, seemingly every 30s: > > client.131831153 at 172.16.212.55 is asking for incrementals from > 1170448..1987355 (see [1]) > > Can you try to evict/kill/block that client and see if your mon load drops? > Before you respond, just noting here ftr that i think there's a possible issue with OSDMonitor::get_removed_snaps_range and clients like this. https://github.com/ceph/ceph/blob/v14.2.19/src/mon/OSDMonitor.cc#L4193 Called by send_incremental: https://github.com/ceph/ceph/blob/v14.2.19/src/mon/OSDMonitor.cc#L4152 When building the incremental it will search the mon's rocksdb for removed snaps across those ~million missing maps. That feature seems removed from octopus onward. -- dan ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Nautilus 14.2.19 mon 100% CPU
On Fri, Apr 9, 2021 at 2:04 PM Dan van der Ster wrote: > > On Fri, Apr 9, 2021 at 9:37 PM Dan van der Ster wrote: > > > > On Fri, Apr 9, 2021 at 8:39 PM Robert LeBlanc wrote: > > > > > > On Fri, Apr 9, 2021 at 11:49 AM Dan van der Ster > > > wrote: > > > > > > > > Thanks. I didn't see anything ultra obvious to me. > > > > > > > > But I did notice the nearfull warnings so I wonder if this cluster is > > > > churning through osdmaps? Did you see a large increase in inbound or > > > > outbound network traffic on this mon following the upgrade? > > > > Totally speculating here, but maybe there is an issue where you have > > > > some old clients, which can't decode an incremental osdmap from a > > > > nautilus mon, so the single mon is busy serving up these maps to the > > > > clients. > > > > > > > > Does the mon load decrease if you stop the osdmap churn?, e.g. by > > > > setting norebalance if that is indeed ongoing. > > > > > > > > Could you also share debug_ms = 1 for a minute of busy cpu mon? > > > > > > Here are the new logs with the debug_ms=1 for a bit. > > > https://owncloud.leblancnet.us/owncloud/index.php/s/1hvtJo3s2oLPpWn > > > > Something strange in this is there is one hammer client that is asking > > for nearly a million incremental osdmaps, seemingly every 30s: > > > > client.131831153 at 172.16.212.55 is asking for incrementals from > > 1170448..1987355 (see [1]) > > > > Can you try to evict/kill/block that client and see if your mon load drops? > > > > Before you respond, just noting here ftr that i think there's a > possible issue with OSDMonitor::get_removed_snaps_range and clients > like this. > > https://github.com/ceph/ceph/blob/v14.2.19/src/mon/OSDMonitor.cc#L4193 > > Called by send_incremental: > > https://github.com/ceph/ceph/blob/v14.2.19/src/mon/OSDMonitor.cc#L4152 > > When building the incremental it will search the mon's rocksdb for > removed snaps across those ~million missing maps. > > That feature seems removed from octopus onward. I evicted that client and CPU hasn't gone down significantly. There may be other clients also causing the issue. Was it the `osdmap=1170448` part of the line that says how many OSDmaps it's trying to get? I can look for others in the logs and evict them as well. Maybe if that code path isn't needed in Nautilus it can be removed in the next point release? Thank you, Robert LeBlanc ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Nautilus 14.2.19 mon 100% CPU
On Fri, Apr 9, 2021 at 11:50 PM Robert LeBlanc wrote:
>
> On Fri, Apr 9, 2021 at 2:04 PM Dan van der Ster wrote:
> >
> > On Fri, Apr 9, 2021 at 9:37 PM Dan van der Ster wrote:
> > >
> > > On Fri, Apr 9, 2021 at 8:39 PM Robert LeBlanc
> > > wrote:
> > > >
> > > > On Fri, Apr 9, 2021 at 11:49 AM Dan van der Ster
> > > > wrote:
> > > > >
> > > > > Thanks. I didn't see anything ultra obvious to me.
> > > > >
> > > > > But I did notice the nearfull warnings so I wonder if this cluster is
> > > > > churning through osdmaps? Did you see a large increase in inbound or
> > > > > outbound network traffic on this mon following the upgrade?
> > > > > Totally speculating here, but maybe there is an issue where you have
> > > > > some old clients, which can't decode an incremental osdmap from a
> > > > > nautilus mon, so the single mon is busy serving up these maps to the
> > > > > clients.
> > > > >
> > > > > Does the mon load decrease if you stop the osdmap churn?, e.g. by
> > > > > setting norebalance if that is indeed ongoing.
> > > > >
> > > > > Could you also share debug_ms = 1 for a minute of busy cpu mon?
> > > >
> > > > Here are the new logs with the debug_ms=1 for a bit.
> > > > https://owncloud.leblancnet.us/owncloud/index.php/s/1hvtJo3s2oLPpWn
> > >
> > > Something strange in this is there is one hammer client that is asking
> > > for nearly a million incremental osdmaps, seemingly every 30s:
> > >
> > > client.131831153 at 172.16.212.55 is asking for incrementals from
> > > 1170448..1987355 (see [1])
> > >
> > > Can you try to evict/kill/block that client and see if your mon load
> > > drops?
> > >
> >
> > Before you respond, just noting here ftr that i think there's a
> > possible issue with OSDMonitor::get_removed_snaps_range and clients
> > like this.
> >
> > https://github.com/ceph/ceph/blob/v14.2.19/src/mon/OSDMonitor.cc#L4193
> >
> > Called by send_incremental:
> >
> > https://github.com/ceph/ceph/blob/v14.2.19/src/mon/OSDMonitor.cc#L4152
> >
> > When building the incremental it will search the mon's rocksdb for
> > removed snaps across those ~million missing maps.
> >
> > That feature seems removed from octopus onward.
>
> I evicted that client and CPU hasn't gone down significantly. There
> may be other clients also causing the issue. Was it the
> `osdmap=1170448` part of the line that says how many OSDmaps it's
> trying to get? I can look for others in the logs and evict them as
> well.
>
Here's what you should look for, with debug_mon=10. It shows clearly
that it takes the mon 23 seconds to run through
get_removed_snaps_range.
So if this is happening every 30s, it explains at least part of why
this mon is busy.
2021-04-09 17:07:27.238 7f9fc83e4700 10 mon.sun-storemon01@0(leader)
e45 handle_subscribe
mon_subscribe({mdsmap=3914079+,monmap=0+,osdmap=1170448})
2021-04-09 17:07:27.238 7f9fc83e4700 10
mon.sun-storemon01@0(leader).osd e1987355 check_osdmap_sub
0x55e2e2133de0 next 1170448 (onetime)
2021-04-09 17:07:27.238 7f9fc83e4700 5
mon.sun-storemon01@0(leader).osd e1987355 send_incremental
[1170448..1987355] to client.131831153
2021-04-09 17:07:28.590 7f9fc83e4700 10
mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 0
[1~3]
2021-04-09 17:07:29.898 7f9fc83e4700 10
mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 5 []
2021-04-09 17:07:31.258 7f9fc83e4700 10
mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 6 []
2021-04-09 17:07:32.562 7f9fc83e4700 10
mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 20
[]
2021-04-09 17:07:33.866 7f9fc83e4700 10
mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 21
[]
2021-04-09 17:07:35.162 7f9fc83e4700 10
mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 22
[]
2021-04-09 17:07:36.470 7f9fc83e4700 10
mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 23
[]
2021-04-09 17:07:37.778 7f9fc83e4700 10
mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 24
[]
2021-04-09 17:07:39.090 7f9fc83e4700 10
mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 25
[]
2021-04-09 17:07:40.398 7f9fc83e4700 10
mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 26
[]
2021-04-09 17:07:41.706 7f9fc83e4700 10
mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 27
[]
2021-04-09 17:07:43.006 7f9fc83e4700 10
mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 28
[]
2021-04-09 17:07:44.322 7f9fc83e4700 10
mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 29
[]
2021-04-09 17:07:45.630 7f9fc83e4700 10
mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 30
[]
2021-04-09 17:07:46.938 7f9fc83e4700 10
mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 31
[]
2021-04-09 17:07:48.246 7f9fc83e4700 10
mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 32
[]
2021-04-09 17:07:49.562 7f9fc83e4700 10
mon.sun-storemon01@0(leader).osd e1987355 get_removed_
[ceph-users] working ansible based crush map?
I'm trying to follow the directions in ceph-ansible for having it automatically set up the crush map. I've also looked at https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html/installation_guide/installing-red-hat-ceph-storage-using-ansible But.. my setup isnt working. ansible complains about: [WARNING]: While constructing a mapping from /etc/ansible/hosts.yml, line 5, column 9, found a duplicate dict key (hostA2). Using last defined value only. Could anyone explain what I'm doing wrong, please? Here's the relevant hosts.yml snippets all: children: osds: hosts: hostA1: osd_crush_location: host: 'hostA1' chassis: 'hostA' root: 'default' hostA2: osd_crush_location: host: 'hostA2' chassis: 'hostA' root: 'default' -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbr...@medata.com| www.medata.com ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: working ansible based crush map?
Ooops. I found the problem with duplicates.. .. there were actually duplicates elsewhere, that were presenting as if they were in this section. So on the one hand, i got rid of the runtime errors. But on the other hand... this isnt actually modding the crush map for me? the "chassis: hostA" doesnt get created as a bucket all: children: osds: hosts: hostA1: osd_crush_location: host: 'hostA1' chassis: 'hostA' root: 'default' hostA2: osd_crush_location: host: 'hostA2' chassis: 'hostA' root: 'default' -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbr...@medata.com| www.medata.com ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: working ansible based crush map?
AAAND final update: problem fixed. I had enabled create_crush_tree: true in group_vars/osd.yml but I had neglected to ALSO set crush_rule_config: true So now it's all happy ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Nautilus 14.2.19 mon 100% CPU
On Fri, Apr 9, 2021 at 4:04 PM Dan van der Ster wrote:
>
> Here's what you should look for, with debug_mon=10. It shows clearly
> that it takes the mon 23 seconds to run through
> get_removed_snaps_range.
> So if this is happening every 30s, it explains at least part of why
> this mon is busy.
>
> 2021-04-09 17:07:27.238 7f9fc83e4700 10 mon.sun-storemon01@0(leader)
> e45 handle_subscribe
> mon_subscribe({mdsmap=3914079+,monmap=0+,osdmap=1170448})
> 2021-04-09 17:07:27.238 7f9fc83e4700 10
> mon.sun-storemon01@0(leader).osd e1987355 check_osdmap_sub
> 0x55e2e2133de0 next 1170448 (onetime)
> 2021-04-09 17:07:27.238 7f9fc83e4700 5
> mon.sun-storemon01@0(leader).osd e1987355 send_incremental
> [1170448..1987355] to client.131831153
> 2021-04-09 17:07:28.590 7f9fc83e4700 10
> mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 0
> [1~3]
> 2021-04-09 17:07:29.898 7f9fc83e4700 10
> mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 5 []
> 2021-04-09 17:07:31.258 7f9fc83e4700 10
> mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 6 []
> 2021-04-09 17:07:32.562 7f9fc83e4700 10
> mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 20
> []
> 2021-04-09 17:07:33.866 7f9fc83e4700 10
> mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 21
> []
> 2021-04-09 17:07:35.162 7f9fc83e4700 10
> mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 22
> []
> 2021-04-09 17:07:36.470 7f9fc83e4700 10
> mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 23
> []
> 2021-04-09 17:07:37.778 7f9fc83e4700 10
> mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 24
> []
> 2021-04-09 17:07:39.090 7f9fc83e4700 10
> mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 25
> []
> 2021-04-09 17:07:40.398 7f9fc83e4700 10
> mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 26
> []
> 2021-04-09 17:07:41.706 7f9fc83e4700 10
> mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 27
> []
> 2021-04-09 17:07:43.006 7f9fc83e4700 10
> mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 28
> []
> 2021-04-09 17:07:44.322 7f9fc83e4700 10
> mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 29
> []
> 2021-04-09 17:07:45.630 7f9fc83e4700 10
> mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 30
> []
> 2021-04-09 17:07:46.938 7f9fc83e4700 10
> mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 31
> []
> 2021-04-09 17:07:48.246 7f9fc83e4700 10
> mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 32
> []
> 2021-04-09 17:07:49.562 7f9fc83e4700 10
> mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 34
> []
> 2021-04-09 17:07:50.862 7f9fc83e4700 10
> mon.sun-storemon01@0(leader).osd e1987355 get_removed_snaps_range 35
> []
> 2021-04-09 17:07:50.862 7f9fc83e4700 20
> mon.sun-storemon01@0(leader).osd e1987355 send_incremental starting
> with base full 1986745 664086 bytes
> 2021-04-09 17:07:50.862 7f9fc83e4700 10
> mon.sun-storemon01@0(leader).osd e1987355 build_incremental
> [1986746..1986785] with features 107b84a842aca
>
> So have a look for that client again or other similar traces.
So, even though I blacklisted the client and we remounted the file
system on it, it wasn't enough for it to keep performing the same bad
requests. We found another node that had two sessions to the same
mount point. We rebooted both nodes and the CPU is now back at a
reasonable 4-6% and the cluster is running at full performance again.
I've added in back both MONs to have all 3 mons in the system and
there are no more elections. Thank you for helping us track down the
bad clients out of over 2,000 clients.
> > Maybe if that code path isn't needed in Nautilus it can be removed in
> > the next point release?
>
> I think there were other major changes in this area that might make
> such a backport difficult. And we should expect nautilus to be nearing
> its end...
But ... we just got to Nautilus... :)
Thank you,
Robert LeBlanc
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
