[ceph-users] Re: MTU mismatch error in Ceph dashboard

On 04.08.2021 20:31, Ernesto Puerta wrote: Could you please go to the Prometheus UI and share the output of the following query "node_network_mtu_bytes"? That'd be useful to understand the issue. If you can open a tracker issue here: https://tracker.ceph.com/projects/dashboard/issues/new ? Fo

[ceph-users] All OSDs on one host down

Hi all, Bit of a panic. Woke this morning to find on of my dedicated mon hosts showing as down. I did a reboot on the host and it came back up fine (ceph-005). Then all the OSDs (5) on host ceph-004 went down. This host is also a mon and the mon daemon is showing as up. I’m running octopus u

[ceph-users] Re: All OSDs on one host down

> > I’ve tried restarting on of the osds but that fails, journalctl shows > osd not found.not convinced I’ve got the systemctl command right. > You are not mixing 'not container commands' with 'container commands'. As in, if you execute this journalctl outside of the container it will not f

[ceph-users] Re: Bucket deletion is very slow.

Hi, We use: radosgw-admin bucket rm --purge-objects --bypass-gc --max-concurrent-ios=2048 --bucket=bucket You can try with greater max-concurrent-ios. YMMV. On 05.08.2021 at 17∶25 +0300, user mhnx wrote: > Hello again. > > I couldn't find any answer on the issue. Is there any way to speed up > b

[ceph-users] Re: MDS crash on FAILED ceph_assert(cur->is_auth())

Le 28/06/2021 à 10:52, Peter van Heusden a écrit : I am running Ceph 15.2.13 on CentOS 7.9.2009 and recently my MDS servers have started failing with the error message In function 'void Server::handle_client_open(MDRequestRef&)' thread 7f0ca9908700 time 2021-06-28T09:21:11.484768+0200 /home/jen

[ceph-users] Re: All OSDs on one host down

Hi Marc, Yes i’m probably doing just that. The ceph admin guides aren’t exactly helpful on this. The cluster was deployed using cephadm and it’s been running perfectly until now. Wouldn’t running “journalctl -u ceph-osd@5” on host ceph-004 show me the logs for osd.5 on that host? Cheers, A

[ceph-users] Re: All OSDs on one host down

> > > Yes i’m probably doing just that. > > > > The ceph admin guides aren’t exactly helpful on this. The cluster was > deployed using cephadm and it’s been running perfectly until now. > That is why I am not using the ceph containers. I would even say nobody without container experience,

[ceph-users] Re: All OSDs on one host down

On 08/06 07:59, Andrew Walker-Brown wrote: > Hi Marc, > > Yes i’m probably doing just that. > > The ceph admin guides aren’t exactly helpful on this. The cluster was > deployed using cephadm and it’s been running perfectly until now. > > Wouldn’t running “journalctl -u ceph-osd@5” on host ceph

[ceph-users] Re: All OSDs on one host down

This is what I get from cephadm logs for the specific OSDin fact all OSDs on that host. bash[2681]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compa> Aug 06 06:52:37 uk1-ceph-004 bash[2681]: *

[ceph-users] Re: MDS crash on FAILED ceph_assert(cur->is_auth())

Hi Yann So I resolved the problem by taking clients offline till I found the one - i think the one where conda had been running - that seemed to be causing the problem. I then was able to restart the MDS daemons and things came right. It is certainly a troubling issue. And sorry, I didn't open a

[ceph-users] Broken pipe error on Rados gateway log

Hi everyone, Recently, I just noticed that there is a lot of log about Broken pipe error from all RGW nodes. Log: 2021-08-04T06:25:05.997+ 7f4f15f7b700 1 == starting new request req=0x7f4fac3d7670 = 2021-08-04T06:25:05.997+ 7f4f15f7b700 0 ERROR: client_io->complete_request()

[ceph-users] 回复: ceph csi issues

Upgrade the client software. 发件人: 峰 发送时间: 2021年8月6日 17:05 收件人: ceph-users 主题: [ceph-users] ceph csi issues hi,When the ceph cluster is set to safe mode，(ceph config set mon auth_allow_insecure_global_id_reclaim false). kubernetes deploy ceph cs

[ceph-users] Re: MDS crash on FAILED ceph_assert(cur->is_auth())

Le 06/08/2021 à 10:46, Peter van Heusden a écrit : Hi Yann So I resolved the problem by taking clients offline till I found the one - i think the one where conda had been running - that seemed to be causing Hi Peter, thanks for the answer, I'm afraid I'll have to do this too. the problem. I

[ceph-users] cephfs_metadata pool unexpected space utilization

Hi, I observe strange behavior on my Ceph MDS cluster, where cephfs_metadata pool is filling out without obvious reason. It's getting +15% by day even when there are no I/O on the cluster. I have separate SSD disks for metadata pool, each 112G with pool replica size 3. `ceph fs status` shows

[ceph-users] Re: All OSDs on one host down

A reboot of the host has fixed the problem but I still want to find the root cause. Looking at the logs I can see the original mon went down because the docker engine shutdown in response to a network event. That network event seems to appears to be systemd wait-on-network timeout related and

[ceph-users] Cephadm Upgrade from Octopus to Pacific

I'm attempting to upgrade my large cephadm deployed cluster with 1600 osd from octopus to pacific. Given this did not work very well when I first tried I decided to break off 3 nodes and create a small "test" cluster to see how badly it would fail. This is upgrading from 15.2.13 to 16.2.5 with a

[ceph-users] ceph csi issues

hi,When the ceph cluster is set to safe mode??(ceph config set mon auth_allow_insecure_global_id_reclaim false). kubernetes deploy ceph csi, regardless of whether I use the admin key or independently authorized user keys, pvc responds to permissions issues. What to do?? __

[ceph-users] Re: Cephadm Upgrade from Octopus to Pacific

Peter, I had the same error and my workaround was to manually create /usr/lib/sysctl.d directory on all nodes, then resume the upgrade Arnaud Martel - Mail original - De: "Peter Childs" À: "ceph-users" Envoyé: Vendredi 6 Août 2021 15:03:20 Objet: [ceph-users] Cephadm Upgrade from Octop

[ceph-users] Re: we're living in 2005.

Anyone know how best to get confirmation from the Ceph team if they would have any issue with a user forum being set up? --> I am toying with the idea of setting one up. Josh On Thu, Aug 5, 2021 at 1:34 AM Janne Johansson wrote: > > Den mån 26 juli 2021 kl 16:56 skrev : > > and there's an irc c

[ceph-users] Re: Cephadm Upgrade from Octopus to Pacific

Looks related to https://tracker.ceph.com/issues/51620 Hopefully this will be backported to Pacific and included in 16.2.6 Regards, Dimitri On Fri, Aug 6, 2021 at 9:21 AM Arnaud MARTEL < arnaud.mar...@i2bc.paris-saclay.fr> wrote: > Peter, > > I had the same error and my workaround was to manua

[ceph-users] Re: we're living in 2005.

There's reddit - https://old.reddit.com/r/ceph/ - that's what I've been using for months now to get my cluster set up. Zach Heise Social Sciences Computing Cooperative Work and Contact Information On 2021-08-06 8:29 AM, j...@cayk.ca wrote: Anyone kno

[ceph-users] Re: Unable to enable dashboard sso with cert file

Glad it worked. I created a PR to clarify this in the docs: https://github.com/ceph/ceph/pull/42701 Thanks! Kind Regards, Ernesto On Fri, Aug 6, 2021 at 1:51 AM Adam Zheng wrote: > Hello Ernesto, > > > > Thank you very much, that was my mistake. > > I was running this on a standby manager con

[ceph-users] Re: Cephadm Upgrade from Octopus to Pacific

Yes that works for me, looping through the nodes and doing a is a quick work around. ssh $node mkdir /usr/lib/sysctl.d does allow the upgrade to complete. As I've probably said before I'm using Debian Buster as CentOS 7 was not 100% happy even with Octopus (although it just about worked) and Cent

[ceph-users] Re: MTU mismatch error in Ceph dashboard

Thanks, Kai! We moved the Dashboard tickets to a separate subproject. I just moved that tracker. It should be easy to remove at least the NICs down. Kind Regards, Ernesto On Fri, Aug 6, 2021 at 9:13 AM Kai Stian Olstad wrote: > On 04.08.2021 20:31, Ernesto Puerta wrote: > > Could you please go

[ceph-users] Re: MTU mismatch error in Ceph dashboard

Thank you Ernesto. Yes - so I see that all the eno1, eno2, and docker0 interfaces show up with an MTU of 1500 which is correct, but since these interfaces are not being used at all, they shouldn’t be flagged as a problem. I’ll just ignore the errors for now, but would be good to have a way to i

[ceph-users] Re: we're living in 2005.

Hi Joshua, I asked like 1 year ago, and didn’t have good sense for setup from ceph foundation. Looks like all must be mailist or IRC but anyway I think the forum can be setup without any permission. The community is free to share technical question whatever we want or share resoures in a b

[ceph-users] Re: PSA: upgrading older clusters without CephFS

Hello Linh, On Thu, Aug 5, 2021 at 9:12 PM Linh Vu wrote: > Without personally knowing the history of a cluster, is there a way to check > and see when and which release it began life as? Or check whether such legacy > data structures still exist in the mons? I'm not aware of an easy way to ch

[ceph-users] Re: 回复：[ceph-users]

According to release note at https://github.com/ceph/ceph-csi/releases (specifically, "Update ceph to 15.2.11"), you need: * >= 3.3.1 or * >= 3.2.5 and < 3.3 ?? 2021/8/7 9:47, ?? : Is it to update the ceph csi version? I suspected that it was a version problem, so I updated from ceph c