[ceph-users] Re: [SPAM] Re: Ceph crash :-(

… this is LTS version :/ BR, Sebastian > On 13 Jun 2024, at 19:47, David C. wrote: > > In addition to Robert's recommendations, > > Remember to respect the update order (mgr->mon->(crash->)osd->mds->...) > > Before everything was containerized, it was not

[ceph-users] Re: cephadm bootstrap on 3 network clusters

Hi, check routing table and default gateway and eventually fix it. use IP instead of dns name. I have more complicated situation :D I have more than 3 public networks and cluster networks… BR, Sebastian > On Jan 3, 2024, at 16:40, Luis Domingues wrote: > > >> Why? The

[ceph-users] Re: [SPAM] Re: Ceph RBD, MySQL write IOPs - what is possible?

stdev=2038.60 BR, Sebastian > On 11 Jun 2024, at 02:23, Mark Lehrer wrote: > > If they can do 1 TB/s with a single 16K write thread, that will be > quite impressive :DOtherwise not really applicable. Ceph scaling > has always been good. > > More seriously, would you m

[ceph-users] Re: Tuning CephFS on NVME for HPC / IO500

iommu=pt should help. On beginning everything looks like something in kernel stack slowdown packets. BR, Sebastian > On 2 Dec 2022, at 16:03, Manuel Holtgrewe wrote: > > Dear Mark. > > Thank you very much for all of this information. I learned a lot! In > particular that I

[ceph-users] Re: P420i Raid Controller HBA Mode for Ceph

eriod. BR, Sebastian > On 1 Jan 2023, at 16:48, hosseinz8...@yahoo.com wrote: > > Hi Experts,For my new ceph cluster, my existing storage nodes have Smart > Array Raid Controller P420i (HP G8). I have 6 Enterprise-SSD Disks for every > storage Node.From your experiences, activatin

[ceph-users] Re: HBA or RAID-0 + BBU

I want add one thing to what other says, we discussed this between Cephalocon sessions, avoid HP controllers p210/420, or upgrade firmware to latest. These controllers has strange bug, during high workload they restart itself. BR, Sebastian > On 19 Apr 2023, at 08:39, Janne Johansson wr

[ceph-users] Re: Module 'dashboard' has failed: '_cffi_backend.CDataGCP' object has no attribute 'type'

Sounds like a bug. mind creating a tracker issue? https://tracker.ceph.com/projects/mgr/issues/new Am 17.11.20 um 17:39 schrieb Marcelo: > Hello all. > > I'm trying to deploy the dashboard (Nautilus 14.2.8), and after I run ceph > dashboard create-self-signed-cert, the cluster started to show th

[ceph-users] Re: performance degredation every 30 seconds

Hi, check your rbd cache, by default it's enabled, for ssd/nvme better is to disable it. Looks like your cache/buffers are full and need flush. It could harmful your env. BR, Sebastian On 11.12.2020 19:08, Philip Brown wrote: I have a new 3 node octopus cluster, set up on SSDs.

[ceph-users] Re: Python API mon_comand()

Am 15.01.21 um 09:24 schrieb Robert Sander: > Hi, > > I am trying to get some statistics via the Python API but fail to run the > equivalent of "ceph df detail". > > > ...snip... cluster.mon_command(json.dumps({'prefix': 'df detail', 'format': 'json'}), b'') > (-22, '', u'command n

[ceph-users] CephFS Octopus snapshots / kworker at 100% / kernel vs. fuse client

M 10 TiB 12.57 56 TiB Snapshots are only on cephfs.cephfs.data pool. Intended snapshot rotation: 4 quarter-hourly snapshots 24 hourly snapshots 14 daily snapshots 8 weekly snapshots Cheers Sebastian ___ ceph-users mailing list -- ceph-users@

[ceph-users] Re: 10G stackabe lacp switches

3y ago I bought it on ebay to my home lab for 750$ with transport and duty and additional tax, so it's possible https://www.ebay.com/sch/i.html?_from=R40&_trksid=p2334524.m570.l1313&_nkw=7050QX-32&_sacat=0&LH_TitleDesc=0&_osacat=0&_odkw=7050QX-32S BR, Sebastia

[ceph-users] Re: bug in latest cephadm bootstrap: got an unexpected keyword argument 'verbose_on_failure'

Indeed. That is going to be fixed by https://github.com/ceph/ceph/pull/39633 Am 03.03.21 um 07:31 schrieb Philip Brown: > Seems like someone is not testing cephadm on centos 7.9 > > Just tried installing cephadm from the repo, and ran > cephadm bootstrap --mon-ip=xxx > > it blew up, with > >

[ceph-users] CephFS: side effects of not using ceph-mgr volumes / subvolumes

or my specific use case I require snapshots on the subvolume group layer. It therefore seems better to just forego the abstraction as a whole and work on bare CephFS. Cheers Sebastian [1] https://docs.ceph.com/en/octopus/cephfs/fs-volumes/ ___ ceph-

[ceph-users] Re: Alertmanager not using custom configuration template

Hi Mark, Indeed. I just merged https://github.com/ceph/ceph/pull/39932 which fixes the names of those config keys. Might want to try again (with slashes instead of underscores). Thanks for reporting this, Sebastian Am 10.03.21 um 15:34 schrieb Marc 'risson' Schmitt: > Hi, >

[ceph-users] Re: Cephadm: Upgrade 15.2.5 -> 15.2.9 stops on non existing OSD

Hi Kai, looks like $ ssh pech-hd-009 # cephadm ls is returning this non-existent OSDs. can you verify that `cephadm ls` on that host doesn't print osd.355 ? Best, Sebastian Am 11.03.21 um 12:16 schrieb Kai Stian Olstad: > Before I started the upgrade the cluster was healthy but one &

[ceph-users] Re: cephadm (curl master)/15.2.9:: how to add orchestration

Hi Adrian, Am 11.03.21 um 13:55 schrieb Adrian Sevcenco: > Hi! After an initial bumpy bootstrapping (IMHO the defaults should be > whatever is already defined in .ssh of the user and custom values setup > with cli arguments) now i'm stuck adding any service/hosts/osds because > apparently i lack

[ceph-users] Re: Cephadm: Upgrade 15.2.5 -> 15.2.9 stops on non existing OSD

yes Am 11.03.21 um 15:46 schrieb Kai Stian Olstad: > Hi Sebastian > > On 11.03.2021 13:13, Sebastian Wagner wrote: >> looks like >> >> $ ssh pech-hd-009 >> # cephadm ls >> >> is returning this non-existent OSDs. >> >> can you verif

[ceph-users] Re: Unhealthy Cluster | Remove / Purge duplicate osds | Fix daemon

Hi Oliver, # ssh gedaopl02 # cephadm rm-daemon osd.0 should do the trick. Be careful to remove the broken OSD :-) Best, Sebastian Am 11.03.21 um 22:10 schrieb Oliver Weinmann: > Hi, > > On my 3 node Octopus 15.2.5 test cluster, that I haven't used for quite > a while,

[ceph-users] Re: Container deployment - Ceph-volume activation

Am 11.03.21 um 18:40 schrieb 胡 玮文: > Hi, > > Assuming you are using cephadm? Checkout this > https://docs.ceph.com/en/latest/cephadm/osd/#activate-existing-osds > > > ceph cephadm osd activate ... Might not be backported. see https://tracker.ceph.com/issues/46691#note-1 for the workaround

[ceph-users] Re: Unhealthy Cluster | Remove / Purge duplicate osds | Fix daemon

Hi Oliver, I don't know how you managed to remove all MGRs from the cluster, but there is the documentation to manually recover from this: > https://docs.ceph.com/en/latest/cephadm/troubleshooting/#manually-deploying-a-mgr-daemon Hope that helps, Sebastian Am 15.03.21 um 18:24 schrie

[ceph-users] Re: cephadm custom mgr modules

You want to build a custom container for that user case indeed. On Mon, Apr 12, 2021 at 2:18 PM Rob Haverkamp wrote: > Hi there, > > I'm developing a custom ceph-mgr module and have issues deploying this on > a cluster deployed with cephadm. > With a cluster deployed with ceph-deploy, I can just

[ceph-users] Re: How to disable ceph-grafana during cephadm bootstrap

cephadm bootstrap --skip-monitoring-stack should to the trick. See man cephadm On Tue, Apr 13, 2021 at 6:05 PM mabi wrote: > Hello, > > When bootstrapping a new ceph Octopus cluster with "cephadm bootstrap", > how can I tell the cephadm bootstrap NOT to install the ceph-grafana > container? > >

[ceph-users] Re: cephadm: how to create more than 1 rgw per host

Hi Ivan, this is a feature that is not yet released in Pacific. It seems the documentation is a bit ahead of time right now. Sebastian On Fri, Apr 16, 2021 at 10:58 PM i...@z1storage.com wrote: > Hello, > > According to the documentation, there's count-per-host key to 'c

[ceph-users] Re: one of 3 monitors keeps going down

Right, here are the docs for that workflow: https://docs.ceph.com/en/latest/cephadm/mon/#mon-service Am 29.04.21 um 13:13 schrieb Eugen Block: Hi, instead of copying MON data to this one did you also try to redeploy the MON container entirely so it gets a fresh start? Zitat von "Robert W.

[ceph-users] Re: mon vanished after cephadm upgrade

Hi Ashley, is sn-m01 listed in `ceph -s`? Which hosts are listed in `ceph orch ps --daemon-type mon ? Otherwise, there are a two helpful commands now: * `cpeh orch daemon rm mon.sn-m01` to remove the mon * `ceph orch daemon start mon.sn-m01` to start it again Am 14.05.21 um 14:14 schrieb

[ceph-users] Re: XFS on RBD on EC painfully slow

during the first sync) - Take old storage off-line for users / read-only - Final rsync with --delete (i.e. rsync -aS --numeric-ids --delete ...) - Mount cephfs at location of old storage, adjust /etc/exports with fsid entries where necessary, turn system back on-line / read-write Cheers Sebastian

[ceph-users] Re: Cephfs metadta pool suddenly full (100%) !

w *exactly* what you are doing. Cheers & Best of luck Sebastian [1] https://docs.ceph.com/en/latest/rados/operations/balancer/ [2] https://docs.ceph.com/en/latest/rados/operations/placement-groups/ Replace latest in the URIs with your Ceph version string (i.e. octopus, nautilus) for ve

[ceph-users] Re: Cephfs metadta pool suddenly full (100%) !

column in `ceph df` output are identical. Is that because of Nautilus (I myself am running Octopus, where USED is the expected multiple of STORED depending on replication factor / EC configuration in the pool) or are you running a specific configuration that might cause that? C

[ceph-users] Re: OT: How to Build a poor man's storage with ceph

2800.00 osd_scrub_sleep 0.10 My data is in a replicated pool with n=3 without compression. You might also consider EC and then want to aim for more nodes. Cheers Sebastian ___ ceph-users mailing list -- ceph-users@ceph.io To

[ceph-users] Re: lib remoto in ubuntu

Hi Alfredo, if you don't use cephadm, then I'd recommend to not install the ceph-mgr-cephadm package. If you use cephadm with an ubuntu based container, you'll have to make sure that the MGR properly finds the remoto package within the container. Thanks, Sebastian Am 11

[ceph-users] Re: Module 'devicehealth' has failed:

Hi Torkil, you should see more information in the MGR log file. Might be an idea to restart the MGR to get some recent logs. Am 15.06.21 um 09:41 schrieb Torkil Svensgaard: Hi Looking at this error in v15.2.13: " [ERR] MGR_MODULE_ERROR: Module 'devicehealth' has failed:     Module 'devicehea

[ceph-users] Re: cephadm shell fails to start due to missing config files?

that helps, Sebastian Am 02.07.21 um 19:04 schrieb Vladimir Brik: Hello I am getting an error on one node in my cluster (other nodes are fine) when trying to run "cephadm shell". Historically this machine has been used as the primary Ceph management host, so it would be nice if this

[ceph-users] Wrong hostnames in "ceph mgr services" (Octopus)

of the monitoring stack documentation. Cheers Sebastian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph Pacific mon is not starting after host reboot

any operator interference? Obviously no one :-). We already have over 750 tests for the cephadm scheduler and I can foresee that we'll get some additional ones for this case as well. Kind regards, Sebastian Regards ___ ceph-users mailing

[ceph-users] Re: "ceph orch ls", "ceph orch daemon rm" fail with exception "'KeyError: 'not'" on 15.2.10

Hi, you managed to hit https://tracker.ceph.com/issues/51176 which will be fixed by https://github.com/ceph/ceph/pull/42177 . https://tracker.ceph.com/issues/51176#note-9 contains a list of steps for you to recover from this. Hope that helps, Sebastian Am 09.08.21 um 13:11 schrieb

[ceph-users] Re: Docker container snapshots accumulate until disk full failure?

Sebastian On 11.08.21 15:35, Harry G. Coin wrote: Does ceph remove container subvolumes holding previous revisions of daemon images after upgrades? I have a couple servers using btrfs to hold the containers.   The number of docker related sub-volumes just keeps growing, way beyond the number of

[ceph-users] CephFS Octopus mv: Invalid cross-device link [Errno 18] / slow move

I was not using the volume/subvolume features and reorganised the data - the directory inode is still the same. Cheers Sebastian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: CephFS Octopus mv: Invalid cross-device link [Errno 18] / slow move

Hi Luís, Am 18.08.2021 um 19:02 schrieb Luis Henriques: > Sebastian Knust writes: > >> Hi, >> >> I am running a Ceph Oc,topus (15.2.13) cluster mainly for CephFS. Moving >> (with >> mv) a large directory (mail server backup, so a few million small fi

[ceph-users] Re: Brand New Cephadm Deployment, OSDs show either in/down or out/down

Could you run 1. ceph orch ls --service-type osd --format yaml 2. cpeh orch ps --daemon-type osd --format yaml 3. try running the `ceph auth add` call form https://docs.ceph.com/en/mimic/rados/operations/add-or-rm-osds/#adding-an-osd-manual Am 30.08.21 um 14:49 schrieb Alcatraz: Hello al

[ceph-users] Re: Very beginner question for cephadm: config file for bootstrap and osd_crush_chooseleaf_type

Try running `cephadm bootstrap --single-host-defaults` Am 20.08.21 um 18:23 schrieb Eugen Block: Hi, you can just set the config option with 'ceph config set ...' after your cluster has been bootstrapped. See [1] for more details about the config store. [1] https://docs.ceph.com/en/latest/

[ceph-users] Re: Brand New Cephadm Deployment, OSDs show either in/down or out/down

Am 30.08.21 um 17:39 schrieb Alcatraz: Sebastian, Thanks for responding! And of course. 1. ceph orch ls --service-type osd --format yaml Output: service_type: osd service_id: all-available-devices service_name: osd.all-available-devices placement:   host_pattern: '*' unmanaged:

[ceph-users] Re: podman daemons in error state - where to find logs?

We have a troubleshooting section here: https://docs.ceph.com/en/latest/cephadm/troubleshooting/#checking-cephadm-logs a ceph user should not be required for the containers to log to systemd. Did things end up in

[ceph-users] Re: cephadm 15.2.14 - mixed container registries?

Am 02.09.21 um 02:54 schrieb Nigel Williams: I managed to upgrade to 15.2.14 by doing: ceph orch upgrade start --image quay.io/ceph/ceph:v15.2.14 (anything else I tried would fail) When I look in ceph orch ps output though I see quay.io for most image sources, but alertmanager, grafana, node-

[ceph-users] Re: Brand New Cephadm Deployment, OSDs show either in/down or out/down

Can you verify that the `/usr/lib/sysctl.d/` folder exists on your debian machines? Am 01.09.21 um 15:19 schrieb Alcatraz: Sebastian, I appreciate all your help. I actually (out of desperation) spun up another cluster, same specs, just using Ubuntu 18.04 rather than Debian 10. All the OSDs

[ceph-users] Re: cephadm Pacific bootstrap hangs waiting for mon

by chance do you still have the logs of the mon the never went up? https://docs.ceph.com/en/latest/cephadm/troubleshooting/#checking-cephadm-logs <https://docs.ceph.com/en/latest/cephadm/troubleshooting/#checking-cephadm-logs> Sebastian Am 31.08.21 um 23:51 schrieb Matthew Pounsett:

[ceph-users] Re: Very beginner question for cephadm: config file for bootstrap and osd_crush_chooseleaf_type

?: mon_allow_pool_size_one = 1 osd_pool_default_size = 1 Ignacio El 30/8/21 a las 17:31, Sebastian Wagner escribió: Try running `cephadm bootstrap --single-host-defaults` Am 20.08.21 um 18:23 schrieb Eugen Block: Hi, you can just set the config option with 'ceph config set ...' after your cluste

[ceph-users] Re: Cephadm cannot aquire lock

Am 31.08.21 um 04:05 schrieb fcid: Hi ceph community, I'm having some trouble trying to delete an OSD. I've been using cephadm in one of our clusters and it's works fine, but lately, after an OSD failure, I cannot delete it using the orchestrator. Since the orchestrator is not working (for s

[ceph-users] Re: mon startup problem on upgrade octopus to pacific

Could you please verify that the mon_map of each mon contains all and correct mons? Am 30.08.21 um 21:45 schrieb Chris Dunlop: Hi, Does anyone have any suggestions? Thanks, Chris On Mon, Aug 30, 2021 at 03:52:29PM +1000, Chris Dunlop wrote: Hi, I'm stuck, mid upgrade from octopus to pacif

[ceph-users] Re: Kworker 100% with ceph-msgr (after upgrade to 14.2.6?)

on CentOS8. Client is CentOS8 with elrepo kernel 5.12. My workaround is to not use cephfs snapshots at all, although I really would like to use them. Cheers Sebastian On 07.09.21 14:12, Frank Schilder wrote: Hi Marc, did you ever get a proper solution for this problem? We are having exactly the

[ceph-users] Re: How to purge/remove rgw from ceph/pacific

Yeah, looks like this was missing from the docs. See https://github.com/ceph/ceph/pull/43141 Am 11.09.21 um 12:46 schrieb Eugen Block: Edit your rgw service specs and set „unmanaged“ to true so cephadm won’t redeploy a daemon, then remove it as you did before. See [1] for more details. [1]

[ceph-users] Re: Docker & CEPH-CRASH

ceph-crash should work, as crash dumps aren't namespaced in the kernel. Note that you need a pid1 process in your containers in order for crash dumps to be created. Am 16.09.21 um 08:57 schrieb Eugen Block: I haven't tried it myself but it would probably work to run the crash services apart fr

[ceph-users] Re: Restore OSD disks damaged by deployment misconfiguration

Hi Phil, Am 27.09.21 um 10:06 schrieb Phil Merricks: Hey folks, A recovery scenario I'm looking at right now is this: 1: In a clean 3-node Ceph cluster (pacific, deployed with cephadm), the OS Disk is lost from all nodes 2: Trying to be helpful, a self-healing deployment system reinstalls the

[ceph-users] Re: How you loadbalance your rgw endpoints?

Hi Szabo, I think you can have a look at https://docs.ceph.com/en/latest/cephadm/rgw/#high-availability-service-for-rgw even if you don't deploy ceph using cephadm. Am 24.09.21 um 07:59 schrieb Szabo, Istvan (Ag

[ceph-users] Re: Remoto 1.1.4 in Ceph 16.2.6 containers

Thank you David! Am 24.09.21 um 00:41 schrieb David Galloway: I just repushed the 16.2.6 container with remoto 1.2.1 in it. On 9/22/21 4:19 PM, David Orman wrote: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-4b2736a28c ^^ if people want to test and provide feedback for a potential

[ceph-users] Re: Error ceph-mgr on fedora 36

looks like you should create a tracker issue for this. https://tracker.ceph.com/projects/mgr/issues/new Am 18.09.21 um 14:34 schrieb Igor Savlook: OS: Fedora 36 (rawhide) Ceph: 16.2.6 Python: 3.10 When start ceph-mgr he is try load core pyth

[ceph-users] Re: Cephadm set rgw SSL port

Am 28.09.21 um 15:12 schrieb Daniel Pivonka: > Hi, > > 1. I believe the field is called 'rgw_frontend_port' > 2. I don't think something like that exists but probably should At least for RGWs, we have: https://docs.ceph.com/en/pacific/cephadm/rgw/#service-specification > > -Daniel Pivonka > >

[ceph-users] Re: Cephadm set rgw SSL port

Here you go: https://github.com/ceph/ceph/pull/43332 Am 28.09.21 um 15:49 schrieb Sebastian Wagner: > Am 28.09.21 um 15:12 schrieb Daniel Pivonka: >> Hi, >> >> 1. I believe the field is called 'rgw_frontend_port' >> 2. I don't think something like that e

[ceph-users] Re: cephadm cluster behing a proxy

ve to merge a PR that adds this use case to the docs. Sebastian Am 14.10.21 um 10:18 schrieb Luis Domingues: > Hello, > > We have a cluster deployed with cephadm that sits behind a proxy. It has no > direct access to internet. > > Deploying was not an issue, we did cephadm p

[ceph-users] MDS_DAMAGE in 17.2.7 / Cannot delete affected files

15: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3fe5d5f381] 16: /lib64/libpthread.so.0(+0x81ca) [0x7f3fe4a0b1ca] 17: clone() Deleting the file

[ceph-users] Re: MDS_DAMAGE in 17.2.7 / Cannot delete affected files

ately however, this seems to be the most sensible solution to me, at least with regards to cluster downtime. Cheers Sebastian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MDS_DAMAGE in 17.2.7 / Cannot delete affected files

ort new damage if it comes in. Cheers Sebastian -- Dr. Sebastian Knust | Bielefeld University IT Administrator | Faculty of Physics Office: D2-110 | Universitätsstr. 25 Phone: +49 521 106 5234 | 33615 Bielefeld ___ ceph-users mailin

[ceph-users] Re: User + Dev Meetup Tomorrow!

you run bare-metal or virtualized? Best, Sebastian Am 24.05.24 um 12:28 schrieb Frédéric Nass: Hello everyone, Nice talk yesterday. :-) Regarding containers vs RPMs and orchestration, and the related discussion from yesterday, I wanted to share a few things (which I wasn't able to

[ceph-users] Re: Centos 7 Kernel clients on ceph Quincy -- experiences??

m not sure. Running a mainline kernel from elrepo prevents this issue. I imagine that you might possibly run into these issues with a Quincy cluster as well, if you are using CephFS snapshots at all. Cheers Sebastian On 20.09.22 13:34, Ackermann, Christoph wrote: Hello all, i would li

[ceph-users] Re: Expose rgw using consul or service discovery

Am 20.10.21 um 09:12 schrieb Pierre GINDRAUD: > Hello, > > I'm migrating from puppet to cephadm to deploy a ceph cluster, and I'm > using consul to expose radosgateway. Before, with puppet, we were > deploying radosgateway with "apt install radosgw" and applying upgrade > using "apt upgrade radosg

[ceph-users] Re: MDS and OSD Problems with cephadm@rockylinux solved

In case you still have the error messages and additional info, do you want to create a tracker issue for this? https://tracker.ceph.com/projects/orchestrator/issues/new . To me this sounds like a network issue and not like a rockylinux issue. Am 26.10.21 um 13:17 schrieb Magnus Harlander: > Hi, >

[ceph-users] Re: cephadm does not find podman objects for osds

Some thoughts: * Do you have any error messages form the MDS daemons? https://docs.ceph.com/en/latest/cephadm/troubleshooting/#gathering-log-files  * Do you have any error messages form the OSDs? * What do you mean by "osd podman object"? * Try downgrading to 3.0.1 Am 25.10.21 um 23

[ceph-users] One cephFS snapshot kills performance

y interested in an explanation for this behaviour. Of course I would be very thankful for a solution of the problem or an advice that could help. Thanks in advance. Best wishes, Sebastian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscrib

[ceph-users] Re: Optimal Erasure Code profile?

5 active servers. Instead of 66% with the simple Host failure domain on 6 aktive servers and “k=4, m=2” erasure coding. But I don't advise you to do that! Best Sebastian > On 05.11.2021, at 06:14, Zakhar Kirpichenko wrote: > > Hi! > > I've got a CEPH 16.2.6 cluste

[ceph-users] Re: One cephFS snapshot kills performance

nterested in investigating this > issue we are more than happy to provide more info. I also would be happy to provide more infos. Best wishes, Sebastian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Expose rgw using consul or service discovery

S, >> SMB and S3. >> Upgrades are done live via apt upgrade We do not use cephadm, we >> provide a web based deployment ui (wizard like steps) as well as ui >> for cluster management. >> For nginx, we use the upstream method to configure the load balancing >> of

[ceph-users] Re: OSD repeatedly marked down

f the system. Maybe you could also work with some Ceph config setting, e.g. lowering osd_max_backfills (although I'm definitely not an expert on which parameters would give you the best result). Adding swap will most likely only produce other issues, but might be a method of last r

[ceph-users] Re: 16.2.7 pacific QE validation status, RC1 available for testing

Am 29.11.21 um 18:23 schrieb Yuri Weinstein: > Details of this release are summarized here: > > https://tracker.ceph.com/issues/53324 > Release Notes - https://github.com/ceph/ceph/pull/44131 > > Seeking approvals for: > > rados - Neha rados/cephadm looks good. Except for https://tracker.ceph.com/

[ceph-users] Re: v16.2.7 Pacific released

ation automatically, in case something goes wrong: https://github.com/ceph/ceph/pull/44252 I hope that helps! Best, Sebastian Am 08.12.21 um 10:42 schrieb Robert Sander: > Am 08.12.21 um 01:11 schrieb David Galloway: > >> * Cephadm & Ceph Dashboard: NFS management has been completely rewor

[ceph-users] Re: cephfs kernel client + snapshots slowness

. Cheers Sebastian On 10.12.21 17:33, Andras Pataki wrote: Hi, We've recently started using cephfs snapshots and are running into some issues with the kernel client.  It seems like traversing the file system and stat'ing files have become extremely slow.  Some (not all) stat operations

[ceph-users] Re: cephfs quota used

really like it! Just beware that some software might have issues with this behaviour - alpine is the only example (bug report and patch proposal have been submitted) that I know of. Cheers Sebastian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Octopus: conversion from ceph-ansible to Cephadm causes unexpected 15.2.15→.13 downgrade for MDSs and RGWs

he adopt playbook. Not because this path is better or worse, but because it's better tested. Guillaume, should we recommend this somehow in the ceph-ansible docs? Best, Sebastian > > When freshly upgraded to Octopus with ceph-ansible, the entire cluster > is at version 15.2.15. A

[ceph-users] Re: airgap install

Hi Zoran, I'd like to have this properly documented in the Ceph documentation as well.  I just created https://github.com/ceph/ceph/pull/44346 to add the monitoring images to that section. Feel free to review this one. Sebastian Am 17.12.21 um 11:06 schrieb Zoran Bošnjak: > Kai, thank

[ceph-users] 3 OSDs can not be started after a server reboot - rocksdb Corruption

e OSD logs from the time around the server reboot at the following link: https://we.tl/t-fArHXTmSM7 Ceph version: 16.2.6 Thanks, Sebastian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 3 OSDs can not be started after a server reboot - rocksdb Corruption

4.004041] systemd[1]: Reached target ceph target allowing to start/stop all ceph-osd@.service instances at once. I try to execute `ceph-bluestore-tool fsck --path /var/lib/ceph/osd/ceph-7` which again result in fsck failed: (5) Input/output error but this does not produce a single

[ceph-users] Re: 3 OSDs can not be started after a server reboot - rocksdb Corruption

Hi Igor, I now fixed my wrong OSD debug config to: [osd.7] debug bluefs = 20 debug bdev = 20 and you can download the debug log from: https://we.tl/t-3e4do1PQGj Thanks, Sebastian > On 21.12.2021, at 19:44, Igor Fedotov wrote: > > Hi Sebastian, > > first

[ceph-users] Re: 3 OSDs can not be started after a server reboot - rocksdb Corruption

storage server. So the data loss was not really a problem. best, Sebastian > On 29.12.2021, at 03:21, Mazzystr wrote: > > Did you check the status of your osd softlinks? > > See this mail thread > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/L47BFQPIPAA3J7

[ceph-users] Re: Single Node Cephadm Upgrade to Pacific

Hi Nathan, Should work, as long as you have two MGRs deployed. Please have a look at ceph config set mgr mgr/mgr_standby_modules = False Best, Sebastian Am 08.01.22 um 17:44 schrieb Nathan McGuire: > Hello! > > I'm running into an issue with upgrading Cephadm v15 to v16 on

[ceph-users] Re: Ideas for Powersaving on archive Cluster ?

. The current APM level can be checkt by $ hdparm -I /dev/sda | grep level > Advanced power management level: 128 You can set it by $ hdparm -B 128 /dev/sda The man page of hdparam states ..“values 128 through 254 (which do not permit spin-down)”… Best regards, Sebastian > On 12.01.202

[ceph-users] Re: 3 OSDs can not be started after a server reboot - rocksdb Corruption

g new snapshots unexpectedly. It my have something to do with a reboots of the ceph FS MDS, but I’m not sure. Best regards, Sebastian > On 24.12.2021, at 13:05, Igor Fedotov wrote: > > Hey Sebastian, > > On 12/22/2021 1:53 AM, Sebastian Mazza wrote: >> >>> 9)

[ceph-users] Re: Ideas for Powersaving on archive Cluster ?

d tear of the motors from spin ups on an archive system that are only used once in a view days. However, it could be that it heavily depends on the drives and I was only extraordinary lucky with all the WD, HGST and Seagate drives in our backup machines. Best regards, Sebastian ___

[ceph-users] Re: Ideas for Powersaving on archive Cluster ?

> When having software raid solutions, I was also thinking about spinning them > down and researching how to do this. I can't exactly remember, but a simple > hdparm/sdparm command was not sufficient. Now I am bit curious if you solved > this problem with mdadm / software raid? > On the first

[ceph-users] Re: Ideas for Powersaving on archive Cluster ?

seconds and spines up again. The OSD daemon doesn't seem to care at all about the spin down. @Gregorys: Thank you for you explanation. Best regards, Sebastian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to

[ceph-users] Re: 3 OSDs can not be started after a server reboot - rocksdb Corruption

hat none of the servers ever faced a non clean shutdown or reboot. Best regards, Sebastian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ideas for Powersaving on archive Cluster ?

> Hmm, I see on the man page > > -B > Get/set Advanced Power Management feature, if the drive > supports it. A low value means aggressive power management > and a high value means better performance. Possible > settings range from values 1 through

[ceph-users] Re: 3 OSDs can not be started after a server reboot - rocksdb Corruption

Corruption: Bad table magic number: expected 9863518390377041911, found 0 in db/002182.sst 2021-12-12T21:21:44.577+0100 7f6d18d2bf00 -1 bluestore(/var/lib/ceph/osd/ceph-8) _open_db erroring opening db: Best regards, Sebastian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Is it possible to stripe rados object?

libradosstriper ? Am 26.01.22 um 10:16 schrieb lin yunfan: > Hi, > I know with rbd and cephfs there is a stripe setting to stripe data > into multiple rodos object. > Is it possible to use librados api to stripe a large object into many > small ones? > > linyunfan > ___

[ceph-users] CephFS Snapshot Scheduling stops creating Snapshots after a restart of the Manager

eat INT NOT NULL, schedule TEXT NOT NULL, created_count INT DEFAULT 0, pruned_count INT DEFAULT 0, active INT NOT NULL, FOREIGN KEY(schedule_id) REFERENCES schedules(id) ON DELETE CASCADE, UNIQUE (schedule_id, start, repeat) ); INSERT IN

[ceph-users] Re: CephFS Snapshot Scheduling stops creating Snapshots after a restart of the Manager

hedule/log_level debug`, and I would be happy to share it. > Could you please file a tracker here: > https://tracker.ceph.com/projects/cephfs/issues/new I signed up for an account, but need to wait for being approved by an administrator. Cheer

[ceph-users] Re: CephFS Snapshot Scheduling stops creating Snapshots after a restart of the Manager

Hey Venky, I would be happy if you create the issue. Under this link: https://www.filemail.com/d/skgyuyszdlgrkxw you can download the log file and also my description of the problem. The txt also includes the most interesting lines of the log. Cheers, Sebastian > On 28.01.2022, at 11

[ceph-users] Re: 3 OSDs can not be started after a server reboot - rocksdb Corruption

shutdown of the cluster and OSD failing happened. I’m now thinking of how I could add more storage space for the log files to each node, so that I can leave on the debug logging all the time. Best regards, Sebastian ___ ceph-users mailing list -- ceph

[ceph-users] Re: 3 OSDs can not be started after a server reboot - rocksdb Corruption

Hi Igor, please find the the startup log under the following link: https://we.tl/t-E6CadpW1ZL It also includes the “normal" log of that OSD from the the day before the crash and the RocksDB sst file with the “Bad table magic number” (db/001922.sst) Best regards, Sebastian > On 21

[ceph-users] Re: 3 OSDs can not be started after a server reboot - rocksdb Corruption

rly 2 minutes before it receives the termination request. Shouldn't this be enough time for flushing every imaginable write cache? I hope this helps you. Best wishes, Sebastian ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send a

[ceph-users] Re: 3 OSDs can not be started after a server reboot - rocksdb Corruption

debug bluestore = 20 ``` in ceph.conf? Thanks, Sebastian > On 21.02.2022, at 21:19, Igor Fedotov wrote: > > Hey Sebastian, > > thanks a lot for the new logs - looks like they provides some insight. At > this point I think the root cause is apparently a race betw

[ceph-users] Re: 3 OSDs can not be started after a server reboot - rocksdb Corruption

. Everything look perfectly fine. So I think we can now be sure that it is not a memory issue. Thanks for reminding me to check memory! Best regards, Sebastian > On 23.02.2022, at 02:03, Alexander E. Patrakov wrote: > > I have another suggestion: check the RAM, just in case, with

[ceph-users] Re: 3 OSDs can not be started after a server reboot - rocksdb Corruption

it down. I'm really looking forward to your interpretation of the logs. Best Regards, Sebastian > On 22.02.2022, at 11:44, Igor Fedotov wrote: > > Hi Sebastian, > > On 2/22/2022 3:01 AM, Sebastian Mazza wrote: >> Hey Igor! >> >> >>> thanks a

  1   2   >