OK, I think I am good now. I have completed the rest of the steps (exit shell,
start the osd.10 daemon back up again, waited for it to be marked as "Up") and
then I ran:
ceph osd metadata 10
and all the bluestore_db items are pointing to the SSD now.
OK, so now to note these steps/commands in
where
you want to move DBs out of the main device. But if you want to create OSDs
from scratch, use the orchestrator, it does all of the work. And don't forget
the --dry-run flag if you want to apply a new spec (ceph orch apply -i
osd-spec.yaml --dry-run).
Regards,
Eugen
Zitat von Alan Murr
s an
empty test cluster). Also note that this is a single-node cluster, so the
orchestrator commands and shell commands are all executed on the same host.
Let me know how it goes.
Zitat von Alan Murrell :
> Ok, just gave it a try and I am still running into an error. Here is
> exactly wh
> ceph-volume lvm new-db --osd-id 10 --osd-fsid
> fb0e0a45-75a0-4400-9b1f-7568f185544c --target cephdb03/ceph-osd-db1
OMG, that worked. I hit a snag on the "migrate" command, but I think I know
what I need to do, but wanted to confirm.
When I ran:
ceph-volume lvm migrate --osd-id 10 --osd-fsi
> Oh, was my formatting really that bad as it looks in your response? I
> apologize for that!
It was, actually; all the commands were one after the other. It's OK; I was
able to make it out. Might have something to do with how my MUA (Outlook)
formatted it.
> Can you show the output of:
>
>
> Alright, and does either cephadm.log (/var/log/ceph/cephadm.log) or
> ceph-volume.log (/var/log/ceph/{FSID}/ceph-volume.log) provide more details?
> Which OS is the host running on?
The host OS is Debian 12.
In the cephadm.log file, I don't see anything around the time I did the
ceph-volume
Hello,
I am running a Ceph cluster installed with cephadm. Version is 18.2.2 reef
(stable).
I am moving the DB/WAL from my HDDs to SSD, and have been doing fine on all the
OSDs until I got to one in particular (osd.14)
>From the cephadm shell, when I run 'ceph orch daemon stop osd.14', nothin
Hello,
I ran the commands from the "regular" cephadm shell ('cephadm shell --') but
did not specify an OSD (I wasn't aware you could do that)
When you ran the ceph-volume command, did you do it within the container of the
OSD? Running it with the container of the OSD would make sense as to why
> Definitely stop the daemon (and set the noout flag), otherwise you won’t be
> able to modify anything.
OK, so then basically just run through the steps/commands in that article that
I linked in my original post, but just make sure to run the 'ceph-volume'
commands from within the specific OSD
Ok, just gave it a try and I am still running into an error. Here is exactly
what I did:
I logged on to my host where osd.10 is
Deleted my current VG and LV's on my NVME that will hold the WAL/DB, as I kind
of liked what you used. My VG is called "cephdb03" and my LVs are called
"ceph-osd-db
Ah, I didn’t notice the release it was for. Sorry ☹ I now know to pay closer
attention to that when I am on the Ceph docs ☹
From: Anthony D'Atri
Sent: February 4, 2025 16:19
To: Anthony D'Atri
Cc: Alan Murrell ; ceph-users@ceph.io
Subject: Re: [ceph-users] Spec file: Possible typo
Sorry, I just got on this list and am asking a number of questions… ☹
It was suggested in one of my other threads to create and apply a spec file
that would have cephadm put the DB for rotational drives onto SSD when it
creates the OSD on those drives. I came up with he following spec file:
--
> I strongly recommend to let cephadm handle the entire process of OSD
> creation, just ensure you have a fitting spec file. Don't prepare any LVs
> manually beforehand, except for such a case as right now where you want to
> move DBs out of the main device.
So if I create a spec file so Ceph a
specific), but I couldn't find them either. I'll
ping Zac about it.
Zitat von Alan Murrell :
> OK, I think I am good now. I have completed the rest of the steps
> (exit shell, start the osd.10 daemon back up again, waited for it to
> be marked as "Up") and then I ran:
> What's the output of
>
> ceph osd ok-to-stop 14
Here it is:
--- START ---
root@cephnode01:/# ceph osd ok-to-stop 14
{"ok_to_stop":true,"osds":[14],"num_ok_pgs":10,"num_not_ok_pgs":0,"ok_become_degraded":["18.0","18.2","18.3","18.f","18.1b","18.26","18.27","18.29","18.2e","18.3a"]}
--- END ---
> You can always fail the mgr (ceph mgr fail) and retry.
I failed the current active mgr (cephnode01) and the secondary took over, but
running the 'ceph orch daemon stop osd.14' still did not work.
Something *seems* to wonky with this OSD< but everything appears healthy. I
even ran a SMART te
Hello,
I am not sure if this is the right list to put this, but I was just looking
over the documentation for the Service Spec file:
https://docs.ceph.com/en/octopus/cephadm/drivegroups/
and under one of the "Advanced Cases" example, I believe there *might* be an
error? Here is the example (f
> Is there anything in the OSD logs?
When I issue 'ceph orch daemon stop osd.14', nothing is showing up in '
/var/log/ceph/474264fe-b00e-11ee-b586-ac1f6b0ff21a/ceph-osd.14.log' (I was
doing a 'tail -f' on the log file at the time I issued to daemon stop command
and absolutely nothing showed up
Hello,
I Googled for this answer but could not find it. I am running Reef 18.2.2 .
I need to find out if krbd is enabled on my cluster but I cannot figure out
what command to check that. I have checked the Pool settings in the GUI but I
see nothing about krbd.
I suspect it is enabled, but I
> cephadm unit stop --name osd.14
>
> Does that work? If not, you could check the cephadm.log for more hints. If it
> works, you can start the OSD again with:
Yep! That does indeed work! I will add that to my notes as an alternate
command should an OSD not respond to the 'ceph orch daemon' com
> Isn't krbd (as opposed to using librbd) a client-side thing?
Yup, you are right. I wasn't seeing the "KRBD" option in the storage options
in my Proxmox, but I was looking in the wrong view. I see it now.
Thanks! 😊
___
ceph-users mailing list -- cep
Hello,
I posted this on the Reddit sub, but it was suggested I might get better
responses here.
I am running a 5-node Ceph cluster (v18.2.2) installed using "cephadm".
I am trying to migrate the DB/WAL on our slower HDDs to NVME; I am following
this article:
https://docs.clyso.com/blog/ceph-v
Hello,
Thanks for the response.
OK, good to know about the 5% misplaced objects report 😊
I just checked 'ceph -s' and the misplaced objects is showing 1.948%, but I
suspect I will see this up to 5% or so later on 😊
It does finally look like there is progress being made, as my "active+clean" is
gh we
technically still have three "very thin" NVME slots available as well (u.3
connector types, I think?), but we are currently using with m.2 adapters for
the WAL/DB SSDs.
-Original Message-
From: Anthony D'Atri
Sent: March 23, 2025 22:57
To: Alan Murrell
Cc: cep
On Mon, 2025-03-24 at 15:35 -0700, Anthony D'Atri wrote:
> So probably all small-block RBD?
Correct. I am using RBD pools.
> Since you’re calling them thin, I’m thinking that they’re probably
> E3.S. U.3 is the size of a conventional 2.5” SFF SSD or HDD.
Hrm, my terminology is probably confusi
Hi Frederic,
> We've been successfully using PetaSAN [1] iSCSI gateways with an external
> Ceph cluster (not the one deployed by PetaSAN itself) on VMware hypervisors
> (135 VMs, 75TB) for a year and a half.
I am interested in this.
We are currently using a TrueNAS VM on our Proxmox cluster th
On Fri, 2025-02-28 at 08:49 +1300, Christian Wuerdig wrote:
> Only if you add more nodes and keep the replication factor at 3 your
> available space will increase.
I would imagine adding more drives to the existing nodes (once you hit
the three node number) would also increase space?
_
On Thu, 2025-02-27 at 20:50 +0100, Janne Johansson wrote:
> In the end, you need to wait until rebalancing has moved the data
> into
> the new OSDs so that one of the previous OSDs get more free space.
> This can take a while.
I can attest to this. PG Autoscaler had an issue, so I had an
extremel
Hello,
We have a 5-node cluster that each have the following drives:
* 4 x 16TB HDD
* 4 x 2TB NVME
* 1 x 1TB NVME (for the WAL/DB for the HDDs)
The nodes don't have any more room to add more NVMEs, but they do have room to
add four more HDDs. I know adding more HDDs are able to make the
Hello,
We had a drive (OSD) failed innour 5 node cluster three days ago (late
afternoon of Mar 20). The PGs have sorted themselves out, but the cluster has
has been recovering with backfill since then. Every time I run a 'ceph -s' it
shows a little over 5% misplaced objects with several jobs
OK, so just an update that the recovery did finally complete, and I am pretty
sure that the "inconsistent" PGs were PGs that the failed OSD were part of.
Running 'ceph pg repair' has them sorted out, along with the 600+ "scrub
errors" I had.
I was able to remove the OSD from the cluster, and a
Thank you everyone, for your responses and advice. VERY much
appreciated.
> Unless you have to reboot a node after OS upgrade. In this case send
> a node into maintenance mode (ceph orch host maintenance enter
> --force) before rebooting.
Ah, I will add that one in to my Ceph notes. If I have
Hello,
I just wanted to check quickly, but feel I am being a little *too*
cautious...
We have a Ceph cluster (installed via cephadm) that is currently
version 18.2.2. The "Upgrade" tab gives me an option to upgrade to
18.2.7 (its default selection), or I can upgrade to another version
where I ca
OS and Ceph are
basically two very separate(d) things?
Regards,
Alan
On Tue, 2025-07-22 at 18:34 -0400, Anthony D'Atri wrote:
> Yes.
>
> > On Jul 22, 2025, at 6:33 PM, Alan Murrell wrote:
> >
> > Hello,
> >
> > I just wanted to check quickly, but fe
34 matches
Mail list logo
