; On Aug 1, 2025, at 11:03 PM, Devender Singh wrote:
>
> No all’s are ssd and none
>
> Regards
> Dev
>
>
> On Fri, 1 Aug 2025 at 7:58 PM, Anthony D'Atri
> wrote:
>
>> What is this the output of? Is this reading from HDDs?
>>
>> > On Aug 1,
Hello All
Using R3 pool with images running ubuntu vms.
Having three nodes…
I have default settings for ms_asyc threads to 3 and rbd_op_thread to 1
How to improve below latency…mainly write
# sed 's/^.\{60\}//' a
WRRDWR_BYTESRD_BYTES WR_LAT RD_LAT
50/s 0/s
e pr. Although it seems to be present in 19.2.3. I'm not quite
> sure if there is a workaround for this in the meantime.
>
>
> On Thu, 24 Jul 2025 at 11:03, Devender Singh <mailto:deven...@netskrt.io>> wrote:
>> Thanks for your reply…
>>
>> Currently
> mount -t ceph 10.101.1.246,10.101.1.247,10.101.1.248:/ /mnt/cephfs -o
> name=proxmoxvolfs,fs=van2-dev,secret=AQBw8X9o8b3GHRAAw58xzwqCP6Zt+WZZbjdb/A==
>
> Did you check you key file content (I've done this mistake with an empty
> key file generated by a wrong command 🙁)
>
> Patrick
&g
Hello
Thanks for your reply…
Just to clear.. currently no load, nothing running on ceph… the fit test was
done on a fresh installed vm to have a fit test output for reference before
deploying any further vms..
Below is the bench result, yes its replicated pool…
# rados bench -p vms-os-r3
Thanks for your reply…
Currently I disabled that option,, but below are the logs….
# zcat ceph-osd.*.log.1.gz |egrep -i "faile|error|down"
2025-07-11T17:57:27.303+ 7f2aeacb4680 4 rocksdb:
Options.error_if_exists: 0
2025-07-11T17:57:27.303+ 7f2aeacb4680 4 rocks
Hello all
Using 19.2.2, when enabling bdev_ioring and rebooting host then osd never comes
online as below…. Why?
Also Dasboard not matching with cli?
Host output in dashboard… showing running. Tried failing manager too but it
same.

Dashboard showing down

# ceph health detail
HEALTH
Hello all
I tried doing some fio test on local disk(NVME) and ceph rbd. Why ceph is
having low IO whereas it’s also on all NVME.
What to tune to reach equal amount of IO?
root@node01:~/fio-cdm# python3 fio-cdm ./
tests: 5, size: 1.0GiB, target: /root/fio-cdm 6.3GiB/64.4GiB
|Name| Read
Dev
> On Jul 22, 2025, at 1:19 PM, Devender Singh wrote:
>
> Thanks Patrick
>
> But still the same issue
>
> # mount -t ceph
> 10.101.1.246,10.101.1.247,10.101.1.248:/volumes/_nogroup/test/e393e142-977f-4d4a-817d-41bb721cb09c
>/mnt/cephfs -o name=proxmoxv
Hello all
Need some help to clean up below…
# ceph orch ps |grep -v running
NAME HOST
PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION
IMAGE ID CONTAINER ID
osd.0
my keefile is /etc/ceph/ceph.client.sebfolder.keyring
> * my cephfs volume is seb_vol
>
>
>>
>> It works
>>
>> Patrick
>>
>> Le 21/07/2025 à 23:32, Devender Singh a écrit :
>>> Hello Dario
>>>
>>> Thanks for your reply.
>
-t ceph admin@.van2-dev=/ /cephfs/ -o
> noatime,ms_mode=prefer-crc,_netdev
>
> Regards.
>
> [1]
> https://www.google.com/url?q=https://docs.ceph.com/en/squid/man/8/mount.ceph/&source=gmail-imap&ust=175345708700&usg=AOvVaw3RPvx2e7p7nZypeiYOOf9c
>
> On Wed,
On Jul 15, 2025, at 4:02 PM, Devender Singh wrote:
>
> Hello
>
>
> Tried creating EC pool, still facing same issue…
>
> https://bugzilla.redhat.com/show_bug.cgi?id=2255030
>
> ceph osd erasure-code-profile set ec-21-profile-isa k=2 m=1
> crush-failure-domain=ho
e=admin,fs=van2devcephfs
# mount -t ceph :/ /new_cephfs -o name=admin,fs=van2devcephfs
mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you
may not be authorized
Regards
Dev
> On Jul 15, 2025, at 3:08 PM, Devender Singh wrote:
>
> Tried below too but
-t ceph :/ /cephfs/ -o noatime,ms_mode=prefer-crc,_netdev,fs=van2-dev
mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you
may not be authorized
Regards
Dev
> On Jul 15, 2025, at 2:17 PM, Devender Singh wrote:
>
> Hello all
>
> I have cephfs wo
Hello all
I have cephfs working fine with running ads but still not able to mount,
getting error mount error: no mds (Metadata Server) is up
Below are details
# ceph osd dump | grep min_compat_client
require_min_compat_client reef
min_compat_client reef
# ceph balancer status
{
"active"
Hello
How to lower this load?
Cephfs is on EC pool sitting over replicated data and meta..
What settings can improve it?

Regards
Dev
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
Sorry !
Yes its 3+2, Now I remember still one is not yet replaced which went down…
Regards
Dev
> On Jul 10, 2025, at 3:51 PM, Peter Eisch wrote:
>
> ill likely be juggling % to get you
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscri
Hello all
Need some help why nothing is happening… version is 19.2.2.
Tried repairing the pgs, repair and force-backfill and force-recovery but
nothing initiating backfills.
# ceph -v
ceph version 19.2.2
# ceph -s
cluster:
id: 79d4fade-ffa9-11eb-a57d-f398f000cfc1
health: HEAL
Hello Peter
I hope you are not keeping images on EC Pool?
In my case my all data and images majorly on ec pools
Regards
Dev
On Wed, 2 Jul 2025 at 1:29 PM, Anthony D'Atri wrote:
> We need to see `ceph osd crush rule dump` and `ceph osd pool ls detail` to
> see which pools are using which CRUSH r
Hello
Seeking some help to recover these incomplete pgs…
I made osd, down then tried but nothing seems working ….
Ceph version 18.2.7, we have multiple disk failed in ceph cluster on different
nodes…
for i in 19.5a 19.5d 19.5e 19.6b 19.6f 19.74 19.7b 34.48 34.69; do
ceph pg map $i | grep
ght one to query an OSD for the
> cluster network, it just shows the public address of an OSD (like a client
> would need to). Just use 'ceph osd dump' and look at the OSD output.
>
>
> Zitat von Devender Singh :
>
>> Hello
>>
>> I checked on my al
s not the right one to query an OSD for the
> cluster network, it just shows the public address of an OSD (like a client
> would need to). Just use 'ceph osd dump' and look at the OSD output.
>
>
> Zitat von Devender Singh :
>
>> Hello
>>
>> I chec
Hello
I checked on my all clusters everywhere OSD’s not using cluster network.
Here is another example in my lab where I have three hosts in vlan1 and vlan2
running on one proxmox server and here also same thing…
No MTU change, these are default to 1500.
I don’t understand what I am missing?
ph orch device ls --hostname=phl-prod-host04n.example.comis --refresh
>
> and see if the upgrade resumes. It if doesn't, check
>
> $ ceph log last 1000 debug cephadm
>
> and run
>
> $ ceph orch upgrade pause
> $ ceph orch upgrade resume
>
> again, see if the upg
Hello All
Found some network issue on OSD’s, why they are not using cluster network? Only
using public network? My mons are on public network…
How can I resolve it? Changing this network will lead any issue like need to
stop all osd’s or restart cluster or rolling restart osd’s & how, please
ou need to compact OSD.
>>
>> I am not sure what you adjusted but as long as the cluster works, I would
>> not have changed parameters and try to fix the mentioned problems.
>>
>> Good luck,
>>
>> Michel
>> Sent from my mobile
>&
Thanks for your reply..
There were multiple OSD’s failure on different hosts and had to zap and read.
Which lead us to this issue… and upgrade was running too..
Now I see three options for upgrade “Back, Pause & Stop” — is back for
rollback? And which one I should use the best option….
Tried r
s",
"services_complete": [
"crash",
"mgr",
"mon"
],
"progress": "74/113 daemons upgraded",
"message": "",
"is_paused": false
}
Regards
Dev
> On Jun 19, 2025, at 8:06 PM
Hello all
I have a cluster where my cluster is in hung state, Some back fills are there
but I reduced it to 1 but still upgrade not progressing…
Please help…
```# ceph health detail
HEALTH_WARN 8 OSD(s) experiencing slow operations in BlueStore; Failed to apply
2 service(s): osd.all-availab
Hello All
Any resolution found for this?
I am also facing same issue, with slow Ops, tried restating odd’s but no luck?
Regards
Dev
> On Nov 26, 2024, at 3:46 AM, Sergio Rabellino wrote:
>
> hen I have strange (and bad) behaviours like these, I double check name
> resolution/DNS configurat
Hello all
Need urgent help on below…
I tried reducing min_size but still showing same…
cluster:
id: 15688cb4-044a-11ec-942e-516035adea04
health: HEALTH_ERR
3 failed cephadm daemon(s)
1 filesystem is degraded
1 MDSs report slow metadata IOs
Dear All
I upgrade my clusters from 18.2.4 & 18.2.6 to 18.2.7.
1. I started facing multiple disks went down..
2. Fio test showing 5-35 IOPS on cephfs?
$ fio --name=latency-test --ioengine=libaio --rw=randread
--bs=4k --size=512M --numjobs=1 --iodepth=1 --direct=1 --runtime=60
May 2025 at 9:14 AM, Devender Singh wrote:
> Hello All
>
>
> Need some help to tune scrubbing.
>
> 1. *How to control not run scrubbing and deep scrubbing together and
> which one should be started first?*
>
> * 124 active+clean+remapped*
>
> *
Hello all
Seeking some help to solve below…
1. # ceph health detail
HEALTH_WARN 2 OSD(s) experiencing slow operations in BlueStore
[WRN] BLUESTORE_SLOW_OP_ALERT: 2 OSD(s) experiencing slow operations in
BlueStore
osd.10 observed slow operation indications in BlueStore
osd.21 observe
Hello All
Need some help to tune scrubbing.
1. How to control not run scrubbing and deep scrubbing together and which one
should be started first?
124 active+clean+remapped
35 active+clean+scrubbing+deep
24 active+clean+scrubbing
4acti
Sorry some typo. It is mclock.
Exact parameter osd_recovery_max_active_ssd/hdd. Is 10, to reduce you have
to override mclock to true.
Restarting osd daemon alone will solve your issue.
Regards
Dev
On Fri, 2 May 2025 at 9:07 AM, Devender Singh wrote:
> Hello
>
> Try restarting osd
Hello
Try restarting osds showing slow ops.
Also if any recovery going on then max recovery drives for Malcom is 10 try
reducing it. Will resolve this issue.
If it persist for a drive then check for smart TK for errors and replace
that drive.
Regards
Dev
On Fri, 2 May 2025 at 8:45 AM, Maged Mokh
Hello Alex
As per Dan bug is in Reef which is v18.2.6 in open ceph.
I also upgraded my cluster to 18.2.6 before I saw the first message of this
mail chain and I am having 120osds but yet I have not seen any issue
where’s as one of my host remain down for 24hrs with 24 osds on it and now
I joined
Hello
Using boto3 trying to upload files to s3 bucket, when I am using boto version
1.35 it works fine, when upgrading it to 1.36 doesn’t work, facing below error,
python3 s3_upload.py
Error testing the new bucket: An error occurred (XAmzContentSHA256Mismatch)
when calling the PutObject opera
Hello all
Do we have a good cluster design calculator which can suggest failure
domain and pool size and min size according the number of nodes and drive
and their size for both replicated and EC pools.
Regards
Dev
___
ceph-users mailing list -- ceph-u
Hello all
Recently upgraded to 19.2.1 and started facing ..
Invalid grafana certificate on hostnode1: Invalid certificate key:
[('PEM routines', '', 'no start line’)]
Tried suggest steps, even re-deployed grafana but no solution?
1.
[WRN] CEPHADM_CERT_ERROR: Invalid grafana certific
👌🥳🕺🕺
On Thu, 6 Feb 2025 at 2:56 PM, Yuri Weinstein wrote:
> We're happy to announce the 1st backport release in the Squid series.
>
> https://ceph.io/en/news/blog/2025/v19-2-1-squid-released/
>
> Notable Changes
> ---
> * CephFS: The command `fs subvolume create` now allows tagging
>
Hello all
Thanks for your reply.
I already tried few things..
1. Tried deleting the old ec profile and thought to create it with same name
but with osd failure domain, it did not allow deleting from pool.
2. Changed crush rule with osd too. It reverted automatically to host.
3. Then Create
Hello all
I tried disabling osd_mclock_override_recovery_settings to true and increased
backfill from 1 to 15 after disabling scrub and deep-scrub.
There was no change I started from 1 then 3 then 5 then 7 then 10 then 13 and
15. But no change.
So it a default behaviour or bug?
# ceph config
Hello all
Is there a way to change ec pool profile from host to osd?
Regrds
Dev
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
uot;
"config-history/5150/+client.ceph-exporter.node03/container_image"
"config-history/5156/-client.ceph-exporter.node03/container_image"
"config-history/5179/+osd/host:node03/osd_memory_target"
"config-history/5179/-osd/host:node03/osd_memory_target&qu
Please ignore, worked after setting blank file with [].
Regards
Dev
> On Jan 31, 2025, at 10:55 PM, Devender Singh wrote:
>
> Hello
>
> Need some help..
>
> Tried draining host but it got stuck and now orchestrator is not running.
> But cluster health is OK
Hello
Need some help..
Tried draining host but it got stuck and now orchestrator is not running. But
cluster health is OK , also I adde host back.
# ceph health detail
HEALTH_OK
Tried adding blank queue file,, ceph config-key set
mgr/cephadm/osd_remove_queue -I osd_remove_queue_blank.json
Hello all
Few more suggestions.. if can be added to further releases.
1. We faced some issue, can we add more command to control clients using
watcher,
rbd status pool/image
Watchers:
watcher=10.160.0.245:0/2076588905 client.12541259 cookie=140446370329088
Some commands to control w
Hello Please find attached again..

Regards
Dev
> On Jan 29, 2025, at 10:58 AM, Afreen Misbah wrote:
>
> Hi Devender,
>
> Seems like the graphs (screenhots) are not visible. Can you attach those
> again ?
>
> Thanks
>
> Afreen
>
> On Thu, Jan
Hello Friends
Seeking some help on OSD write latency in "osd performance graph" whereas "osd
graph on dashboard" is not showing any high latency.
So, is dashboard graph showing only read latency as I can see low latency in
osd perf graph which matches the dashboard osd perf graph?
OSD per
to add it for
> the sake of completeness. I strongly suggest to re-add the fifth host (and
> think about adding a sixth).
>
> Regards,
> Eugen
>
> Zitat von Devender Singh mailto:deven...@netskrt.io>>:
>
>> +Eugen
>> Lets follow “No recovery after removing
+Eugen
Lets follow “No recovery after removing node - active+undersized+degraded--
removed osd using purge…”. Here.
Sorry I missed ceph version which is 18.2.4. (with 5 nodes, 22osd each, where I
removed one node and all mess.)
Regards
Dev
> On Jan 25, 2025, at 11:34 AM, Devender Si
Hello Eugen
Thanks for your reply.
ceph osd set nodeep-scrub is not stopping if repairs are running.
Reapir started another set for deepscrub+repair which is not controlled using
this command.
When I started my cluster utilization as 74% and when it finished now my
cluster is showin
t;,
"config-history/9710/-osd/host:node3/osd_memory_target",
"config/osd/host:node3/osd_memory_target”,
Regards
Dev
> On Jan 25, 2025, at 4:39 AM, Frédéric Nass
> wrote:
>
> Hi,
>
> I've seen this happening on a test cluster after draining a host
Hello all
Urgent help needed. No recovery happening.
Tried repairing pg and redeploy or create.
Rebooted cluster but no luck..
data:
volumes: 2/2 healthy
pools: 18 pools, 817 pgs
objects: 6.06M objects, 20 TiB
usage: 30 TiB used, 302 TiB / 332 TiB avail
pgs: 284
Hello all
Sorry for late reply..
I tried two things,
1. My cluster was using swap, I made swap off.
2. I started repair command on pool/image and seems it worked .
But after that there was no command found to pause repair as it started deep
scrubs too. How to unpause the repairs…
Regard
big error is Error ENOENT:
Regards
Dev
> On Jan 24, 2025, at 4:59 PM, Fnu Virender Kumar
> wrote:
>
> Did you try
>
> Ceph mgr module enable orchestrator
> Ceph orch set backend
> Ceph orch ls
>
> Check the mgr service daemon as well
> Ceph -s
>
&
Hello all
Any quick fix for …
root@sea-devnode1:~# ceph orch ls
Error ENOENT: Module not found
Regards
Dev
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
9 - /dev/rbd0
> sh-4.4# umount /dev/rbd/k8s-rgnl-disks/csi-vol-945c6a66-9129
> sh-4.4# rbd unmap /dev/rbd/k8s-rgnl-disks/csi-vol-945c6a66-9129
> sh-4.4# rbd device list
> sh-4.4#
>
> Hope there's no typo.
>
> Regards,
> Frédéric.
>
> - Le 21 Jan 25,
Hello Laura
Few more suggestions..
1. As we are facing some issue, can we add more command to control clients
using watcher,
rbd status pool/image
Watchers:
watcher=10.160.0.245:0/2076588905 client.12541259 cookie=140446370329088
Some commands to control watcher and kill client.id
Dear all
Sending again. Our mail server was having some issue, not sure if you have
received it or not.
Exactly Similar issue,
https://github.com/ceph/ceph-csi/discussions/4410
Regards
Dev
> On Jan 21, 2025, at 2:33 PM, Devender Singh wrote:
>
> Hello Eugen
>
> Thanks
Similar issue
https://github.com/ceph/ceph-csi/discussions/4410
Regards
Dev
On Tue, 21 Jan 2025 at 2:33 PM, Devender Singh wrote:
> Hello Eugen
>
> Thanks for your reply.
> I have the image available and it’s not under trash.
>
> When scaling a pod to different node usin
list the client
> to release the watcher, then delete the image again.
>
> I have to do that from time to time on a customer’s openstack cluster.
>
> Zitat von Devender Singh :
>
>> Hello
>>
>> Seeking some help if I can clean the client mounting my volume?
>>
Hello
Seeking some help if I can clean the client mounting my volume?
rbd status pool/image
Watchers:
watcher=10.160.0.245:0/2076588905 client.12541259 cookie=140446370329088
Issue: pod is failing in init- state.
Events:
Type Reason Age From Message
You moved some oSD’s I believe it’s looking for peer osds data too. But as
long as you keep
nobackfill,norebalance,norecover it will take longer and keep showing more
data to balance, and kept filling or writing data to volumes it will
accumulate. So unset and wait for some time to finish it.
Rega
Hello all
I tried configuring cephfs mirroring between cluster as per doc
https://docs.ceph.com/en/reef/dev/cephfs-mirroring/ using bootstrap.
But My replication is not working.
Using two mirror daemons on each cluster. [Enabled module/mirroring/snap
mirroring/aded directory/allow_new_snaps]
Few More If you like.
Metaverse
Jumbo
Regards
Dev
> On Aug 15, 2024, at 10:12 AM, Tarrago, Eli (RIS-BCT)
> wrote:
>
> Additional names that could be considered:
>
> Sea Monster
> Sea Bear
> Star (for Patrick star)
>
> And for the best underwater squirrel friend, Sandy.
>
>
> From: Boris
What is the fix for above error.
My OS is : ubuntu 20.04 and ceph version is 18.2.4
Currently I disabled the prediction. : ceph mgr module disable
diskprediction_local
Regards
Dev
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send
70 matches
Mail list logo
