Hi,
I used this but all returns "directory inode not in cache"
ceph tell mds.* dirfrag ls path
I would like to pin some subdirs to a rank after dynamic subtree
partitioning. Before that, I need to know where are they exactly
Thank you,
Ben
__
Hi,
this directory is very busy:
ceph tell mds.* dirfrag ls
/volumes/csi/csi-vol-3a69d51a-f3cd-11ed-b738-964ec15fdba7/
while running it, all mds output:
[
{
"value": 0,
"bits": 0,
"str": "0/0"
}
]
Thank you,
Ben
Patrick
Dear cephers,
we have a cephfs volume, that will be mounted by many clients with
concurrent read/write capability. From time to time, maybe when concurrency
goes as high as 100 clients' access, accessing it will become very slow to
be useful at all.
the cluster has multiple active mds. All disks a
e was a recent thread [1] where Zakhar explained nicely how many
> IOPS you can expect from a hdd-only cluster. Maybe that helps.
>
> [1]
>
> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/FPMCNPYIHBHIJLWVVG2ECI2DSTR6DZIO/
>
> Zitat von Ben :
>
> > Dear ce
with the same mds configuration, we see exactly the same(problem, log and
solution) with 17.2.5, constantly happening again and again in couples days
intervals. MDS servers are stuck somewhere, ceph status reports no issue
however. We need to restart some of the mds (if not all of them) to restore
Dear Cephers,
With improvement on dmclock in 17.2.7, we are considering upgrade from
17.2.5 to 17.2.7.
seeing this which is worrysome:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/SG7CKALU3AIWEIVN7QENIY3KRETUQKM7/
any suggestions for an easy upgrade?
Regards,
Ben
ssl.c:897)'),))
debug 2022-10-21T09:07:22.079+ 7f019f728700 0 [dashboard INFO
dashboard.module] Engine started.
It looks like the rook module complains the k8s api server has a self
sighed certification. Any configuration can be done to resolve the issue?
Thanks,
Ben
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
ceph nfs export delete nfs4rgw /bucketexport
Ganesha servers go back normal:
rook-ceph-nfs-nfs1-a-679fdb795-82tcx 2/2 Running
0 4h30m
rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 2/2 Running
10 4h33m
Any ideas to make it work?
Thanks
d that in the
course.
Thanks,
Ben
osd0 log:
debug 2023-03-04T04:41:50.620+ 7f824d7043c0 0 set uid:gid to 167:167
(ceph:ceph)
debug 2023-03-04T04:41:50.620+ 7f824d7043c0 0 ceph version 17.2.5
(98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process
ceph-osd, pid 1
debug 202
restore that in the
course.
Thanks,
Ben
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
initialize
The fact is original monitor clusters ids are k,m,o, however the new ones
are a,b,d. It was deployed by rook. Any ideas to make this work?
Ben
Eugen Block 于2023年3月9日周四 16:00写道：
> Hi,
>
> there's no attachment to your email, please use something like
> pastebin to
I could start a mon daemon cluster with exact ID as original one( something
like k,m,o). Any thoughts on this?
Ben
Eugen Block 于2023年3月9日周四 20:56写道：
> Hi,
>
> I'm not familiar with rook so the steps required may vary. If you try
> to reuse the old mon stores you'll have
d.service', 'ntpd.service',
'ntp.service', 'ntpsec.service', 'openntpd.service']
ERROR: Distro uos version 20 not supported
uname -a
Linux 4.19.0-91.82.42.uelc20.x86_64 #1 SMP Sat May 15 13:50:04 CST
2021 x86_64 x86_64 x86_64 GNU/Linux
Thank you in
)
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in
_make_cd_request
self._fs.basename(path))
File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in
make_request
raise exc
Any ideas on this?
Thanks,
Ben
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
.
Ben
Eugen Block 于2023年5月8日周一 21:01写道：
> Hi,
>
> could you provide some more details about your host OS? Which cephadm
> version is it? I was able to bootstrap a one-node cluster with both
> 17.2.5 and 17.2.6 with a non-root user with no such error on openSUSE
> Leap 15.4:
>
Get one of the time services up and running, you then get through this. The
error message is quite of misleading?
Ben 于2023年4月26日周三 15:07写道：
> Hi,
> This seems not very relevant since all ceph components are running in
> containers. Any ideas to get over this issue? Any other
appen when
> they had changed the umask to 027. We had patched in
> https://github.com/ceph/ceph/pull/50736 to address it, which I don't
> think was merged too late for the 17.2.6 release.
>
> On Mon, May 8, 2023 at 5:24 AM Ben wrote:
>
>> Hi,
>>
>> with foll
rtunately, I
> can't use the curl-based cephadm but the contents are identical, I
> compared. Not sure what it could be at the moment.
>
> Zitat von Ben :
>
> > Hi, It is uos v20(with kernel 4.19), one linux distribution among others.
> > no matter since cephadm depl
mgr up running again through cephadm?
Thanks,
Ben
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
hem for obvious reasons (these are just on a test cluster
> I'll tear back down so it's fine for me) but those are the first couple
> things I'd check. You could also try to make adjustments directly to the
> unit.run file if you have other things you'd like to
you could check owner of /var/lib/ceph on host with grafana container
running. If its owner is root, change to 167:167 recursively.
Then systemctl daemon-reload and restart the service. Good luck.
Ben
Adiga, Anantha 于2023年5月17日周三 03:57写道：
> Hi
>
> Upgraded from Pacific 16.2.5 to 17.2
use this to get relevant long lines in log:
journalctl -u ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e@grafana-fl31ca104ja0201
| less -S
it is '--user 472' by content of unit.run, not the default ceph user
167. Maybe set the directory owner to 472 could help.
Hope it helps
Ben
Adig
0 PM
[INF]
Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow
metadata IOs)
5/30/23 8:12:40 PM
[INF]
MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 30
secs, oldest blocked for 38 secs
However, random write test is performing very good.
Any sugg
something with cephfs
kernel client?
Any other thoughts?
Patrick Donnelly 于2023年5月31日周三 00:58写道：
> On Tue, May 30, 2023 at 8:42 AM Ben wrote:
> >
> > Hi,
> >
> > We are performing couple performance tests on CephFS using fio. fio is
> run
> > in k8s pod an
Hi,
I got many critical alerts in ceph dashboard. Meanwhile the cluster shows
health ok status.
See attached screenshot for detail. My questions are, are they real alerts?
How to get rid of them?
Thanks
Ben
___
ceph-users mailing list -- ceph-users
CephFS filesystem is degraded
CephFilesystemFailureNoStandby critical mds 1 minute MDS daemon failed, no
further standby available
Meanwhile the cluster status is green ok. What should we do for this?
Thanks,
Ben
___
ceph-users mailing list -- ceph-users
Hi,
see many of this in cluster log channel. many are blocked with long period
of seconds. It should hurt client access performance. Any ideas to get rid
of them?
Thanks,
Ben
-
7/7/23 4:48:50 PM
[WRN]
Health check update: 8 MDSs report slow metadata IOs
Hi,
In our cluster monitors' log grows to couple GBs in days. There are quite
many debug message from rocksdb, osd, mgr and mds. These should not be
necessary with a well-run cluster. How could I close these logging?
Thanks,
Ben
___
ceph-users ma
just rechecked debug_mon is by default 1/5. mgr/cephadm log_to_cluster
level has been set to critical from debug. Wonder how to set others' level.
Haven't got a clue to do that.
Thanks,
Ben
Wesley Dillingham 于2023年7月10日周一 23:21写道：
> At what level do you have logging set to for yo
Hi,
We have a cluster running for a while. From grafana ceph dashboard, I saw
OSD onode hits ratio 92% when cluster was just up and running. After couple
month, it says now 70%. This is not a good trend I think. Just wondering
what should be done to stop this trend.
Many thank,
Ben
Hi,
It is running 17.2.5. there are slow requests warnings in cluster log.
ceph tell mds.5 dump_ops_in_flight,
get the following.
These look like outdated and clients were k8s pods. There are warning of
the kind in other mds as well. How could they be cleaned from warnings
safely?
Many thanks.
.
It looks like bugs with client library. And rebooting nodes with problem
will fix it for short period of time? Any suggestions from community for
fixing?
Thanks,
Ben
[root@8cd2c0657c77 /]# ceph health detail
HEALTH_WARN 6 hosts fail cephadm check; 2 clients failing to respond to
capability
195341004/893374309813 to expire
debug 2023-09-27T15:27:33.024+ 7f8f047cf700 10 mds.5.log
_trim_expired_segments waiting for 189942575/642272326541 to expire
Regards,
Ben
Ben 于2023年9月26日周二 20:31写道：
> Hi,
> see below for details of warnings.
> the cluster is running 17.2.5. the warn
definitely interrupt business. Any
thoughts to stop the warnings?
Best wishes,
Ben
Venky Shankar 于2023年9月28日周四 11:56写道：
> Hi Ben,
>
> On Tue, Sep 26, 2023 at 6:02 PM Ben wrote:
> >
> > Hi,
> > see below for details of warnings.
> > the cluster is running 17.2.5. th
Dear cephers:
more log captures(see below) show the full segments list(more than 3 to
be trimmed stuck, growing over time). any ideas to get out of this?
Thanks,
Ben
debug 2023-09-30T14:34:14.557+ 7f9c29bb1700 5 mds.4.log trim already
expiring segment 195341004/893374309813, 180 events
Yes, I am. 8 active + 2 standby, no subtree pinning. What if I restart the
mds with trimming issues? Trying to figure out what happens with restarting.
Venky Shankar 于2023年10月3日周二 12:39写道：
> Hi Ben,
>
> Are you using multimds without subtree pinning?
>
> On Tue, Oct 3, 2023
egment list
continually grows overtime. Any ideas for having the segment list to be
processed well normal again?
The issue has been around for weeks and haven't seen complaints from
storage client side so far.
Best wishes,
Ben
Venky Shankar 于2023年10月4日周三 13:31写道：
> Hi Ben,
>
> On T
draw
concerns overtime.
Any other ideas?
Thanks,
Ben
Eugen Block 于2023年10月4日周三 16:44写道：
> Hi,
>
> is this still an issue? If so, I would try to either evict the client
> via admin socket:
>
> ceph tell mds.5 client evict [...] --- Evict client
> session(s) based on a filter
&g
Hi,
It get cleared by restarting ceph client with issues. It works. to do that,
you would umount problematic cephfs volume and remount. All ceph warning is
gone in couple minutes, trimming well now. Indeed I wouldn't restart mds
unless I had to.
Many thanks for help,
Ben
Eugen Block 于2023年
,
deployed by cephadm. The slowness can do harm in peak hours of usage.
Best wishes,
Ben
-
volumes: 1/1 healthy
pools: 8 pools, 209 pgs
objects: 93.04M objects, 4.8 TiB
usage: 15 TiB used, 467 TiB / 482 TiB avail
pgs
not aware of?
胡 玮文 于2023年10月11日周三 00:18写道：
> Hi Ben,
>
> Please see this thread
> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/PWHG6QJ6N2TJEYD2U4AXJAJ23CRPJG4E/#7ZMBM23GXYFIGY52ZWJDY5NUSYSDSYL6
> for
> possible workaround.
>
> 发自我的 iPad
>
> 在 2023年1
I have a 3 node ceph cluster for my house that I have been using for a few
years now without issue. Each node is a MON, MGR, and MDS, and has 2-3 OSDs
on them. It has, however been slow. I decided to finally move the bluestore
DBs to SSDs. I did one OSD as a test case to make sure everything was go
A colleague asked me to take a look at a Ceph cluster that has stopped working.
The "ceph -s" command (any ceph command) just times out.
Of three monitors two are crashing with:
(gdb) bt
#0 0x7fffee17b7bb in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x7fffee166535 in abort () f
Thanks for your reply.
I tried something similar (but wrong) based on your messages in the
referenced threads, but I was missing the "gdb commands...end"
sequence so I just kept hitting my breakpoint over and over. However,
I was able to get the monitor running with your guidance.
Indeed, ceph rm
Yes, I have all MDS running now and I am in the process of building
ceph sources with your patch from the original thread. I also tried
the command:
ceph fs compat prod add_incompat 7 "mds uses inline data"
Since I noticed that option was defined for two of the three file
systems (the working one
I am having little luck building ceph on Debian bullseye. First of all
if I install the package ninja-build and run ninja in the build
directory, I get an error that ninja.build file is not found after
running ./do_cmake.sh. I am thus just executing make, which is failing
to compile some tests. I a
Building without tests worked and I am now operational once again.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
h (~6 months) and I haven’t emailed this list before. Sorry
in advance if I’ve mistakenly broken any roles and thanks for the help!
- Ben M
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
skip)
4, delete the export:
ceph nfs export delete nfs4rgw /bucketexport
Ganesha servers go back normal:
rook-ceph-nfs-nfs1-a-679fdb795-82tcx 2/2 Running 0
4h30m
rook-ceph-nfs-nfs4rgw-a-5c594d67dc-nlr42 2/2 Running
10 4h
Hi All,
We have a RGW cluster running Luminous (12.2.11) that has one object with an
extremely large OMAP database in the index pool. Listomapkeys on the object
returned 390 Million keys to start. Through bilog trim commands, we’ve whittled
that down to about 360 Million. This is a bucket index
. Thanks again!
-Ben
From: Dan van der Ster
Date: Wednesday, April 26, 2023 at 11:11 AM
To: Ben.Zieglmeier
Cc: ceph-users@ceph.io
Subject: [EXTERNAL] Re: [ceph-users] Massive OMAP remediation
Hi Ben,
Are you compacting the relevant osds periodically? ceph tell osd.x
compact (for the three osds
/nautilus/#upgrading-from-mimic-or-luminous
the procedure is to upgrade each daemon one by one (MON -> MGR -> OSD).
However, it seems that the Debian packages are all interdependent on
ceph-base which means that upgrading one at a time is not possible.
I would be grateful for any advice.
seem to be a solution mentioned there.
https://bugzilla.redhat.com/show_bug.cgi?id=2056935 )
Below are some parts of the MDS logs that seem relevant to us for this
issue.
We are thankful for any ideas. :-)
Best
Ben
Log excerpt of Active MDS (replay):
-140> 2023-06-14T07:51:59.585+ 7feb58
ialization timeout to 600, and tried
removing all non-essential config from ceph.conf. All produce the same results.
I would greatly appreciate any other ideas or insight.
Thanks,
Ben
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe
I was able to get what I needed from http://mirrors.gigenet.com/ceph/ (one
of the mirrors listed in the Ceph doco).
On Thu, Nov 14, 2024, 6:05 AM Frank Schilder wrote:
> Hi all,
>
> +1 from me
>
> this is a really bad issue. We need access to these packages very soon.
> Please restore this folde
55 matches
Mail list logo
