Hi Jorge,
On 4/6/23 07:09, Jorge Garcia wrote:
We have a ceph cluster with a cephfs filesystem that we use mostly for
backups. When I do a "ceph -s" or a "ceph df", it reports lots of space:
data:
pools: 3 pools, 4104 pgs
objects: 1.09 G objects, 944 TiB
usage: 1.5 Pi
On 4/11/23 03:24, Thomas Widhalm wrote:
Hi,
If you remember, I hit bug https://tracker.ceph.com/issues/58489 so I
was very relieved when 17.2.6 was released and started to update
immediately.
Please note, this fix is not in the v17.2.6 yet in upstream code.
Thanks
- Xiubo
But now I'm s
On 4/11/23 15:59, Thomas Widhalm wrote:
On 11.04.23 09:16, Xiubo Li wrote:
On 4/11/23 03:24, Thomas Widhalm wrote:
Hi,
If you remember, I hit bug https://tracker.ceph.com/issues/58489 so
I was very relieved when 17.2.6 was released and started to update
immediately.
Please note, this
On 5/1/23 17:35, Frank Schilder wrote:
Hi all,
I think we might be hitting a known problem
(https://tracker.ceph.com/issues/57244). I don't want to fail the mds yet,
because we have troubles with older kclients that miss the mds restart and hold
on to cache entries referring to the killed in
Hi Emmanuel,
This should be one known issue as https://tracker.ceph.com/issues/58392
and there is one fix in https://github.com/ceph/ceph/pull/49652.
Could you just stop all the clients first and then set the 'max_mds' to
1 and then restart the MDS daemons ?
Thanks
On 5/3/23 16:01, Emmanue
From: Xiubo Li
Sent: Friday, May 5, 2023 2:40 AM
To: Frank Schilder; ceph-users@ceph.io
Subject: Re: [ceph-users] client isn't responding to mclientcaps(revoke),
pending pAsLsXsFsc issued pAsLsXsFsc
On 5/1/23 17:35, Frank Schilder wrote:
Hi all,
I think we might be hitting a
Hey Frank,
On 5/10/23 21:44, Frank Schilder wrote:
The kernel message that shows up on boot on the file server in text format:
May 10 13:56:59 rit-pfile01 kernel: WARNING: CPU: 3 PID: 34 at
fs/ceph/caps.c:689 ceph_add_cap+0x53e/0x550 [ceph]
May 10 13:56:59 rit-pfile01 kernel: Modules linked in
this.
Thanks
It would be great to have light-weight tools available to rectify such simple
conditions in an as non-disruptive as possible way.
Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
From: Xiubo Li
by reading the ceph
and kceph code. In theory you can reproduce this by making the directory
to do the migrating during stress IOs.
Thanks
Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________
From: Xiubo Li
Sent: Thursda
the metadata in cephfs.
Thanks
Thanks a lot and best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
____
From: Frank Schilder
Sent: Thursday, May 11, 2023 12:26 PM
To: Xiubo Li;ceph-users@ceph.io
Subject: [ceph-users] Re: mds d
how to fix it.
Firstly we need to know where the corrupted metadata is.
I think the mds debug logs and the above corrupted snaptrace could help.
Need to parse that corrupted binary data.
Thanks
Thanks and best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
___
Thanks
Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________
From: Xiubo Li
Sent: Friday, May 12, 2023 3:44 PM
To: Frank Schilder; ceph-users@ceph.io
Subject: Re: [ceph-users] Re: mds dump inode crashes file system
On 5/12
__
From: Frank Schilder
Sent: Monday, May 15, 2023 6:33 PM
To: Xiubo Li; ceph-users@ceph.io
Subject: [ceph-users] Re: mds dump inode crashes file system
Dear Xiubo,
I uploaded the cache dump, the MDS log and the dmesg log containing the
snaptrace dump to
ceph-post-file: 763955a3-7d37-408a-
the mds dump
inode command.
Thanks a lot and best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
From: Frank Schilder
Sent: Thursday, May 11, 2023 12:26 PM
To: Xiubo Li; ceph-users@ceph.io
Subject: [ceph-users] Re: mds dump inode cr
On 5/16/23 21:55, Gregory Farnum wrote:
On Fri, May 12, 2023 at 5:28 AM Frank Schilder wrote:
Dear Xiubo and others.
I have never heard about that option until now. How do I check that and how to
I disable it if necessary?
I'm in meetings pretty much all day and will try to send some more i
On 5/24/23 12:23, Mark Kirkwood wrote:
I am looking at using an iscsi gateway in front of a ceph setup.
However the warning in the docs is concerning:
The iSCSI gateway is in maintenance as of November 2022. This means
that it is no longer in active development and will not be updated to
ad
Yeah, it's a bug.
I have raised on ceph tracker to follow this:
https://tracker.ceph.com/issues/61584
And I have found the root cause, more detail please see my comments on
the above tracker.
I am still going through the code to find one way to fix it.
Thanks
- Xiubo
On 6/5/23 13:42, San
Raised one PR to fix this, please see
https://github.com/ceph/ceph/pull/51931.
Thanks
- Xiubo
On 5/24/23 23:52, Sandip Divekar wrote:
Hi Team,
I'm writing to bring to your attention an issue we have encountered with the
"mtime" (modification time) behavior for directories in the Ceph filesy
On 6/10/23 05:35, Denis Polom wrote:
Hi
I'm running latest Ceph Pacific 16.2.13 with Cephfs. I need to collect
performance stats per client, but getting empty list without any numbers
I even run dd on client against mounted ceph fs, but output is only
like this:
#> ceph fs perf stats 0 4
On 7/20/23 22:09, Frank Schilder wrote:
Hi all,
we had a client with the warning "[WRN] MDS_CLIENT_OLDEST_TID: 1 clients failing to
advance oldest client/flush tid". I looked at the client and there was nothing going
on, so I rebooted it. After the client was back, the message was still there
On 7/20/23 22:09, Frank Schilder wrote:
Hi all,
we had a client with the warning "[WRN] MDS_CLIENT_OLDEST_TID: 1 clients failing to
advance oldest client/flush tid". I looked at the client and there was nothing going
on, so I rebooted it. After the client was back, the message was still there
On 7/20/23 11:36, dxo...@naver.com wrote:
This issue has been closed.
If any rook-ceph users see this, when mds replay takes a long time, look at the
logs in mds pod.
If it's going well and then abruptly terminates, try describing the mds pod,
and if liveness probe terminated, try increasing
ecoverable state. I did not observe unusual RAM consumption and there were
no MDS large cache messages either. Seems like our situation was of a more
harmless nature. Still, the fail did not go entirely smooth.
Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
__
On 7/26/23 22:13, Frank Schilder wrote:
Hi Xiubo.
... I am more interested in the kclient side logs. Just want to
know why that oldest request got stuck so long.
I'm afraid I'm a bad admin in this case. I don't have logs from the host any
more, I would have needed the output of dmesg and thi
#x27;s okay till now.
Thanks
- Xiubo
Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________
From: Xiubo Li
Sent: Friday, July 28, 2023 11:37 AM
To: Frank Schilder; ceph-users@ceph.io
Subject: Re: [ceph-users] Re: MDS stuck
for the dmesg logs from the client node.
Yeah, after the client's sessions are closed the corresponding warning
should be cleared.
Thanks
Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
____
From: Xiubo Li
Sent: Mo
______
From: Xiubo Li
Sent: Monday, July 31, 2023 12:14 PM
To: Frank Schilder; ceph-users@ceph.io
Subject: Re: [ceph-users] Re: MDS stuck in rejoin
On 7/31/23 16:50, Frank Schilder wrote:
Hi Xiubo,
its a kernel client. I actually made a mistake when trying to evict the client
and my co
On 9/13/23 20:58, Ilya Dryomov wrote:
On Wed, Sep 13, 2023 at 9:20 AM Stefan Kooman wrote:
Hi,
Since the 6.5 kernel addressed the issue with regards to regression in
the readahead handling code... we went ahead and installed this kernel
for a couple of mail / web clusters (Ubuntu 6.5.1-060501
Have you tried to backup and then remove the 'mds%d_openfiles.%x' object
to see could you start the MDS ?
Thanks.
On 2/23/22 7:07 PM, Wolfgang Mair wrote:
Update: I managed to clear the inode errors by deleting the parent
directory entry from the metadata pool. However the MDS still refuses
Hi xuchenhuig
> "client_metadata": {
> "client_features": {
> "feature_bits": "0x00ff"
> },
From the above feature_bits, it says you are still using the old
kernel, which hasn't support the io read/write metrics yet.
Please make sure the ker
On 4/19/22 3:43 AM, Vladimir Brik wrote:
Does anybody know why cephfs-top may only display header lines (date,
client types, metric names) but no actual data?
When I run it, cephfs-top consumes quite a bit of the CPU and
generates quite a bit of network traffic, but it doesn't actually
disp
On 4/19/22 10:56 PM, Vladimir Brik wrote:
Yeah, this must be the bug. I have about 180 clients
Yeah, a new bug. Maybe too many clients and couldn't show them correctly
in the terminal.
-- Xiubo
Vlad
On 4/18/22 23:52, Jos Collin wrote:
Do you have mounted clients? How many clients do you
Hi Vladimir,
This issue looks like the one I am working on now in [1], which is also
a infinitely stuck bug when creating a new file and then writes
something to it.
The issue [1] was caused by setting the max_mds > 1 and enabling the
inline_data and then create a file and then write to it.
On 4/25/22 5:25 AM, Vladimir Brik wrote:
Hello
We are experiencing an issue where, sometimes, when users write to
cephfs an empty file is created and then the application hangs,
seemingly indefinitely. I am sometimes able to reproduce with dd.
Does anybody know what might be going on?
Som
stuff until we fix that issue.
I'll let you know when that happens
Vlad
On 4/24/22 23:40, Xiubo Li wrote:
Hi Vladimir,
This issue looks like the one I am working on now in [1], which is
also a infinitely stuck bug when creating a new file and then writes
something to it.
The issue
kernel version I can test on Centos7. It
seems Alma 8.5 is not affected by this issue (it has other issues
though), but we need cephfs to work on Centos7.
Seems only existing in old kernel.
Thanks for the logs, will check it later.
-- Xiubo
This is Ceph 16.2.7.
Vlad
On 4/25/22 23:56,
4
kernel. This is the highest kernel version I can test on Centos7. It
seems Alma 8.5 is not affected by this issue (it has other issues
though), but we need cephfs to work on Centos7.
This is Ceph 16.2.7.
Vlad
On 4/25/22 23:56, Xiubo Li wrote:
On 4/26/22 2:06 AM, Vladimir Brik wrote:
&g
On 4/29/22 5:17 AM, Vladimir Brik wrote:
Some new information:
It looks like the hang happens at the 4194304 byte boundary. For example:
`dd if=/dev/zero of=zero bs=4194304 count=1` succeeds, but
`dd if=/dev/zero of=zero bs=4194305 count=1` hangs with an empty file.
If bs is set lower, e.g. 8
On 5/12/22 12:06 AM, Stefan Kooman wrote:
Hi List,
We have quite a few linux kernel clients for CephFS. One of our
customers has been running mainline kernels (CentOS 7 elrepo) for the
past two years. They started out with 3.x kernels (default CentOS 7),
but upgraded to mainline when those k
On 6/20/22 3:45 PM, Arnaud M wrote:
Hello to everyone
I have looked on the internet but couldn't find an answer.
Do you know the maximum size of a ceph filesystem ? Not the max size of a
single file but the limit of the whole filesystem ?
For example a quick search on zfs on google output :
A Z
101 - 140 of 140 matches
Mail list logo
