On 8/7/24 09:40, Konstantin Shalygin wrote:
Hi,
On 7 Aug 2024, at 10:31, Nicola Mori wrote:
Unfortunately I'm on bare metal, with very old hardware so I cannot do much.
I'd try to build a Ceph image based on Rocky Linux 8 if I could get the
Dockerfile of the current image to start with, but
Hi Alwin,
On 7/10/24 15:55, Alwin Antreich wrote:
Hi Dietmar,
On Wed, 10 Jul 2024 at 11:22, Dietmar Rieder <mailto:dietmar.rie...@i-med.ac.at>> wrote:
Hi,
Am 9. Juli 2024 13:37:58 MESZ schrieb Dietmar Rieder
mailto:dietmar.rie...@i-med.ac.at>>:
>Hi,
Hi,
On 7/10/24 12:07, Loïc Tortay wrote:
On 10/07/2024 11:20, Dietmar Rieder wrote:
Hi,
Am 9. Juli 2024 13:37:58 MESZ schrieb Dietmar Rieder
:
Hi,
we noticed the following ceph errors in the kernel messages (dmesg -T):
[Tue Jul 9 11:59:24 2024] ceph: ceph_fill_inode
10003683698
Hi,
Am 9. Juli 2024 13:37:58 MESZ schrieb Dietmar Rieder
:
>Hi,
>
>we noticed the following ceph errors in the kernel messages (dmesg -T):
>
>[Tue Jul 9 11:59:24 2024] ceph: ceph_fill_inode 10003683698.fffe
>BAD symlink size 0
>
>Is this something that we
Hi,
we noticed the following ceph errors in the kernel messages (dmesg -T):
[Tue Jul 9 11:59:24 2024] ceph: ceph_fill_inode
10003683698.fffe BAD symlink size 0
Is this something that we should be worried about?
We are currently trying to identify the inode/file using:
# find /d
Hi Stefan,
On 7/1/24 10:34, Stefan Kooman wrote:
Hi Dietmar,
On 29-06-2024 10:50, Dietmar Rieder wrote:
Hi all,
finally we were able to repair the filesystem and it seems that we did
not lose any data. Thanks for all suggestions and comments.
Here is a short summary of our journey
nerves.
Is there an obvious place that I missed were such known issues are
prominently made public? (The bug tracker maybe, but I think it is easy
to miss the important among all others)
Thanks again for all the help
Dietmar
On 6/19/24 09:43, Dietmar Rieder wrote:
Hello cephers,
we have a
The reason why you don't see it in /proc/mounts is because it is the
default in recent kernels (see
https://github.com/gregkh/linux/commit/f7a67b463fb83a4b9b11ceaa8ec4950b8fb7f902)
If you set 'wsync' among your mount options, this will show up in
/proc/mounts
Cheers,
Enrico
Can anybody comment on my questions below? Thanks so much in advance
Am 26. Juni 2024 08:08:39 MESZ schrieb Dietmar Rieder
:
>...sending also to the list and Xiubo (were accidentally removed from
>recipients)...
>
>On 6/25/24 21:28, Dietmar Rieder wrote:
>> Hi Patric
...sending also to the list and Xiubo (were accidentally removed from
recipients)...
On 6/25/24 21:28, Dietmar Rieder wrote:
Hi Patrick, Xiubo and List,
finally we managed to get the filesystem repaired and running again!
YEAH, I'm so happy!!
Big thanks for your support Patrick and
trick Donnelly wrote:
On Mon, Jun 24, 2024 at 5:22 PM Dietmar Rieder
wrote:
(resending this, the original message seems that it didn't make it through
between all the SPAM recently sent to the list, my apologies if it doubles at
some point)
Hi List,
we are still struggeling to get our c
00 00 00 00 00 00 00 00 ||
0190 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a0 70 |...p|
01a0 72 66 75 78 90 32 01 00 00 00 00 00 00 00 ff ff |rfux.2..|
01b0 ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ||
01c0 00 00 00 0
Hi Patrick,
thanks for your message, see my comments below.
(BTW it seem that there is an issue with the ceph mailing list, my
previous message did not go through yet, so this may be redundant)
On 6/19/24 17:27, Patrick Donnelly wrote:
Hi Dietmar,
On Wed, Jun 19, 2024 at 3:44 AM Dietmar
On 6/19/24 11:15, Dietmar Rieder wrote:
On 6/19/24 10:30, Xiubo Li wrote:
On 6/19/24 16:13, Dietmar Rieder wrote:
Hi Xiubo,
[...]
0> 2024-06-19T07:12:39.236+ 7f90fa912700 -1 *** Caught
signal (Aborted) **
in thread 7f90fa912700 thread_name:md_log_replay
ceph version 18.
Hi,
On 6/19/24 12:14, Stefan Kooman wrote:
Hi,
On 19-06-2024 11:15, Dietmar Rieder wrote:
Please follow
https://docs.ceph.com/en/nautilus/cephfs/disaster-recovery-experts/#disaster-recovery-experts.
OK, when I run the cephfs-journal-tool I get an error:
# cephfs-journal-tool journal
On 6/19/24 10:30, Xiubo Li wrote:
On 6/19/24 16:13, Dietmar Rieder wrote:
Hi Xiubo,
[...]
0> 2024-06-19T07:12:39.236+ 7f90fa912700 -1 *** Caught
signal (Aborted) **
in thread 7f90fa912700 thread_name:md_log_replay
ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf
Hi Xiubo,
On 6/19/24 09:55, Xiubo Li wrote:
Hi Dietmar,
On 6/19/24 15:43, Dietmar Rieder wrote:
Hello cephers,
we have a degraded filesystem on our ceph 18.2.2 cluster and I'd need
to get it up again.
We have 6 MDS daemons and (3 active, each pinned to a subtree, 3 standby)
It st
I did that after I have seen that it seems to be more severe, see
my action "log" below.
Dietmar
On 6/19/24 10:05, Dietmar Rieder wrote:
Hi Joachim,
I suppose that setting the filesystem down will block all clients:
ceph fs set cephfs down true
right?
Dietmar
On 6/1
: joachim.kraftma...@clyso.com
<mailto:joachim.kraftma...@clyso.com>
a: Loristr. 8 | 80335 Munich | Germany | w: https://clyso.com
<https://clyso.com> |
Utting a. A. | HR: Augsburg | HRB 25866 | USt. ID: DE275430677
Am Mi., 19. Juni 2024 um 09:44 Uhr schrieb Dietmar Rieder
mailto:diet
Hello cephers,
we have a degraded filesystem on our ceph 18.2.2 cluster and I'd need to
get it up again.
We have 6 MDS daemons and (3 active, each pinned to a subtree, 3 standby)
It started this night, I got the first HEALTH_WARN emails saying:
HEALTH_WARN
--- New ---
[WARN] MDS_CLIENT_RECA
On 4/26/24 23:51, Erich Weiler wrote:
As Dietmar said, VS Code may cause this. Quite funny to read,
actually, because we've been dealing with this issue for over a year,
and yesterday was the very first time Ceph complained about a client
and we saw VS Code's remote stuff running. Coincidence.
dules/*/**": true,
"**/.cache/**": true,
"**/.conda/**": true,
"**/.local/**": true,
"**/.nextflow/**": true,
"**/work/**": true,
"**/cephfs/**": true
}
}
On 4/27/24 12:24 AM, Dietmar Rieder wrote:
Hi
#x27;ll suggest they make the mods you referenced! Thanks for the tip.
>
>cheers,
>erich
>
>On 4/24/24 12:58 PM, Dietmar Rieder wrote:
>> Hi Erich,
>>
>> in our case the "client failing to respond to cache pressure" situation
>> is/was often caused
Hi Erich,
in our case the "client failing to respond to cache pressure" situation
is/was often caused by users how have vscode connecting via ssh to our
HPC head node. vscode makes heavy use of file watchers and we have seen
users with > 400k watchers. All these watched files must be held in t
o with reef (we see massive writes as
well there)?
Unfortunately, I can't comment on Reef as we're still using Pacific.
/Z
On Tue, 16 Apr 2024 at 18:08, Dietmar Rieder <mailto:dietmar.rie...@i-med.ac.at>> wrote:
Hi Zakhar, hello List,
I just wanted to follow up on
Hi Zakhar, hello List,
I just wanted to follow up on this and ask a few quesitions:
Did you noticed any downsides with your compression settings so far?
Do you have all mons now on compression?
Did release updates go through without issues?
Do you know if this works also with reef (we see massiv
Hello,
we a run CentOS 7.9 client to access cephfs on a Ceph Reef (18.2.2)
Cluster and it works just fine using the kernel client that comes with
CentOS 7.9 + updates.
Best
Dietmar
On 4/15/24 16:17, Dario Graña wrote:
Hello everyone!
We deployed a platform with Ceph Quincy and now we nee
ar
On Tue, Jan 30, 2024 at 2:03 AM Dietmar Rieder
wrote:
Hello,
I have a question regarding the default pool of a cephfs.
According to the docs it is recommended to use a fast ssd replicated
pool as default pool for cephfs. I'm asking what are the space
requirements for storing the in
On 1/31/24 20:13, Patrick Donnelly wrote:
On Tue, Jan 30, 2024 at 5:03 AM Dietmar Rieder
wrote:
Hello,
I have a question regarding the default pool of a cephfs.
According to the docs it is recommended to use a fast ssd replicated
pool as default pool for cephfs. I'm asking what ar
Hello,
I have a question regarding the default pool of a cephfs.
According to the docs it is recommended to use a fast ssd replicated
pool as default pool for cephfs. I'm asking what are the space
requirements for storing the inode backtrace information?
Let's say I have a 85 TiB replicated
...nevermind, after restart of the managers I was getting the metrics.
sorry for the noise
Dietmar
On 1/6/24 13:45, Dietmar Rieder wrote:
Hi,
I just freshly deployed a new cluster (v18.2.1) using cephadm. Now
before creating pools, cephfs and so on I wanted to check if the
dashboard is
Hi,
I just freshly deployed a new cluster (v18.2.1) using cephadm. Now
before creating pools, cephfs and so on I wanted to check if the
dashboard is working and if I get some metrics.
If I navigate to Cluster >> Hosts and open one of the OSD hosts the
"Performance Details" tab is shown but
Hi,
this is on our nautilus cluster, not sure if it is relevant, however
here are the results:
1) iotop results:
TID PRIO USER DISK READ DISK WRITE SWAPIN IOCOMMAND
TID PRIO USER DISK READ DISK WRITE SWAPIN IOCOMMAND
1801 be/4 ceph 0.00 B
I thought so too, but now I'm a bit confused. We are planning to
setup a new ceph cluster and initially opted for a el9 system, which is
supposed to be stable, should we rather use a stream trail version?
Dietmar
On 8/4/23 09:04, Marc wrote:
But Rocky Linux 9 is the continuation of what
On 5/23/23 15:58, Gregory Farnum wrote:
On Tue, May 23, 2023 at 3:28 AM Dietmar Rieder
wrote:
Hi,
can the cephfs "max_file_size" setting be changed at any point in the
lifetime of a cephfs?
Or is it critical for existing data if it is changed after some time? Is
there anything t
On 5/23/23 15:53, Konstantin Shalygin wrote:
Hi,
On 23 May 2023, at 13:27, Dietmar Rieder
wrote:
can the cephfs "max_file_size" setting be changed at any point in the
lifetime of a cephfs?
Or is it critical for existing data if it is changed after some time?
Is there anything t
Hi,
can the cephfs "max_file_size" setting be changed at any point in the
lifetime of a cephfs?
Or is it critical for existing data if it is changed after some time? Is
there anything to consider when changing, let's say, from 1TB (default)
to 4TB ?
We are running the latest Nautilus release
On 5/27/21 2:33 PM, Adrian Sevcenco wrote:
Hi! is is (technically) possible to instruct cephfs to store files <
1Mib on a (replicate) pool
and the others files on another (ec) pool?
And even more, is it possible to take the same kind of decision on the
path of the file?
(let's say that critic
: Dietmar Rieder
Sent: 19 January 2021 13:24:15
To: Frank Schilder; ceph-users@ceph.io
Subject: Re: [ceph-users] Re: cephfs: massive drop in MDS requests per second
with increasing number of caps
Hi Frank,
you don't need to remount the fs. The kernel driver should react to the
change on th
regarding updating config settings in the manual pages.
I would be most interested in further updates in this matter and also if you
find other flags with positive performance impact.
Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________
ll keep this for now and observe if it has any impact on other
operations or situations.
Still I wonder why a higher number (i.e. >64k) of caps on the client
destroys the performance completely.
Thanks again
Dietmar
On 1/18/21 6:20 PM, Dietmar Rieder wrote:
Hi Burkhard,
thanks so much f
Hi Burkhard,
thanks so much for the quick reply and the explanation and suggestions.
I'll check these settings and eventually change them and report back.
Best
Dietmar
On 1/18/21 6:00 PM, Burkhard Linke wrote:
Hi,
On 1/18/21 5:46 PM, Dietmar Rieder wrote:
Hi all,
we noticed a ma
Hi all,
we noticed a massive drop in requests per second a cephfs client is able
to perform when we do a recursive chown over a directory with millions
of files. As soon as we see about 170k caps on the MDS, the client
performance drops from about 660 reqs/sec to 70 reqs/sec.
When we then cl
On 2020-06-12 16:35, Marc Roos wrote:
>
> Will there be a ceph release available on rhel7 until the eol of rhel7?
much needed here as well
+1
Would be really great, Thanks a lot.
Dietmar
--
_
D i e t m a r R i e d e r, Mag.Dr.
Innsbruck Medical Univers
On 2020-04-02 16:40, konstantin.ilya...@mediascope.net wrote:
> I have done it.
> I am not sure, if i didn’t miss something, but i upgraded test cluster from
> CentOs7.7.1908+Ceph14.2.8 to Debian10.3+Ceph15.2.0.
>
> Preparations:
> - 6 nodes with OS CentOs7.7.1908, Ceph14.2.8:
> - cephtest01,ce
On 2020-04-02 12:24, Paul Emmerich wrote:
> Safe to ignore/increase the warning threshold. You are seeing this
> because the warning level was reduced to 200k from 2M recently.
>
> The file will be sharded in a newer version which will clean this up
>
Thanks Paul,
would that "newer version" be
Hi,
I'm trying to understand the "LARGE_OMAP_OBJECTS 1 large omap objects"
warning for out cephfs metadata pool.
It seems that pg 5.26 has a large omap object with > 200k keys
[WRN] : Large omap object found. Object:
5:654134d2:::mds0_openfiles.0:head PG: 5.4b2c82a6 (5.26) Key count:
286083 Size
On 2020-03-24 23:37, Sage Weil wrote:
> On Tue, 24 Mar 2020, konstantin.ilya...@mediascope.net wrote:
>> Is it poosible to provide instructions about upgrading from CentOs7+
>> ceph 14.2.8 to CentOs8+ceph 15.2.0 ?
>
> You have ~2 options:
>
> - First, upgrade Ceph packages to 15.2.0. Note that
tting Octopus ready.
>
> ta ta
>
> Jake
>
>
> On 3/16/20 4:58 PM, Dietmar Rieder wrote:
>> On 2020-03-03 13:36, Abhishek Lekshmanan wrote:
>>>
>>> This is the eighth update to the Ceph Nautilus release series. This release
>>> fixes issues a
On 2020-03-03 13:36, Abhishek Lekshmanan wrote:
>
> This is the eighth update to the Ceph Nautilus release series. This release
> fixes issues across a range of subsystems. We recommend that all users upgrade
> to this release. Please note the following important changes in this
> release; as alwa
Oh, didn't realize, Thanks
Dietmar
On 2020-03-16 09:44, Ashley Merrick wrote:
> This was a bug in 14.2.7 and calculation for EC pools.
>
> It has been fixed in 14.2.8
>
>
> On Mon, 16 Mar 2020 16:21:41 +0800 *Dietmar Rieder
> * wrote
>
> Hi,
&g
Hi,
I was planing to activate the pg_autoscaler on a EC (6+3) pool which I
created two years ago.
Back then I calculated the total # of pgs for this pool with a target
per ods pg # of 150 (this was the recommended /osd pg number as far as I
recall).
I used the RedHat ceph pg per pool calculator [
keep an eye on it.
Thanks again
Dietmar
On 2020-02-13 09:37, Dietmar Rieder wrote:
> Hi,
>
> they were not down as far as I can tell form the affected osd logs at
> the time in question.
> I'll try to play with those values, thanks. Is there anything else that
> might help
Hi,
they were not down as far as I can tell form the affected osd logs at
the time in question.
I'll try to play with those values, thanks. Is there anything else that
might help?
The kernel crash is something that makes me nervous.
Dietmar
On 2020-02-13 09:16, thoralf schulze wrote:
> hi Dietma
Hi,
the attachments got removed from my previous message, here the pastebins:
client vmcore-dmesg:
https://pastebin.com/AFZgkpaK
mds.log:
https://pastebin.com/FUU6hyya
Best
Dietmar
On 2020-02-13 05:00, Dietmar Rieder wrote:
> Hi,
>
> now we got a kernel crash (Oops) probably relat
]
[281203.017510] RSP
[281203.019743] CR2: 0010
# uname -a
Linux zeus.icbi.local 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4
23:02:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
More dmesg extract attached.
Should I file a bug report?
Dietmar
On 2020-02-12 13:32, Dietmar Rieder wrote:
>
Hi,
we sometimes loose access to our cephfs mount and get permission denied
if we try to cd into it. This happens apparently only on some of our HPC
cephfs-client nodes (fs mounted via kernel client) when they are busy
with calculation and I/O.
When we then manually force unmount the fs and remou
worked fine for us as well
D.
On 2020-02-12 09:33, Massimo Sgaravatto wrote:
> We skipped from Luminous to Nautilus, skipping Mimic
> This is supported and documented
>
> On Wed, Feb 12, 2020 at 9:30 AM Eugen Block wrote:
>
>> Hi,
>>
>> we also skipped Mimic when upgrading from L --> N and it
58 matches
Mail list logo
