So here is the answer from the docs
https://docs.ceph.com/docs/master/cephfs/app-best-practices/
There isnt a physical limit but you may well hit operational issues with some
tools.
Best practice would be to create some sort of directory tree and not just dump
all the files you have into a sin
Hi Francois,
I have seen reports of poor performance from Nautilus onwards and you might be
hit by this. This might require a ticket. There is a hypothesis that a
regression occurred that affects the cluster's ability to run background
operations properly.
What you observe should not happen an
HI Ben,
yes we have the same issues and switched to seagate for those reasons.
you can fix at least a big part of it by disabling the write cache of
those drives - generally speaking it seems the toshiba firmware is broken.
I was not able to find a newer one.
Greets,
Stefan
Am 24.06.20 um 09:4
Hi,
We have a Nautilus (14.2.9) Ceph cluster with two types of HDDs:
- TOSHIBA MG07ACA14TE [1]
- HGST HUH721212ALE604 [2]
They're all bluestore OSDs with no separate DB+WAL and part of the same pool.
We noticed that while the HGST OSDs have a commit latency of about 15ms, the
Toshiba OSDs h
Hello,
I am running ceph nautilus 14.2.8
I had to remove 2 pools (old cephfs data and metadata pool with 1024 pgs).
The removal of the pools seems to take a incredible time to free the
space (the data pool I deleted was more than 100 TB and in 36h I got
back only 10TB). In the meantime, the clus
Am 24.06.20 um 05:15 schrieb steven prothero:
> Hello,
>
> I am new to CEPH and on a few test servers attempting to setup and
> learn a test ceph system.
>
> I started off the install with the "Cephadm" option and it uses podman
> containers.
> Followed steps here:
> https://docs.ceph.com/docs/
This isn't the first time I've seen drive cache cause problematic
latency issues, and not always from the same manufacturer. Unfortunately
it seems like you really have to test the drives you want to use before
deploying them them to make sure you don't run into issues.
Mark
On 6/24/20 6:36
Benoit, wondering what are the write cache settings in your case?
And do you see any difference after disabling it if any?
Thanks,
Igor
On 6/24/2020 3:16 PM, Mark Nelson wrote:
This isn't the first time I've seen drive cache cause problematic
latency issues, and not always from the same man
Here is a thread that seems most relevant:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/W4M5XQRDBLXFGJGDYZALG6TQ4QBVGGAJ/#W4M5XQRDBLXFGJGDYZALG6TQ4QBVGGAJ
I do not see this issue on mimic, but it seems to be a problem from nautilus
onwards.
Best regards,
=
Fra
Benoit, thanks for the update.
for the sake of completeness one more experiment please if possible:
turn off write cache for HGST drives and measure commit latency once again.
Kind regards,
Igor
On 6/24/2020 3:53 PM, Benoît Knecht wrote:
Thank you all for your answers, this was really helpf
Hello,
If you do it, like Sebastian told you, you would automatically deploy osd's.
For a beginner I would recommend to do it semi-automated, so you know a bit
more what's going on.
So first do the "ceph orch device ls" which should print every disk, in all
nodes.
Then I recommend to zap device
fyi, there is an interesting note on disabling the write cache here:
https://yourcmc.ru/wiki/index.php?title=Ceph_performance&mobileaction=toggle_view_desktop#Drive_cache_is_slowing_you_down
On Wed, Jun 24, 2020 at 9:45 AM Benoît Knecht wrote:
>
> Hi Igor,
>
> Igor Fedotov wrote:
> > for the sak
Hello,
After two months of the "ceph try and error game", I finally managed to get an
Octopuss cluster up and running.
The unconventional thing about it is, it's just for hot backups, no virtual
machines on there.
All the nodes are without any caching ssd's, just plain hdd's.
At the moment ther
Has anyone ever encountered a drive with a write cache that actually
*helped*?
I haven't.
As in: would it be a good idea for the OSD to just disable the write cache
on startup? Worst case it doesn't do anything, best case it improves
latency.
Paul
--
Paul Emmerich
Looking for help with your Ce
Have a look at cephfs subvolumes:
https://docs.ceph.com/docs/master/cephfs/fs-volumes/#fs-subvolumes
They are internally just directories with quota/pool placement
layout/namespace with some mgr magic to make it easier than doing that all
by hand
Paul
--
Paul Emmerich
Looking for help with you
Well, what I was saying was "does it hurt to unconditionally run hdparm -W
0 on all disks?"
Which disk would suffer from this? I haven't seen any disk where this would
be a bad idea
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Frese
> I run the corresponding smartctl command on every drive just before
OSD daemon start.
How/where did you do this?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
Just throwing my hat in here with a small bit of anecdotal experience.
In the early days of experimenting with ceph, I had 24x 8T disk, all behind
RAID controllers as R0 vd's with no BBU (so controller cache is WT, default
value), and pdcache (disk write cache) enabled (default value).
We had a
> Sorry for the spam, but I need to add this disclaimer:
> Although it is documented as safe to disable volatile write cache on a
disk in use, I would
> probably not do it. The required cache flush might be erroneous in the
firmware.
I can remember reading this before. I was hoping you maybe
Thank you all for your answers, this was really helpful!
Stefan Priebe wrote:
> yes we have the same issues and switched to seagate for those reasons.
> you can fix at least a big part of it by disabling the write cache of
> those drives - generally speaking it seems the toshiba firmware is
> brok
All;
This conversation has been fascinating.
I'm throwing my hat in the ring, though I know almost nothing about systemd...
Completely non-portable, but...
Couldn't you write a script to issue the necessary commands to the desired
drives, then create a system unit that calls it before OSD initi
a udev rule may be easier
On Wed, Jun 24, 2020 at 1:17 PM wrote:
>
> All;
>
> This conversation has been fascinating.
>
> I'm throwing my hat in the ring, though I know almost nothing about systemd...
>
> Completely non-portable, but...
> Couldn't you write a script to issue the necessary command
>> I can remember reading this before. I was hoping you maybe had some
>> setup with systemd scripts or maybe udev.
>
> Yeah, doing this on boot up would be ideal. I was looking really hard into
> tuned and other services that claimed can do it, but required plugins or
> other stuff did/does n
Hi Igor,
Igor Fedotov wrote:
> for the sake of completeness one more experiment please if possible:
>
> turn off write cache for HGST drives and measure commit latency once again.
I just did the same experiment with HGST drives, and disabling the write cache
on those drives brought the latency do
Thanks Ramana and David.
So we are using the Shaman search API to get the latest build for
ceph_nautilus flavor of NFS Ganesha, and that's how we get to the mentioned
build. We are doing this since it's part of our CI and it's better for
automation.
Should we use different repos?
Thanks,
V
On
Just an anecdotal answer from me...
You want as few files as possible. I wouldn't go beyond a few hundred files in
a dir.
Seeing ~1s for each 1,000 files when I "ls".
But this is in a pretty idle directory. When there were files actively being
written to those dirs and being read, just doing "l
Yes, non-volatile write cache helps as described in the wiki. When you disable
write cache with hdparm, it actually only disables volatile write cache. That's
why SSDs with power loss protection are recommended for ceph.
A SAS/SATA SSD without any write cache will perform poorly no matter what.
Ah, OK, misunderstood the question.
In my experience, no. I run the corresponding smartctl command on every drive
just before OSD daemon start. I use smartctl because it applies to SAS and SATA
drives with the same command (otherwise, you need to select between hdparm and
sdparm). All SAS drive
I use the stable ceph/daemon containers and introduced my own startup script
for the container entrypoint. On the action "disk activate", it does a smartctl
on the device argument before executing entrypoint.sh.
Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
Sorry for the spam, but I need to add this disclaimer:
Although it is documented as safe to disable volatile write cache on a disk in
use, I would probably not do it. The required cache flush might be erroneous in
the firmware.
Therefore, the method I use will not necessarily apply to OSD set-u
> I can remember reading this before. I was hoping you maybe had some
> setup with systemd scripts or maybe udev.
Yeah, doing this on boot up would be ideal. I was looking really hard into
tuned and other services that claimed can do it, but required plugins or other
stuff did/does not exist and
We have a cluster, running Octopus 15.2.2, with the same exact issue described
originally.
Confirmed, setting debug_rgw logs to "20/1" fixed the issue for us as well.
What information would be needed to begin a preliminary bug report? As with
others, I haven't found a way to easily replicate t
Hi all.
Is there anyway to completely health check one OSD host or instance?
For example rados bech just on that OSD or do some checks for disk and
front and back netowrk?
Thanks.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an
The benefit of disabling on-drive cache may be at least partly dependent on the
HBA; I’ve done testing of one specific drive model and found no difference,
where someone else reported a measurable difference for the same model.
> Good to know that we're not alone :) I also looked for a newer fir
On 25/06/2020 3:17 am, dhils...@performair.com wrote:
Completely non-portable, but...
Couldn't you write a script to issue the necessary commands to the desired
drives, then create a system unit that calls it before OSD initialization?
Couldn't we just set (uncomment)
write_cache = off
in /e
Hi, https://yourcmc.ru/wiki/Ceph_performance author here %)
Disabling write cache is REALLY bad for SSDs without capacitors
[consumer SSDs], also it's bad for HDDs with firmwares that don't have
this bug-o-feature. The bug is really common though. I have no idea
where it comes from, but it's r
I did a quick test with wcache off[1]. And have the impression the
simple rados bench of 2 minutes performed a bit worse on my slow hdd's.
[1]
IFS=$'\n' && for line in `mount | grep 'osd/ceph'| awk '{print $1"
"$3}'| sed -e 's/1 / /' -e 's#/var/lib/ceph/osd/ceph-##'`;do IFS=' '
arr=($line); s
37 matches
Mail list logo
