Re: [ceph-users] ceph 12.2.5 - atop DB/WAL SSD usage 0%

Shouldn't Steven see some data being written to the block/wal for object metadata? Though that might be negligible with 4MB objects On 27-04-18 16:04, Serkan Çoban wrote: rados bench is using 4MB block size for io. Try with with io size 4KB, you will see ssd will be used for write operations.

Re: [ceph-users] ceph luminous 12.2.4 - 2 servers better than 3 ?

Write Cache : Disk's Default > Adapter 0-VD 2(target id: 2): Disk Write Cache : Disk's Default > Adapter 0-VD 3(target id: 3): Disk Write Cache : Disk's Default > > > On Thu, 19 Apr 2018 at 14:22, Hans van den Bogert > wrote: > >> I see, the second one i

Re: [ceph-users] ceph luminous 12.2.4 - 2 servers better than 3 ?

DB ( on separate SSD or same HDD) Thanks Steven On Thu, 19 Apr 2018 at 12:06, Hans van den Bogert wrote: > I take it that the first bench is with replication size 2, the second > bench is with replication size 3? Same for the 4 node OSD scenario? > > Also please let us know how you

Re: [ceph-users] ceph luminous 12.2.4 - 2 servers better than 3 ?

4194304 > Bandwidth (MB/sec): 44.0793 > Stddev Bandwidth: 55.3843 > Max bandwidth (MB/sec): 232 > Min bandwidth (MB/sec): 0 > Average IOPS: 11 > Stddev IOPS:13 > Max IOPS: 58 > Min IOPS: 0 > Average Latency(s

Re: [ceph-users] ceph luminous 12.2.4 - 2 servers better than 3 ?

Hi Steven, There is only one bench. Could you show multiple benches of the different scenarios you discussed? Also provide hardware details. Hans On Apr 19, 2018 13:11, "Steven Vacaroaia" wrote: Hi, Any idea why 2 servers with one OSD each will provide better performance than 3 ? Servers are

Re: [ceph-users] scalability new node to the existing cluster

I keep seeing these threads where adding nodes has such an impact on the cluster as a whole, that I wonder what the rest of the cluster looks like. Normally I’d just advise someone to put a limit on the concurrent backfills that can be done, and `osd max backfills` by default already is 1. Could

Re: [ceph-users] Luminous 12.2.3 release date?

Hi Wido, Did you ever get an answer? I'm eager to know as well. Hans On Tue, Jan 30, 2018 at 10:35 AM, Wido den Hollander wrote: > Hi, > > Is there a ETA yet for 12.2.3? Looking at the tracker there aren't that many > outstanding issues: http://tracker.ceph.com/projects/ceph/roadmap > > On Git

[ceph-users] Retrieving ceph health from restful manager plugin

Hi All, I might really be bad at searching, but I can't seem to find the ceph health status through the new(ish) restful api. Is that right? I know how I could retrieve it through a Python script, however I'm trying to keep our monitoring application as layer cake free as possible -- as such a res

[ceph-users] Redirect for restful API in manager

Hi all, In the release notes of 12.2.2 the following is stated: > Standby ceph-mgr daemons now redirect requests to the active messenger, easing configuration for tools & users accessing the web dashboard, restful API, or other ceph-mgr module services. However, it doesn't seem to be the cas

Re: [ceph-users] Fwd: Ceph team involvement in Rook (Deploying Ceph in Kubernetes)

Should I summarize this is ceph-helm being being EOL? If I'm spinning up a toy cluster for a homelab, should I invest time in Rook, or stay with ceph-helm for now? On Fri, Jan 19, 2018 at 11:55 AM, Kai Wagner wrote: > Just for those of you who are not subscribed to ceph-users. > > > For

Re: [ceph-users] Increasing PG number

Please refer to standard documentation as much as possible, http://docs.ceph.com/docs/jewel/rados/operations/placement-groups/#set-the-number-of-placement-groups Han’s is also incomplet

Re: [ceph-users] The way to minimize osd memory usage?

There’s probably multiple reasons. However I just wanted to chime in that I set my cache size to 1G and I constantly see OSD memory converge to ~2.5GB. In [1] you can see the difference between a node with 4 OSDs, v12.2.2, on the left; and a node with 4 OSDs v12.2.1 on the right. I really hoped

[ceph-users] osd/bluestore: Get block.db usage

Hi all, Is there a way to get the current usage of the bluestore's block.db? I'd really like to monitor this as we have a relatively high number of objects per OSD. A second question related to the above, are there mechanisms to influence which objects' metadata gets spilled once the block.db is

Re: [ceph-users] ceps-deploy won't install luminous

verify that you did that part? > On Nov 15, 2017, at 10:41 AM, Hans van den Bogert > wrote: > > Hi, > > Can you show the contents of the file, /etc/yum.repos.d/ceph.repo ? > > Regards, > > Hans >> On Nov 15, 2017, at 10:27 AM, Ragan, Tj (Dr.) >> wr

Re: [ceph-users] ceps-deploy won't install luminous

Hi, Can you show the contents of the file, /etc/yum.repos.d/ceph.repo ? Regards, Hans > On Nov 15, 2017, at 10:27 AM, Ragan, Tj (Dr.) > wrote: > > Hi All, > > I feel like I’m doing something silly. I’m spinning up a new cluster, and > followed the instructions on the pre-flight and quick s

Re: [ceph-users] Fwd: Luminous RadosGW issue

config show | grep log_file > "log_file": "/var/log/ceph/ceph-client.rgw.radosgw.log", > > > [root@radosgw system]# cat /etc/ceph/ceph.client.radosgw.keyring > [client.radosgw.gateway] > key = AQCsywNaqQdDHxAAC24O8CJ0A9Gn6qeiPalEYg== > caps mon = "all

Re: [ceph-users] Fwd: Luminous RadosGW issue

Are you sure you deployed it with the client.radosgw.gateway name as well? Try to redeploy the RGW and make sure the name you give it corresponds to the name you give in the ceph.conf. Also, do not forget to push the ceph.conf to the RGW machine. On Wed, Nov 8, 2017 at 11:44 PM, Sam Huracan wrote

Re: [ceph-users] Ceph versions not showing RGW

Just to get this really straight, Jewel OSDs do send this metadata? Otherwise I'm probably mistaken that I ever saw 10.2.x versions in the output. Thanks, Hans On 2 Nov 2017 12:31 PM, "John Spray" wrote: > On Thu, Nov 2, 2017 at 11:16 AM, Hans van den Bogert &g

[ceph-users] Ceph versions not showing RGW

Hi all, During our upgrade from Jewel to Luminous I saw the following behaviour, if my memory serves me right: When upgrading for example monitors and OSDs, we saw that the `ceph versions` command correctly showed at one that some OSDs were still on Jewel (10.2.x) and some were already upgraded a

Re: [ceph-users] PGs inconsistent, do I fear data loss?

Never mind, I should’ve read the whole thread first. > On Nov 2, 2017, at 10:50 AM, Hans van den Bogert wrote: > > >> On Nov 1, 2017, at 4:45 PM, David Turner > <mailto:drakonst...@gmail.com>> wrote: >> >> All it takes for data loss is that an osd on

Re: [ceph-users] PGs inconsistent, do I fear data loss?

> On Nov 1, 2017, at 4:45 PM, David Turner wrote: > > All it takes for data loss is that an osd on server 1 is marked down and a > write happens to an osd on server 2. Now the osd on server 2 goes down > before the osd on server 1 has finished backfilling and the first osd > receives a reque

Re: [ceph-users] announcing ceph-helm (ceph on kubernetes orchestration)

Very interesting. I've been toying around with Rook.io [1]. Did you know of this project, and if so can you tell if ceph-helm and Rook.io have similar goals? Regards, Hans [1] https://rook.io/ On 25 Oct 2017 21:09, "Sage Weil" wrote: > There is a new repo under the ceph org, ceph-helm, which

[ceph-users] Drive write cache recommendations for Luminous/Bluestore

Hi All, For Jewel there is this page about drive cache: http://docs.ceph.com/docs/jewel/rados/configuration/filesystem-recommendations/#hard-drive-prep For Bluestore I can't find any documentation or discussions about drive write cache, while I can imagine that revisiting this subject might be ne

Re: [ceph-users] Ceph delete files and status

My experience with RGW is that actual freeing up of space is asynchronous to the a S3 client’s command to delete an object. I.e., it might take a while before it’s actually freed up. Can you redo your little experiment and simply wait for an hour to let the garbage collector to do its thing, or

Re: [ceph-users] High mem with Luminous/Bluestore

> Memory usage is still quite high here even with a large onode cache! > Are you using erasure coding? I recently was able to reproduce a bug in > bluestore causing excessive memory usage during large writes with EC, > but have not tracked down exactly what's going on yet. > > Mark No, this is

Re: [ceph-users] High mem with Luminous/Bluestore

ke HDDs and monitor the memory usage. Thanks, Hans On Wed, Oct 18, 2017 at 11:56 AM, Wido den Hollander wrote: > > > Op 18 oktober 2017 om 11:41 schreef Hans van den Bogert < > hansbog...@gmail.com>: > > > > > > Hi All, > > > > I've c

[ceph-users] High mem with Luminous/Bluestore

Hi All, I've converted 2 nodes with 4 HDD/OSDs each from Filestore to Bluestore. I expected somewhat higher memory usage/RSS values, however I see, imo, a huge memory usage for all OSDs on both nodes. Small snippet from `top` PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ C

Re: [ceph-users] How to get current min-compat-client setting

t; >> Op 13 oktober 2017 om 10:22 schreef Hans van den Bogert >> : >> >> >> Hi, >> >> I’m in the middle of debugging some incompatibilities with an upgrade of >> Proxmox which uses Ceph. At this point I’d like to know what my current >>

[ceph-users] How to get current min-compat-client setting

Hi, I’m in the middle of debugging some incompatibilities with an upgrade of Proxmox which uses Ceph. At this point I’d like to know what my current value is for the min-compat-client setting, which would’ve been set by: ceph osd set-require-min-compat-client … AFAIK, there is no direct g

Re: [ceph-users] Gracefully reboot OSD node

Aug 3, 2017 at 1:55 PM, Hans van den Bogert wrote: > What are the implications of this? Because I can see a lot of blocked > requests piling up when using 'noout' and 'nodown'. That probably makes > sense though. > Another thing, no when the OSDs come back onli

Re: [ceph-users] Gracefully reboot OSD node

cted? On Thu, Aug 3, 2017 at 1:36 PM, linghucongsong wrote: > > > set the osd noout nodown > > > > > At 2017-08-03 18:29:47, "Hans van den Bogert" > wrote: > > Hi all, > > One thing which has bothered since the beginning of using ceph is that a >

[ceph-users] Gracefully reboot OSD node

Hi all, One thing which has bothered since the beginning of using ceph is that a reboot of a single OSD causes a HEALTH_ERR state for the cluster for at least a couple of seconds. In the case of planned reboot of a OSD node, should I do some extra commands in order not to go to HEALTH_ERR state?

[ceph-users] Linear space complexity or memory leak in `Radosgw-admin bucket check --fix`

Hi All, I don't seem to be able to fix a bucket, a bucket which has become inconsistent due to the use of the `inconsistent-index` flag 8). My ceph-admin VM has 4GB of RAM, but that doesn't seem to be enough to do a `radosgw-admin bucket check --fix` which holds 6M items, as the radosgw-admin pro

[ceph-users] Crash on startup

Hi All, I'm clueless as to why an OSD crashed. I have a log at [1]. If anyone can explain how this should be interpreted, then please let me know. I can only see generic errors probably started by a false assert. Restarting the OSD fails with the same errors as in [1]. It seems like, though co