Re: [ceph-users] block db sizing and calculation

i'm plannung to split the block db to a seperate flash device which i also would like to use as an OSD for erasure coding metadata for rbd devices. If i want to use 14x 14TB HDDs per Node https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#sizing recommends a minimum size

Re: [ceph-users] Install specific version using ansible

Hello all! I'm trying to install a specific version of luminous (12.2.4). In the directory group_vars/all.yml I can specify the luminous version, but i didn't find a place where I can be more specific about the version. The ansible installs the latest version (12.2.12 at this time). I'm using ce

Re: [ceph-users] Use telegraf/influx to detect problems is very difficult

But it is very difficult/complicated to make simple queries because, for example I have osd up and osd total but not osd down metric. To determine how much osds down you don't need special metric, because you already have osd_up and osd_in metrics. Just use math. k ___

Re: [ceph-users] Is a scrub error (read_error) on a primary osd safe to repair?

I tried to dig in the mailinglist archives but couldn't find a clear answer to the following situation: Ceph encountered a scrub error resulting in HEALTH_ERR Two PG's are active+clean+inconsistent. When investigating the PG i see a "read_error" on the primary OSD. Both PG's are replicated PG's

Re: [ceph-users] rbd image size

Hello , I use ceph as block storage in kubernetes. I want to get the rbd usage by command "rbd diff image_id | awk '{ SUM += $2 } END { print SUM/1024/1024 " MB" }’”, but I found it is a lot bigger than the value which I got by command “df -h” in the pod. I do not know the reason and need you

Re: [ceph-users] Impact of a small DB size with Bluestore

I have an Ceph cluster which was designed for file store. Each host have 5 SSDs write intensive of 400GB and 20 HDD of 6TB. So each HDD have a WAL of 5 GB on SSD If i want to put Bluestore on this cluster, i can only allocate ~75GB of WAL and DB on SSD for each HDD which is far below the 4% limit

Re: [ceph-users] Strange CEPH_ARGS problems

I found a typo in my post: Of course I tried export CEPH_ARGS="-n client.rz --keyring=" and not export CEPH_ARGS=="-n client.rz --keyring=" try `export CEPH_ARGS="--id rz --keyring=..."` k ___ ceph-users mailing list ceph-users@lists.ce

Re: [ceph-users] changing set-require-min-compat-client will cause hiccup?

On 10/31/19 2:12 PM, Philippe D'Anjou wrote: Hi, it is NOT safe. All clients fail to mount rbds now :( Your clients is upmap compatible? k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] changing set-require-min-compat-client will cause hiccup?

Hi,I need to change set-require-min-compat-clientto use upmap mode for the PG balancer. Will this cause a disconnect of all clients? We're talking cephfs and RBD images for VMs. Or is it save to switch that live? Is safe. k ___ ceph-users mailing

Re: [ceph-users] ceph balancer do not start

connections coming from qemu vm clients. It's generally easy to upgrade. Just switch your Ceph yum repo from jewel to luminous. Then update `librbd` on your hypervisors and migrate your VM's. It's fast and without downtime of your VM's. k ___ ce

Re: [ceph-users] ceph balancer do not start

Hi, ceph features { "mon": { "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 3 } }, "osd": { "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous",

Re: [ceph-users] ceph balancer do not start

root at ceph-mgr :~# ceph balancer mode upmap root at ceph-mgr :~# ceph balancer optimize myplan root at ceph-mgr :~# ceph b

Re: [ceph-users] How to reset compat weight-set changes caused by PG balancer module?

Apparently the PG balancer crush-compat mode adds some crush bucket weights. Those cause major havoc in our cluster, our PG distribution is all over the place. Seeing things like this:... 97 hdd 9.09470 1.0 9.1 TiB 6.3 TiB 6.3 TiB 32 KiB 17 GiB 2.8 TiB 69.03 1.08 28 up 98 hd

Re: [ceph-users] Pool statistics via API

Currently I am getting the pool statistics (especially USED/MAX AVAIL) via the command line: ceph df -f json-pretty| jq '.pools[] | select(.name == "poolname") | .stats.max_avail' ceph df -f json-pretty| jq '.pools[] | select(.name == "poolname") | .stats.bytes_used' Command "ceph df" does not sh

Re: [ceph-users] hanging/stopped recovery/rebalance in Nautilus

Hi,I often observed now that the recovery/rebalance in Nautilus starts quite fast but gets extremely slow (2-3 objects/s) even if there are like 20 OSDs involved. Right now I am moving (reweighted to 0) 16x8TB disks, it's running since 4 days and since 12h it's kind of stuck now at   cluster:

Re: [ceph-users] Local Device Health PG inconsistent

I was able to get OSDs to boot by updating from 14.2.2 to 14.2.4. Unclear why this would improve things, but it at least got me running again. I guess it was covered by this PR [1]. [1] https://github.com/ceph/ceph/pull/29115 k ___ ceph-users maili

Re: [ceph-users] multiple RESETSESSION messages

We have a 5 node Luminous cluster on which we see multiple RESETSESSION messages for OSDs on the last node alone. 's=STATE_CONNECTING_WAIT_CONNECT_REPLY_AUTH pgs=2613 cs=1 l=0).handle_connect_reply connect got RESETSESSION' We found the below fix for this issue, but not able to identify the corr

Re: [ceph-users] iostat and dashboard freezing

On 9/13/19 4:51 AM, Reed Dier wrote: I would love to deprecate the multi-root, and may try to do just that in my next OSD add, just worried about data shuffling unnecessarily. Would this in theory help my distribution across disparate OSD topologies? May be. Actually I don't know where is bala

Re: [ceph-users] How to create multiple Ceph pools, based on drive type/size/model etc?

Right - but what is you have two types of NVMe drives? I thought that there's only a fixed enum of device classes - hdd, ssd, or nvme. You can't add your own ones, right? Indeed you can: `ceph osd crush set-device-class nvme2 osd.0`. k ___ cep

Re: [ceph-users] How to create multiple Ceph pools, based on drive type/size/model etc?

Right - but what is you have two types of NVMe drives? I thought that there's only a fixed enum of device classes - hdd, ssd, or nvme. You can't add your own ones, right? Indeed you can: `ceph set-device-class nvme2 osd.0`. k ___ ceph-users mailin

Re: [ceph-users] Ceph Balancer Limitations

We're using Nautilus 14.2.2 (upgrading soon to 14.2.3) on 29 CentOS osd servers. We've got a large variation of disk sizes and host densities. Such that the default crush mappings lead to an unbalanced data and pg distribution. We enabled the balancer manager module in pg upmap mode. The balance

Re: [ceph-users] How to create multiple Ceph pools, based on drive type/size/model etc?

I have a 3-node Ceph cluster, with a mixture of Intel Optane 905P PCIe disks, and normal SATA SSD drives. I want to create two Ceph pools, one with only the Optane disks, and the other with only the SATA SSDs. When I checked "ceph osd tree", all the drives had device class "ssd". As a hack - I

Re: [ceph-users] AutoScale PG Questions - EC Pool

On 9/10/19 1:17 PM, Ashley Merrick wrote: So I am correct in 2048 being a very high number and should go for either 256 or 512 like you said for a cluster of my size with the EC Pool of 8+2? Indeed. I suggest stay at 256. k ___ ceph-users mailin

Re: [ceph-users] AutoScale PG Questions - EC Pool

I have a EC Pool (8+2) which has 30 OSD (3 Nodes), grown from the orginal 10 OSD (1 Node). I originally set the pool with a PG_NUM of 300, however the AutoScale PG is showing a warn saying I should set this to 2048, I am not sure if this is a good suggestion or if the Autoscale currently is

Re: [ceph-users] iostat and dashboard freezing

On 9/2/19 5:47 PM, Jake Grimmett wrote: Hi Konstantin, To confirm, disabling the balancer allows the mgr to work properly. I tried re-enabling the balancer, it briefly worked, then locked up the mgr again. Here it's working OK... [root@ceph-s1 ~]# time ceph balancer optimize new real0m1.6

Re: [ceph-users] iostat and dashboard freezing

On 8/29/19 9:56 PM, Reed Dier wrote: "config/mgr/mgr/balancer/active", "config/mgr/mgr/balancer/max_misplaced", "config/mgr/mgr/balancer/mode", "config/mgr/mgr/balancer/pool_ids", This is useless keys, you may to remove it. https://pastebin.com/bXPs28h1 Issues that you have: 1. Multi-root.

Re: [ceph-users] iostat and dashboard freezing

Just a follow up 24h later, and the mgr's seem to be far more stable, and have had no issues or weirdness after disabling the balancer module. Which isn't great, because the balancer plays an important role, but after fighting distribution for a few weeks and getting it 'good enough' I'm taking

Re: [ceph-users] Ceph + SAMBA (vfs_ceph)

On 8/29/19 1:32 AM, Salsa wrote: This is the result: # testparm -s Load smb config files from /etc/samba/smb.conf rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384) Processing section "[homes]" Processing section "[cephfs]" Processing section "[printers]" Processing secti

Re: [ceph-users] health: HEALTH_ERR Module 'devicehealth' has failed: Failed to import _strptime because the import lockis held by another thread.

On 8/28/19 8:16 PM, Peter Eisch wrote: Thank you for your reply. The I receive an error as the module can't be disabled. I may have worked through this by restarting the nodes in a rapid succession. What exactly error? May be you catches a bug and should be create redmine ticket for thi

Re: [ceph-users] Ceph + SAMBA (vfs_ceph)

I'm running a ceph installation on a lab to evaluate for production and I have a cluster running, but I need to mount on different windows servers and desktops. I created an NFS share and was able to mount it on my Linux desktop, but not a Win 10 desktop. Since it seems that Windows server 2016

Re: [ceph-users] health: HEALTH_ERR Module 'devicehealth' has failed: Failed to import _strptime because the import lockis held by another thread.

What is the correct/best way to address a this? It seems like a python issue, maybe it's time I learn how to "restart" modules? The cluster seems to be working beyond this. Restart of single module is: `ceph mgr module disable devicehealth ; ceph mgr module enable devicehealth`. k ___

Re: [ceph-users] Ceph capacity versus pool replicated size discrepancy?

On 8/14/19 6:19 PM, Kenneth Van Alstyne wrote: Got it!  I can calculate individual clone usage using “rbd du”, but does anything exist to show total clone usage across the pool?  Otherwise it looks like phantom space is just missing. rbd du for each snapshot, I think... k ___

Re: [ceph-users] Ceph capacity versus pool replicated size discrepancy?

Hey guys, this is probably a really silly question, but I’m trying to reconcile where all of my space has gone in one cluster that I am responsible for. The cluster is made up of 36 2TB SSDs across 3 nodes (12 OSDs per node), all using FileStore on XFS. We are running Ceph Luminous 12.2.8 on t

Re: [ceph-users] Nautilus - Balancer is always on

ceph mgr module disable balancer Error EINVAL: module 'balancer' cannot be disabled (always-on) Whats the way to restart balanacer? Restart MGR service? I wanna suggest to Balancer developer to setup a ceph-balancer.log for this module get more information about whats doing. Maybe

Re: [ceph-users] New CRUSH device class questions

On 8/7/19 2:30 PM, Robert LeBlanc wrote: ... plus 11 more hosts just like this Interesting. Please paste full `ceph osd df tree`. What is actually your NVMe models? Yes, our HDD cluster is much like this, but not Luminous, so we created as separate root with SSD OSD for the metadata and set

Re: [ceph-users] New CRUSH device class questions

On 8/7/19 1:40 PM, Robert LeBlanc wrote: Maybe it's the lateness of the day, but I'm not sure how to do that. Do you have an example where all the OSDs are of class ssd? Can't parse what you mean. You always should paste your `ceph osd tree` first. Yes, we can set quotas to limit space usage

Re: [ceph-users] New CRUSH device class questions

Is it possible to add a new device class like 'metadata'? Yes, but you don't need this. Just use your existing class with another crush ruleset. If I set the device class manually, will it be overwritten when the OSD boots up? Nope. Classes assigned automatically when OSD is created, not

Re: [ceph-users] Is the admin burden avoidable? "1 pg inconsistent" every other day?

Question:  If you have enough osds it seems an almost daily thing when you get to work in the morning there' s a "ceph health error"  "1 pg inconsistent"   arising from a 'scrub error'.   Or 2, etc.   Then like most such mornings you look to see there's two or more valid instances of the pg and on

Re: [ceph-users] Balancer in HEALTH_ERR

Two weeks ago, we started a data migration from one old ceph node to a new one. For task we added a 120TB Host to the cluster and evacuated the old one with the ceph osd crush reweight osd.X 0.0 that move near 15 TB per day. After 1 week and few days we found that balancer module don't work

Re: [ceph-users] Nautilus: significant increase in cephfs metadata pool usage

we just recently upgraded our cluster from luminous 12.2.10 to nautilus 14.2.1 and I noticed a massive increase of the space used on the cephfs metadata pool although the used space in the 2 data pools basically did not change. See the attached graph (NOTE: log10 scale on y-axis) Is there any re

Re: [ceph-users] Ceph OSD daemon possibly causes network card issues

On 7/19/19 5:59 PM, Geoffrey Rhodes wrote: Holding thumbs this helps however I still don't understand why the issue only occurs on ceph-osd nodes. ceph-mon and ceph-mds nodes and even a cech client with the same adapters do not have these issues. Because osd hosts actually do data storage wo

Re: [ceph-users] Legacy BlueStore stats reporting?

Using Ceph-Ansible stable-4.0 I did a rolling update from latest Mimic to Nautilus 14.2.2 on a cluster yesterday, and the update ran to completion successfully. However, in ceph status I see a warning of the form "Legacy BlueStore stats reporting detected” for all OSDs in the cluster. Can any

Re: [ceph-users] Ceph OSD daemon causes network card issues

On 7/18/19 7:43 PM, Geoffrey Rhodes wrote: Sure, also attached. Try to disable flow control via `ethtool -K rx off tx off`. k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Changing the release cadence

Arch Linux packager for Ceph here o/ I take this opportunity to consider the possibility of the appearance not in Ceph packaging, but Archlinux+Ceph related. Currently with Archlinux packaging impossible to build "Samba CTDB Cluster with CephFS backend". This caused by lack of build options,

Re: [ceph-users] Ceph OSD daemon causes network card issues

I've been having an issue since upgrading my cluster to Mimic 6 months ago (previously installed with Luminous 12.2.1). All nodes that have the same PCIe network card seem to loose network connectivity randomly. (frequency ranges from a few days to weeks per host node) The affected nodes only have

Re: [ceph-users] BlueStore bitmap allocator under Luminous and Mimic

On 5/28/19 5:16 PM, Marc Roos wrote: I switched first of may, and did not notice to much difference in memory usage. After the restart of the osd's on the node I see the memory consumption gradually getting back to as before. Can't say anything about latency. Anybody else? Wido? I see many pa

Re: [ceph-users] 3 OSDs stopped and unable to restart

I'll give that a try. Is it something like... ceph tell 'osd.*' bluestore_allocator stupid ceph tell 'osd.*' bluefs_allocator stupid And should I expect any issues doing this? You should place this to ceph.conf and restart your osds. Otherwise, this should fix new bitmap allocator issue via s

Re: [ceph-users] Faux-Jewel Client Features

Hi all, Starting to make preparations for Nautilus upgrades from Mimic, and I'm looking over my client/session features and trying to fully grasp the situation. >/$ ceph versions />/{ />/"mon": { />/"ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)": 3 }, />/"mgr":

Re: [ceph-users] RGW: Is 'radosgw-admin reshard stale-instances rm' safe?

On 6/25/19 12:46 AM, Rudenko Aleksandr wrote: Hi, Konstantin. Thanks for the reply. I know about stale instances and that they remained from prior version. I ask about “marker” of bucket. I have bucket “clx” and I can see his current marker in stale-instances list. As I know, stale-instan

Re: [ceph-users] Thoughts on rocksdb and erasurecode

Hi Have been thinking a bit about rocksdb and EC pools: Since a RADOS object written to a EC(k+m) pool is split into several minor pieces, then the OSD will receive many more smaller objects, compared to the amount it would receive in a replicated setup. This must mean that the rocksdb will als

Re: [ceph-users] rebalancing ceph cluster

Hello everyone, We have some osd on the ceph. Some osd's usage is more than 77% and another osd's usage is 39% in the same host. I wonder why osd’s usage is different.(Difference is large) and how can i fix it? ID CLASS WEIGHTREWEIGHT SIZEUSE AVAIL %USE VAR PGS TYPE NAME

Re: [ceph-users] RGW: Is 'radosgw-admin reshard stale-instances rm' safe?

Hi, folks. I have Luminous 12.2.12. Auto-resharding is enabled. In stale instances list I have: # radosgw-admin reshard stale-instances list | grep clx "clx:default.422998.196", I have the same marker-id in bucket stats of this bucket: # radosgw-admin bucket stats --bucket clx | grep mar

Re: [ceph-users] Possible to move RBD volumes between pools?

Both pools are in the same Ceph cluster. Do you have any documentation on the live migration process? I'm running 14.2.1 Something like: ``` rbd migration prepare test1 rbd2/test2 rbd migration execute test1 rbd migration commit test1 --force ``` k __

Re: [ceph-users] Any way to modify Bluestore label ?

Hello, I would like to modify Bluestore label of an OSD, is there a way to do this ? I so that we could diplay them with "ceph-bluestore-tool show-label" but i did not find anyway to modify them... Is it possible ? I changed LVM tags but that don't help with bluestore labels.. # ceph-bluestor

Re: [ceph-users] SSD Sizing for DB/WAL: 4% for large drives?

We have a similar setup, but 24 disks and 2x P4800X. And the 375GB NVME drives are _not_ large enough: 2019-05-29 07:00:00.000108 mon.bcf-03 [WRN] overall HEALTH_WARN BlueFS spillover detected on 22 OSD(s) root at bcf-10 :~# parted /dev/

Re: [ceph-users] SSD Sizing for DB/WAL: 4% for large drives?

Dear All, Quick question regarding SSD sizing for a DB/WAL... I understand 4% is generally recommended for a DB/WAL. Does this 4% continue for "large" 12TB drives, or can we economise and use a smaller DB/WAL? Ideally I'd fit a smaller drive providing a 266GB DB/WAL per 12TB OSD, rather than

Re: [ceph-users] Luminous OSD: replace block.db partition

On 5/28/19 5:16 PM, Igor Fedotov wrote: LVM volume and raw file resizing is quite simple, while partition one might need manual data movement to another target via dd or something. This also possible and tested, how-to is here https://bit.ly/2UFVO9Z k

Re: [ceph-users] BlueStore bitmap allocator under Luminous and Mimic

Hi, With the release of 12.2.12 the bitmap allocator for BlueStore is now available under Mimic and Luminous. [osd] bluestore_allocator = bitmap bluefs_allocator = bitmap Before setting this in production: What might the i

Re: [ceph-users] Luminous OSD: replace block.db partition

Hello - I have created an OSD with 20G block.db, now I wanted to change the block.db to 100G size. Please let us know if there is a process for the same. PS: Ceph version 12.2.4 with bluestore backend. You should upgrade to 12.2.11+ first! Expand your block.db via `ceph-bluestore-tool bluefs-

Re: [ceph-users] large omap object in usage_log_pool

in the config. ```"rgw_override_bucket_index_max_shards": "8",```. Should this be increased? Should be decreased to default `0`, I think. Modern Ceph releases resolve large omaps automatically via bucket dynamic resharding: ``` {     "option": {     "name": "rgw_dynamic_resharding",

Re: [ceph-users] RGW metadata pool migration

What are the metadata pools in an RGW deployment that need to sit on the fastest medium to better the client experience from an access standpoint ? Also is there an easy way to migrate these pools in a PROD scenario with minimal to no-outage if possible ? Just change crush rule to place defaul

Re: [ceph-users] How do you deal with "clock skew detected"?

how do you deal with the "clock skew detected" HEALTH_WARN message? I think the internal RTC in most x86 servers does have 1 second resolution only, but Ceph skew limit is much smaller than that. So every time I reboot one of my mons (for kernel upgrade or something), I have to wait for several m

Re: [ceph-users] Major ceph disaster

On 5/15/19 1:49 PM, Kevin Flöh wrote: since we have 3+1 ec I didn't try before. But when I run the command you suggested I get the following error: ceph osd pool set ec31 min_size 2 Error EINVAL: pool min_size must be between 3 and 4 What is your current min size? `ceph osd pool get ec31

Re: [ceph-users] Major ceph disaster

peering does not seem to be blocked anymore. But still there is no recovery going on. Is there anything else we can try? Try to reduce min_size for problem pool as 'health detail' suggested: `ceph osd pool set ec31 min_size 2`. k ___ ceph-user

Re: [ceph-users] CRUSH rule device classes mystery

Hi List, I'm playing around with CRUSH rules and device classes and I'm puzzled if it's working correctly. Platform specifics: Ubuntu Bionic with Ceph 14.2.1 I created two new device classes "cheaphdd" and "fasthdd". I made sure these device classes are applied to the right OSDs and that the (sh

Re: [ceph-users] reshard list

Hello, I am have been managing a ceph cluster running 12.2.11. This was running 12.2.5 until the recent upgrade three months ago. We build another cluster running 13.2.5 and synced the data between clusters and now would like to run primarily off the 13.2.5 cluster. The data is all S3 buckets.

Re: [ceph-users] op_w_latency

Hello Ceph Users, I am finding that the write latency across my ceph clusters isn't great and I wanted to see what other people are getting for op_w_latency. Generally I am getting 70-110ms latency. I am using: ceph --admin-daemon /var/run/ceph/ceph-osd.102.asok perf dump | grep -A3 '\"op_w_l

Re: [ceph-users] Nautilus upgrade but older releases reported by features

We recently updated a cluster to the Nautlius release by updating Debian packages from the Ceph site. Then rebooted all servers. ceph features still reports older releases, for example the osd "osd": [ { "features": "0x3ffddff8ffac", "release": "lumino

Re: [ceph-users] [Bluestore] Some of my osd's uses BlueFS slow storage for db - why?

On 3/23/19 12:20 AM, Mazzystr wrote: inline... On Fri, Mar 22, 2019 at 1:08 PM Konstantin Shalygin <mailto:k0...@k0ste.ru>> wrote: On 3/22/19 11:57 PM, Mazzystr wrote: > I am also seeing BlueFS spill since updating to Nautilus.  I also see > high slow

Re: [ceph-users] [Bluestore] Some of my osd's uses BlueFS slow storage for db - why?

On 3/22/19 11:57 PM, Mazzystr wrote: I am also seeing BlueFS spill since updating to Nautilus.  I also see high slow_used_bytes and slow_total_bytes metrics.  It sure looks to me that the only solution is to zap and rebuilt the osd.  I had to manually check 36 osds some of them traditional proc

Re: [ceph-users] SSD Recovery Settings

I setup an SSD Luminous 12.2.11 cluster and realized after data had been added that pg_num was not set properly on the default.rgw.buckets.data pool ( where all the data goes ). I adjusted the settings up, but recovery is going really slow ( like 56-110MiB/s ) ticking down at .002 per log entry(c

Re: [ceph-users] v14.2.0 Nautilus released

On 3/19/19 2:52 PM, Benjamin Cherian wrote: >/Hi, />//>/I'm getting an error when trying to use the APT repo for Ubuntu bionic. />/Does anyone else have this issue? Is the mirror sync actually still in />/progress? Or was something setup incorrectly? />//>/E: Failed to fetch />/https://download.ce

Re: [ceph-users] Constant Compaction on one mimic node

I am getting a huge number of messages on one out of three nodes showing Manual compaction starting all the time. I see no such of log entries on the other nodes in the cluster. Mar 16 06:40:11 storage1n1-chi docker[24502]: debug 2019-03-16 06:40:11.441 7f6967af4700 4 rocksdb: [/home/jenkins

Re: [ceph-users] [Bluestore] Some of my osd's uses BlueFS slow storage for db - why?

Yes, I was in a similar situation initially where I had deployed my OSD's with 25GB DB partitions and after 3GB DB used, everything else was going into slowDB on disk. From memory 29GB was just enough to make the DB fit on flash, but 30GB is a safe round figure to aim for. With a 30GB DB partit

Re: [ceph-users] Error in Mimic repo for Ubunut 18.04

This seems to be still a problem... Is anybody looking into it? Anybody of Ubuntu users is created ticket to devops [1] project? No... [1] http://tracker.ceph.com/projects/devops/activity k ___ ceph-users mailing list ceph-users@lists.ceph.com htt

Re: [ceph-users] Move from own crush map rule (SSD / HDD) to Luminous device class

in the beginning, I create separate crush rules for SSD and HDD pool ( six Ceph nodes), following this HOWTO: https://www.sebastien-han.fr/blog/2014/08/25/ceph-mix-sata-and-ssd-within-the-same-box/ Now I want to migrate to the standard crush rules, which comes with Luminous. What is the procedur

Re: [ceph-users] Need clarification about RGW S3 Bucket Tagging

On 3/14/19 8:58 PM, Matt Benjamin wrote: Sorry, object tagging. There's a bucket tagging question in another thread :) Luminous works fine with object tagging, at least on 12.2.11 getObjectTagging and putObjectTagging. [k0ste@WorkStation]$ curl -s https://rwg_civetweb/my_bucket/empty-file

Re: [ceph-users] Need clarification about RGW S3 Bucket Tagging

On 3/14/19 8:36 PM, Casey Bodley wrote: The bucket policy documentation just lists which actions the policy engine understands. Bucket tagging isn't supported, so those requests were misinterpreted as normal PUT requests to create a bucket. I opened https://github.com/ceph/ceph/pull/26952 to re

[ceph-users] Need clarification about RGW S3 Bucket Tagging

Hi. I CC'ed Casey Bodley as new RGW tech lead. Luminous doc [1] tells that s3:GetBucketTagging & s3:PutBucketTagging methods is supported.But actually PutBucketTagging fails on Luminous 12.2.11 RGW with "provided input did not specify location constraint correctly", I think is issue [2], but

Re: [ceph-users] weight-set defined for some OSDs and not defined for the new installed ones

On 3/14/19 2:15 PM, Massimo Sgaravatto wrote: I have some clients running centos7.4 with kernel 3.10 I was told that the minimum requirements are kernel >=4.13 or CentOS >= 7.5. Yes, this is correct. k ___ ceph-users mailing list ceph-users@list

Re: [ceph-users] weight-set defined for some OSDs and not defined for the new installed ones

On 3/14/19 2:10 PM, Massimo Sgaravatto wrote: I am using Luminous everywhere I'm mean, what is version of your kernel clients? k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] weight-set defined for some OSDs and not defined for the new installed ones

On 3/14/19 2:09 PM, Massimo Sgaravatto wrote: I plan to use upmap after having migrated all my clients to CentOS 7.6 What is your current release? k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-user

Re: [ceph-users] weight-set defined for some OSDs and not defined for the new installed ones

On 3/14/19 2:02 PM, Massimo Sgaravatto wrote: Oh, I missed this information. So this means that, after having run once the balancer in compat mode, if you add new OSDs you MUST manually defined the weight-set for these newly added OSDs if you want to use the balancer, right ? This is an impo

Re: [ceph-users] weight-set defined for some OSDs and not defined for the new installed ones

On 3/14/19 1:53 PM, Massimo Sgaravatto wrote: So if I try to run the balancer in the current compat mode, should this define the weight-set also for the new OSDs ? But if I try to create a balancer plan, I get an error [*] (while it worked before adding the new OSDs). Nope, balancer creates

Re: [ceph-users] weight-set defined for some OSDs and not defined for the new installed ones

On 3/14/19 1:11 PM, Massimo Sgaravatto wrote: Thanks I will try to set the weight-set for the new OSDs But I am wondering what I did wrong to be in such scenario. You don't. You just use legacy. But why? Jewel clients? Old kernels? Is it normal that a new created OSD has no weight-set defi

Re: [ceph-users] weight-set defined for some OSDs and not defined for the new installed ones

On 3/14/19 12:42 PM, Massimo Sgaravatto wrote: [root@c-mon-01 /]# ceph osd df tree ID  CLASS WEIGHT  REWEIGHT SIZE    USE     AVAIL   %USE VAR  PGS TYPE NAME  -1       1.95190        - 1.95TiB 88.4GiB 1.87TiB    0 0   - root default  -2             0        -      0B      0B      0B    0 0   - 

Re: [ceph-users] weight-set defined for some OSDs and not defined for the new installed ones

I have a cluster where for some OSD the weight-set is defined, while for other OSDs it is not [*]. The OSDs with weight-set defined are Filestore OSDs created years ago using "ceph-disk prepare" The OSDs where the weight set is not defined are Bluestore OSDs installed recently using ceph-volume

Re: [ceph-users] S3 data on specific storage systems

I have a cluster with SSD and HDD storage. I wonder how to configure S3 buckets on HDD storage backends only. Do I need to create pools on this particular storage and define radosgw placement with those or there is a better or easier way to achieve this ? Just assign your "crush hdd rule" to you

Re: [ceph-users] how to identify pre-luminous rdb images

Hello list, I upgraded to mimic some time ago and want to make use of the upmap feature now. But I can't do "ceph osd set-require-min-compat-client luminous" as there are still pre-luminous clients connected. The cluster was originally created from jewel release. When I run "ceph features", I

Re: [ceph-users] 3-node cluster with 3 x Intel Optane 900P - very low benchmarked performance (200 IOPS)?

These results (800 MB/s writes, 1500 Mb/s reads, and 200 write IOPS, 400 read IOPS) seems incredibly low - particularly considering what the Optane 900p is meant to be capable of. Is this in line with what you might expect on this hardware with Ceph though? Or is there some way to find out the s

Re: [ceph-users] ceph bug#2445 hitting version-12.2.4

Hi - we are using ceph 12.2.4 and bug#24445 hitting, which caused 10 min IO pause on ceph cluster.. Is this bug fixed? bug:https://tracker.ceph.com/issues/24445/ Seems this is a network issue, not ceph. Reporter of this ticket was never come backs. k __

Re: [ceph-users] How to use straw2 for new buckets

A few weeks ago I converted everything from straw to straw2 (to be able to use the balancer) using the command: ceph osd crush set-all-straw-buckets-to-straw2 I have now just added a new rack bucket, and moved a couple of new osd nodes in this rack, using the commands: ceph osd crush add-bucket

Re: [ceph-users] [Bluestore] Some of my osd's uses BlueFS slow storage for db - why?

Bluestore/RocksDB will only put the next level up size of DB on flash if the whole size will fit. These sizes are roughly 3GB,30GB,300GB. Anything in-between those sizes are pointless. Only ~3GB of SSD will ever be used out of a 28GB partition. Likewise a 240GB partition is also pointless as onl

Re: [ceph-users] radosgw-admin reshard stale-instances rm experience

My advise: Upgrade to 12.2.11 and run the stale-instances list asap and see if you need to rm data. This isn't available in 13.2.4, but should be in 13.2.5, so on Mimic you will need to wait. But this might bite you at some point. I hope I can prevent some admins from having sleepless nights abo

Re: [ceph-users] How to change/anable/activate a different osd_memory_target value

we run into some OSD node freezes with out of memory and eating all swap too. Till we get more physical RAM I’d like to reduce the osd_memory_target, but can’t find where and how to enable it. We have 24 bluestore Disks in 64 GB centos nodes with Luminous v12.2.11 Just set value for `osd_memory

Re: [ceph-users] [Bluestore] Some of my osd's uses BlueFS slow storage for db - why?

On 2/19/19 11:46 PM, David Turner wrote: I don't know that there's anything that can be done to resolve this yet without rebuilding the OSD.  Based on a Nautilus tool being able to resize the DB device, I'm assuming that Nautilus is also capable of migrating the DB/WAL between devices.  That fu

Re: [ceph-users] krbd: Can I only just update krbd module without updating kernal?

Because of some reasons, I can update the kernal to higher version. So I wonder if I can only just update krbd kernal module ? Has anyone done this before? Of course you can. You "just" need a make krbd patch from upstream kernel and apply it to your kernel tree. It's a lot of work and ma

Re: [ceph-users] [Bluestore] Some of my osd's uses BlueFS slow storage for db - why?

On 2/18/19 9:43 PM, David Turner wrote: Do you have historical data from these OSDs to see when/if the DB used on osd.73 ever filled up?  To account for this OSD using the slow storage for DB, all we need to do is show that it filled up the fast DB at least once.  If that happened, then somethi

Re: [ceph-users] Placing replaced disks to correct buckets.

I recently replaced failed HDDs and removed them from their respective buckets as per procedure. But I’m now facing an issue when trying to place new ones back into the buckets. I’m getting an error of ‘osd nr not found’ OR ‘file or directory not found’ OR command sintax error. I have been using

Re: [ceph-users] Openstack RBD EC pool

### ceph.conf [global] fsid = b5e30221-a214-353c-b66b-8c37b4349123 mon host = ceph-mon.service.i.ewcs.ch auth cluster required = cephx auth service required = cephx auth client required = cephx ### ## ceph.ec.conf [global] fsid = b5e30221-a214-353c

  1   2   3   4   >