Re: [ceph-users] oVirt/RHEV and Ceph

ine-backup and OpenStack database backup once at hour/day. There <http://lists.ovirt.org/pipermail/users/2017-October/084796.html>, on ovirt-users mail list Matthias Leopold perform first tests with Luminous. I'm on the way to Luminous, too.

Re: [ceph-users] rocksdb: Corruption: missing start of fragmented record

Fair point. I just tried with 12.2.1 (on pre-release Ubuntu bionic now). Doesn't change anything - fsck doesn't fix rocksdb, the bluestore won't mount, the OSD won't activate and the error is the same. Is there any fix in .2 that might address this, or do you just mean that in general there will

Re: [ceph-users] rocksdb: Corruption: missing start of fragmented record

Which isn't released yet, yes. I could try building the development repository if you think that has a chance of resolving the issue? For tests - yes... This ML tells that 12.2.2 should be based on commit 1071fdcf73faa387d0df18489ab7b0359a

Re: [ceph-users] features required for live migration

I'd like to use the live migration feature of KVM. In this scenario, what features may be enabled in the rbd base image? and in my EV (snapshot clone)? You can use live migration without features. For KVM I can recommend minimal "rbd default features = 3" (layering, striping).

Re: [ceph-users] features required for live migration

d features? -- Best regards, Konstantin Shalygin ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] features required for live migration

cuted "yum upgrade -y" in order to update all operating system components. And please, apollogize me but In your lines I am not able to find the answer to my questions. Please, can you clarify? -- Best regards, Konstantin Shalygin ___ ceph-use

Re: [ceph-users] features required for live migration

rwise - you can set minimal features (like '3') - this is enough for virtualization (snapshots, clones). And start your project. -- Best regards, Konstantin Shalygin ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/li

[ceph-users] Is the 12.2.1 really stable? Anybody have production cluster with Luminous Bluestore?

Hi cephers. Some thoughts... At this time my cluster on Kraken 11.2.0 - works smooth with FileStore and RBD only. I want upgrade to Luminous 12.2.1 and go to Bluestore because this cluster want grows double with new disks, so is best opportunity migrate to Bluestore. In ML I was found two pro

Re: [ceph-users] Is the 12.2.1 really stable? Anybody have production cluster with Luminous Bluestore?

We upgraded from firefly to 12.2.1 You still on FileStore? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Is the 12.2.1 really stable? Anybody have production cluster with Luminous Bluestore?

My cluster (55 OSDs) runs 12.2.x since the release, and bluestore too All good so far This is cleanly deployed cluster or upgrade from some version? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph

Re: [ceph-users] Is the 12.2.1 really stable? Anybody have production cluster with Luminous Bluestore?

I was told at the Openstack Summit that 12.2.2 should drop "In a few days." That was a week ago yesterday. If you have a little leeway, it may be best to wait. I know I am, but I'm paranoid. There was also a performance regression mentioned recently that's supposed to be fixed. As we can see Er

Re: [ceph-users] Journal / WAL drive size?

When I look at the drive partitions the DB / WAL partitions are only 576Mb & 1GB respectively. This feels a bit small. Before deploy bluestore OSD you should define your db/wal size via: ceph_bluestore_block_db_size: '32212254720' # 30Gb ceph_bluestore_block_wal_size: '1073741824' # 1Gb

Re: [ceph-users] question pool usage vs. pool usage raw

What is the difference between the "usage" and the "raw usage" of a pool? Usage - is your data. Raw - is what actually your data use with all copies (pool 'size' option). I.e. if your data is 1000G - your raw is 3000G. ___ ceph-users mailing list ceph-

Re: [ceph-users] question pool usage vs. pool usage raw

The way is: "rbd help disk-usage" -- Best regards, Konstantin Shalygin ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Is the 12.2.1 really stable? Anybody have production cluster with Luminous Bluestore?

Hi, We're running 12.2.1 on production and facing some memory & cpu issues --> http://tracker.ceph.com/issues/4?next_issue_id=3&prev_issue_id=5 http://tracker.ceph.com/issues/21933 Try 12.2.2 http://ceph.com/releases/v12-2-2-luminous-released/ _

Re: [ceph-users] Luminous 12.2.2 rpm's not signed?

Total size: 51 M Is this ok [y/d/N]: y Downloading packages: Package ceph-common-12.2.2-0.el7.x86_64.rpm is not signed http://tracker.ceph.com/issues/22311 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph

Re: [ceph-users] Memory leak in OSDs running 12.2.1 beyond the buffer_anon mempool leak

We are trying out Ceph on a small cluster and are observing memory leakage in the OSD processes. Try new 12.2.2 - this release should fix memory issues with Bluestore. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listin

Re: [ceph-users] Hangs with qemu/libvirt/rbd when one host disappears

the rule of thumb is 3 for small to mid-sized cluster. 3 mons works with 1+ OSD with Luminous: http://ceph.com/community/new-luminous-scalability/ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-

Re: [ceph-users] Hangs with qemu/libvirt/rbd when one host disappears

as the 1Gb network is completely busy in such a scenario i would assume maybe the problem is that some network communication got stuck somewhere 1Gbit is nothing for ceph OSD hosts. Even if you use only spinners. Don't forget 1Gbit is much more latency and less speed (obviously) compared with 1

Re: [ceph-users] Removing a ceph node and ceph documentation.

Hi, 1. Is there a clear procedure documented for removing a failed ceph-node ? I have read and performed removing ODSs successfully, but are there commands that can be used to remove the node itself ? 2. On the ceph documentation website, how can I find Jewel, Kraken or Luminous specific docs ?

Re: [ceph-users] The way to minimize osd memory usage?

I am testing running ceph luminous(12.2.1-249-g42172a4 (42172a443183ffe6b36e85770e53fe678db293bf) on ARM server. Try new 12.2.2 - this release should fix memory issues with Bluestore. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.

[ceph-users] ceph.com/logos: luminous missed.

Luminous logo is absent. I saw here http://ceph.com/releases/v12-2-0-luminous-released/ luminous logo exists, but this one is very low resolution. k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ce

Re: [ceph-users] 1 osd Segmentation fault in test cluster

>/Is this useful for someone? / Yes! Seehttp://tracker.ceph.com/issues/21259 The latest luminous branch (which you can get from https://shaman.ceph.com/builds/ceph/luminous/) has some additional debugging on OSD shutdown that should help me figure out what is causing this. If this is somethin

Re: [ceph-users] Problems understanding 'ceph features' output

So I assume there are 3 ceph applications (e.g. three VMs) on the jewel host, and 5 applications on the two luminous hosts. To be clear - client is not VM, client is disk. If one VM have 3 disks - 'ceph features' show 3 clients. k ___ ceph-users mai

Re: [ceph-users] [Luminous 12.2.2] Cluster peformance drops after certain point of time

I am testing luminous 12.2.2 and find a strange behavior of my cluster. Check your block.db usage. Luminous 12.2.2 is affected http://tracker.ceph.com/issues/22264 [root@ceph-osd0]# ceph daemon osd.46 perf dump | jq '.bluefs' | grep -E '(db|slow)'   "db_total_bytes": 30064762880,   "db_used_

Re: [ceph-users] determining the source of io in the cluster

I can see, that the io/read ops come from the pool where we store VM volumes, but i can't source this issue to a particular volume. You can use this script https://github.com/cernceph/ceph-scripts/blob/master/tools/rbd-io-stats.pl This is for filestore only. I adapted it to use bluestore for my

Re: [ceph-users] Ceph disk failure causing outage/ stalled writes

he writes to the cluster gets stalled for a good 10-15 mins and actually the machine just shut itself down. If your writes is stalled due host shutdown than your cluster have wrong configuration - like 2 osd-hosts with min_size 2 and failure domain - host. As is usually difficult to give confi

Re: [ceph-users] Proper way of removing osds

Is this the correct way to removes OSDs, or am I doing something wrong ? Generic way for maintenance (e.g. disk replace) is rebalance by change osd weight: ceph osd crush reweight osdid 0 cluster migrate data "from this osd" When HEALTH_OK you can safe remove this OSD: ceph osd out osd_id

Re: [ceph-users] POOL_NEARFULL

Update your ceph.conf file This is also not help. I was create ticket http://tracker.ceph.com/issues/22520 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] qemu-img convert vs rbd import performance

It's already in qemu 2.9 http://git.qemu.org/?p=qemu.git;a=commit;h=2d9187bc65727d9dd63e2c410b5500add3db0b0d " This patches introduces 2 new cmdline parameters. The -m parameter to specify the number of coroutines running in parallel (defaults to 8). And the -W parameter to allow qemu-img to w

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

So finally it logs "scrub ok", but what does " _scan_snaps no head for ..." mean? Does this indicate a problem? Ceph 12.2.2 with bluestore on lvm I think this is because you have snaps created by client before 11.2.1. See http://tracker.ceph.com/issues/19413 I have already come across this o

Re: [ceph-users] Running Jewel and Luminous mixed for a longer period

Performance as well - in my testing FileStore was much quicker than BlueStore. Proof? k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

This is still a pre-production cluster. Most tests have been done using rbd. We did make some rbd clones / snapshots here and there. What clients you used? k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ce

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

On 01/04/2018 11:38 PM, Stefan Kooman wrote: Only luminous clients. Mostly rbd (qemu-kvm) images. Who is managed your images? May be OpenStack Cinder? k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-u

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

On 01/04/2018 11:53 PM, Stefan Kooman wrote: OpenNebula 5.4.3 (issuing rbd commands to ceph cluster). Yes! And what librbd is installed on "commands issuer"? k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi

Re: [ceph-users] Bad crc causing osd hang and block all request.

What could cause this problem?Is this caused by a faulty HDD? what data's crc didn't match ? This may be caused due faulty drive. Check your dmesg. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph

Re: [ceph-users] Real life EC+RBD experience is required

Hello. My real life experience tells me that this kind of setup will use much more hardware resources and will show lower benchmarks compared to recommended replicated pools on the same hardware. Writes to ec in some cases better than replicated pools. http://en.community.dell.com/cfs-file/__

Re: [ceph-users] replace failed disk in Luminous v12.2.2

Now wonder what is the correct way to replace a failed OSD block disk? Generic way for maintenance (e.g. disk replace) is rebalance by change osd weight: ceph osd crush reweight osdid 0 cluster migrate data "from this osd" When HEALTH_OK you can safe remove this OSD: ceph osd out osd_id syst

Re: [ceph-users] Suggestion fur naming RBDs

Hi, I was wondering what naming scheme you use for naming RBDs in different pools. There are no strict rules I know, so what might be a best practice? Something like the target service like fileserver_students or webservers_xen, webservers_vmware? A good naming scheme might be helpful :) Mo

Re: [ceph-users] What is the should be the expected latency of 10Gbit network connections

ping -c 10 -f 10.0.1.12 Intel X710-DA2 -> Switch -> Intel X710-DA2: --- 172.16.16.3 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 1932ms rtt min/avg/max/mdev = 0.013/0.014/0.131/0.004 ms, ipg/ewma 0.019/0.014 ms k __

[ceph-users] luminous rbd feature 'striping' is deprecated or just a bug?

Anybody know about changes in rbd feature 'striping'? May be is deprecated feature? What I mean: I have volume created by Jewel client on Luminous cluster. # rbd --user=cinder info solid_rbd/volume-12b5df1e-df4c-4574-859d-22a88415aaf7 rbd image 'volume-12b5df1e-df4c-4574-859d-22a88415aaf7':  

Re: [ceph-users] POOL_NEARFULL

On 01/29/2018 04:25 PM, Karun Josy wrote: In Luminous version, we have to use osd set command Yep. Since Luminous _full options saved in osdmap. k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-

Re: [ceph-users] luminous rbd feature 'striping' is deprecated or just a bug?

On 01/29/2018 06:40 PM, Ilya Dryomov wrote: Unless you specify a non-default stripe_unit/stripe_count, striping feature bit is not set and striping-related fields aren't displayed. This behaviour is new in luminous, but jewel and older clients still work with luminous images. Yes, I see it... I

Re: [ceph-users] luminous rbd feature 'striping' is deprecated or just a bug?

On 01/29/2018 07:49 PM, Jason Dillaman wrote: To me, it didn't make sense to set the striping feature bit if fancy striping wasn't really being used. The same logic was applied to the "data-pool" feature bit -- it does make sense to set it if the data pool is really not different from the base i

Re: [ceph-users] luminous rbd feature 'striping' is deprecated or just a bug?

On 01/29/2018 08:33 PM, Jason Dillaman wrote: OK -- but that is the normal case of RBD w/o the need for fancy striping (i.e. no need for the special feature bit). The striping feature is only needed when using stripe counts != 1 and stripe units != object size. When you specify the "--stripe-unit

[ceph-users] Migration from "classless pre luminous" to "device classes" CRUSH.

Hi cephers. I have typical double root crush - for nvme pools and hdd pools created on Kraken cluster (what I mean: http://cephnotes.ksperis.com/blog/2015/02/02/crushmap-example-of-a-hierarchical-cluster-map). Now cluster upgraded to Luminous and going to devices classes crush rules and I l

Re: [ceph-users] Migration from "classless pre luminous" to"deviceclasses" CRUSH.

We had a MASSIVE data movement upon changing the crush rules to device class based one. I'm not sure about the exact reasons, but I assume that the order of hosts in the crush tree has changed (hosts are ordered lexically now...), which resulted in about 80% of data being moved around. What is

Re: [ceph-users] Migration from "classless pre luminous" to"deviceclasses" CRUSH.

On 02/01/2018 08:56 PM, David Turner wrote: You can attempt to mitigate this by creating new, duplicate rules and change 1 pool at a time to start using them. Yes, I'm already prepared to this strategy. k ___ ceph-users mailing list ceph-users@lis

Re: [ceph-users] Migration from "classless pre luminous" to"deviceclasses" CRUSH.

Migration was complete flawless without any issues and slow requests. Thanks. k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] New Ceph-cluster and performance "questions"

/offtopic When and where did you get those? I wonder if they're available again, had 0 luck getting any last year. I was behold P3700 in Russia since December 2017 with real quantity on stock, not just a "price with out of stock". https://market.yandex.ru/catalog/55316/list?text=intel%20p3

Re: [ceph-users] max number of pools per cluster

And if for any reason even single PG was damaged and for example stuck inactive - then all RBDs will be affected. First that come to mind is to create a separate pool for every RBD. I think this is insane. Is better to think how Kipod save data in CRUSH. Plan your failure domains and perform f

Re: [ceph-users] Ceph-mgr Python error with prometheus plugin

i just try to get the prometheus plugin up and runing Use module from master. From this commit should work with 12.2.2, just wget it and replace stock module. https://github.com/ceph/ceph/blob/d431de74def1b8889ad568ab99436362833d063e/src/pybind/mgr/prometheus/module.py k __

Re: [ceph-users] Ceph-mgr Python error with prometheus plugin

On 02/18/2018 09:20 AM, Sergey Malinin wrote: All I got with script replacement is the following: Oooh. Try something earlier, without this function https://github.com/ceph/ceph/blob/1adf4325455ccdbf03da63046c510bb041ac8520/src/pybind/mgr/prometheus/module.py To be clear: prometheus module

Re: [ceph-users] Erasure coding RBD pool for OpenStack Glance, Nova and Cinder

Does someone have used EC pools with OpenStack in production ? By chance, I found that link : https://www.reddit.com/r/ceph/comments/72yc9m/ceph_openstack_with_ec/ Yes, this good post. My configuration is: cinder.conf: [erasure-rbd-hdd] volume_driver = cinder.volume.drivers.rbd.RBDDriver

Re: [ceph-users] Rotating Cephx Keys

If I want to rotate the keys for that I can simply do that ceph cluster side, but then I also need to do that on the client side (in my case virtual machine hypervisors). DUring this window (which might be tiny with decent tooling, but still non-zero) my clients can't do new connections to the cep

Re: [ceph-users] Rotating Cephx Keys

On 07/10/2018 11:41 AM, Graeme Gillies wrote: I think you are missing the part where if you update a key in ceph, in the space between that and when you update it in ovirt-engine any new connections to ceph by any ovirt nodes will fail Yes, this is should be seconds. But, actually startup will

Re: [ceph-users] Erasure coding RBD pool for OpenStack Glance, Nova and Cinder

So if you want, two more questions to you : - How do you handle your ceph.conf configuration (default data pool by user) / distribution ? Manually, config management, openstack-ansible... ? - Did you made comparisons, benchmarks between replicated pools and EC pools, on the same hardware / drives

Re: [ceph-users] Journel SSD recommendation

I have lots of Samsung 850 EVO but they are consumer, Do you think consume drive should be good for journal? No. Since the fall of 2017purchase of Intel P3700 is not difficult, you should buy it if you can. k ___ ceph-users mailing list ceph-us

Re: [ceph-users] SSDs for data drives

In a recent thread the Samsung SM863a was recommended as a journal SSD. Are there any recommendations for data SSDs, for people who want to use just SSDs in a new Ceph cluster? Take a look to HGST SN260, this is MLC NVMe's [1] [1] https://www.hgst.com/products/solid-state-solutions/ultrast

Re: [ceph-users] OSD tuning no longer required?

I saw this in the Luminous release notes: "Each OSD now adjusts its default configuration based on whether the backing device is an HDD or SSD. Manual tuning generally not required" Which tuning in particular? The ones in my configuration are osd_op_threads, osd_disk_threads, osd_recovery

Re: [ceph-users] Increase queue_depth in KVM

I've seen some people using 'num_queues' but I don't have this parameter in my schemas(libvirt version = 1.3.1, qemu version = 2.5.0 num-queues is available from qemu 2.7 [1] [1] https://wiki.qemu.org/ChangeLog/2.7 k ___ ceph-users mailing list

Re: [ceph-users] Slow requests during OSD maintenance

2. What is the best way to remove an OSD node from the cluster during maintenance? ceph osd set noout is not the way to go, since no OSD's are out during yum update and the node is still part of the cluster and will handle I/O. I think the best way is the combination of "ceph osd set noout" + stop

Re: [ceph-users] Migrating EC pool to device-class crush rules

So mostly I want to confirm that is is safe to change the crush rule for the EC pool. Changing crush rules for replicated or ec pool is safe. One thing is, when I was migrated from multiroot to device-classes I was recreate ec pools and clone images with qemu-img for ec_overwrites feature, so

Re: [ceph-users] Crush Rules with multiple Device Classes

Now my first question is: 1) Is there a way to specify "take default class (ssd or nvme)"? Then we could just do this for the migration period, and at some point remove "ssd". If multi-device-class in a crush rule is not supported yet, the only workaround which comes to my mind right now is

Re: [ceph-users] Error bluestore doesn't support lvm

I am using openstack-ansible with ceph-ansible to deploy my Ceph custer and here is my config in yml file --- osd_objectstore: bluestore osd_scenario: lvm lvm_volumes: - data: /dev/sdb - data: /dev/sdc - data: /dev/sdd - data: /dev/sde This is the error i am getting.. TASK [ceph-os

Re: [ceph-users] 12.2.7 - Available space decreasing when adding disks

Hello Ceph Users, We have added more ssd storage to our ceph cluster last night. We added 4 x 1TB drives and the available space went from 1.6TB to 0.6TB ( in `ceph df` for the SSD pool ). I would assume that the weight needs to be changed but I didn't think I would need to? Should I change t

Re: [ceph-users] Self shutdown of 1 whole system (Derbian stretch/Ceph 12.2.7/bluestore)

I even have no fancy kernel or device, just real standard Debian. The uptime was 6 days since the upgrade from 12.2.6... Nicolas, you should upgrade your 12.2.6 to 12.2.7 due bugs in this release. http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-July/028153.html k _

[ceph-users] radosgw: S3 object retention: high usage of default.rgw.log pool

Hi. For some bucket for backup application is applied S3 retention policy, at 04:00 2+days backups will be deleted from bucket. At this time I see very high usage of default.rgw.log pool. Usage log is enabled, ops log is disabled, index pool on NVMe: - https://ibb.co/dozqPJ - https://ibb.c

Re: [ceph-users] radosgw: S3 object retention: high usage of default.rgw.log pool

I was misled.In fact, this is not an automatic deletion, but the removal of one object per op by application. Reject. k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Error: journal specified but not allowed by osd backend

ceph_disk.main.Error: Error: journal specified but not allowed by osd backend I faced this issue once before. The problem is - function is query for osd.0 instead your osd.21. Change in main.py '-i', '0', to 21 (your osd number) '-i', '21', and try again. k

Re: [ceph-users] Broken multipart uploads

after emtpying the bucket, cannot deleted since there are some aborted multipart uploads radosgw-admin bucket check --bucket=weird_bucket [ "_multipart_DISK_P/collection_1/anonymous/GRLIX/GRLIX_001069.pdf.VOeGNgr-gvhXCrf6dlnhAqhjaFHIF7t.1", "_multipart_DISK_P/collection_1/anonymous/GR

Re: [ceph-users] Broken multipart uploads

On 08/08/2018 01:31 AM, DHD.KOHA wrote: But still, I get No Such key !! s3cmd abortmp s3://weird_bucket 2~CDIJMxZvy8aQejBGBPeNyQK-AJ1lmO4 ERROR: S3 error: 404 (NoSuchKey) s3cmd abortmp s3://weird_bucket 2~alvAZmF5tAlSeiJrUjOwXV7Io22uH0H ERROR: S3 error: 404 (NoSuchKey) Try like this: s3cmd

Re: [ceph-users] Inconsistent PGs every few days

Hi, I run a cluster with 7 OSD. The cluster has no much traffic on it. But every few days, I get a HEALTH_ERR, because of inconsistent PGs: root at Sam ~ # ceph status cluster: id: c4bfc288-8ba8-4c3a-b3a6-ed95503f50b7

Re: [ceph-users] RBD image "lightweight snapshots"

1. Create snapshot of the image we want to backup 2. If there's a previous backup snapshot, export diff and apply it on the backup image 3. If there's no older snapshot, just do a full backup of image So you need incremental backup? Try look to "rbd2qcow2" [1] [1] https://github.com/socketpai

Re: [ceph-users] Help needed for debugging slow_requests

Now here's the thing: Some weeks ago Proxmox upgraded from kernel 4.13 to 4.15. Since then I'm getting slow requests that cause blocked IO inside the VMs that are running on the cluster (but not necessarily on the host with the OSD causing the slow request). If I boot back into 4.13 then Ceph

Re: [ceph-users] Scope of ceph.conf rgw values

I am turning off resharding for Luminous with rgw dynamic resharding = false on the rgw server. When I show the configuration on that server (with ceph daemon), I see that it is false, like I expect. When I show the configuration on the monitor servers, that setting shows up as "true". Do I n

Re: [ceph-users] Invalid Object map without flags set

We are having issues with ensuring that object-map and fast-diff is working correctly. Most of the time when there is an invalid fast-diff map, the flag is set to correctly indicate this. We have a script that checks for this and rebuilds object maps as required. If we don't fix these, snapshot

Re: [ceph-users] How to set the DB and WAL partition size in Ceph-Ansible?

Hi everyone, If I choose to use the "non-collocated" scenario and Bluestore in Ceph-Ansible, how could I define the size of the partitions on a dedicated device used for the DB and WAL by multiple OSDs? Like this in global_vars: ceph_conf_overrides:   global:     bluestore_block_db_size: "{{ ho

Re: [ceph-users] How to set the DB and WAL partition size in Ceph-Ansible?

On 08/20/2018 09:07 AM, Cody wrote: Hi Konstantin, Thank you for the reply. Would the settings in the 'ceph_conf_overrides' in the all.yml get applied to the partitioning process during deployment? Yes, because ceph.conf will be deployed before osds. Look at roles path: - hosts: ceph-mons  

Re: [ceph-users] Questions on CRUSH map

Hi everyone, I am new to Ceph and trying to test out my understanding on the CRUSH map. Attached is a hypothetical cluster diagram with 3 racks. On each rack, the first host runs 3 SSD-based OSDs and the second 3 HDD-based. My goal is to create two rules that separate SSD and HDD performance dom

Re: [ceph-users] QEMU/Libvirt + librbd issue using Luminous 12.2.7

This issue first started while using Luminous 12.2.5, I upgraded to 12.2.7 and it's still present.  This issue is _not_ present in 12.2.4. With Ceph 12.2.4, using QEMU/KVM + Libvirt, I'm able to mount an rbd image using the following syntax and populated xml: 'virsh attach-device $vm foo.xml -

Re: [ceph-users] Questions on CRUSH map

On 08/20/2018 08:15 PM, Cody wrote: Hi Konstantin, Thank you for looking into my question. I was trying to understand how to set up CRUSH hierarchies and set rules for different failure domains. I am particularly confused by the 'step take' and 'step choose|chooseleaf' settings for which I thin

Re: [ceph-users] QEMU/Libvirt + librbd issue using Luminous 12.2.7

On 08/21/2018 11:44 PM, Andre Goree wrote: Thank you for your reply. Interestingly, the same (or similar enough) settings still fail here.  Which version of libvirt are you using?  I think maybe this is a libvirt or QEMU bug, and not specifically Ceph...? ~# qemu-system-x86_64 --version

Re: [ceph-users] Question about 'firstn|indep'

Hi everyone, I read an earlier thread [1] that made a good explanation on the 'step choose|chooseleaf' option. Could someone further help me to understand the 'firstn|indep' part? Also, what is the relationship between 'step take' and 'step choose|chooseleaf' when it comes to define a failure dom

[ceph-users] radosgw: need couple of blind (indexless) buckets, how-to?

I need bucket without index for 5000 objects, how to properly create a indexless bucket in next to indexed buckets? This is "default radosgw" Luminous instance. I was take a look to cli, as far as I understand I will need to create placement rule via "zone placement add" and add this key t

Re: [ceph-users] Migrating from pre-luminous multi-root crush hierachy

We recently upgrade to luminous (you can see the device-classes in the output). So it should be possible to have one single root, no fake hosts and just use the device-class. We added some hosts/osds recently which back a new pools, so we also created a new hierarchy and crush rules for those.

Re: [ceph-users] Migrating from pre-luminous multi-root crush hierachy

On 08/24/2018 01:57 PM, Buchberger, Carsten wrote: Hi Konstantin, sounds easy;-) If i apply the new rule to the existing pools there won't be any osds to satisfy the requirements of the rule - because the osds are not in the new root yet. Isn't that a problem ? Thank you Your IO will stall

Re: [ceph-users] RGW pools don't show up in luminous

I installed a new Ceph cluster with Luminous, after a long time working with Jewel. I created my RGW pools the same as always (pool create default.rgw.buckets.data etc.), but they don't show up in ceph df with Luminous. Has the command changed? Since Luminous you don't need to create pools.

Re: [ceph-users] radosgw: need couple of blind (indexless) buckets, how-to?

Answer to myself. radosgw-admin realm create --rgw-realm=default --default radosgw-admin zonegroup modify --rgw-zonegroup=default --rgw-realm=default radosgw-admin period update --commit radosgw-admin zonegroup placement add --rgw-zonegroup="default" \   --placement-id="indexless-placement" rados

Re: [ceph-users] radosgw: need couple of blind (indexless) buckets, how-to?

Thank you very much! If anyone would like to help update these docs, I would be happy to help with guidance/review. I was make a try half year ago - http://tracker.ceph.com/issues/23081 k ___ ceph-users mailing list ceph-users@lists.ceph.com http

Re: [ceph-users] Why rbd rn did not clean used pool?

Configuration: rbd - erasure pool rbdtier - tier pool for rbd ceph osd tier add-cache rbd rbdtier 549755813888 ceph osd tier cache-mode rbdtier writeback Create new rbd block device: rbd create --size 16G rbdtest rbd feature disable rbdtest object-map fast-diff deep-flatten rbd device map rbdte

Re: [ceph-users] Why rbd rn did not clean used pool?

On 08/26/2018 01:39 PM, Vasiliy Tolstov wrote: Why avoid cache tier? Does this only for erasure or for replicated too? Because cache tier is very uncommon feature. Cepher's was used it to will have rbd writes to EC pools mostly, before Luminous [1] Why this need for replicated? With cache tie

Re: [ceph-users] mount cephfs without tiering

I have cephfs with tiering. Does anyone know if it's possible to mount a file system so that the tiring is not used? I.e. I want mount cephfs on backup server without tiering usage and on samba server with tiering usage. It's possible? https://ceph.com/community/new-luminous-erasure-coding-r

Re: [ceph-users] Is luminous ceph rgw can only run with the civetweb ?

In jewel I use the below config rgw is work well with the nginx. But with luminous the nginx look like can not work with the rgw. In your case use proxy before rgw is totally overhead IMHO (no balancing, no ha). This is working configuration for your case: upstream rados {   server  10.11

Re: [ceph-users] CephFS small files overhead

We are trying to use cephfs as storage for web graphics, such as thumbnails and so on. Is there any way to reduse overhead on storage? On test cluster we have 1 fs, 2 pools (meta and data) with replica size = 2     objects: 1.02 M objects, 1.1 GiB     usage:   144 GiB used, 27 GiB / 172 GiB ava

Re: [ceph-users] Is luminous ceph rgw can only run with the civetweb ?

On 09/20/2018 10:09 AM, linghucongsong wrote: By the way  I use  keepalive+lvs  to loadbalance and ha. This is good. But in that case I wonder why fastcgi+nginx, instead civetweb or beast? k ___ ceph-users mailing list ceph-users@lists.ceph.com h

Re: [ceph-users] Mixing EC and Replicated pools on HDDs in Ceph RGW Luminous

I am setting up RadosGW and Ceph cluster on Luminous. I am using EC for `buckets.data` pool on HDD osds, is it okay to put `buckets.non-ec` pool with replicated ruleset for multi-parts upload on the same HDD osds? Will there be issues with mixing EC and replicated pools on the same disk types?

Re: [ceph-users] v12.2.8 Luminous released

12.2.8 improves the deep scrub code to automatically repair these inconsistencies. Once the entire cluster has been upgraded and then fully deep scrubbed, and all such inconsistencies are resolved; it will be safe to disable the `osd distrust data digest = true` workaround option. Just for clare

Re: [ceph-users] Ceph backfill problem

Has anyone experienced below? 2 of OSD server was down, after bring up 2 of servers, I brought 52 OSD's in with just weight of 0.05, but it causing huge backfilling load, I saw so many blocked requests and a number of pg stuck inactive. some of servers was impact. so I stopped backfilling by mark

Re: [ceph-users] network architecture questions

Hi, I read through the various documentation and had a few questions: - From what I understand cephFS clients reach the OSDs directly, does the cluster network need to be opened up as a public network? All RADOS clients connect to osd's and mon's directly. - Is it still necessary to have a publ

  1   2   3   4   >