Re: [ceph-users] Issue with fstrim and Nova hw_disk_discard=unmap

Hallo Jason, sure here it is! rbd --cluster cephpa1 -p cinder-ceph info volume-80838a69-e544-47eb-b981-a4786be89736 rbd image 'volume-80838a69-e544-47eb-b981-a4786be89736': size 15360 MB in 3840 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.9e7ffe238e1f2

[ceph-users] Ceph see the data size larger than actual stored data in rbd

Hi , I found some thing strange in ceph. The actual stored data in mounted rbd image is 156 G . /dev/rbd0p1 985G 156G 779G 17% /mnt When I run ceph -s , I see the actual data is 316 GB. pgmap v27446382: 1024 pgs, 1 pools, 316 GB data, 81742 objects 952 GB used, 44145 GB

[ceph-users] why we show removed snaps in ceph osd dump pool info?

what is the purpose for we to show the removed snaps? look like the removed snaps no use to the user. we use rbd export and import backup images from one ceph cluster to another ceph cluster. the increment image backup depand on the snap.and we wiil remove the snap after the backup.so it will sh

Re: [ceph-users] Ceph see the data size larger than actual stored data in rbd

Hi, In order to reclaim space in Ceph you will need to use the discard feature of KRBD: https://www.sebastien-han.fr/blog/2015/01/26/ceph-and-krbd-discard/ Kind regards, Caspar 2018-03-14 10:34 GMT+01:00 Mostafa Hamdy Abo El-Maty El-Giar < mostafaha...@mans.edu.eg>: > Hi , > > I found some thi

Re: [ceph-users] Object Gateway - Server Side Encryption

On Tuesday 13 March 2018 10:32 PM, Casey Bodley wrote: On 03/10/2018 12:58 AM, Amardeep Singh wrote: On Saturday 10 March 2018 02:01 AM, Casey Bodley wrote: On 03/08/2018 07:16 AM, Amardeep Singh wrote: Hi, I am trying to configure server side encryption using Key Management Service as per

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

On 2018-02-28T02:38:34, Patrick Donnelly wrote: > I think it will be necessary to reduce the actives to 1 (max_mds -> 1; > deactivate other ranks), shutdown standbys, upgrade the single active, > then upgrade/start the standbys. > > Unfortunately this didn't get flagged in upgrade testing. Thank

Re: [ceph-users] Ceph iSCSI is a prank?

On 2018-03-02T15:24:29, Joshua Chen wrote: > Dear all, > I wonder how we could support VM systems with ceph storage (block > device)? my colleagues are waiting for my answer for vmware (vSphere 5) and > I myself use oVirt (RHEV). the default protocol is iSCSI. Lean on VMWare to stop being diff

Re: [ceph-users] Issue with fstrim and Nova hw_disk_discard=unmap

Hmm -- perhaps as an experiment, can you disable the object-map and fast-diff features to see if they are incorrectly reporting the object as in-use after a discard? $ rbd --cluster cephpa1 -p cinder-ceph feature disable volume-80838a69-e544-47eb-b981-a4786be89736 object-map,fast-diff On Wed, Mar

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

On 03/14/2018 01:48 PM, Lars Marowsky-Bree wrote: > On 2018-02-28T02:38:34, Patrick Donnelly wrote: > >> I think it will be necessary to reduce the actives to 1 (max_mds -> 1; >> deactivate other ranks), shutdown standbys, upgrade the single active, >> then upgrade/start the standbys. >> >> Unfor

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

On Wed, Mar 14, 2018 at 5:48 AM, Lars Marowsky-Bree wrote: > On 2018-02-28T02:38:34, Patrick Donnelly wrote: > >> I think it will be necessary to reduce the actives to 1 (max_mds -> 1; >> deactivate other ranks), shutdown standbys, upgrade the single active, >> then upgrade/start the standbys. >>

Re: [ceph-users] Cephfs MDS slow requests

On Tue, Mar 13, 2018 at 7:17 PM, David C wrote: > Hi All > > I have a Samba server that is exporting directories from a Cephfs Kernel > mount. Performance has been pretty good for the last year but users have > recently been complaining of short "freezes", these seem to coincide with > MDS related

[ceph-users] rctime not tracking inode ctime

Hi all, On our luminous v12.2.4 ceph-fuse clients / mds the rctime is not tracking the latest inode ctime, but only the latest directory ctimes. Initial empty dir: # getfattr -d -m ceph . | egrep 'bytes|ctime' ceph.dir.rbytes="0" ceph.dir.rctime="1521043742.09466372697" Create a file, rctime is

Re: [ceph-users] Luminous | PG split causing slow requests

On Mon, Feb 26, 2018 at 6:08 PM, David Turner wrote: > The slow requests are absolutely expected on filestore subfolder > splitting. You can however stop an OSD, split it's subfolders, and start > it back up. I perform this maintenance once/month. I changed my settings > to [1]these, but I onl

Re: [ceph-users] Cephfs MDS slow requests

Thanks, John. I'm pretty sure the root of my slow OSD issues is filestore subfolder splitting. On Wed, Mar 14, 2018 at 2:17 PM, John Spray wrote: > On Tue, Mar 13, 2018 at 7:17 PM, David C wrote: > > Hi All > > > > I have a Samba server that is exporting directories from a Cephfs Kernel > > mo

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

On Sun, Mar 11, 2018 at 5:10 PM, Mike Christie wrote: > On 03/11/2018 08:54 AM, shadow_lin wrote: > > Hi Jason, > > How the old target gateway is blacklisted? Is it a feature of the target > > gateway(which can support active/passive multipath) should provide or is > > it only by rbd excusive loc

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

Maxim, can you provide steps for a reproducer? On Wed, Mar 14, 2018 at 2:06 PM, Maxim Patlasov wrote: > On Sun, Mar 11, 2018 at 5:10 PM, Mike Christie wrote: >> >> On 03/11/2018 08:54 AM, shadow_lin wrote: >> > Hi Jason, >> > How the old target gateway is blacklisted? Is it a feature of the targ

Re: [ceph-users] Understanding/correcting sudden onslaught of unfound objects

The fix for tracker 20089 undid the changes you're seeing in the 15368 pull request.  The attr name mismatch of 'hinfo_key'  means that key is missing because every erasure coded object should have a key called "hinfo_key." You should try to determine why your extended attributes are getting

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

On Wed, Mar 14, 2018 at 11:13 AM, Jason Dillaman wrote: > Maxim, can you provide steps for a reproducer? > Yes, but it involves adding two artificial delays: one in tcmu-runner and another in kernel iscsi. If you're willing to take pains of recompiling kernel and tcmu-runner on one of gateway no

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

On 03/14/2018 01:06 PM, Maxim Patlasov wrote: > On Sun, Mar 11, 2018 at 5:10 PM, Mike Christie > wrote: > > On 03/11/2018 08:54 AM, shadow_lin wrote: > > Hi Jason, > > How the old target gateway is blacklisted? Is it a feature of the target > > gateway(

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

On 03/14/2018 01:24 PM, Maxim Patlasov wrote: > On Wed, Mar 14, 2018 at 11:13 AM, Jason Dillaman > wrote: > > Maxim, can you provide steps for a reproducer? > > > Yes, but it involves adding two artificial delays: one in tcmu-runner > and another in kernel iscsi.

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

On 03/14/2018 01:26 PM, Michael Christie wrote: > On 03/14/2018 01:06 PM, Maxim Patlasov wrote: >> On Sun, Mar 11, 2018 at 5:10 PM, Mike Christie > > wrote: >> >> On 03/11/2018 08:54 AM, shadow_lin wrote: >> > Hi Jason, >> > How the old target gateway is blac

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

On Wed, Mar 14, 2018 at 11:47 AM, Michael Christie wrote: > > > ... > > Ignore all these questions. I'm pretty sure I know the issue. > > Fine, but can you please also elaborate on: > For this case it would be tcmu_rbd_handle_blacklisted_cmd How does it tell kernel to stop iscsi connection? T

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

On 03/14/2018 01:27 PM, Michael Christie wrote: > On 03/14/2018 01:24 PM, Maxim Patlasov wrote: >> On Wed, Mar 14, 2018 at 11:13 AM, Jason Dillaman > > wrote: >> >> Maxim, can you provide steps for a reproducer? >> >> >> Yes, but it involves adding two artificial del

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

On Wed, Mar 14, 2018 at 12:05 PM, Michael Christie wrote: > On 03/14/2018 01:27 PM, Michael Christie wrote: > > On 03/14/2018 01:24 PM, Maxim Patlasov wrote: > >> On Wed, Mar 14, 2018 at 11:13 AM, Jason Dillaman >> > wrote: > >> > >> Maxim, can you provide steps f

[ceph-users] Disk write cache - safe?

I'm using Ceph on Ubuntu 16.04 on Dell R730xd servers. A recent [1] update to the PERC firmware disabled the disk write cache by default which made a noticable difference to the latency on my disks (spinning disks, not SSD) - by as much as a factor of 10. For reference their change list says: "Ch

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

On 2018-03-14T06:57:08, Patrick Donnelly wrote: > Yes. But the real outcome is not "no MDS [is] active" but "some or all > metadata I/O will pause" -- and there is no avoiding that. During an > MDS upgrade, a standby must take over the MDS being shutdown (and > upgraded). During takeover, metada

Re: [ceph-users] Disk write cache - safe?

Tim, Enabling the drive write cache is a recipe for disaster. In the event of a power interruption, you have in-flight data that is stored in the cache and uncommitted to the disk media itself. Being that the power is interrupted and the drive cache does not have a battery or supercap to keep

[ceph-users] Hybrid pool speed (SSD + SATA HDD)

Hello everyone, I have been thinking about building  a hybrid storage pool (inspiration from this article: http://www.root314.com/ceph/2017/04/30/Ceph-hybrid-storage- tiers/). So instead of 3 replicas on SSD I plan to use 2 SSD and the third one will be plain old SATA HDD. I can easily arran

Re: [ceph-users] Updating standby mds from 12.2.2 to 12.2.4 caused up:active 12.2.2 mds's to suicide

On Wed, Mar 14, 2018 at 12:41 PM, Lars Marowsky-Bree wrote: > On 2018-03-14T06:57:08, Patrick Donnelly wrote: > >> Yes. But the real outcome is not "no MDS [is] active" but "some or all >> metadata I/O will pause" -- and there is no avoiding that. During an >> MDS upgrade, a standby must take ove

Re: [ceph-users] Disk write cache - safe?

Tim, I can corroborate David’s sentiments as it pertains to being a disaster. In the early days of my Ceph cluster, I had 8TB SAS drives behind an LSI RAID controller as RAID0 volumes (no IT mode), with on-drive write-caching enabled (pdcache=default). I subsequently had my the data center whe

Re: [ceph-users] Hybrid pool speed (SSD + SATA HDD)

On Wed, Mar 14, 2018 at 09:50:12PM +0100, mart.v wrote: : But from what I understood so far, during the writing process the : client communicates also only with the primary OSDs but it will wait : until all data are written on all replicas. This is my main concern. : Does this mean that wr

[ceph-users] mount.ceph error 5

Hello there, I am trying to set up Cephfs on and for a testing system which will be running for a couple of months.It is a single node system so far (of course a non-ideal setup only for non prouctive usage). I fail to mount my cephfs with error 5. I set up ceph on centos7 using the followin

Re: [ceph-users] iSCSI Multipath (Load Balancing) vs RBD Exclusive Lock

On Wed, Mar 14, 2018 at 12:05 PM, Michael Christie wrote: > On 03/14/2018 01:27 PM, Michael Christie wrote: >> > On 03/14/2018 01:24 PM, Maxim Patlasov wrote: >> >> On Wed, Mar 14, 2018 at 11:13 AM, Jason Dillaman > >> > wrote: >> >> >> >> Maxim, can you provide st

Re: [ceph-users] rctime not tracking inode ctime

On Wed, Mar 14, 2018 at 9:22 AM, Dan van der Ster wrote: > Hi all, > > On our luminous v12.2.4 ceph-fuse clients / mds the rctime is not > tracking the latest inode ctime, but only the latest directory ctimes. > > Initial empty dir: > > # getfattr -d -m ceph . | egrep 'bytes|ctime' > ceph.dir.rbyt

Re: [ceph-users] mount.ceph error 5

On Wed, Mar 14, 2018 at 1:59 PM, Marc Marschall wrote: > Hello there, > > I am trying to set up Cephfs on and for a testing system which will be > running for a couple of months.It is a single node system so far (of course > a non-ideal setup only for non prouctive usage). I fail to mount my cephf

[ceph-users] Bluestore with CephFS: Recommendations for WAL / DB device for MDS

Dear Cephalopodians, for our MDS data, we are currently using 4 Bluestore-OSDs with classic SATA SSDs (Replica mode). In metadata stress testing, this easily becomes the bottleneck. Also, the existing SSDs are too small (4 x 240 GB, 4 replicas) if the cluster was to become full. My question