Re: [ceph-users] Luminous LTS: `ceph osd crush class create` is gone?

On Fri, Nov 3, 2017 at 4:04 PM, Linh Vu wrote: > Hi all, > > > Back in Luminous Dev and RC, I was able to do this: > > > `ceph osd crush class create myclass` This was removed as part of https://github.com/ceph/ceph/pull/16388 It looks like the set-device-class command is the replacement or equi

[ceph-users] Luminous LTS: `ceph osd crush class create` is gone?

Hi all, Back in Luminous Dev and RC, I was able to do this: `ceph osd crush class create myclass` so I could utilise the new CRUSH device classes feature as described here: http://ceph.com/community/new-luminous-crush-device-classes/ and in use here: http://blog-fromsomedude.rhcloud.com/

Re: [ceph-users] CephFS: clients hanging on write with ceph-fuse

Either ought to work fine. On Thu, Nov 2, 2017 at 4:58 PM Andras Pataki wrote: > I'm planning to test the newer ceph-fuse tomorrow. Would it be better to > stay with the Jewel 10.2.10 client, or would the 12.2.1 Luminous client be > better (even though the back-end is Jewel for now)? > > > Andr

Re: [ceph-users] 回复: Re: [luminous]OSD memory usage increase when writing a lot of data to cluster

On Wed, Nov 1, 2017 at 11:54 PM, Mazzystr wrote: > I experienced this as well on tiny Ceph cluster testing... > > HW spec - 3x > Intel i7-4770K quad core > 32Gb m2/ssd > 8Gb memory > Dell PERC H200 > 6 x 3Tb Seagate > Centos 7.x > Ceph 12.x > > I also run 3 memory hungry procs on the Ceph nodes.

Re: [ceph-users] Ceph RDB with iSCSI multipath

There was a little delay getting things merged in the upstream kernel so we are now hoping for v4.16. You should be able to take a 4.15 rc XYZ kernel and apply the patches from this thread [1]. It's due to this upstream delay that CentOS 7.4. doesn't have the patches backported, but hopefully a for

Re: [ceph-users] iSCSI: tcmu-runner can't open images?

On Thu, Nov 2, 2017 at 11:34 AM, Matthias Leopold < matthias.leop...@meduniwien.ac.at> wrote: > Hi, > > i'm trying to set up iSCSI gateways for a Ceph luminous cluster using > these instructions: > http://docs.ceph.com/docs/master/rbd/iscsi-target-cli/ > > When arriving at step "Configuring: Addin

Re: [ceph-users] PGs inconsistent, do I fear data loss?

I'm not a big expert but the OP said he's suspecting bitrot is at least part of issue in which case you can have the situation where the drive has ACK'ed the write but a later scrub discovered checksum errors Plus you don't need to actually loose a drive to get inconsistent pgs with size=2 min_size

Re: [ceph-users] CephFS: clients hanging on write with ceph-fuse

I'm planning to test the newer ceph-fuse tomorrow.  Would it be better to stay with the Jewel 10.2.10 client, or would the 12.2.1 Luminous client be better (even though the back-end is Jewel for now)? Andras On 11/02/2017 05:54 PM, Gregory Farnum wrote: Have you tested on the new ceph-fuse? T

Re: [ceph-users] Bluestore OSD_DATA, WAL & DB

On 3 November 2017 at 07:45, Martin Overgaard Hansen wrote: > I want to bring this subject back in the light and hope someone can provide > insight regarding the issue, thanks. Thanks Martin, I was going to do the same. Is it possible to make the DB partition (on the fastest device) too big? in

Re: [ceph-users] PGs inconsistent, do I fear data loss?

Hi Greg, Accepting the fact, that an osd with outdated data can never accept write, or io of any kind, how is it possible, that the system goes into this state? -All osds are Bluestore, checksum, mtime etc. -All osds are up and in -No hw failures, lost disks, damaged journals or databases e

Re: [ceph-users] PGs inconsistent, do I fear data loss?

On Thu, Nov 2, 2017 at 1:21 AM koukou73gr wrote: > The scenario is actually a bit different, see: > > Let's assume size=2, min_size=1 > -We are looking at pg "A" acting [1, 2] > -osd 1 goes down > -osd 2 accepts a write for pg "A" > -osd 2 goes down > -osd 1 comes back up, while osd 2 still down

Re: [ceph-users] UID Restrictions

Awesome! Thanks much again. Keane On Thu, Nov 2, 2017 at 5:23 PM, Douglas Fuller wrote: > Hi Keane, > > No problem. A fix for the gids bug should go in shortly. See: > https://github.com/ceph/ceph/pull/18689 > > Cheers, > --Doug > > On Thu, Nov 2, 2017 at 4:24 PM Keane Wolter wrote: > >> Here

Re: [ceph-users] CephFS desync

On Thu, Nov 2, 2017 at 9:05 AM Andrey Klimentyev < andrey.kliment...@flant.com> wrote: > Hi, > > we've recently hit a problem in a production cluster. The gist of it is > that sometimes file will be changed on one machine, but only the "change > time" would propagate to others. The checksum is dif

Re: [ceph-users] CephFS: clients hanging on write with ceph-fuse

Have you tested on the new ceph-fuse? This does sound vaguely familiar and is an issue I'd generally expect to have the fix backported for, once it was identified. On Thu, Nov 2, 2017 at 11:40 AM Andras Pataki wrote: > We've been running into a strange problem with Ceph using ceph-fuse and > the

Re: [ceph-users] UID Restrictions

Hi Keane, No problem. A fix for the gids bug should go in shortly. See: https://github.com/ceph/ceph/pull/18689 Cheers, --Doug On Thu, Nov 2, 2017 at 4:24 PM Keane Wolter wrote: > Here we go. removing the trailing slash and adding the gids parameter in > auth caps works. > > [kwolter@um-test03

Re: [ceph-users] FAILED assert(p.same_interval_since) and unusable cluster

I followed the instructions in the Github repo for cloning and setting up the build environment, checked out the 12.2.0 tag, modified OSD.cc with the fix, and then tried to build with dpkg-buildpackage. I got the following error: "ceph/src/kv/RocksDBStore.cc:593:22: error: ‘perf_context’ is not a m

Re: [ceph-users] Bluestore OSD_DATA, WAL & DB

Hi, it seems like I’m in the same boat as everyone else in this particular thread. I’m also unable to find any guidelines or recommendations regarding sizing of the wal and / or db. I want to bring this subject back in the light and hope someone can provide insight regarding the issue, thanks.

Re: [ceph-users] UID Restrictions

Here we go. removing the trailing slash and adding the gids parameter in auth caps works. [kwolter@um-test03 ~]$ sudo ceph auth get-or-create-key client.kwolter_test1 mon 'allow r' mds 'allow r, allow rw path=/user uid=100026 gids=100026' osd 'allow rw pool=cephfs_osiris, allow rw pool=cephfs_user

Re: [ceph-users] UID Restrictions

Looks like there may be a bug here. Please try: * Removing the trailing slash from path= (needs documentation or fixing) * Adding your gid to a “gids” parameter in the auth caps? (bug: we’re checking the gid when none is supplied) mds “allow r, allow rw path=/user uid=100026 gids=100026” Pleas

Re: [ceph-users] UID Restrictions

Hi Doug, Here is the output: [kwolter@um-test03 ~]$ sudo ceph auth get client.kwolter_test1 exported keyring for client.kwolter_test1 [client.kwolter_test1] key = caps mds = "allow r, allow rw path=/user/ uid=100026" caps mon = "allow r" caps osd = "allow rw pool=c

Re: [ceph-users] UID Restrictions

Hi Keane, Could you include the output of ceph auth get client.kwolter_test1 Also, please take a look at your MDS log and see if you see an error from the file access attempt there. Thanks, —Doug > On Nov 2, 2017, at 2:24 PM, Keane Wolter wrote: > > Hi Doug, > > Here is my current mds line

[ceph-users] CephFS: clients hanging on write with ceph-fuse

We've been running into a strange problem with Ceph using ceph-fuse and the filesystem. All the back end nodes are on 10.2.10, the fuse clients are on 10.2.7. After some hours of runs, some processes get stuck waiting for fuse like: [root@worker1144 ~]# cat /proc/58193/stack [] wait_answer_int

Re: [ceph-users] ceph inconsistent pg missing ec object

Okay, after consulting with a colleague this appears to be an instance of http://tracker.ceph.com/issues/21382. Assuming the object is one that doesn't have snapshots, your easiest resolution is to use rados get to retrieve the object (which, unlike recovery, should work) and then "rados put" it ba

Re: [ceph-users] UID Restrictions

Hi Doug, Here is my current mds line I have for my user: caps: [mds] allow r, allow rw path=/user/ uid=100026. My results are as follows when I mount: sudo ceph-fuse --id=kwolter_test1 -k ./ceph.client.kwolter_test1.keyring -r /user/kwolter --client-die-on-failed-remount=false ceph ceph-fuse[34537

Re: [ceph-users] UID Restrictions

Hi Keane, path= has to come before uid= mds “allow r, allow rw path=/user uid=100026, allow rw path=/project" If that doesn’t work, could you send along a transcript of your shell session in setting up the ceph user, mounting the file system, and attempting access? Thanks, —Doug > On Nov 1, 2

Re: [ceph-users] Deleting large pools

Deletion is throttled, though I don’t know the configs to change it you could poke around if you want stuff to go faster. Don’t just remove the directory in the filesystem; you need to clean up the leveldb metadata as well. ;) Removing the pg via Ceph-objectstore-tool would work fine but I’ve seen

[ceph-users] Deleting large pools

Jewel 10.2.7; XFS formatted OSDs; no dmcrypt or LVM. I have a pool that I deleted 16 hours ago that accounted for about 70% of the available space on each OSD (averaging 84% full), 370M objects in 8k PGs, ec 4+2 profile. Based on the rate that the OSDs are freeing up space after deleting the pool,

[ceph-users] CephFS desync

Hi, we've recently hit a problem in a production cluster. The gist of it is that sometimes file will be changed on one machine, but only the "change time" would propagate to others. The checksum is different. Contents, obviously, differ as well. How can I debug this? In other words, how would I a

Re: [ceph-users] iSCSI: tcmu-runner can't open images?

Hello Matthias, We encountered a similar issue, it turned out to be because we used another pool then rbd with gwcli. We got it fixed and it should be in a pull request upstream. /Heðin - Original Message - From: "Matthias Leopold" To: ceph-users@lists.ceph.com Sent: Thursday, 2 Novem

[ceph-users] iSCSI: tcmu-runner can't open images?

Hi, i'm trying to set up iSCSI gateways for a Ceph luminous cluster using these instructions: http://docs.ceph.com/docs/master/rbd/iscsi-target-cli/ When arriving at step "Configuring: Adding a RADOS Block Device (RBD)" things start to get messy: there is no "disks" entry in my target path,

Re: [ceph-users] Ceph versions not showing RGW

On Thu, Nov 2, 2017 at 1:54 PM, Hans van den Bogert wrote: > Just to get this really straight, Jewel OSDs do send this metadata? > Otherwise I'm probably mistaken that I ever saw 10.2.x versions in the > output. RGW daemons only started sending metadata in Luminous. OSD/mon/MDS daemons already s

Re: [ceph-users] Ceph versions not showing RGW

Just to get this really straight, Jewel OSDs do send this metadata? Otherwise I'm probably mistaken that I ever saw 10.2.x versions in the output. Thanks, Hans On 2 Nov 2017 12:31 PM, "John Spray" wrote: > On Thu, Nov 2, 2017 at 11:16 AM, Hans van den Bogert > wrote: > > Hi all, > > > > Durin

[ceph-users] How would ec profile effect performance?

Hi all, I am wondering how ec profile would effect ceph performance? Will ec profile k=10,m=2 perform better than k=8,m=2 since there would be more chunk to wirte and read concurrently? Will ec profile k=10,m=2 perform need more memory and cpu power than ec profile k=8,m=2? 2017-11-02 lin.yu

Re: [ceph-users] Ceph versions not showing RGW

On Thu, Nov 2, 2017 at 11:16 AM, Hans van den Bogert wrote: > Hi all, > > During our upgrade from Jewel to Luminous I saw the following behaviour, if > my memory serves me right: > > When upgrading for example monitors and OSDs, we saw that the `ceph > versions` command correctly showed at one tha

[ceph-users] Ceph versions not showing RGW

Hi all, During our upgrade from Jewel to Luminous I saw the following behaviour, if my memory serves me right: When upgrading for example monitors and OSDs, we saw that the `ceph versions` command correctly showed at one that some OSDs were still on Jewel (10.2.x) and some were already upgraded a

Re: [ceph-users] PGs inconsistent, do I fear data loss?

Never mind, I should’ve read the whole thread first. > On Nov 2, 2017, at 10:50 AM, Hans van den Bogert wrote: > > >> On Nov 1, 2017, at 4:45 PM, David Turner > > wrote: >> >> All it takes for data loss is that an osd on server 1 is marked down and a >> write happ

Re: [ceph-users] PGs inconsistent, do I fear data loss?

> On Nov 1, 2017, at 4:45 PM, David Turner wrote: > > All it takes for data loss is that an osd on server 1 is marked down and a > write happens to an osd on server 2. Now the osd on server 2 goes down > before the osd on server 1 has finished backfilling and the first osd > receives a reque

Re: [ceph-users] PGs inconsistent, do I fear data loss?

The scenario is actually a bit different, see: Let's assume size=2, min_size=1 -We are looking at pg "A" acting [1, 2] -osd 1 goes down -osd 2 accepts a write for pg "A" -osd 2 goes down -osd 1 comes back up, while osd 2 still down -osd 1 has no way to know osd 2 accepted a write in pg "A" -osd 1

[ceph-users] 回复: 回复: Re: [luminous]OSD memory usage increase when writing^J a lot of data to cluster

Hi Sage, I did some more test and found this: I use ceph tell osd.6 heap stats to found that osd.6 tcmalloc heap stats: MALLOC: 404608432 ( 385.9 MiB) Bytes in use by application MALLOC: + 26599424 ( 25.4 MiB) Bytes in page heap freelist