Re: [ceph-users] PGs per OSD guidance

Here are a few thoughts. The more PGs, the higher memory requirement for the osd process. If you are having problems with scrubs causing problems with customer io, check some of the io priority settings that received a big overhaul with Jewel and again with 10.2.9. The more PGs you have, the smalle

Re: [ceph-users] Problems getting nfs-ganesha with cephfs backend to work.

On 07/20/2017 at 12:02 AM, Daniel Gryniewicz wrote: > On 07/19/2017 05:27 AM, Micha Krause wrote: > > Hi, > > > >> Ganesha version 2.5.0.1 from the nfs-ganesha repo hosted on > >> download.ceph.com > > > > I didn't know about that repo, and compiled ganesha myself. The

Re: [ceph-users] 答复: 答复: How's cephfs going?

On 20 July 2017 at 12:23, 许雪寒 wrote: > May I ask how many users do you have on cephfs? And how much data does the > cephfs store? https://www.redhat.com/en/resources/monash-university-improves-research-ceph-storage-case-study As I said, we don't yet have CephFS in production, just finalising ou

[ceph-users] 答复: 答复: How's cephfs going?

Hi, sir, thanks for your sharing. May I ask how many users do you have on cephfs? And how much data does the cephfs store? Thanks:-) -邮件原件- 发件人: Blair Bethwaite [mailto:blair.bethwa...@gmail.com] 发送时间: 2017年7月17日 11:51 收件人: 许雪寒 抄送: ceph-users@lists.ceph.com 主题: Re: 答复: [ceph-users] How

Re: [ceph-users] PGs per OSD guidance

Anyone able to offer any advice on this? Cheers, Adrian > -Original Message- > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of > Adrian Saul > Sent: Friday, 14 July 2017 6:05 PM > To: 'ceph-users@lists.ceph.com' > Subject: [ceph-users] PGs per OSD guidance > >

Re: [ceph-users] pgs not deep-scrubbed for 86400

This code shows how that all works (part of some new health reporting code). https://github.com/ceph/ceph/blob/master/src/mon/PGMap.cc#L3188-L3203 So the last_deep_scrub_stamp of the pg is compared to deep_cutoff which is the time now minus mon_warn_not_deep_scrubbed + osd_deep_scrub_interval. B

Re: [ceph-users] 答复: How's cephfs going?

Hi, While not necessarily CephFS specific - we somehow seem to manage to frequently end up with objects that have inconsistent omaps. This seems to be replication (as anecdotally it's a replica that ends up diverging, and it's at least a few times something that happened after the osd that held th

Re: [ceph-users] pgs not deep-scrubbed for 86400

Exactly have this issue (or not?) at the moment. Mine says "906 pgs not scrubbed for 86400". But it is decrementing slowly (very slowly). I cannot find any documentation for exact "pgs not srubbed for" phrase on the web but only this. Log like this: 2017-07-19 15:05:10.125041 [INF] 3.5

Re: [ceph-users] 答复: How's cephfs going?

Thanks Greg. I thought it was impossible when I reported 34MB for 52 million files. On Jul 19, 2017 1:17 PM, "Gregory Farnum" wrote: > > > On Wed, Jul 19, 2017 at 10:25 AM David wrote: > >> On Tue, Jul 18, 2017 at 6:54 AM, Blair Bethwaite < >> blair.bethwa...@gmail.com> wrote: >> >>> We are a d

Re: [ceph-users] 答复: How's cephfs going?

On Wed, Jul 19, 2017 at 10:25 AM David wrote: > On Tue, Jul 18, 2017 at 6:54 AM, Blair Bethwaite < > blair.bethwa...@gmail.com> wrote: > >> We are a data-intensive university, with an increasingly large fleet >> of scientific instruments capturing various types of data (mostly >> imaging of one k

Re: [ceph-users] 答复: How's cephfs going?

On Tue, Jul 18, 2017 at 6:54 AM, Blair Bethwaite wrote: > We are a data-intensive university, with an increasingly large fleet > of scientific instruments capturing various types of data (mostly > imaging of one kind or another). That data typically needs to be > stored, protected, managed, share

Re: [ceph-users] 答复: How's cephfs going?

On Wed, Jul 19, 2017 at 4:47 AM, 许雪寒 wrote: > Is there anyone else willing to share some usage information of cephfs? > I look after 2 Cephfs deployments, both Jewel, been in production since Jewel went stable so just over a year I think. We've had a really positive experience, I've not experien

Re: [ceph-users] best practices for expanding hammer cluster

The main seeing you can control is osd_max_backfills. It's default is 1. I watch iostat on my osds are I slowly increment that seeing to leave enough overhead on the disks for client activity while the cluster moves all of its data around. On Wed, Jul 19, 2017, 11:45 AM Richard Hesketh wrote: >

Re: [ceph-users] iSCSI production ready?

On Sat, Jul 15, 2017 at 11:02 PM Alvaro Soto wrote: > Hi guys, > does anyone know any news about in what release iSCSI interface is going > to be production ready, if not yet? > > I mean without the use of a gateway, like a different endpoint connector > to a CEPH cluster. > We very successfully

[ceph-users] pgs not deep-scrubbed for 86400

I just upgraded from Luminous 12.1.0 to 12.1.1 and was greeted with this new "pgs not deep-scrubbed for" warning. Should this resolve itself, or should I get scrubbing? $ ceph health detail HEALTH_WARN 4 pgs not deep-scrubbed for 86400; 15 pgs not scrubbed for 86400 PG_NOT_DEEP_SCRUBBED 4 pgs not

[ceph-users] Ceph kraken: Calamari Centos7

Hi, Anybody has been able to setup Calamari on Centos7?? I've done a lot of Google but I haven't found any good documentation... The command "ceph-deploy calamari connect" does not work! Thanks a lot for your help! ___ ceph-users mailing list ceph-us

Re: [ceph-users] Problems getting nfs-ganesha with cephfs backend to work.

On 07/19/2017 05:27 AM, Micha Krause wrote: Hi, Ganesha version 2.5.0.1 from the nfs-ganesha repo hosted on download.ceph.com I didn't know about that repo, and compiled ganesha myself. The developers in the #ganesha IRC channel pointed me to the libcephfs version

Re: [ceph-users] How's cephfs going?

Hello, Can anyone share their experience with the  bulit-in FSCache support with or without CephFS? Interested in knowing the following:- Are you using FSCache in production environment?- How large is your Ceph deployment?- If with CephFS, how many Ceph clients are using FSCache- which version o

Re: [ceph-users] best practices for expanding hammer cluster

In my case my cluster is under very little active load and so I have never had to be concerned about recovery operations impacting on client traffic. In fact, I generally tune up from the defaults (increase osx max backfills) to improve recovery speed when I'm doing major changes, because there'

Re: [ceph-users] Adding multiple osd's to an active cluster

On Fri, Feb 17, 2017 at 10:42 AM, nigel davies wrote: > How is the best way to added multiple osd's to an active cluster? > As the last time i done this i all most killed the VM's we had running on > the cluster You possibly mean that messing with OSDs caused the cluster to reorganise the date a

[ceph-users] To flatten or not to flatten?

I'm running OpenStack and using Ceph as a backend. As all the tutorials advocate I use the show_image_direct_url option. This creates the new volume with a pointer to original image. As we defined everything with availability zones, we have one pool that's HA in all of the zones for images - if a

Re: [ceph-users] Writing data to pools other than filesystem

Hi, you must add the extra pools to your current file system configuration: ceph fs add_data_pool {fs_name} {pool_name} Once this is done, you just have to create some specific directory layout within CephFS to modify the name of the pool targetted by a specific directory. See http://docs.ceph

Re: [ceph-users] best practices for expanding hammer cluster

Hi Rich, Thank you for your answer. This is good news to hear :) Regarding the reconfiguration you've done: if I understand correctly, you have changed it all at once (like download the crush map, edit it - add all the new OSDs, and upload the new map to the cluster). How did you controlled the

[ceph-users] Writing data to pools other than filesystem

Hello! I want to organize data in pools and therefore created additional pools: ceph osd lspools 0 rbd,1 templates,2 hdb-backup,3 cephfs_data,4 cephfs_metadata, As you can see, pools "cephfs_data" and "cephfs_metadata" belong to a Ceph filesystem. Question: How can I write data to other pools,

Re: [ceph-users] best practices for expanding hammer cluster

On 19/07/17 15:14, Laszlo Budai wrote: > Hi David, > > Thank you for that reference about CRUSH. It's a nice one. > There I could read about expanding the cluster, but in one of my cases we > want to do more: we want to move from host failure domain to chassis failure > domain. Our concern is: h

Re: [ceph-users] undersized pgs after removing smaller OSDs

David, So as I look at logs, it was originally 9.0956 for the 10TB drives and 0.9096 for the 1TB drives. # zgrep -i weight /var/log/ceph/*.log*gz /var/log/ceph/ceph.audit.log.4.gz:...cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":9.0956,... /var/log/ceph/ceph.audit.log.4.gz:...cmd=

Re: [ceph-users] best practices for expanding hammer cluster

Hi David, Thank you for that reference about CRUSH. It's a nice one. There I could read about expanding the cluster, but in one of my cases we want to do more: we want to move from host failure domain to chassis failure domain. Our concern is: how will ceph behave for those PGs where all the th

Re: [ceph-users] undersized pgs after removing smaller OSDs

I would go with the weight that was originally assigned to them. That way it is in line with what new osds will be weighted. On Wed, Jul 19, 2017, 9:17 AM Roger Brown wrote: > David, > > Thank you. I have it currently as... > > $ ceph osd df > ID WEIGHT REWEIGHT SIZE USEAVAIL %USE VAR

Re: [ceph-users] undersized pgs after removing smaller OSDs

David, Thank you. I have it currently as... $ ceph osd df ID WEIGHT REWEIGHT SIZE USEAVAIL %USE VAR PGS 3 10.0 1.0 9313G 44404M 9270G 0.47 1.00 372 4 10.0 1.0 9313G 46933M 9268G 0.49 1.06 372 0 10.0 1.0 9313G 41283M 9273G 0.43 0.93 372

Re: [ceph-users] iSCSI production ready?

On 07/17/2017 10:15 PM, Alvaro Soto wrote: > The second part, nevermind know I see that the solution is to use > the TCMU daemon, I was thinking in a out of the box iSCSI endpoint > directly from CEPH, sorry don't have to much expertise in this area. There is no "native" iSCSI support built i

Re: [ceph-users] How's cephfs going?

I had a corruption issue with the FUSE client on Jewel. I use CephFS for a samba share with a light load, and I was using the FUSE client. I had a power flap and didn't realize my UPS batteries had went bad so the MDS servers were cycled a couple times and some how the file system had become corrup

Re: [ceph-users] Luminous RC OSD Crashing

Logged a bug ticket, let me know if need anything further : http://tracker.ceph.com/issues/20687 From: Ashley Merrick Sent: Wednesday, 19 July 2017 8:05 PM To: ceph-us...@ceph.com Subject: RE: Luminous RC OSD Crashing Also found this error on some of the OSD's crashing: 2017-07-19 12:50:57.5871

Re: [ceph-users] Updating 12.1.0 -> 12.1.1

Thanks! updating all indeed resolved this. -Original Message- From: Gregory Farnum [mailto:gfar...@redhat.com] Sent: dinsdag 18 juli 2017 23:01 To: Marc Roos; ceph-users Subject: Re: [ceph-users] Updating 12.1.0 -> 12.1.1 Yeah, some of the message formats changed (incompatibly) duri

Re: [ceph-users] Luminous RC OSD Crashing

Also found this error on some of the OSD's crashing: 2017-07-19 12:50:57.587194 7f19348f1700 -1 /build/ceph-12.1.1/src/osd/PrimaryLogPG.cc: In function 'virtual void C_CopyFrom_AsyncReadCb::finish(int)' thread 7f19348f1700 time 2017-07-19 12:50:57.583192 /build/ceph-12.1.1/src/osd/PrimaryLogPG.

Re: [ceph-users] ipv6 monclient

> Op 19 juli 2017 om 10:36 schreef Dan van der Ster : > > > Hi Wido, > > Quick question about IPv6 clusters which you may have already noticed. > We have an IPv6 cluster and clients use this as the ceph.conf: > > [global] > mon host = cephv6.cern.ch > > cephv6 is an alias to our three mons,

Re: [ceph-users] Luminous RC OSD Crashing

I have just found : http://tracker.ceph.com/issues/20167 Looks to be the same error in an earlier release : 12.0.2-1883-gb3f5819, is marked as resolved one month ago by Sage, however unable to see how and by what. However would guess this fix would have made it to latest RC? ,Ashley From: ceph

Re: [ceph-users] best practices for expanding hammer cluster

One of the things you need to be aware of when doing this is that the crush map is, more or less, stupid in knowing your network setup. You can configure your crush map with racks, datacenters, etc, but it has no idea where anything is. You have to tell it. You can use placement rules to help when

Re: [ceph-users] How's cephfs going?

Unfortunately no. Using FUSE was discarded due to poor performance. > 19 июля 2017 г., в 13:45, Blair Bethwaite > написал(а): > > Interesting. Any FUSE client data-points? > > On 19 July 2017 at 20:21, Дмитрий Глушенок wrote: >> RBD (via krbd) was in action at the same time - no problems. >>

Re: [ceph-users] How's cephfs going?

Interesting. Any FUSE client data-points? On 19 July 2017 at 20:21, Дмитрий Глушенок wrote: > RBD (via krbd) was in action at the same time - no problems. > > 19 июля 2017 г., в 12:54, Blair Bethwaite > написал(а): > > It would be worthwhile repeating the first test (crashing/killing an > OSD ho

Re: [ceph-users] How's cephfs going?

RBD (via krbd) was in action at the same time - no problems. > 19 июля 2017 г., в 12:54, Blair Bethwaite > написал(а): > > It would be worthwhile repeating the first test (crashing/killing an > OSD host) again with just plain rados clients (e.g. rados bench) > and/or rbd. It's not clear whether

[ceph-users] 答复: How's cephfs going?

I got it, thank you☺ 发件人: Дмитрий Глушенок [mailto:gl...@jet.msk.su] 发送时间: 2017年7月19日 18:20 收件人: 许雪寒 抄送: ceph-users@lists.ceph.com 主题: Re: [ceph-users] How's cephfs going? You right. Forgot to mention that the client was using kernel 4.9.9. 19 июля 2017 г., в 12:36, 许雪寒 написал(а): Hi, thanks

Re: [ceph-users] How's cephfs going?

You right. Forgot to mention that the client was using kernel 4.9.9. > 19 июля 2017 г., в 12:36, 许雪寒 написал(а): > > Hi, thanks for your sharing:-) > > So I guess you have not put cephfs into real production environment, and it's > still in test phase, right? > > Thanks again:-) > > 发件人: Дми

Re: [ceph-users] Luminous RC OSD Crashing

Hello, Seems recovering is fine, only happens when I do ceph osd unset nobackfill, rapidly random OSD's start to fail (I am guessing backfill sources but unable to catch due to speed) The backfilling OSD is a recently re-created OSD using Bluestore. ,Ashley From: ceph-users [mailto:ceph-users

[ceph-users] upgrade ceph from 10.2.7 to 10.2.9

hi *, we are facing some issue with the upgrade of our OSD the updateprogess on ubuntu 16.04 stops at: Setting system user ceph properties..usermod: no changes ..done Fixing /var/run/ceph ownershipdone no more output is given to the system, my permission are ok so how to go ahead

Re: [ceph-users] How's cephfs going?

It would be worthwhile repeating the first test (crashing/killing an OSD host) again with just plain rados clients (e.g. rados bench) and/or rbd. It's not clear whether your issue is specifically related to CephFS or actually something else. Cheers, On 19 July 2017 at 19:32, Дмитрий Глушенок wro

[ceph-users] Luminous RC OSD Crashing

Hello, Getting the following on random OSD's crashing during a backfill/rebuilding on the latest RC, from the log's so far I have seen the following: 172.16.3.10:6802/21760 --> 172.16.3.6:6808/15997 -- pg_update_log_missing(6.19ds12 epoch 101931/101928 rep_tid 59 entries 101931'55683 (0'0) err

[ceph-users] 答复: How's cephfs going?

Hi, thanks for your sharing:-) So I guess you have not put cephfs into real production environment, and it's still in test phase, right? Thanks again:-) 发件人: Дмитрий Глушенок [mailto:gl...@jet.msk.su] 发送时间: 2017年7月19日 17:33 收件人: 许雪寒 抄送: ceph-users@lists.ceph.com 主题: Re: [ceph-users] How's ceph

Re: [ceph-users] How's cephfs going?

Hi, I can share negative test results (on Jewel 10.2.6). All tests were performed while actively writing to CephFS from single client (about 1300 MB/sec). Cluster consists of 8 nodes, 8 OSD each (2 SSD for journals and metadata, 6 HDD RAID6 for data), MON/MDS are on dedicated nodes. 2 MDS at al

Re: [ceph-users] Problems getting nfs-ganesha with cephfs backend to work.

Hi, Ganesha version 2.5.0.1 from the nfs-ganesha repo hosted on download.ceph.com I didn't know about that repo, and compiled ganesha myself. The developers in the #ganesha IRC channel pointed me to the libcephfs version. After recompiling ganesha with a kraken lib

[ceph-users] ipv6 monclient

Hi Wido, Quick question about IPv6 clusters which you may have already noticed. We have an IPv6 cluster and clients use this as the ceph.conf: [global] mon host = cephv6.cern.ch cephv6 is an alias to our three mons, which are listening on their v6 addrs (ms bind ipv6 = true). But those mon hos