Re: [ceph-users] Ceph mgr Prometheus plugin: error when osd is down

2018-11-14 Thread John Spray
On Wed, Nov 14, 2018 at 3:32 PM Gökhan Kocak wrote: > > Hello everyone, > > we encountered an error with the Prometheus plugin for Ceph mgr: > One osd was down and (therefore) it had no class: > ``` > sudo ceph osd tree > ID CLASS WEIGHTTYPE NAME STATUS REWEIGHT PRI-AFF > 28 hdd

Re: [ceph-users] Cephfs / mds: how to determine activity per client?

2018-11-05 Thread John Spray
On Mon, Nov 5, 2018 at 9:45 AM Erwin Bogaard wrote: > > Hi, > > > > Is there any way to determine the activity per cephfs client? > > For example, is there a way to get the requests/sec, bytes/sec, > connections/sec, or any other relevant performance parameters per client? Currently, you're limi

Re: [ceph-users] Should OSD write error result in damaged filesystem?

2018-11-05 Thread John Spray
On Sun, Nov 4, 2018 at 10:24 PM Bryan Henderson wrote: > > >OSD write errors are not usual events: any issues with the underlying > >storage are expected to be handled by RADOS, and write operations to > >an unhealthy cluster should block, rather than returning an error. It > >would not be correc

Re: [ceph-users] Snapshot cephfs data pool from ceph cmd

2018-11-04 Thread John Spray
On Sat, Nov 3, 2018 at 3:43 PM Rhian Resnick wrote: > > is it possible to snapshot the cephfs data pool? CephFS snapshots operate on a per-directory level (rather than per pool), but you can make snapshots of the root of the filesystem if you wish. John > ___

Re: [ceph-users] Should OSD write error result in damaged filesystem?

2018-11-04 Thread John Spray
On Sat, Nov 3, 2018 at 7:28 PM Bryan Henderson wrote: > > I had a filesystem rank get damaged when the MDS had an error writing the log > to the OSD. Is damage expected when a log write fails? > > According to log messages, an OSD write failed because the MDS attempted > to write a bigger chunk t

Re: [ceph-users] Reducing Max_mds

2018-10-30 Thread John Spray
On Tue, Oct 30, 2018 at 6:36 AM Rhian Resnick wrote: > > Evening, > > > I am looking to decrease our max mds servers as we had a server failure and > need to remove a node. > > > When we attempt to decrease the number of mds servers from 5 to 4 (or any > other number) they never transition to st

Re: [ceph-users] [Ceph Days 2017] Short movie from 3D presentation (ceph + blender + python)

2018-10-25 Thread John Spray
On Thu, Oct 25, 2018 at 9:31 AM igor.podo...@ts.fujitsu.com wrote: > > Hello Cephers! > > For those of you who like puzzles, here > https://www.youtube.com/watch?v=vWbK4L77aEE is the movie from our 3D > presentation that we made especially for Ceph Days Warsaw 2017. > > The movie contains 3 east

Re: [ceph-users] ceph-mgr hangs on larger clusters in Luminous

2018-10-18 Thread John Spray
On Thu, Oct 18, 2018 at 10:31 PM Bryan Stillwell wrote: > > I could see something related to that bug might be happening, but we're not > seeing the "clock skew" or "signal: Hangup" messages in our logs. > > > > One reason that this cluster might be running into this problem is that we > appear

Re: [ceph-users] ceph-mgr hangs on larger clusters in Luminous

2018-10-18 Thread John Spray
On Thu, Oct 18, 2018 at 6:17 PM Bryan Stillwell wrote: > > After we upgraded from Jewel (10.2.10) to Luminous (12.2.5) we started seeing > a problem where the new ceph-mgr would sometimes hang indefinitely when doing > commands like 'ceph pg dump' on our largest cluster (~1,300 OSDs). The rest

Re: [ceph-users] Mimic and Debian 9

2018-10-17 Thread John Spray
On Wed, Oct 17, 2018 at 3:23 PM Paul Emmerich wrote: > > John Spray : > > > > Apologies if this has been asked before, but how are you building > > these, some third party compiler backport? If there isn't anything > > too hacky involved in the build perhaps yo

[ceph-users] Ceph BoF at Open Source Summit Europe

2018-10-17 Thread John Spray
Hi everyone, OSS Europe is happening in Edinburgh next week, and on Monday there'll be a Ceph BoF session[1] at 18:00. This is an informal session for Ceph users and developers to meet up and talk about whatever's on our collective mind. Hope to see you there! Thanks, John 1. https://osseu18.s

Re: [ceph-users] Mimic and Debian 9

2018-10-17 Thread John Spray
On Wed, Oct 17, 2018 at 2:37 PM Paul Emmerich wrote: > > I wouldn't hold my breath for an official Stretch release. > > We are running several Mimic clusters in production with Debian, our > Stretch packages are available at our mirror: > > https://croit.io/2018/09/23/2018-09-23-debian-mirror Apo

Re: [ceph-users] Ceph mds is stuck in creating status

2018-10-16 Thread John Spray
public addresses in the 192.168.10.x range rather than the 192.168.40.x range like the others. So my guess would be that you are intending your OSDs to be in the 192.168.40.x range, but are missing some config settings for certain daemons. John > 2018년 10월 16일 (화) 오전 1:18, John Spray 님이 작성:

Re: [ceph-users] Ceph mds is stuck in creating status

2018-10-15 Thread John Spray
something went wrong. Check out the "ceph osd dump --format=json-pretty" and "ceph fs dump --format=json-pretty" outputs and how the pool ID's relate. According to those logs, data pool with ID 1 and metadata pool with ID 2 do not exist. John > -Kisik >

Re: [ceph-users] Ceph mds is stuck in creating status

2018-10-15 Thread John Spray
On Mon, Oct 15, 2018 at 3:34 PM Kisik Jeong wrote: > > Hello, > > I successfully deployed Ceph cluster with 16 OSDs and created CephFS before. > But after rebooting due to mds slow request problem, when creating CephFS, > Ceph mds goes creating status and never changes. > Seeing Ceph status, ther

Re: [ceph-users] ceph dashboard ac-* commands not working (Mimic)

2018-10-15 Thread John Spray
user management command that works is to create a user? In other > words, no way to check the user list, delete users, or control their access > levels at this point? Correct: the code in Mimic just has a single user account. John > Mami > > On Sun, Oct 14, 2018 at 10:38 AM, J

Re: [ceph-users] ceph dashboard ac-* commands not working (Mimic)

2018-10-14 Thread John Spray
The docs you're looking at are from the master (development) version of ceph, so you're seeing commands that don't exist in mimic. You can swap master for mimic in that URL. Hopefully we'll soon have some changes to make this more apparent when looking at the docs. John On Fri, 12 Oct 2018, 17:

Re: [ceph-users] https://ceph-storage.slack.com

2018-10-11 Thread John Spray
On Thu, Oct 11, 2018 at 8:44 AM Marc Roos wrote: > > > Why slack anyway? Just because some people like using it. Don't worry, IRC is still the primary channel and lots of people don't use slack. I'm not on slack, for example, which is either a good or bad thing depending on your perspective :-D

Re: [ceph-users] cephfs set quota without mount

2018-10-11 Thread John Spray
On Thu, Oct 11, 2018 at 9:55 AM Felix Stolte wrote: > > Hey folks, > > I use nfs-ganesha to export cephfs to nfs. nfs-ganesha can talk to > cephfs via libcephfs so there is no need for mounting cephfs manually. I > also like to use directory quotas from cephfs. Anyone knows a way to set > quota on

[ceph-users] Does anyone use interactive CLI mode?

2018-10-10 Thread John Spray
Hi all, Since time immemorial, the Ceph CLI has had a mode where when run with no arguments, you just get an interactive prompt that lets you run commands without "ceph" at the start. I recently discovered that we actually broke this in Mimic[1], and it seems that nobody noticed! So the question

Re: [ceph-users] vfs_ceph ignoring quotas

2018-10-09 Thread John Spray
le to write a little df command. The quota is stored as an > extended attribute, where can i get its utilization? The utilization (recursive statistics) are also xattrs. Specifically, you probably want "ceph.dir.rbytes" John > > Regards Felix > > > On 10/09/2018 10:59

Re: [ceph-users] vfs_ceph ignoring quotas

2018-10-09 Thread John Spray
On Tue, Oct 9, 2018 at 9:14 AM Felix Stolte wrote: > > Hi folks, > > i'm running a luminous cluster on Ubuntu 18.04 an want to share folders > on cephfs with samba using the vfs_ceph. Sharing works fine, but the > quotas I set on the directories is ignored and every share reports its > size as the

Re: [ceph-users] mds_cache_memory_limit value

2018-10-08 Thread John Spray
On Fri, Oct 5, 2018 at 9:33 AM Hervé Ballans wrote: > > Hi all, > > I have just configured a new value for 'mds_cache_memory_limit'. The output > message tells "not observed, change may require restart". > So I'm not really sure, has the new value been taken into account directly or > do I have

Re: [ceph-users] daahboard

2018-10-08 Thread John Spray
Assuming that ansible is correctly running "ceph mgr module enable dashboard", then the next place to look is in "ceph status" (any errors?) and "ceph mgr module ls" (any reports of the module unable to run?) John On Sat, Oct 6, 2018 at 1:53 AM solarflow99 wrote: > > I enabled the dashboard modul

Re: [ceph-users] Bluestore vs. Filestore

2018-10-03 Thread John Spray
On Tue, Oct 2, 2018 at 6:28 PM wrote: > > Hi. > > Based on some recommendations we have setup our CephFS installation using > bluestore*. We're trying to get a strong replacement for "huge" xfs+NFS > server - 100TB-ish size. > > Current setup is - a sizeable Linux host with 512GB of memory - one l

Re: [ceph-users] cephfs issue with moving files between data pools gives Input/output error

2018-10-01 Thread John Spray
10 client.3246747 > > handle_client_session client_session(renewcaps seq 9) v1 from > mds.0 > > 2018-09-28 16:03:41.284915 7f1ac3fff700 10 client.3246750 > renew_caps() > > 2018-09-28 16:03:41.284923 7f1ac3fff700 10 client.3246750 > renew_caps &g

Re: [ceph-users] cephfs issue with moving files between data pools gives Input/output error

2018-10-01 Thread John Spray
ad(faked_ino=0 ref=3 ll_ref=1 cap_refs={} open={1=0} > >> > mode=100644 size=17/0 nlink=1 mtime=2018-09-28 14:45:50.323273 > >> > caps=pAsLsXsFscr(0=pAsLsXsFscr) objectset[0x10001698ac5 ts 0/0 objects > >> > >> > 0 dirty_or_tx 0] parents=0x7f1a780f1

Re: [ceph-users] cephfs issue with moving files between data pools gives Input/output error

2018-09-28 Thread John Spray
On Fri, Sep 28, 2018 at 2:28 PM Marc Roos wrote: > > > Looks like that if I move files between different data pools of the > cephfs, something is still refering to the 'old location' and gives an > Input/output error. I assume this, because I am using different client > ids for authentication. > >

Re: [ceph-users] rados rm objects, still appear in rados ls

2018-09-28 Thread John Spray
On Fri, Sep 28, 2018 at 2:25 PM Frank (lists) wrote: > > Hi, > > On my cluster I tried to clear all objects from a pool. I used the > command "rados -p bench ls | xargs rados -p bench rm". (rados -p bench > cleanup doesn't clean everything, because there was a lot of other > testing going on here)

Re: [ceph-users] cephfs-data-scan tool

2018-09-27 Thread John Spray
ata pool approach. I double checked that MDS > > servers are down and both original and recovery fs are set not joinable. > > > > > >> On 27.09.2018, at 13:10, John Spray wrote: > >> > >> On Thu, Sep 27, 2018 at 11:03 AM Sergey Malinin wrote: &

Re: [ceph-users] cephfs-data-scan tool

2018-09-27 Thread John Spray
On Thu, Sep 27, 2018 at 11:03 AM Sergey Malinin wrote: > > Hello, > Does anybody have experience with using cephfs-data-scan tool? > Questions I have are how long would it take to scan extents on filesystem > with 120M relatively small files? While running extents scan I noticed that > number of

Re: [ceph-users] Cannot write to cephfs if some osd's are not available on the client network

2018-09-27 Thread John Spray
On Thu, Sep 27, 2018 at 10:16 AM Marc Roos wrote: > > > I have a test cluster and on a osd node I put a vm. The vm is using a > macvtap on the client network interface of the osd node. Making access > to local osd's impossible. > > the vm of course reports that it cannot access the local osd's. Wh

Re: [ceph-users] [RGWRados]librados: Objecter returned from getxattrs r=-36

2018-09-20 Thread John Spray
On Thu, Sep 20, 2018 at 9:42 AM fatkun chan wrote: > > OSD : filestore FileSystem: ext4 system: centos7.2 > > I use https://rook.io/ deploy ceph on kubernetes > > > ceph config > -- > mon keyvaluedb = rocksdb > mon_allow_pool_delete= true > mon_max_

Re: [ceph-users] Ceph MDS WRN replayed op client.$id

2018-09-19 Thread John Spray
x27;t knowingly done anything about it, it would be a (pleasant) surprise if it was accidentally resolved in mimic ;-) John > Regards, > Eugen > > > Zitat von John Spray : > > > On Mon, Sep 17, 2018 at 2:49 PM Eugen Block wrote: > >> > >> Hi, > >>

Re: [ceph-users] Ceph MDS WRN replayed op client.$id

2018-09-17 Thread John Spray
n't really > now when to increase the debug log level. > > Any hint would be highly appreciated! > > Regards, > Eugen > > > Zitat von John Spray : > > > On Thu, Sep 13, 2018 at 11:01 AM Stefan Kooman wrote: > >> > >> Hi John, > >&g

Re: [ceph-users] CephFS "authorize" on erasure-coded FS

2018-09-17 Thread John Spray
On Fri, Sep 14, 2018 at 11:31 PM Виталий Филиппов wrote: > > Hi, > > I've recently tried to setup a user for CephFS running on a pair of > replicated+erasure pools, but after I ran > > ceph fs authorize ecfs client.samba / rw > > The "client.samba" user could only see listings, but couldn't read o

Re: [ceph-users] can we drop support of centos/rhel 7.4?

2018-09-14 Thread John Spray
#x27;all not test ceph on 7.3 right now? This email thread really might be > enough to get us off of CentOS for Ceph. There is a set of permutations in qa/distros, used in qa/suites/buildpackages/ -- I'm not sure exactly what's run when though (possibly some only at release time?), perhap

Re: [ceph-users] Standby mgr stopped sending beacons after upgrade to 12.2.8

2018-09-14 Thread John Spray
Thanks, it's clear from that backtrace what's going on. Opened http://tracker.ceph.com/issues/35985 John On Fri, Sep 14, 2018 at 11:33 AM Christian Albrecht wrote: > > 14. September 2018 11:31, "John Spray" schrieb: > > > On Thu, Sep 13, 2018 at 7:55 PM Chr

Re: [ceph-users] Standby mgr stopped sending beacons after upgrade to 12.2.8

2018-09-14 Thread John Spray
On Thu, Sep 13, 2018 at 7:55 PM Christian Albrecht wrote: > > Hi all, > > after upgrading from 12.2.7 to 12.2.8 the standby mgr instances in my cluster > stopped sending beacons. > The service starts and everything seems to work just fine, but after a period > of time the mgr disappears. > All o

Re: [ceph-users] cephfs is growing up rapidly

2018-09-14 Thread John Spray
On Fri, Sep 14, 2018 at 7:25 AM Zhenshi Zhou wrote: > > Hi, > > I have a ceph cluster of version 12.2.5 on centos7. > > I created 3 pools, 'rbd' for rbd storage, as well as 'cephfs_data' > and 'cephfs_meta' for cephfs. Cephfs is used for backing up by > rsync and volumes mounting by docker. > > Th

Re: [ceph-users] can we drop support of centos/rhel 7.4?

2018-09-14 Thread John Spray
On Fri, Sep 14, 2018 at 3:48 AM kefu chai wrote: > > hi ceph-{maintainers,users,developers}, > > recently, i ran into an issue[0] which popped up when we build Ceph on > centos 7.5, but test it on centos 7.4. as we know, the gperftools-libs > package provides the tcmalloc allocator shared library,

Re: [ceph-users] Ceph MDS WRN replayed op client.$id

2018-09-13 Thread John Spray
On Thu, Sep 13, 2018 at 11:01 AM Stefan Kooman wrote: > > Hi John, > > Quoting John Spray (jsp...@redhat.com): > > > On Wed, Sep 12, 2018 at 2:59 PM Stefan Kooman wrote: > > > > When replaying a journal (either on MDS startup or on a standby-replay > > MDS)

Re: [ceph-users] Not all pools are equal, but why

2018-09-13 Thread John Spray
On Thu, Sep 13, 2018 at 9:03 AM Stefan Kooman wrote: > > Hi List, > > TL;DR: what application types are compatible with each other concerning > Ceph Pools? > > I.e. is it safe to mix "RBD" pool with (some) native librados objects? > > RBD / RGW / Cephfs all have their own pools. Since luminous rel

Re: [ceph-users] Ceph MDS WRN replayed op client.$id

2018-09-13 Thread John Spray
On Wed, Sep 12, 2018 at 2:59 PM Stefan Kooman wrote: > > Hi, > > Once in a while, today a bit more often, the MDS is logging the > following: > > mds.mds1 [WRN] replayed op client.15327973:15585315,15585103 used ino > 0x19918de but session next is 0x1873b8b > > Nothing of importance is lo

Re: [ceph-users] Need help

2018-09-10 Thread John Spray
release (28). John On Mon, Sep 10, 2018 at 3:17 PM marc-antoine desrochers wrote: > > What Is the advantages of using ceph-fuse ? and if I stay on kernel client > what kind of distro/kernel are you suggesting ? > > -Message d'origine- > De : John Spray [mailto:jsp...

Re: [ceph-users] Need help

2018-09-10 Thread John Spray
On Mon, Sep 10, 2018 at 1:40 PM marc-antoine desrochers wrote: > > Hi, > > > > I am currently running a ceph cluster running in CEPHFS with 3 nodes each > have 6 osd’s except 1 who got 5. I got 3 mds : 2 active and 1 standby, 3 mon. > > > > > > [root@ceph-n1 ~]# ceph -s > > cluster: > > id:

Re: [ceph-users] Upgrading ceph with HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent

2018-09-05 Thread John Spray
On Wed, Sep 5, 2018 at 8:38 AM Marc Roos wrote: > > > The adviced solution is to upgrade ceph only in HEALTH_OK state. And I > also read somewhere that is bad to have your cluster for a long time in > an HEALTH_ERR state. > > But why is this bad? Aside from the obvious (errors are bad things!), m

Re: [ceph-users] MDS does not always failover to hot standby on reboot

2018-09-04 Thread John Spray
It's mds_beacon_grace. Set that on the monitor to control the replacement of laggy MDS daemons, and usually also set it to the same value on the MDS daemon as it's used there for the daemon to hold off on certain tasks if it hasn't seen a mon beacon recently. John On Mon, Sep 3, 2018 at 9:26 AM W

Re: [ceph-users] MDS not start. Timeout??

2018-08-31 Thread John Spray
On Fri, Aug 31, 2018 at 6:11 AM morf...@gmail.com wrote: > > Hello all! > > I had a electric power problem. After this I have 2 incomplete pg. But all > RBD volumes are work. > > But not work my CephFS. MDS load stop at "replay" state and MDS related > commands hangs: > > cephfs-journal-tool jou

Re: [ceph-users] Mimic prometheus plugin -no socket could be created

2018-08-23 Thread John Spray
On Thu, Aug 23, 2018 at 5:18 PM Steven Vacaroaia wrote: > > Hi All, > > I am trying to enable prometheus plugin with no success due to "no socket > could be created" > > The instructions for enabling the plugin are very straightforward and simple > > Note > My ultimate goal is to use Prometheus w

Re: [ceph-users] Intermittent client reconnect delay following node fail

2018-08-23 Thread John Spray
s > > William Lawton > > -Original Message----- > From: William Lawton > Sent: Wednesday, August 01, 2018 2:05 PM > To: 'John Spray' > Cc: ceph-users@lists.ceph.com; Mark Standley > Subject: RE: [ceph-users] Intermittent client reconnect delay following no

Re: [ceph-users] HDD-only CephFS cluster with EC and without SSD/NVMe

2018-08-22 Thread John Spray
On Wed, Aug 22, 2018 at 1:28 PM Kevin Olbrich wrote: > > Hi! > > I am in the progress of moving a local ("large", 24x1TB) ZFS RAIDZ2 to CephFS. > This storage is used for backup images (large sequential reads and writes). > > To save space and have a RAIDZ2 (RAID6) like setup, I am planning the >

Re: [ceph-users] HEALTH_ERR vs HEALTH_WARN

2018-08-22 Thread John Spray
On Wed, Aug 22, 2018 at 7:57 AM mj wrote: > > Hi, > > This morning I woke up, seeing my ceph jewel 10.2.10 cluster in > HEALTH_ERR state. That helps you getting out of bed. :-) > > Anyway, much to my surprise, all VMs running on the cluster were still > working like nothing was going on. :-) > >

Re: [ceph-users] Removing all rados objects based on a prefix

2018-08-21 Thread John Spray
On Mon, Aug 20, 2018 at 5:40 PM Wido den Hollander wrote: > > > > On 08/20/2018 05:20 PM, David Turner wrote: > > The general talk about the rados cleanup command is to clean things up > > after benchmarking. Could this command also be used for deleting an old > > RGW bucket or an RBD. For insta

Re: [ceph-users] missing dependecy in ubuntu packages

2018-08-20 Thread John Spray
On Mon, Aug 20, 2018 at 6:50 PM Alfredo Daniel Rezinovsky wrote: > > > > On 20/08/18 06:44, John Spray wrote: > > On Sun, Aug 19, 2018 at 9:21 PM Alfredo Daniel Rezinovsky > > wrote: > >> both in ubuntu 16.04 and 18.04 ceph-mgr fail to starts when package &g

Re: [ceph-users] missing dependecy in ubuntu packages

2018-08-20 Thread John Spray
On Sun, Aug 19, 2018 at 9:21 PM Alfredo Daniel Rezinovsky wrote: > > both in ubuntu 16.04 and 18.04 ceph-mgr fail to starts when package > python-routes is not installed I guess you mean that the dashboard doesn't work, as opposed to the whole ceph-mgr process not starting? If it's the latter th

Re: [ceph-users] Make a ceph options persist

2018-08-13 Thread John Spray
On Fri, Aug 10, 2018 at 10:40 AM Willem Jan Withagen wrote: > > Hi, > > The manual of dashboard suggests: > ceph config-key set mgr/dashboard/server_addr ${MGR_IP} > > But that command is required after reboot. config-key settings are persistent. The docs are probably just telling you to

Re: [ceph-users] cephfs - restore files

2018-08-13 Thread John Spray
On Thu, Aug 9, 2018 at 6:34 PM Erik Schwalbe wrote: > > Hi, > > Unfortunately, I deleted a few files and would like to restore them. > For ext4 I would take photorec but that not seem to work for cephfs. > > Is it possible to restore deleted files stored in cephfs? In general, no. There is a sho

Re: [ceph-users] cephfs kernel client hangs

2018-08-08 Thread John Spray
esources (1GB RAM per TB of OSD storage). I'd also be concerned about having only one quarter of a CPU core for each OSD. Sounds like you've got your settings tuned to something that's working in practice though, so I wouldn't mess with it :-) John > > many thanks! >

Re: [ceph-users] CephFS - Mounting a second Ceph file system

2018-08-08 Thread John Spray
On Tue, Aug 7, 2018 at 11:41 PM Scott Petersen wrote: > We are using kernel 4.15.17 and we keep receiving this error > mount.ceph: unrecognized mount option "mds_namespace", passing to kernel. > That message is harmless -- it just means that the userspace mount.ceph utility doesn't do anything w

Re: [ceph-users] cephfs kernel client hangs

2018-08-07 Thread John Spray
On Tue, Aug 7, 2018 at 5:42 PM Reed Dier wrote: > > This is the first I am hearing about this as well. This is not a Ceph-specific thing -- it can also affect similar systems like Lustre. The classic case is when under some memory pressure, the kernel tries to free memory by flushing the client'

Re: [ceph-users] Intermittent client reconnect delay following node fail

2018-08-01 Thread John Spray
do reliably reconnect, there's just some variation in the time it takes? Or are you still losing some clients entirely? John > > William Lawton > > -Original Message- > From: John Spray > Sent: Tuesday, July 31, 2018 11:17 AM > To: William Lawton > Cc: ceph

Re: [ceph-users] Force cephfs delayed deletion

2018-08-01 Thread John Spray
a known issue? And is there a workaround to give more priority to > the objects purging operations? > > > > Thanks in advance, > > Nitin > > > > *From: *ceph-users on behalf of > Alexander Ryabov > *Date: *Thursday, July 19, 2018 at 8:09 AM > *To: *Joh

Re: [ceph-users] CephFS Snapshots in Mimic

2018-07-31 Thread John Spray
On Tue, Jul 31, 2018 at 3:45 PM Kenneth Waegeman wrote: > > Hi all, > > I updated an existing Luminous cluster to Mimic 13.2.1. All daemons were > updated, so I did ceph osd require-osd-release mimic, so everything > seems up to date. > > I want to try the snapshots in Mimic, since this should be

Re: [ceph-users] Intermittent client reconnect delay following node fail

2018-07-31 Thread John Spray
On Tue, Jul 31, 2018 at 12:33 AM William Lawton wrote: > > Hi. > > > > We have recently setup our first ceph cluster (4 nodes) but our node failure > tests have revealed an intermittent problem. When we take down a node (i.e. > by powering it off) most of the time all clients reconnect to the cl

Re: [ceph-users] Mgr cephx caps to run `ceph fs status`?

2018-07-31 Thread John Spray
On Tue, Jul 31, 2018 at 3:36 AM Linh Vu wrote: > > Hi all, > > > I want a non-admin client to be able to run `ceph fs status`, either via the > ceph CLI or a python script. Adding `mgr "allow *"` to this client's cephx > caps works, but I'd like to be more specific if possible. I can't find the

Re: [ceph-users] Enable daemonperf - no stats selected by filters

2018-07-30 Thread John Spray
On Mon, Jul 30, 2018 at 10:27 PM Marc Roos wrote: > > > Do you need to enable the option daemonperf? This looks strange, it's supposed to have sensible defaults -- what version are you on? John > [@c01 ~]# ceph daemonperf mds.a > Traceback (most recent call last): > File "/usr/bin/ceph", line

Re: [ceph-users] cephfs tell command not working

2018-07-30 Thread John Spray
ohn > > Thank You, > Scott > On Mon, Jul 30, 2018 at 7:19 AM John Spray wrote: >> >> On Fri, Jul 27, 2018 at 8:35 PM Scottix wrote: >> > >> > ceph tell mds.0 client ls >> > 2018-07-27 12:32:40.344654 7fa5e27fc700 0 client.89408629 ms_handle_re

Re: [ceph-users] cephfs tell command not working

2018-07-30 Thread John Spray
On Fri, Jul 27, 2018 at 8:35 PM Scottix wrote: > > ceph tell mds.0 client ls > 2018-07-27 12:32:40.344654 7fa5e27fc700 0 client.89408629 ms_handle_reset on > 10.10.1.63:6800/1750774943 > Error EPERM: problem getting command descriptions from mds.0 You need "mds allow *" capabilities (the defaul

Re: [ceph-users] Preventing pool from allocating PG to OSD belonging not beloning to the device class defined in crush rule

2018-07-26 Thread John Spray
On Thu, Jul 26, 2018 at 4:57 PM Benoit Hudzia wrote: > HI, > > We currently segregate ceph pool PG allocation using the crush device > class ruleset as described: > https://ceph.com/community/new-luminous-crush-device-classes/ > simply using the following command to define the rule : ceph osd cr

Re: [ceph-users] Cephfs kernel driver availability

2018-07-23 Thread John Spray
On Sun, Jul 22, 2018 at 9:03 PM Bryan Henderson wrote: > > Is there some better place to get a filesystem driver for the longterm > stable Linux kernel (3.16) than the regular kernel.org source distribution? The general advice[1] on this is not to try and use a 3.x kernel with CephFS. The only e

Re: [ceph-users] Force cephfs delayed deletion

2018-07-19 Thread John Spray
On Thu, Jul 19, 2018 at 1:58 PM Alexander Ryabov wrote: > Hello, > > I see that free space is not released after files are removed on CephFS. > > I'm using Luminous with replica=3 without any snapshots etc and with > default settings. > > > From client side: > $ du -sh /mnt/logs/ > 4.1G /mnt/logs

Re: [ceph-users] ls operation is too slow in cephfs

2018-07-17 Thread John Spray
On Tue, Jul 17, 2018 at 8:26 AM Surya Bala wrote: > > Hi folks, > > We have production cluster with 8 nodes and each node has 60 disks of size > 6TB each. We are using cephfs and FUSE client with global mount point. We are > doing rsync from our old server to this cluster rsync is slow compared

Re: [ceph-users] CephFS with erasure coding, do I need a cache-pool?

2018-07-16 Thread John Spray
On Sun, Jul 15, 2018 at 12:46 PM Oliver Schulz wrote: > > Dear all, > > we're planning a new Ceph-Clusterm, with CephFS as the > main workload, and would like to use erasure coding to > use the disks more efficiently. Access pattern will > probably be more read- than write-heavy, on average. > > I

Re: [ceph-users] MDS damaged

2018-07-11 Thread John Spray
o look for corresponding errors on your OSD logs, system logs, and possibly also check things like the SMART counters on your hard drives for possible root causes. John > > > Can this be recovered someway? > > Thanks, > > > Alessandro > > > Il 11/07/18 18:33, J

Re: [ceph-users] MDS damaged

2018-07-11 Thread John Spray
On Wed, Jul 11, 2018 at 4:10 PM Alessandro De Salvo wrote: > > Hi, > > after the upgrade to luminous 12.2.6 today, all our MDSes have been > marked as damaged. Trying to restart the instances only result in > standby MDSes. We currently have 2 filesystems active and 2 MDSes each. > > I found the f

Re: [ceph-users] CephFS - How to handle "loaded dup inode" errors

2018-07-11 Thread John Spray
ant because we don't know what originally went wrong with your cluster. You'd also need to add an "fs reset" before starting up again if you had multiple active MDS ranks to begin with. John > > Many thanks, I've learned a lot more about this process. > > Cheers,

Re: [ceph-users] CephFS - How to handle "loaded dup inode" errors

2018-07-10 Thread John Spray
On Tue, Jul 10, 2018 at 3:14 PM Dennis Kramer (DBS) wrote: > > Hi John, > > On Tue, 2018-07-10 at 10:11 +0100, John Spray wrote: > > On Tue, Jul 10, 2018 at 12:43 AM Linh Vu wrote: > > > > > > > > > We're affected by something like this right

Re: [ceph-users] CephFS - How to handle "loaded dup inode" errors

2018-07-10 Thread John Spray
to avoid re-using any inode numbers that had been claimed by journal entries that we threw away. John > > Cheers, > > Linh > > > From: ceph-users on behalf of Wido den > Hollander > Sent: Saturday, 7 July 2018 12:26:15 AM > To:

Re: [ceph-users] CephFS - How to handle "loaded dup inode" errors

2018-07-10 Thread John Spray
picked up and converted into explicit damage handling, but this one appears to have been missed -- so yes, it's a bug that the MDS asserts out. John > > From: ceph-users on behalf of Wido den > Hollander > Sent: Saturday, 7 July 2018 12:26:15

Re: [ceph-users] FYI - Mimic segv in OSD

2018-07-09 Thread John Spray
On Mon, Jul 9, 2018 at 2:37 PM Steffen Winther Sørensen wrote: > > Dunno if this has been seen before so just for info, 1 in 24 OSD just did > this: > > Jul 9 15:13:35 n4 ceph-osd: *** Caught signal (Segmentation fault) ** > Jul 9 15:13:35 n4 ceph-osd: in thread 7ff209282700 thread_name:msgr-wo

Re: [ceph-users] luminous ceph-fuse with quotas breaks 'mount' and 'df'

2018-07-09 Thread John Spray
On Fri, Jul 6, 2018 at 6:30 PM Chad William Seys wrote: > > Hi all, >I'm having a problem that when I mount cephfs with a quota in the > root mount point, no ceph-fuse appears in 'mount' and df reports: > > Filesystem 1K-blocks Used Available Use% Mounted on > ceph-fuse 0

Re: [ceph-users] CephFS - How to handle "loaded dup inode" errors

2018-07-06 Thread John Spray
On Fri, Jul 6, 2018 at 12:19 PM Wido den Hollander wrote: > > > > On 07/05/2018 03:36 PM, John Spray wrote: > > On Thu, Jul 5, 2018 at 1:42 PM Dennis Kramer (DBS) wrote: > >> > >> Hi list, > >> > >> I have a serious problem now... I think.

Re: [ceph-users] Ceph mon quorum problems under load

2018-07-06 Thread John Spray
On Fri, Jul 6, 2018 at 11:10 AM Marcus Haarmann wrote: > > Hi experts, > > we have setup a proxmox cluster with a minimal environment for some testing. > We have put some VMs on the cluster and encountered mon quorum problems > while backups are executed. (possibly polluting either hard disk I/O o

Re: [ceph-users] CephFS - How to handle "loaded dup inode" errors

2018-07-05 Thread John Spray
On Thu, Jul 5, 2018 at 1:42 PM Dennis Kramer (DBS) wrote: > > Hi list, > > I have a serious problem now... I think. > > One of my users just informed me that a file he created (.doc file) has > a different content then before. It looks like the file's inode is > completely wrong and points to the

Re: [ceph-users] Spurious empty files in CephFS root pool when multiple pools associated

2018-07-03 Thread John Spray
On Tue, Jul 3, 2018 at 12:24 PM Jesus Cea wrote: > > On 03/07/18 13:08, John Spray wrote: > > Right: as you've noticed, they're not spurious, they're where we keep > > a "backtrace" xattr for a file. > > > > Backtraces are lazily updated pa

Re: [ceph-users] Spurious empty files in CephFS root pool when multiple pools associated

2018-07-03 Thread John Spray
On Tue, Jul 3, 2018 at 11:53 AM Jesus Cea wrote: > > Hi there. > > I have an issue with cephfs and multiple datapools inside. I have like > SIX datapools inside the cephfs, I control where files are stored using > xattrs in the directories. > > The "root" directory only contains directories with "

Re: [ceph-users] mgr modules not enabled in conf

2018-07-03 Thread John Spray
On Tue, Jul 3, 2018 at 9:37 AM Gökhan Kocak wrote: > > Hello everyone, > > I tried to enable the Prometheus module (and later with the same result > the Dashboard module) as outlined in the docs here: > http://docs.ceph.com/docs/mimic/mgr/dashboard/#enabling > [mon] > mgr initial m

Re: [ceph-users] commend "ceph dashboard create-self-signed-cert " ERR

2018-07-03 Thread John Spray
On Tue, Jul 3, 2018 at 9:18 AM John Spray wrote: > > On Tue, Jul 3, 2018 at 6:25 AM jaywaychou wrote: > > > > > > HI,Cephers: > > > > I just use Mimic Ceph for Dashboard. I just do as > > http://docs.ceph.com/docs/mimic/mgr/dashboard/ > >

Re: [ceph-users] commend "ceph dashboard create-self-signed-cert " ERR

2018-07-03 Thread John Spray
On Tue, Jul 3, 2018 at 6:25 AM jaywaychou wrote: > > > HI,Cephers: > > I just use Mimic Ceph for Dashboard. I just do as > http://docs.ceph.com/docs/mimic/mgr/dashboard/ > > When install a self-signed certificate as build-in commend , it stuck ERR > like as bellow: > > [root@localhost ~]

Re: [ceph-users] MDS: journaler.pq decode error

2018-06-21 Thread John Spray
_pos but the journal inspect command gives an > > 'OK' for overall integrity. > > > > As recommended I did take an export of the journal first and I'll take > > a stab at using a hex editor on it near future. Worst case we go > > through the tag/scan if

Re: [ceph-users] MDS: journaler.pq decode error

2018-06-21 Thread John Spray
quot;expire_pos": 6822392, > "trimmed_pos": 4194304, > "stream_format": 1, > "layout": { > "stripe_unit": 4194304, > "stripe_count": 1, > "object_size": 4194304, > "

Re: [ceph-users] separate monitoring node

2018-06-19 Thread John Spray
On Tue, Jun 19, 2018 at 1:17 PM Denny Fuchs wrote: > > Hi, > > at the moment, we use Icinga2, check_ceph* and Telegraf with the Ceph > plugin. I'm asking what I need to have a separate host, which knows all > about the Ceph cluster health. The reason is, that each OSD node has > mostly the exact s

Re: [ceph-users] MDS: journaler.pq decode error

2018-06-15 Thread John Spray
On Fri, Jun 15, 2018 at 2:55 PM, Benjeman Meekhof wrote: > Have seen some posts and issue trackers related to this topic in the > past but haven't been able to put it together to resolve the issue I'm > having. All on Luminous 12.2.5 (upgraded over time from past > releases). We are going to upg

Re: [ceph-users] ceph pg dump

2018-06-14 Thread John Spray
On Thu, Jun 14, 2018 at 6:31 PM, Ranjan Ghosh wrote: > Hi all, > > we have two small clusters (3 nodes each) called alpha and beta. One node > (alpha0/beta0) is on a remote site and only has monitor & manager. The two > other nodes (alpha/beta-1/2) have all 4 services and contain the OSDs and > ar

Re: [ceph-users] About "ceph balancer": typo in doc, restrict by class

2018-05-29 Thread John Spray
On Mon, May 28, 2018 at 1:50 PM, Fulvio Galeazzi wrote: > Hallo, > I am using 12.2.4 and started using "ceph balancer". Indeed it does a > great job, thanks! > > I have few comments: > > - in the documentation http://docs.ceph.com/docs/master/mgr/balancer/ >I think there is an error, si

Re: [ceph-users] CephFS "move" operation

2018-05-25 Thread John Spray
On Fri, May 25, 2018 at 1:10 PM, Oliver Freyermuth wrote: > Dear Cephalopodians, > > I was wondering why a simple "mv" is taking extraordinarily long on CephFS > and must note that, > at least with the fuse-client (12.2.5) and when moving a file from one > directory to another, > the file appear

Re: [ceph-users] How high-touch is ceph?

2018-05-25 Thread John Spray
On Fri, May 25, 2018 at 1:17 PM, Rhugga Harper wrote: > > I've been evaluating ceph as a solution for persistent block in our > kubrenetes clusters for low-iops requirement applications. It doesn't do too > terribly bad with 32k workloads even though it's object storage under the > hood. > > Howev

Re: [ceph-users] Web panel is failing where create rpm

2018-05-22 Thread John Spray
On Tue, May 22, 2018 at 6:38 PM, Antonio Novaes wrote: > Hi people, > I need help for you. > I trying create package rpm ceph calamari but i get error. > The package of calamari server is OK, but Diamond get error. > I send log > My ceph is OK, create pool, put file remove file and remove pool wit

Re: [ceph-users] ceph osd status output

2018-05-18 Thread John Spray
On Fri, May 18, 2018 at 9:55 AM, Marc Roos wrote: > > Should ceph osd status not be stdout? Oops, that's a bug. http://tracker.ceph.com/issues/24175 https://github.com/ceph/ceph/pull/22089 John > So I can do something like this > > [@ ~]# ceph osd status |grep c01 > > And don't need to do this

  1   2   3   4   5   6   7   8   >