[ceph-users] ceph plugin balancer error

2018-07-05 Thread Chris Hsiang
Hi, I am running test on ceph mimic 13.0.2.1874+ge31585919b-lp150.1.2 using openSUSE-Leap-15.0 when I ran "ceph balancer status", it errors out. g1:/var/log/ceph # ceph balancer status Error EIO: Module 'balancer' has experienced an error and cannot handle commands: 'dict' object has no attribu

Re: [ceph-users] ceph plugin balancer error

2018-07-05 Thread Chris Hsiang
weird thing is that, g1:~ # locate /bin/python /usr/bin/python /usr/bin/python2 /usr/bin/python2.7 /usr/bin/python3 /usr/bin/python3.6 /usr/bin/python3.6m g1:~ # ls /usr/bin/python* -al lrwxrwxrwx 1 root root 9 May 13 07:41 /usr/bin/python -> python2.7 lrwxrwxrwx 1 root root 9 May 13 07:4

Re: [ceph-users] ceph plugin balancer error

2018-07-05 Thread Chris Hsiang
I have tried to modify /usr/lib64/ceph/mgr/balancer/module.py replace iteritems () with items(), but I still got following error g1:/usr/lib64/ceph/mgr/balancer # ceph balancer status Error EINVAL: Traceback (most recent call last): File "/usr/lib64/ceph/mgr/balancer/module.py", line 297, in

[ceph-users] CephFS - How to handle "loaded dup inode" errors

2018-07-05 Thread Dennis Kramer (DT)
Hi, I'm getting a bunch of "loaded dup inode" errors in the MDS logs. How can this be fixed? logs: 2018-07-05 10:20:05.591948 mds.mds05 [ERR] loaded dup inode 0x1991921 [2,head] v160 at , but inode 0x1991921.head v146 already exists at _

Re: [ceph-users] CephFS MDS server stuck in "resolve" state

2018-07-05 Thread Dennis Kramer (DBS)
Hi, On Thu, 2018-07-05 at 09:55 +0800, Yan, Zheng wrote: > On Wed, Jul 4, 2018 at 7:02 PM Dennis Kramer (DBS) > wrote: > > > > > > Hi, > > > > I have managed to get cephfs mds online again...for a while. > > > > These topics covers more or less my symptoms and helped me get it > > up > > and

Re: [ceph-users] ceph plugin balancer error

2018-07-05 Thread Nathan Cutler
Hi Chris: I suggest you raise your openSUSE Ceph-related questions on the openSUSE Ceph mailing list instead of ceph-users. For info on how to join, go to https://en.opensuse.org/openSUSE:Ceph#Communication The version of Ceph currently shipping in Leap 15.0 is built against Python 3 and thi

Re: [ceph-users] ceph plugin balancer error

2018-07-05 Thread Nathan Cutler
Update: opened http://tracker.ceph.com/issues/24779 to track this bug, and am in the process of fixing it. The fix will make its way into a future mimic point release. Thanks, Chris, for bringing the issue to my attention! Nathan On 07/05/2018 11:27 AM, Nathan Cutler wrote: Hi Chris: I sugge

Re: [ceph-users] CephFS - How to handle "loaded dup inode" errors

2018-07-05 Thread Dennis Kramer (DBS)
Hi list, I have a serious problem now... I think. One of my users just informed me that a file he created (.doc file) has a different content then before. It looks like the file's inode is completely wrong and points to the wrong object. I myself have found another file with the same symptoms. I'

[ceph-users] corrupt OSD: BlueFS.cc: 828: FAILED assert

2018-07-05 Thread Jake Grimmett
Dear All, I have a Mimic (13.2.0) cluster, which, due to a bad disk controller, corrupted three Bluestore OSD's on one node. Unfortunately these three OSD's crash when they try to start. systemctl start ceph-osd@193 (snip) /BlueFS.cc: 828: FAILED assert(r != q->second->file_map.end()) Full log

[ceph-users] jemalloc / Bluestore

2018-07-05 Thread Uwe Sauter
Hi all, is using jemalloc still recommended for Ceph? There are multiple sites (e.g. https://ceph.com/geen-categorie/the-ceph-and-tcmalloc-performance-story/) from 2015 where jemalloc is praised for higher performance but I found a bug report that Bluestore crashes when used with jemalloc. Re

Re: [ceph-users] corrupt OSD: BlueFS.cc: 828: FAILED assert

2018-07-05 Thread Igor Fedotov
Hi Jake, IMO it doesn't make sense to recover from this drive/data as the damage coverage looks pretty wide. By modifying BlueFS code you can bypass that specific assertion but most probably BlueFS and  other BlueStore stuff are pretty inconsistent and most probably are unrecoverable at the

Re: [ceph-users] jemalloc / Bluestore

2018-07-05 Thread Igor Fedotov
Hi Uwe, AFAIK jemalloc isn't recommended for use with BlueStore anymore. tcmalloc is the right way so far. Thanks, Igor On 7/5/2018 4:08 PM, Uwe Sauter wrote: Hi all, is using jemalloc still recommended for Ceph? There are multiple sites (e.g. https://ceph.com/geen-categorie/the-ceph-an

[ceph-users] RGW User Stats Mismatch

2018-07-05 Thread Ryan Leimenstoll
Hi all, We currently have a Ceph Luminous 12.2.5 cluster running, among other functions, an RGW service for users in our organization. The cluster has been upgraded through a few major versions, at least farther back than Hammer. For some time, we were bitten by the RGW user quota sync issue t

Re: [ceph-users] CephFS - How to handle "loaded dup inode" errors

2018-07-05 Thread John Spray
On Thu, Jul 5, 2018 at 1:42 PM Dennis Kramer (DBS) wrote: > > Hi list, > > I have a serious problem now... I think. > > One of my users just informed me that a file he created (.doc file) has > a different content then before. It looks like the file's inode is > completely wrong and points to the

Re: [ceph-users] corrupt OSD: BlueFS.cc: 828: FAILED assert

2018-07-05 Thread Jake Grimmett
Hi Igor, Many thanks for the quick reply. Your advice concurs with my own thoughts, given the damage, probably safest to wipe the OSD's and start over. thanks again, Jake On 05/07/18 14:28, Igor Fedotov wrote: > Hi Jake, > > IMO it doesn't make sense to recover from this drive/data as the da

Re: [ceph-users] Performance tuning for SAN SSD config

2018-07-05 Thread Matthew Stroud
Bump. I’m hoping I can get people more knowledgeable than me to take a look. From: ceph-users on behalf of Matthew Stroud Date: Friday, June 29, 2018 at 10:31 AM To: ceph-users Subject: [ceph-users] Performance tuning for SAN SSD config We back some of our ceph clusters with SAN SSD disk, par

Re: [ceph-users] jemalloc / Bluestore

2018-07-05 Thread Mark Nelson
Hi Uwe, As luck would have it we were just looking at memory allocators again and ran some quick RBD and RGW tests that stress memory allocation: https://drive.google.com/uc?export=download&id=1VlWvEDSzaG7fE4tnYfxYtzeJ8mwx4DFg The gist of it is that tcmalloc looks like it's doing pretty we

Re: [ceph-users] jemalloc / Bluestore

2018-07-05 Thread Uwe Sauter
Ah, thanks… I'm currently trying to diagnose a performace regression that occurs with the Ubuntu 4.15 kernel (on a Proxmox system) and thought that jemalloc, given the old reports, could help with that. But than I ran into that bug report. I'll take from your info that I'm gonna stick to tcmal

Re: [ceph-users] pool has many more objects per pg than average

2018-07-05 Thread Stefan Kooman
Quoting Brett Chancellor (bchancel...@salesforce.com): > The error will go away once you start storing data in the other pools. Or, > you could simply silence the message with mon_pg_warn_max_object_skew = 0 Ran into this issue myself (again). Note to self: You need to restart the _active_ MGR to

Re: [ceph-users] Performance tuning for SAN SSD config

2018-07-05 Thread Steffen Winther Sørensen
> On 5 Jul 2018, at 16.51, Matthew Stroud wrote: > > Bump. I’m hoping I can get people more knowledgeable than me to take a look. > We back some of our ceph clusters with SAN SSD disk, particularly VSP G/F and > Purestorage. I’m curious what are some settings we should look into modifying > t