[ceph-users] High IOWAIT On OpenStack Instance

Hi All, We are testing Ceph with OpenStack and installed 3 Mon (This three monitor nodes are also OpenStack controller and network node), 6 OSD (3 of the OSDs are also Nova Computer Node). There are total 24 OSDs (21 SAS, 3 SSD and all journals are in SSD). There is no cache tiering for now. Be

Re: [ceph-users] SSD Journal

Hi, unfortunately I'm not a dev, so it's gonna be someone else ripping the journal out and trying. But I came to understand that getting rid of journal is not that easy of a task. To me, more important would be if the devs understood what I'm trying to say :-) because without that any new develop

Re: [ceph-users] Ceph Stats back to Calamari

I can see the is ok files are there *root@ceph1:/var/run/ceph# ls -la* *total 0* *drwxrwx--- 2 ceph ceph 80 Feb 1 10:51 .* *drwxr-xr-x 18 root root 640 Feb 1 10:52 ..* *srwxr-xr-x 1 ceph ceph 0 Feb 1 10:51 ceph-mon.ceph1.asok* *srwxr-xr-x 1 root root 0 Jan 27 15:08 ceph-osd.0.asok* *roo

Re: [ceph-users] Ceph Stats back to Calamari

The "assert path[-1] == 'type'" is the error you get when using the calamari diamond branch with a >= infernalis version of Ceph (where new fields were added to the perf schema output). No idea if anyone has worked on updating Calamari+Diamond for latest ceph. John On Mon, Feb 1, 2016 at 12:09 P

[ceph-users] remove rbd volume with watchers

Hi, I accidentally terminated an instance in openstack that was running a system upgrade. It's boot volume was a cinder volume backed by ceph cache tier pool. Now i can not delete the volume neither from cinder of directly from ceph. If i try to delete it from cinder i get "rbd volume busy. Try a

Re: [ceph-users] remove rbd volume with watchers

I managed to delete it after running "rados -p cache-ssd listwatchers rbd_header.". It was one of the monitors that was watching and keeping the image busy. Any light on why did the monitor kept keeping the image busy? I would like to know! Thanks! 2016-02-01 14:32 GMT+02:00 mcapsali : > Hi,

Re: [ceph-users] Ceph Stats back to Calamari

hmmm we are running infernalis... root@ceph1:~# ceph -v ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) root@ceph1:~# So you think everything should work with hammer ? As this is all just a proof of concept on some vm's so we know how to build into prod Regards, Daniel On Mon, F

[ceph-users] Install ceph with infiniband

Hello, I am going to install ceph with infiniband and RDMA support. However, it seems there is no documents for doing that. I can only find 'RDMA support' in ceph's release notes, and it seems using Accelio (libxio) library. Most of the documents are based on Ethernet configuration. So are

[ceph-users] CephFS is not maintianing conistency

Hi guys, This is sort of rebuttal. I have a CephFS deployed and mounted on a couple of clients via ceph-fuse (due to quota support and possibility to kill the ceph-fuse process to avoid stale mounts). So the problems is that some times the changes made on one client are not visible on the o

[ceph-users] download.ceph.com not reachable over IPv6

Hi, My monitoring just showed me that download.ceph.com isn't reachable over IPv6. I tried from multipe locations, but on all I get a connection refused: root@soekris:~# telnet -6 download.ceph.com 80 Trying 2607:f298:6050:51f3:f816:3eff:fe50:5ec... telnet: Unable to connect to remote host: Conn

[ceph-users] Need help to develop CEPH EC Plugin for array type of Erasure Code

Hi, I've been working to develop a CEPH EC plgin for array type erasure code, for example RAID-DP (i.e RDP). Later I realized that I can't continue with (k+m) format in CEPH as like normal RS code if I want to save disk IO (or Read data). Since I need to ready the individual symbols from each dev

Re: [ceph-users] CephFS is not maintianing conistency

On Monday, February 1, 2016, Mykola Dvornik wrote: > Hi guys, > > This is sort of rebuttal. > > I have a CephFS deployed and mounted on a couple of clients via ceph-fuse > (due to quota support and possibility to kill the ceph-fuse process to > avoid stale mounts). > > So the problems is that som

Re: [ceph-users] CephFS - Trying to understand direct OSD connection to ceph-fuse cephfs clients

On Sunday, January 31, 2016, Goncalo Borges wrote: > Dear CephFS experts... > > We are using Ceph and CephFS 9.2.0. CephFS clients are being mounted via > ceph-fuse. > > We recently noticed the firewall from certain CephFS clients dropping > connections with OSDs as SRC. This is something which i

Re: [ceph-users] Install ceph with infiniband

-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 RDMA for Ceph is not quite production ready yet. For now, you will have to go with IPoIB and when RDMA is available, you hopefully can migrate pretty easy. Matt Benjamin should be able to give you a better idea about timeline. I haven't heard a lot a

Re: [ceph-users] attempt to access beyond end of device on osd prepare

Has nobody else encountered this (or can explain it away) then? That's a bit of a worry, I wish I'd never spotted it now. I've just built the second OSD server and it produces exactly the same "attempt to access beyond end of device" message for each osd prepared. Regards, Simon. On 27/01/16

Re: [ceph-users] attempt to access beyond end of device on osd prepare

I can at least say that I've seen this. (a lot) Running Infernalis with Btrfs on Cent 7.2. I haven't seen any other issue in the cluster that I would say is related. Take it for what you will. Best Regards, Wade On Mon, Feb 1, 2016 at 12:39 PM Simon Ironside wrote: > Has nobody else encount

Re: [ceph-users] CephFS is not maintianing conistency

What version are you running on your servers and clients? On the clients: ceph-fuse --version ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) MDS/OSD/MON: ceph --version ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299) Exactly what changes are you making that are

[ceph-users] Remove MDS

My cluster is 0.94.5 At first I installed the MDS role on an OSD node. I now have a dedicated server for MDS. How do I remove from the OSD node? TIA, Don ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-user

Re: [ceph-users] Remove MDS

You can get rid of a daemon by simply stopping it and removing its directory in /var/lib/ceph/mds -- no important state is stored locally on the servers running MDS daemons. Hopefully your new server is already up and running, so you can see both the new and the old servers' daemons in "ceph mds d

[ceph-users] Switching cache from writeback to forward causes I/O error in Firefly

Hello, Latest Firefly (and no, I can't safely upgrade before the cache is in). While running some high-intensity I/O (for the crappy test cluster that is) with "rados -p rbd bench 60 write -t 32 -b 4096" turning a cache-tier on (from forward to writeback mode) works as expected and without any

[ceph-users] virsh, virt-filesystems, guestmount, virt-install not working well with ceph rbd yet?

Hello everybody, This is a cross post to libvirt-users, libguestfs and ceph-users. I came back from FOSDEM 2016 and this was my 7th year or so and seen the awesome development around visualization going on and want to thank everybody for there contributions. I seen presentations from oVirt, Open

[ceph-users] Ceph Developer Monthly (CDM)

Hey cephers, Just as a reminder, the new format for our periodic developer meetings is starting this Wed @ 12:30p EST. http://tracker.ceph.com/projects/ceph/wiki/Planning This meeting will be replacing the once quarterly (ish) Ceph Developer Summits. Our hope is that a less formal, and more freq

Re: [ceph-users] attempt to access beyond end of device on osd prepare

On 01/02/16 17:47, Wade Holler wrote: I can at least say that I've seen this. (a lot) Running Infernalis with Btrfs on Cent 7.2. I haven't seen any other issue in the cluster that I would say is related. Take it for what you will. Thanks Wade, That's reassuring at least that it hasn't cause

Re: [ceph-users] attempt to access beyond end of device on osd prepare

327.3.1 as I recall. Thank for the additional information / confirmation. On Mon, Feb 1, 2016 at 6:05 PM Simon Ironside wrote: > On 01/02/16 17:47, Wade Holler wrote: > > I can at least say that I've seen this. (a lot) > > > > Running Infernalis with Btrfs on Cent 7.2. I haven't seen any other >

Re: [ceph-users] Upgrading Ceph

What if the upgrade fails, what is the rollback scenario? On Wed, Jan 27, 2016 at 10:10 PM, wrote: > I just upgraded from firefly to infernalis (firefly to hammer to > infernalis) my cluster > All came like a charm > I upgraded mon first, then the osd, one by one, restarting the daemon > after

Re: [ceph-users] CephFS - Trying to understand direct OSD connection to ceph-fuse cephfs clients

Hi Greg. We are using Ceph and CephFS 9.2.0. CephFS clients are being mounted via ceph-fuse. We recently noticed the firewall from certain CephFS clients dropping connections with OSDs as SRC. This is something which is not systematic but we noticed happening at least once. Here is an example

Re: [ceph-users] Upgrading Ceph

Upgrades have been easy for me, following the steps. I would say to be careful not to 'miss' one OSD, or forget to restart it after updating, since having an OSD on a different version than the rest of the cluster for too long during an upgrade started to cause issues when i missed one once. -Ben

[ceph-users] CEPHFS: standby-replay mds crash

Hi CephFS experts. 1./ We are using Ceph and CephFS 9.2.0 with an active mds and a standby-replay mds (standard config) # ceph -s cluster health HEALTH_OK monmap e1: 3 mons at {mon1=:6789/0,mon2=:6789/0,mon3=:6789/0} election epoch 98, quorum 0,1,2 mon1,mon3,mon2

Re: [ceph-users] CEPHFS: standby-replay mds crash

Hi... Seems very similar to http://tracker.ceph.com/issues/14144 Can you confirm it is the same issue? Cheers G. From: Goncalo Borges Sent: 02 February 2016 15:30 To: ceph-us...@ceph.com Cc: rct...@coepp.org.au Subject: CEPHFS: standby-replay mds crash Hi Ce

Re: [ceph-users] CEPHFS: standby-replay mds crash

On Tue, Feb 2, 2016 at 12:52 PM, Goncalo Borges wrote: > Hi... > > Seems very similar to > > http://tracker.ceph.com/issues/14144 > > Can you confirm it is the same issue? should be the same. it's fixed by https://github.com/ceph/ceph/pull/7132 Regards Yan, Zheng > > Cheers > G. > > > > ___

Re: [ceph-users] Upgrading Ceph

Hi Vlad, I just upgraded my ceph cluster from firefly to hammer and all right. Please do it according to the manuals in www.ceph.com and restart monitors and then osds. I restarted osds one by one, which means restart one OSD and waits for it runs normal and then r

Re: [ceph-users] CephFS is not maintianing conistency

On Tue, Feb 2, 2016 at 2:27 AM, Mykola Dvornik wrote: > What version are you running on your servers and clients? > Are you using 4.1 or 4.2 kernel? https://bugzilla.kernel.org/show_bug.cgi?id=104911. Upgrade to 4.3+ kernel or 4.1.17 kernel or 4.2.8 kernel can resolve this issue. > > On the clie