Re: [ceph-users] monitor quorum

Is anyone able to offer any advice on how to fix this? I've tried re-injecting the monmap into mon03 as that was mentioned in the mon troubleshooting docs, but that has not helped at all. mon03 is still stuck in the same electing state :( I've increased the debug level on mon03 and it is reportin

Re: [ceph-users] Still seing scrub errors in .80.5

Hi, we did run a deep scrub on everything yesterday, and a repair afterwards. Then a new deep scrub today, which brought new scrub errors. I did check the osd config, they report "filestore_xfs_extsize": "false", as it should be if I understood things correctly. FTR the deep scrub has been initi

[ceph-users] Newbie Ceph Design Questions

Hello Ceph-Community, we are considering to use a Ceph Cluster for serving VMs. We need goog performance and absolute stability. Regarding Ceph I have a few questions. Presently we use Solaris ZFS Boxes as NFS Storage for VMs. The zfs boxes are totally fast, because they use all free ram for r

[ceph-users] Frequent Crashes on rbd to nfs gateway Server

Hi, I have build an NFS Server based on Sebastiens Blog Post here: http://www.sebastien-han.fr/blog/2012/07/06/nfs-over-rbd/ Im using Kernel 3.14-0.bpo.1-amd64 on Debian wheezy, the host is a VM on Vmware. Using rsync im writing data via nfs from one client to this Server. The NFS Server crash

[ceph-users] three way replication on pool a failed

Hello Sebastien, I am configuring ceph with 3 node storage cluster + one ceph admin. I have few questions. I have created pool name ' storage' with the replication size 3 on it and I have set the CRUSH rule . root@node1:/home/oss# ceph osd dump | grep -E 'storage' pool 9 'storage' replicated

Re: [ceph-users] Newbie Ceph Design Questions

Hello, On Thu, 18 Sep 2014 13:07:35 +0200 Christoph Adomeit wrote: > > Hello Ceph-Community, > > we are considering to use a Ceph Cluster for serving VMs. > We need goog performance and absolute stability. > I really don't want to sound snarky here, but you get what you pay for, as in the old

Re: [ceph-users] [Single OSD performance on SSD] Can't go over 3, 2K IOPS

>>Have anyone ever testing multi volume performance on a *FULL* SSD setup? I known that Stefan Priebe run full ssd clusters in production, and have done benchmark. (Ad far I remember, he have benched around 20k peak with dumpling) >>We are able to get ~18K IOPS for 4K random read on a single vo

Re: [ceph-users] ceph issue: rbd vs. qemu-kvm

thanks Luke, I will try that. Steve On Wed, 17 Sep 2014, Luke Jing Yuan wrote: Hi, From the ones we managed to configure in our lab here. I noticed that using image format "raw" instead of "qcow2" worked for us. Regards, Luke -Original Message- From: ceph-users [mailto:ceph-users-

Re: [ceph-users] ceph issue: rbd vs. qemu-kvm

With the default user "libvirt" (corresponding to client.libvirt ceph token) and with the permissions that were suggested both in the Ceph manual and the OpenNebula manual, I get a different error, namely permission denied. I am not sure why that is.. I then tried with the full ceph admin privilg

Re: [ceph-users] ceph issue: rbd vs. qemu-kvm

On Thu, 18 Sep 2014, Osier Yang wrote: On 2014年09月18日 10:38, Luke Jing Yuan wrote: Hi, From the ones we managed to configure in our lab here. I noticed that using image format "raw" instead of "qcow2" worked for us. Regards, Luke -Original Message- From: ceph-users [mailto:

Re: [ceph-users] [Single OSD performance on SSD] Can't go over 3, 2K IOPS

Couple of questions: Are those client IOPS or real IOPS after replication and journal writes? Also, how's CPU usage? Are interrupts being distributed to all cores? Mark On 09/18/2014 01:12 AM, Zhang, Jian wrote: Have anyone ever testing multi volume performance on a *FULL* SSD setup? We ar

Re: [ceph-users] getting ulimit set error while installing ceph in admin node

Thanks John, I tried the same you suggested. Disabled the SELinux and requiretty also. But still getting the same issue. Attached the log for debug. Please help to resolve. Regards, Subhadip

[ceph-users] Timeout on ceph-disk activate

I've hit a timeout issue on calls to ceph-disk activate. Initially, I followed the 'Storage Cluster Quick Start' on the CEPH website to get a cluster up and running. I wanted to tweak the configuration however and decided to blow away the initial setup using the purge / purgedata / forgetkeys comm

Re: [ceph-users] three way replication on pool a failed

On 18/09/2014 13:50, m.channappa.nega...@accenture.com wrote: Even after setting replication size 3 , my data is not getting replicated on all the 3 nodes. Example: root@Cephadmin:/home/oss# ceph osd map storage check1 osdmap e122 pool 'storage' (9) object 'check1' -> pg 9.7c9c5619 (9.1) -> up

Re: [ceph-users] [Single OSD performance on SSD] Can't go over 3, 2K IOPS

Hi Mark It's client IOPS and we use replica = 2, journal and OSD are hosted in the same SSDs so the real IOPS is 23K * 2 * 2 =90K, still far from HW limit (30K+ for a single DCS3700) CPU % is ~62% in peak (2VM ), interrupt distributed. An additional information, seems the cluster is in a kind

Re: [ceph-users] ceph mds unable to start with 0.85

On Wed, Sep 17, 2014 at 9:59 PM, 廖建锋 wrote: > dear, > my ceph cluster worked for about two weeks, mds crashed every 2-3 > days, > Now it stuck on replay , looks like replay crash and restart mds process > again > what can i do for this? > > 1015 => # ceph -s > cluster 07df7765-c2e7-44de-9b

Re: [ceph-users] Still seing scrub errors in .80.5

On Thu, Sep 18, 2014 at 3:09 AM, Marc wrote: > Hi, > > we did run a deep scrub on everything yesterday, and a repair > afterwards. Then a new deep scrub today, which brought new scrub errors. > > I did check the osd config, they report "filestore_xfs_extsize": "false", > as it should be if I under

Re: [ceph-users] CephFS : rm file does not remove object in rados

On Thu, Sep 18, 2014 at 10:39 AM, Florent B wrote: > On 09/12/2014 07:38 PM, Gregory Farnum wrote: >> On Fri, Sep 12, 2014 at 6:49 AM, Florent Bautista >> wrote: >>> Hi all, >>> >>> Today I have a problem using CephFS. I use firefly last release, with >>> kernel 3.16 client (Debian experimental)

Re: [ceph-users] [Ceph-community] Can't Start-up MDS

None of your PGs exist. Since you only have one OSD, they're probably not capable of fulfilling their default size requirements. You should go through the generic quick start guides and configuration before moving on to using the MDS. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.c

Re: [ceph-users] ceph issue: rbd vs. qemu-kvm

Using image type raw actually got kvm to create the VM but then the virt-viewer console shows Booting from Hard Disk Geom Error --- We do not even get as far as GRUB. Below is the network stanza from XML.

Re: [ceph-users] ceph mds unable to start with 0.85

if i turn on debug=20, the log will be more than 100G, looks no way to put, do you have any other good way to figure it out? would you like to log into the server to check? From: Gregory Farnum Date: 2014-09-19 02:33 To: 廖建锋 CC: ceph-users

Re: [ceph-users] RGW hung, 2 OSDs using 100% CPU

No, removing the snapshots didn't solve my problem. I eventually traced this problem to XFS deadlocks caused by [osd] "osd mkfs options xfs": "-l size=1024m -n size=64k -i size=2048 -s size=4096" Changing to just "-s size=4096", and reformatting all OSDs solved this problem. Since then, I ran

Re: [ceph-users] osd going down every 15m blocking recovery from degraded state

The magic in Sage's steps was really setting noup. That gives the OSD time to apply the osdmap changes, without starting the timeout. Set noup, nodown, noout, restart the OSD, and wait until the CPU usage goes to zero. Some of mine took 5 minutes. Once it's done, unset noup, and restart again.

Re: [ceph-users] ceph issue: rbd vs. qemu-kvm

Hi Steven, Assuming the original image was in qcow2 format, did you convert it back to raw before registering it? Another tweak I did was enabling and NFS shared the system datastore (id: 0) from the frontend to the other hosts: nebula@z4-hn01:~$ onedatastore list ID NAMESIZE

[ceph-users] do you have any test case that lost data mostlikely

hi,all i want to test some cases that lost data mostlikely. now i just test killing osds. do you have any such test cases? thanks___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Can ceph-deploy be used with 'osd objectstore = keyvaluestore-dev' in config file ?

I noticed ceph added key/value store OSD backend feature in firefly, but i can hardly get any documentation about how to use it. At last I found that i can add a line in ceph.conf: osd objectstore = keyvaluestore-dev but got failed with ceph-deploy creating OSDs. According to the log, ceph-

Re: [ceph-users] ceph issue: rbd vs. qemu-kvm

Yes--the image was converted back to raw. Since the image is mapped via rbd I can run fdisk on it and see both the partition tables and a normal set of files inside of it. My system datastore is local to each node. Have been in that mode for quite some time. Steve Timm ___

[ceph-users] confusion when kill 3 osds that store the same pg

hi,all in order to test ceph stability. i try to kill osds. in this case ,i kill 3 osds(osd3,2,0) that store the same pg 2.30. ---crush--- osdmap e1342 pool 'rbd' (2) object 'rbd_data.19d92ae8944a.' -> pg 2.c59a45b0 (2.30) -> up ([3,2,0], p3) acting ([3,2,0], p3) [root@cephosd5

Re: [ceph-users] Can ceph-deploy be used with 'osd objectstore = keyvaluestore-dev' in config file ?

Sorry for the poor document, it's in progress. KeyValueStore doesn't need journal, so we may need some change for ceph-disk. But I'm not familiar to it. KeyValueStore is a experiment backend, so it's still exists some bugs in Firefly version. And there no existing bug in master branch. On Fri, S

Re: [ceph-users] ceph mds unable to start with 0.85

On Thu, Sep 18, 2014 at 5:35 PM, 廖建锋 wrote: > if i turn on debug=20, the log will be more than 100G, > > looks no way to put, do you have any other good way to figure it out? It should compress well and you can use ceph-post-file if you don't have a place to host it yourself. -Greg > > would yo

[ceph-users] ceph health related message

Hi, I've setup a cluster with 3 monitors and 2 OSD nodes with 2 disks each.Cluster is in active+clean state. But, "ceph -s" keeps throwing the following message, every other time "ceph -s" is run. #ceph -s 2014-09-19 04:13:07.116662 7fc88c3f9700 0 -- :/1011833 >> *192.168.240.200

Re: [ceph-users] Can ceph-deploy be used with 'osd objectstore = keyvaluestore-dev' in config file ?

On 19/09/14 15:11, Aegeaner wrote: I noticed ceph added key/value store OSD backend feature in firefly, but i can hardly get any documentation about how to use it. At last I found that i can add a line in ceph.conf: osd objectstore = keyvaluestore-dev but got failed with ceph-deploy creating O

[ceph-users] Troubleshooting down OSDs: Invalid command: ceph osd start osd.1

Has the command for manually starting and stopping OSDs changed? The documentation for troubleshooting OSDs (http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-osd/) mentions restarting OSDs with the command; ceph osd start osd.{num} Yet I find, using Firefly 0.80.5 piers

[ceph-users] Fwd: Troubleshooting down OSDs: Invalid command: ceph osd start osd.1

Also, using the init.d framework seams to fail. piers@sol:/etc/ceph$ sudo service ceph start osd.1 /etc/init.d/ceph: osd.1 not found (/etc/ceph/ceph.conf defines , /var/lib/ceph defines ) The disks are mounted piers@sol:~$ cat /etc/mtab | sort cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpu

Re: [ceph-users] Can ceph-deploy be used with 'osd objectstore = keyvaluestore-dev' in config file ?

On 19/09/14 18:02, Mark Kirkwood wrote: On 19/09/14 15:11, Aegeaner wrote: I noticed ceph added key/value store OSD backend feature in firefly, but i can hardly get any documentation about how to use it. At last I found that i can add a line in ceph.conf: osd objectstore = keyvaluestore-dev b

Re: [ceph-users] ceph issue: rbd vs. qemu-kvm

Hi Steven, I am not sure what else would be different, the stanza you shown is similar to those I have. The only possibility maybe that I am using a different linux distro? FYI, I am using Ubuntu 12.04 but I had Ubuntu CloudArchive's Havana repo enabled (https://wiki.ubuntu.com/ServerTeam/Clou