Re: [ceph-users] Ceph mds laggy and failed assert in function replay mds/journal.cc

Hi, Assuming we got MDS working back on track, should we still leave the mds_wipe_sessions in the ceph.conf or remove it and restart MDS. Thanks. Regards, Luke On Tue, Apr 29, 2014 at 2:12 PM, Yan, Zheng wrote: > On Tue, Apr 29, 2014 at 11:24 AM, Jingyuan Luke wrote: >> Hi, >> >> We had appli

Re: [ceph-users] Ceph mds laggy and failed assert in function replay mds/journal.cc

On Tue, Apr 29, 2014 at 3:13 PM, Jingyuan Luke wrote: > Hi, > > Assuming we got MDS working back on track, should we still leave the > mds_wipe_sessions in the ceph.conf or remove it and restart MDS. > Thanks. No. It has been several hours. the MDS still does not finish replaying the journal? R

Re: [ceph-users] Ceph mds laggy and failed assert in function replay mds/journal.cc

Hi, MDS did finish the replay and working after that but we are wondering should we leave the mds_wipe_sessions in ceph.conf or remove it. Regards, Luke -Original Message- From: ceph-users-boun...@lists.ceph.com [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Yan, Zheng Sent: T

Re: [ceph-users] Access denied error

Hi Cedric/Yehuda, I have generated the signature dynamically as like in this documentation http://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthentication.html by using a reference http://birkoff.net/blog/amazon-s3-query-string-authentication-using-php/ and my code is like

[ceph-users] Regarding Rados Gateway access Info

Hi Yehuda, I have configured cluster and its health is active and clean like below *root@mon:/etc/ceph# ceph statuscluster a7f64266-0894-4f1e-a635-d0aeaca0e993 health HEALTH_WARN 110 pgs stuck unclean monmap e1: 1 mons at {mon=192.168.0.102:6789/0

[ceph-users] Kernel-client crash when adding OSDs

Hi! A few days ago I added a bunch of new OSDs to hour production servers, we have 3 hosts that maps and unmaps hundreds of RBD devices every day. with aprox 100 RBD devs mapped at each host at any given time. Adding OSDs is usually quite smooth if you do it the right way. I usually follow the Ma

Re: [ceph-users] What happened if rbd lose a block?

On 04/28/2014 05:59 PM, Timofey Koolin wrote: Is a setting for change the behavior to return read error instead of block read? I think it is more reasonable behavior because it is similar to bad block on HDD: it can’t be read. Or may be a timeout some seconds, then return read error for the bl

Re: [ceph-users] Regarding Rados Gateway access Info

this means that there is no limit to the size of buckets created by this user, bucket quota is not enabled. you can enable with: radosgw-admin quota enable --uid shrinivas radosgw-admin quota set --uid shrinivas On 04/29/2014 09:12 AM, Srinivasa Rao Ragolu wrote: Hi Yehuda, I have config

[ceph-users] Hey, Where can I find the source code of " class ObjectOperationImpl " ?

Hey, I can find a declaration in librados.hpp , but when I try to find the source code of ObjectOperatoinImpl , I find nothing .. Is it a ghost class?? Confused.. Appreciate any help .___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] Hey, Where can I find the source code of " class ObjectOperationImpl " ?

It's not a real class, just a type definition used for the ObjectOperation::impl pointer. The actual object is an ObjectOperation. src/librados/librados.cc 1797: impl = (ObjectOperationImpl *)new ::ObjectOperation; John On Tue, Apr 29, 2014 at 10:49 AM, peng wrote: > Hey, > I can find a dec

Re: [ceph-users] What happened if rbd lose a block?

>From time to time old monitor can’t connect to new monitors. For example upgrade from 0.66 to 0.67 http://ceph.com/docs/master/release-notes/#upgrading-from-v0-66 • There is monitor internal protocol change, which means that v0.67 ceph-mon daemons cannot talk to v0.66 or older daemons.

[ceph-users] Unable to bring cluster up

After a simple "service ceph restart" on a server, i'm unable to get my cluster up again http://pastebin.com/raw.php?i=Wsmfik2M suddenly, some OSDs goes UP and DOWN randomly. I don't see any network traffic on cluster interface. How can I detect what ceph is doing ? From the posted output there i

[ceph-users] Information needed

Hi All, I have created a ceph volume in a pool which I have defined. A hard drive is also attached to the ceph node. How do we know on which hard drive ceph created this volume on ? What is its path ? Thanks Kumar This message is for the designated recipie

Re: [ceph-users] What happened if rbd lose a block?

> Le 28 avr. 2014 à 17:59, Timofey Koolin a écrit : > > Now if I will upgrade monitors and upgrade will fail on second (of three) > monitor - cluster will down. Becouse it will have > 1 new monitor > 1 down monitor > 1 old monitor As of my compréhension, migration path is planned, between cep

Re: [ceph-users] What happened if rbd lose a block?

See to my next message: >From time to time old monitor can’t connect to new monitors. For example upgrade from 0.66 to 0.67 http://ceph.com/docs/master/release-notes/#upgrading-from-v0-66 • There is monitor internal protocol change, which means that v0.67 ceph-mon daemons cannot talk to

Re: [ceph-users] What happened if rbd lose a block?

On 04/29/2014 11:59 AM, Timofey Koolin wrote: From time to time old monitor can’t connect to new monitors. For example upgrade from 0.66 to 0.67 http://ceph.com/docs/master/release-notes/#upgrading-from-v0-66 • There is monitor internal protocol change, which means that v0.67 ceph-mon

Re: [ceph-users] What happened if rbd lose a block?

On Tue, Apr 29, 2014 at 01:13:25PM +0200, Wido den Hollander wrote: > When you go from the major release to another one there is no > problem. Dumpling -> Emperor -> Firefly, etc. > > That should all work without downtime. I can confirm that upgrading production instances since bobtail did never

Re: [ceph-users] What happened if rbd lose a block?

Is this mean about I can stop updates for any potential downtime operation, then update to next release without downtime? > On Tue, Apr 29, 2014 at 01:13:25PM +0200, Wido den Hollander wrote: >> When you go from the major release to another one there is no >> problem. Dumpling -> Emperor -> Firef

[ceph-users] RAID implementation.

Hi All, I have setup a three node ceph storage cluster on Ubunut. I want to implement RAID 5, RAID 1 and RAID 1+0 volumes using ceph. Any information or link providing information on this will help a lot. Thanks Kumar This message is for the designated recipie

Re: [ceph-users] rbd Problem with disk full

Hello, 1) RBD is sparsely allocated, meaning that creating a 10TB volume would initially take up close to no space at all. 2) It is also completely file system agnostic (as the B for block indicates, a misconception that creeps up with DRBD all the time, too). That means that when you write 8T

Re: [ceph-users] Slow RBD Benchmark Compared To Direct I/O Test

On Thu, 24 Apr 2014 13:51:49 +0800 Indra Pramana wrote: > Hi Christian, > > Good day to you, and thank you for your reply. > > On Wed, Apr 23, 2014 at 11:41 PM, Christian Balzer wrote: > > > > > > Using 32 concurrent writes, result is below. The speed really > > > > > fluctuates. > > > > > > >

Re: [ceph-users] Only one OSD log available per node?

Hmm, log handling is a bit outside my expertise. You should be able to get a "config dump" from the OSD's admin socket; can you paste that? (It includes the log file locations and log levels.) If you restart a particular OSD, does it write anything to its log file at that time? -Greg Software Engin

[ceph-users] Problem with OSD not activating

Hi, My OSD disks wont go past "prepared" state, it's been like this for a while now, anyone knows why, i tried repeating the steps from disk zapping but still won't go past "prepared" state http://paste.openstack.org/show/77580/ ___ ceph-users mailing l

[ceph-users] (no subject)

___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] osd can not start

hello, My many osds can't not start , now I lost lots of data , how can I solve this issue? all the failed osd have a "bad op 39" signal. what happen ? help me ! the osd log are list below: --- -17> 2014-04-29 22:16:06.432347 7f1006d2b7a0 3 journal journal_replay:

[ceph-users] Mons deadlocked after they all died

Hi, still working on a troubled ceph cluster running .61.2-1raring consisting of (currently) 4 monitors a,b,c,g with g being a newly added monitor that failed/fails to sync up, so consider that one down. Now mon a and b died because for some (currently unknown) reason linux created a core dump on

Re: [ceph-users] osd can not start

op 39 is a setallochint op, it seems that you run a very new release. Is any info can be provided before you stop your osd and start it then? On Tue, Apr 29, 2014 at 10:19 PM, 飞 wrote: > hello, My many osds can't not start , now I lost lots of data , how can I > solve this issue? > all the faile

Re: [ceph-users] RAID implementation.

No sense doing RAID for ceph! 2014-04-29 15:58 GMT+04:00 : > Hi All, > > > > I have setup a three node ceph storage cluster on Ubunut. I want to > implement RAID 5, RAID 1 and RAID 1+0 volumes using ceph. > > Any information or link providing information on this will help a lot. > > > > > > Tha

Re: [ceph-users] packages for Trusty

Hello list, It appears there are Trusty packages available from the ceph.com repos as I installed Emperor (0.72) from them on Trusty yesterday. The sources.list entry was: deb http://ceph.com/debian-emperor/ trusty main I also had to pin the packages from this repo to avoid installing the T

[ceph-users] Automating radosgw provisioning lifecycle

Hi there, I am getting to the Ceph party a little late but I am trying to find out if any work has already been done on trying to automate the provisioning lifecycle of users, etc in radosgw? A few days ago I started trying to write a PHP script that created a user using the adminops API and I

Re: [ceph-users] osd can not start

On Tue, 29 Apr 2014, Haomai Wang wrote: > op 39 is a setallochint op, it seems that you run a very new release. > Is any info can be provided before you stop your osd and start it > then? Perhaps you ran a new release for a while and then downgraded? Upgrading to the firefy 0.80-rc will make tha

Re: [ceph-users] Information needed

That's not quite how Ceph works. I recommend perusing some of the introductory documentation at ceph.com/docs, but in short: When you set up a ceph pool, you are specifying groups of hard drives which will be used together. When you create an RBD volume in a pool, you are saying "I want this volume

Re: [ceph-users] Mons deadlocked after they all died

Monitor keys don't change; I think something else must be going on. Did you remove any of their stores? Are the local filesystems actually correct (fsck)? The ceph-create-keys is a red herring and will stop as soon as. The monitors do get into a quorum. -Greg On Tuesday, April 29, 2014, Marc wro

Re: [ceph-users] Mons deadlocked after they all died

'ls' on the respective stores in /var/lib/ceph/mon/ceph.X/store.db returns a list of files (i.e. still present), fsck seems fine. I did notice that one of the nodes has different contents in the /var/lib/ceph/mon/ceph-b/keyring i.e. its key is different from the other 2 nodes'. That shouldn't be th

Re: [ceph-users] Copying RBD images between clusters?

'rbd export | rbd import' is pretty slow. Any other ideas here? It's significantly faster for me to export an image with qemu-img, scp it over, and import it again with qemu-img. I'm not sure why the rbd tools are so much slower then the qemu tools (the backend remains the same, so it has t

Re: [ceph-users] Mons deadlocked after they all died

On Tue, Apr 29, 2014 at 9:48 AM, Marc wrote: > 'ls' on the respective stores in /var/lib/ceph/mon/ceph.X/store.db > returns a list of files (i.e. still present), fsck seems fine. I did > notice that one of the nodes has different contents in the > /var/lib/ceph/mon/ceph-b/keyring i.e. its key is d

Re: [ceph-users] Unable to bring cluster up

You'll need to go look at the individual OSDs to determine why they aren't on. All the cluster knows is that the OSDs aren't communicating properly. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com On Tue, Apr 29, 2014 at 3:06 AM, Gandalf Corvotempesta wrote: > After a simple "

Re: [ceph-users] OSD could not start and PG down & incomplete

It looks like the OSD is expecting a file to be there, and it is, but it's incorrectly empty or something. Did you lose power to the node? Have you run fsck on the local filesystem? -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com On Tue, Apr 29, 2014 at 3:05 AM, vernon1...@126.

Re: [ceph-users] Ceph unstable when upgrading from emperor (v0.72.2) to firefly (v0.80-rc1-16-g2708c3c)

Hmm, I think this might actually be another instance of http://tracker.ceph.com/issues/8232, which was just reported yesterday. That said, I think that if you restart one OSD at a time, you should be able to avoid the race condition. It was restarting all of them simultaneously that got you into tr

Re: [ceph-users] Only one OSD log available per node?

Is there a way I can verify if the logs are actually being written by the ceph-osd processes? Try bumping up the logging level temporarily. This makes my make osd.0 write to it's log every second: ceph tell osd.0 injectargs '--debug-osd 5/5' Turn it off with: ceph tell osd.0 injectarg

Re: [ceph-users] Automating radosgw provisioning lifecycle

On 4/29/14 08:29 , Drew Weaver wrote: I am getting to the Ceph party a little late but I am trying to find out if any work has already been done on trying to automate the provisioning lifecycle of users, etc in radosgw? I started writing some Chef recipes. I intended to manage users, pool

[ceph-users] Ceph 0.72.2 installation on Ubuntu 12.04.4 LTS never got active + сlean

Hello all, I have tried to install subj. almost 10 times from the beginning using official guides: http://ceph.com/docs/master/start/ and older one http://eu.ceph.com/docs/wip-6919/start/quick-start/ The result is always the same. At the end of installation I

Re: [ceph-users] Ceph 0.72.2 installation on Ubuntu 12.04.4 LTS never got active + сlean

Hi Vadim, The default distribution rule is now to split over hosts. If you only have one host Ceph will not be able to replicate your data. If you need to replicate within a single host you will have to update your crush rules from "step chooseleaf firstn 0 type host" to "step chooseleaf fir

Re: [ceph-users] Ceph 0.72.2 installation on Ubuntu 12.04.4 LTS never got active + сlean

Have just looked at the documentation Vadim was trying to use to set up a cluster and http://eu.ceph.com/docs/wip-6919/start/quick-start/ should really be updated or removed as it will not result in a working cluster with recent Ceph versions. -Michael On 29/04/2014 21:09, Michael wrote: Hi

Re: [ceph-users] Automating radosgw provisioning lifecycle

Le 29/04/2014 20:39, Craig Lewis a écrit : > On 4/29/14 08:29 , Drew Weaver wrote: >> I am getting to the Ceph party a little late but I am trying to find >> out if any work has already been done on trying to automate the >> provisioning lifecycle of users, etc in radosgw? > I started writing some

Re: [ceph-users] Automating radosgw provisioning lifecycle

On 4/29/14 13:56 , Cedric Lemarchand wrote: Le 29/04/2014 20:39, Craig Lewis a écrit : On 4/29/14 08:29 , Drew Weaver wrote: I am getting to the Ceph party a little late but I am trying to find out if any work has already been done on trying to automate the provisioning lifecycle of users, etc

Re: [ceph-users] Mons deadlocked after they all died

Thank you for the help so far! I went for option 1 and that did solve that problem. However quorum has not been restored. Here's the information I can get: mon a+b are in state Electing and have been for more than 2 hours now. mon c does reply to "help" by using the socket, but it does not respond

Re: [ceph-users] Mons deadlocked after they all died

On Tue, Apr 29, 2014 at 3:28 PM, Marc wrote: > Thank you for the help so far! I went for option 1 and that did solve > that problem. However quorum has not been restored. Here's the > information I can get: > > mon a+b are in state Electing and have been for more than 2 hours now. > mon c does rep

[ceph-users] Ceph Object Storage front-end?

Hi all, Is there some kind of web-based or WebDAV-based front-end for accessing a Ceph cluster? Our situation is sometimes we have big blobs that we'd like to stash somewhere safe, things like customer database backups, etc. Things other than disk images. We haven't deployed CephFS at this stag

[ceph-users] primary osd question

hello, I want to change a pg's primary osd Manually, how to do it ? thank you___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph Benchmark 0.79

Hi, A little confused over how to perform benchmark tests on Ceph 0.79. Would someone help clarify the operation please? Per this article 'http://ceph.com/docs/master/rados/operations/control/?highlight=bench', the benchmark syntax should be: ceph osd tell N bench [BYTES_PER_WRITE] [T

[ceph-users] mkcephfs questions

Hi, I setup ceph cluster thru mkcephfs command, after I enter "ceph -s", it always returns 4950 stuck unclean pgs. I tried the same "ceph -s" after 12 hrs, there still returns the same unclean pgs number, nothing changed. Does mkcephfs always has the problem or I did something wrong? I attach

Re: [ceph-users] mkcephfs questions

The result of "ceph -s" should tell you the reason. There only exists 21 OSD up but we need 24 OSDs On Wed, Apr 30, 2014 at 11:21 AM, Cao, Buddy wrote: > Hi, > > > > I setup ceph cluster thru mkcephfs command, after I enter “ceph –s”, it > always returns 4950 stuck unclean pgs. I tried the same “

[ceph-users] ceph-deploy - how to get in-memory ceph.conf

Hi, After I setup ceph cluster via ceph-deploy, how do I get the ceph.conf? I did not see any ceph.conf generated on storage node, is it in memory? Wei Cao (Buddy) ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.c

[ceph-users] Ceph 0.80 and Ubuntu 14.04

Latest Ceph release is Firefly v0.80 right? Or is it still in beta? And Ubuntu is on 14.04. Will I be able to install ceph 0.80 on Ubuntu 14.04 for production? If not, what is the time frame on when can I install the ceph v0.80 on ubuntu 14.04? -- Thanks, Kenneth _

Re: [ceph-users] mkcephfs questions

Thanks your reply, Haomai. What I don't understand is that, why the stuck unclean pgs keep the same numbers after 12 hours. It's the common behavior or not? Wei Cao (Buddy) -Original Message- From: Haomai Wang [mailto:haomaiw...@gmail.com] Sent: Wednesday, April 30, 2014 11:36 AM To:

[ceph-users] [SOLVED] RE: Ceph 0.72.2 installation on Ubuntu 12.04.4 LTS never got active + сlean

Thank you Michael, It works like a charm ! I got cluster c887f570-bd8d-4aa3-bc63-b8ff959bd03f health HEALTH_OK monmap e1: 1 mons at {storage=172.16.3.2:6789/0}, election epoch 2, quorum 0 storage osdmap e18: 2 osds: 2 up, 2 in pgmap v33: 192 pgs, 3 pools, 0 b

Re: [ceph-users] Ceph 0.80 and Ubuntu 14.04

hi, Ubuntu 14.04 currently ships ceph 0.79. After firefly release ubuntu maintainer will update ceph version in ubuntu's repos. On 2014.04.30 07:08, Kenneth wrote: > > Latest Ceph release is Firefly v0.80 right? Or is it still in beta? > And Ubuntu is on 14.04. > > Will I be able to install ceph

Re: [ceph-users] primary osd question

On Wed, Apr 30, 2014 at 6:52 AM, 飞 wrote: > hello, I want to change a pg's primary osd Manually, how to do it ? thank > you If you want to move the load off of that osd, you can use the primary affinity knob: ceph osd primary-affinity osd. 0 It shifts "primaryness" away from overloaded osds and

Re: [ceph-users] ceph-deploy - how to get in-memory ceph.conf

/etc/ceph/ceph.conf in any ceph node. On Wed, Apr 30, 2014 at 11:36 AM, Cao, Buddy wrote: > Hi, > > > > After I setup ceph cluster via ceph-deploy, how do I get the ceph.conf? I > did not see any ceph.conf generated on storage node, is it in memory? > > > > > > Wei Cao (Buddy) > > > > >

Re: [ceph-users] mkcephfs questions

Hmm, it should be another problem plays. Maybe more logs could explain it. ceph.log ceph-mon.log On Wed, Apr 30, 2014 at 12:06 PM, Cao, Buddy wrote: > Thanks your reply, Haomai. What I don't understand is that, why the stuck > unclean pgs keep the same numbers after 12 hours. It's the common be

Re: [ceph-users] ceph-deploy - how to get in-memory ceph.conf

Thanks for your reply Haomai. There is no /etc/ceph/ceph.conf on any ceph nodes, that is why I raised the question at beginning. It's what ceph-deploy shall do? Some people tell me ceph-deploy don't distribute ceph.conf to ceph node, but stored in memory, is it true? Wei Cao (Buddy) -Orig