[ceph-users] radosgw create user using API

[Alvaro Izquierdo Jimeno]

Hi,

Reading the URL http://ceph.com/docs/next/radosgw/adminops/#create-user , I'm 
trying to create a new user with:

curl -v -X PUT  -d '{"uid": "alvaro", "display-name": "alvaro"} 
http://myradosgw/admin/user?format=json

and a 403 response is received :(

So, Do I need a token? A token of who? Do I need a superuser with a special  
feature? Or I have  to setup something special in radosgw part into ceph.conf?

Many thanks in advanced and best regards,
Álvaro.


Verificada la ausencia de virus por G Data AntiVirus
Versión: AVA 22.11060 del 18.07.2013
Noticias de virus: www.antiviruslab.com___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] mon down for 3 hours after clocksource glitch

[Dan van der Ster]

Hi,  
Last night our cluster became unhealthy for 3 hours after one of the mons (a 
qemu-kvm VM) had this glitch:

Jul 18 00:12:43 andy03 kernel: Clocksource tsc unstable (delta = -60129537028 
ns).  Enable clocksource failover by adding clocksource_failover kernel 
parameter.

shortly afterwords the mon.2 log said:

2013-07-18 00:13:01.147770 7feca7a5d700  1 mon.2@1(probing).data_health(52) 
service_dispatch not in quorum -- drop message
2013-07-18 00:13:01.148622 7feca7a5d700  1 mon.2@1(synchronizing sync( 
requester state start )) e3 sync_obtain_latest_monmap
2013-07-18 00:13:01.148786 7feca7a5d700  1 mon.2@1(synchronizing sync( 
requester state start )) e3 sync_obtain_latest_monmap obtained monmap e3
2013-07-18 00:14:01.208569 7feca845e700  0 mon.2@1(synchronizing sync( 
requester state chunks )).data_health(52) update_stats avail 59% total 8547036 
used 2993036 avail 5119824
2013-07-18 00:15:01.298686 7feca845e700  0 mon.2@1(synchronizing sync( 
requester state chunks )).data_health(52) update_stats avail 58% total 8547036 
used 3074968 avail 5037892
2013-07-18 00:16:08.455941 7feca845e700  0 mon.2@1(synchronizing sync( 
requester state chunks )).data_health(52) update_stats avail 58% total 8547036 
used 3147732 avail 4965128
…


and that continued for over three hours until:

2013-07-18 03:32:33.991232 7f334ecd0700  0 mon.2@1(synchronizing sync( 
requester state chunks )).data_health(0) update_stats avail 56% total 8547036 
used 3260064 avail 4852796
2013-07-18 03:33:34.314538 7f334ecd0700  0 mon.2@1(synchronizing sync( 
requester state chunks )).data_health(0) update_stats avail 56% total 8547036 
used 3294832 avail 4818028
2013-07-18 03:34:05.285568 7f334e2cf700  0 log [INF] : mon.2 calling new 
monitor election
2013-07-18 03:34:05.285747 7f334e2cf700  1 mon.2@1(electing).elector(52) init, 
last seen epoch 52


In the meantime I tried restarting each ceph-mon daemon, but that didn't speed 
up the recovery.

We are talking with our OpenStack team to understand if they can provide a more 
stable clocksource, but we wanted to better understand why Ceph was so 
sensitive to this glitch. It is good that mon.2 managed to recover eventually, 
but does anyone have an idea why it took 3 hours??!!
thx, Dan

--  
Dan van der Ster
CERN IT-DSS

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] How to remove stale pgs?

[Ta Ba Tuan]

Hi all,

I have 4 (stale+inactive) pgs, how to delete those pgs?

pgmap v59722: 21944 *pgs: 4 stale,* 12827 active+clean, 9113 
active+degraded; 45689 MB data, 1006 GB used, 293 TB / 294 TB avail;


I found on google a long time, still can't resolve it.
Please, help me!

Thank you so much.

--tuantaba
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Problem executing ceph-deploy on RHEL6

[jose.valeriooropeza]

Hello,

I've deployed a Ceph cluster consisting of 5 server nodes and a Ceph client 
that will hold the mounted CephFS.

>From the client I want to deploy the 5 servers with the ceph-deploy tool.

I installed Ceph from this repository: http://ceph.com/rpm-cuttlefish/el6/x86_64
And ceph-deploy from this one: http://ceph.com/rpm-cuttlefish/el6/noarch/
(BTW: I found myself the second repository, there's no sign of it in the 
instructions - as far as I know)

I made sure that the client can automatically SSH the servers with the command 
"ssh cephserverX", where X goes from 1 to 5. There's a "ceph" user in each of 
them, with passwordless sudo.

I executed first: ceph-deploy new cephserver2

Then, I executed this command: ceph-deploy install cephserver2

I received the following error:

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pushy/client.py", line 383, in __init__
self.modules = AutoImporter(self)
  File "/usr/lib/python2.6/site-packages/pushy/client.py", line 236, in __init__
remote_compile = self.__client.eval("compile")
  File "/usr/lib/python2.6/site-packages/pushy/client.py", line 478, in eval
return self.remote.eval(code, globals, locals)
  File "/usr/lib/python2.6/site-packages/pushy/protocol/connection.py", line 
54, in eval
return self.send_request(MessageType.evaluate, args)
  File "/usr/lib/python2.6/site-packages/pushy/protocol/baseconnection.py", 
line 315, in send_request
m = self.__waitForResponse(handler)
  File "/usr/lib/python2.6/site-packages/pushy/protocol/baseconnection.py", 
line 420, in __waitForResponse
m = self.__recv()
  File "/usr/lib/python2.6/site-packages/pushy/protocol/baseconnection.py", 
line 601, in __recv
m = self.__istream.receive_message()
  File "/usr/lib/python2.6/site-packages/pushy/protocol/baseconnection.py", 
line 104, in receive_message
return Message.unpack(self.__file)
  File "/usr/lib/python2.6/site-packages/pushy/protocol/message.py", line 96, 
in unpack
header = read(file, Message.PACKING_SIZE)
  File "/usr/lib/python2.6/site-packages/pushy/protocol/message.py", line 60, 
in read
raise IOError, "End of file"
IOError: End of file

[remote] sudo: sorry, you must have a tty to run sudo

Traceback (most recent call last):
  File "/usr/bin/ceph-deploy", line 21, in 
main()
  File "/usr/lib/python2.6/site-packages/ceph_deploy/cli.py", line 112, in main
return args.func(args)
  File "/usr/lib/python2.6/site-packages/ceph_deploy/install.py", line 364, in 
install
sudo = args.pushy(get_transport(hostname))
  File "/usr/lib/python2.6/site-packages/pushy/client.py", line 583, in connect
return PushyClient(target, **kwargs)
  File "/usr/lib/python2.6/site-packages/pushy/client.py", line 383, in __init__
self.modules = AutoImporter(self)
  File "/usr/lib/python2.6/site-packages/pushy/client.py", line 236, in __init__
remote_compile = self.__client.eval("compile")
  File "/usr/lib/python2.6/site-packages/pushy/client.py", line 478, in eval
return self.remote.eval(code, globals, locals)
  File "/usr/lib/python2.6/site-packages/pushy/protocol/connection.py", line 
54, in eval
return self.send_request(MessageType.evaluate, args)
  File "/usr/lib/python2.6/site-packages/pushy/protocol/baseconnection.py", 
line 315, in send_request
m = self.__waitForResponse(handler)
  File "/usr/lib/python2.6/site-packages/pushy/protocol/baseconnection.py", 
line 420, in __waitForResponse
m = self.__recv()
  File "/usr/lib/python2.6/site-packages/pushy/protocol/baseconnection.py", 
line 601, in __recv
m = self.__istream.receive_message()
  File "/usr/lib/python2.6/site-packages/pushy/protocol/baseconnection.py", 
line 104, in receive_message
return Message.unpack(self.__file)
  File "/usr/lib/python2.6/site-packages/pushy/protocol/message.py", line 96, 
in unpack
header = read(file, Message.PACKING_SIZE)
  File "/usr/lib/python2.6/site-packages/pushy/protocol/message.py", line 60, 
in read
raise IOError, "End of file"
IOError: End of file

What should I do?

Thanks in advance for your help.

José Valerio

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Problem executing ceph-deploy on RHEL6

[Oliver Fuckner]

*snip*

 raise IOError, "End of file"
IOError: End of file
[remote] sudo: sorry, you must have a tty to run sudo



Did you comment out #Defaults requiretty in /etc/sudoers?
That worked for me.

 Oliver

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Problem executing ceph-deploy on RHEL6

[jose.valeriooropeza]

I did it now and it partially solved the problem. Thanks!

However, now I face another error:

curl: (7) couldn't connect to host
error: https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc: import 
read failed(2).
Traceback (most recent call last):
  File "/usr/bin/ceph-deploy", line 21, in 
main()
  File "/usr/lib/python2.6/site-packages/ceph_deploy/cli.py", line 112, in main
return args.func(args)
  File "/usr/lib/python2.6/site-packages/ceph_deploy/install.py", line 384, in 
install
version=version,
  File "/usr/lib/python2.6/site-packages/pushy/protocol/proxy.py", line 255, in 

(conn.operator(type_, self, args, kwargs))
  File "/usr/lib/python2.6/site-packages/pushy/protocol/connection.py", line 
66, in operator
return self.send_request(type_, (object, args, kwargs))
  File "/usr/lib/python2.6/site-packages/pushy/protocol/baseconnection.py", 
line 323, in send_request
return self.__handle(m)
  File "/usr/lib/python2.6/site-packages/pushy/protocol/baseconnection.py", 
line 639, in __handle
raise e
pushy.protocol.proxy.ExceptionProxy: Command 'su -c 'rpm --import 
"https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc";'' returned 
non-zero exit status 1



when I type the command 'su -c 'rpm --import 
"https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc";' directly as 
user ceph in cephserver2, it works.

Thanks in advance for your help

José

> -Original Message-
> From: Oliver Fuckner [mailto:o...@fuckner.net]
> Sent: Thursday, July 18, 2013 1:44 PM
> To: Valerio Oropeza José, ITS-CPT-DEV-TAD
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Problem executing ceph-deploy on RHEL6
> 
> 
> *snip*
> >  raise IOError, "End of file"
> > IOError: End of file
> > [remote] sudo: sorry, you must have a tty to run sudo
> 
> 
> Did you comment out #Defaults requiretty in /etc/sudoers?
> That worked for me.
> 
>   Oliver

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Problem executing ceph-deploy on RHEL6

[Terekhov, Mikhail]

> -Original Message-
> From: ceph-users-boun...@lists.ceph.com [mailto:ceph-users-
> boun...@lists.ceph.com] On Behalf Of jose.valerioorop...@swisscom.com
> Sent: Thursday, July 18, 2013 8:12 AM
> To: o...@fuckner.net
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Problem executing ceph-deploy on RHEL6
> 
> I did it now and it partially solved the problem. Thanks!
> 
> However, now I face another error:
> 
> curl: (7) couldn't connect to host
> error: https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc:
> import read failed(2).
> Traceback (most recent call last):
>   File "/usr/bin/ceph-deploy", line 21, in 
> main()
>   File "/usr/lib/python2.6/site-packages/ceph_deploy/cli.py", line 112, in 
> main
> return args.func(args)
>   File "/usr/lib/python2.6/site-packages/ceph_deploy/install.py", line 384, in
> install
> version=version,
>   File "/usr/lib/python2.6/site-packages/pushy/protocol/proxy.py", line 255,
> in 
> (conn.operator(type_, self, args, kwargs))
>   File "/usr/lib/python2.6/site-packages/pushy/protocol/connection.py", line
> 66, in operator
> return self.send_request(type_, (object, args, kwargs))
>   File "/usr/lib/python2.6/site-packages/pushy/protocol/baseconnection.py",
> line 323, in send_request
> return self.__handle(m)
>   File "/usr/lib/python2.6/site-packages/pushy/protocol/baseconnection.py",
> line 639, in __handle
> raise e
> pushy.protocol.proxy.ExceptionProxy: Command 'su -c 'rpm --import
> "https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc";''
> returned non-zero exit status 1
> 
> 
> 
> when I type the command 'su -c 'rpm --import
> "https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc";' directly
> as user ceph in cephserver2, it works.
> 

I had similar problem on SLES11 SP2 but in my case it didn't work from the 
command line either. When I run the command under strace I've found that WEB 
server returns an error saying that my client tries to access https resource 
but can't talk SSL protocol. Changing https to http in this URL solved the 
problem.

Regards,
Mikhail
 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Problems with tgt with ceph support

[Toni F. [ackstorm]]

Hi Dan, thanks for your reply

Yes, as you has said:

"RBD Device /dev/rbd/rbd/iscsi-image-part1 exported with tgt"  is kernel 
block device and "the TGT-RBD connector" is stgt/tgtd.


My tests was been done exporting it with tgt without multipath. It seems 
that are no very big difference in this tests. The real problem is not that.


I want to use tgt to connect the storage to an ESX but it presents a few 
problems.


With kernel block device, i can't use multipath (it turns inestable and 
turns a corrupted data)
With stgt/tgtd rbd implementation i can use multipath but the 
performance is very poor.


To imaginate the difference, Deploy a VM with 1st (3min) but with 2nd 
option (83min).


I hope your understand me. I'm newbie with this concepts.

A lot of thanks

On 16/07/13 04:23, Dan Mick wrote:
Apologies; I don't really understand the results.  They're labeled 
"RBD Device /dev/rbd/rbd/iscsi-image-part1 exported with tgt" and
"TGT-RBD connector".  Is the former nothing to do with tgt (i.e., just 
the kernel block device), and the latter is stgt/tgtd?


Do you interpret them to say the first run gave avg 258MB/s, and the 
second run 198MB/s?


On 07/15/2013 02:41 AM, Toni F. [ackstorm] wrote:

Here's my results.

The performance test was seq write

On 15/07/13 10:12, Toni F. [ackstorm] wrote:

I'm going to do a performance test with fio to see the difference.

Regards

On 12/07/13 18:15, Dan Mick wrote:


Ceph performance is a very very complicated subject. How does that
compare to other access methods?  Say, rbd import/export for an easy
test?

On Jul 12, 2013 8:22 AM, "Toni F. [ackstorm]"
mailto:toni.fuen...@ackstorm.es>> wrote:

It works, but the performance is very poor. 100MB/s or less

Which are your performance experience?

Regards

On 12/07/13 13:56, Toni F. [ackstorm] wrote:

It works!

Thanks for all

On 12/07/13 11:23, Toni F. [ackstorm] wrote:

Yes! it seems that i wasn't compiled the rbd support.

System:
State: ready
debug: off
LLDs:
iscsi: ready
Backing stores:
bsg
sg
null
ssc
aio
rdwr (bsoflags sync:direct)
Device types:
disk
cd/dvd
osd
controller
changer
tape
passthrough
iSNS:
iSNS=Off
iSNSServerIP=
iSNSServerPort=3205
iSNSAccessControl=Off

I'm going to recompile it
Thanks a lot!

On 11/07/13 07:45, Dan Mick wrote:



On 07/10/2013 04:12 AM, Toni F. [ackstorm] wrote:

Hi all,

I have installed the v0.37 of tgt.

To test this feature i follow the

http://ceph.com/dev-notes/adding-support-for-rbd-to-stgt/

guide

When i launch the command:

tgtadm --lld iscsi --mode logicalunit --op new
--tid 1 --lun 0
--backing-store iscsi-image --bstype rbd

fails

First i think that lun cannot be 0 because lun 0
is used by the
controller (previous command)


This worked when I first wrote the backend, but tgt
may have changed; I'll investigate and change the
blog entry if so.  Thanks.


If i launch the correct command with lun 1 i have
this error:

tgtadm: invalid request

In syslog:

Jul 10 12:54:03 datastore-lnx001 tgtd:
device_mgmt(245) sz:28
params:path=iscsi-image,bstype=rbd
Jul 10 12:54:03 datastore-lnx001 tgtd:
tgt_device_create(532) failed to
find bstype, rbd

What's wrong? not supported?



Where did you get your tgtd?  Was it built with rbd
support (CEPH_RBD defined
in the environment for make)?

sudo ./tgtadm --lld iscsi --op show --mode system

should tell you.

How did you set up access to ceph.conf?








--

Toni Fuentes Rico
toni.fuen...@ackstorm.es 
Administración de Sistemas

Oficina central: 902 888 345

ACK STORM, S.L.
ISO 9001:2008 (Cert.nº. 536932)
http://ackstorm.es

Este mensaje electrónico contiene información de ACK STORM, S.L.
que es privada y confidencial, siendo para el uso exclusivo de la
persona(s) o entidades arriba  mencionadas. Si usted no es el
destinatario señalado, le informamos que cualquier d

Re: [ceph-users] mon down for 3 hours after clocksource glitch

[Sage Weil]

Hi Dan,

On Thu, 18 Jul 2013, Dan van der Ster wrote:
> Hi,
> Last night our cluster became unhealthy for 3 hours after one of the mons (a
> qemu-kvm VM) had this glitch:
> 
> Jul 18 00:12:43 andy03 kernel: Clocksource tsc unstable (delta =
> -60129537028 ns).  Enable clocksource failover by adding
> clocksource_failover kernel parameter.
> 
> shortly afterwords the mon.2 log said:
> 
> 2013-07-18 00:13:01.147770 7feca7a5d700  1 mon.2@1(probing).data_health(52)
> service_dispatch not in quorum -- drop message
> 2013-07-18 00:13:01.148622 7feca7a5d700  1 mon.2@1(synchronizing sync(
> requester state start )) e3 sync_obtain_latest_monmap
> 2013-07-18 00:13:01.148786 7feca7a5d700  1 mon.2@1(synchronizing sync(
> requester state start )) e3 sync_obtain_latest_monmap obtained monmap e3
> 2013-07-18 00:14:01.208569 7feca845e700  0 mon.2@1(synchronizing sync(
> requester state chunks )).data_health(52) update_stats avail 59% total
> 8547036 used 2993036 avail 5119824
> 2013-07-18 00:15:01.298686 7feca845e700  0 mon.2@1(synchronizing sync(
> requester state chunks )).data_health(52) update_stats avail 58% total
> 8547036 used 3074968 avail 5037892
> 2013-07-18 00:16:08.455941 7feca845e700  0 mon.2@1(synchronizing sync(
> requester state chunks )).data_health(52) update_stats avail 58% total
> 8547036 used 3147732 avail 4965128
> ?
> 
> and that continued for over three hours until:
> 
> 2013-07-18 03:32:33.991232 7f334ecd0700  0 mon.2@1(synchronizing sync(
> requester state chunks )).data_health(0) update_stats avail 56% total
> 8547036 used 3260064 avail 4852796
> 2013-07-18 03:33:34.314538 7f334ecd0700  0 mon.2@1(synchronizing sync(
> requester state chunks )).data_health(0) update_stats avail 56% total
> 8547036 used 3294832 avail 4818028
> 2013-07-18 03:34:05.285568 7f334e2cf700  0 log [INF] : mon.2 calling new
> monitor election
> 2013-07-18 03:34:05.285747 7f334e2cf700  1 mon.2@1(electing).elector(52)
> init, last seen epoch 52
> 
> In the meantime I tried restarting each ceph-mon daemon, but that didn't
> speed up the recovery.
>
> We are talking with our OpenStack team to understand if they can provide a
> more stable clocksource, but we wanted to better understand why Ceph was so
> sensitive to this glitch. It is good that mon.2 managed to recover
> eventually, but does anyone have an idea why it took 3 hours??!!

A couple of possibilities.  There are some sync improvements in 0.61.5 
(just out) that can prevent sync loops, which might explain things (hard 
to say from the log fragments above).  Were there sync start messages in 
that 3 hour interval, or just the update_stats messages?

If the clock skew peristed, the mons won't be able to hold a quorum, but 
that would happen after the election.  A 60s skew is definitely enough to 
cause problems...

sage


> thx, Dan
> 
> -- 
> Dan van der Ster
> CERN IT-DSS
> 
> 
> ___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] v0.61.5 Cuttlefish update released

[Sage Weil]

We've prepared another update for the Cuttlefish v0.61.x series. This 
release primarily contains monitor stability improvements, although there 
are also some important fixes for ceph-osd for large clusters and a few 
important CephFS fixes. We recommend that all v0.61.x users upgrade.

 * mon: misc sync improvements (faster, more reliable, better tuning)
 * mon: enable leveldb cache by default (big performance improvement)
 * mon: new scrub feature (primarily for diagnostic, testing purposes)
 * mon: fix occasional leveldb assertion on startup
 * mon: prevent reads until initial state is committed
 * mon: improved logic for trimming old osdmaps
 * mon: fix pick_addresses bug when expanding mon cluster
 * mon: several small paxos fixes, improvements
 * mon: fix bug osdmap trim behavior
 * osd: fix several bugs with PG stat reporting
 * osd: limit number of maps shared with peers (which could cause domino 
failures)
 * rgw: fix radosgw-admin buckets list (for all buckets)
 * mds: fix occasional client failure to reconnect
 * mds: fix bad list traversal after unlink
 * mds: fix underwater dentry cleanup (occasional crash after mds restart)
 * libcephfs, ceph-fuse: fix occasional hangs on umount
 * libcephfs, ceph-fuse: fix old bug with O_LAZY vs O_NOATIME confusion
 * ceph-disk: more robust journal device detection on RHEL/CentOS
 * ceph-disk: better, simpler locking
 * ceph-disk: do not inadvertantely mount over existing osd mounts
 * ceph-disk: better handling for unusual device names
 * sysvinit, upstart: handle symlinks in /var/lib/ceph/*

Please also refer to the complete release notes:

   http://ceph.com/docs/master/release-notes/#v0-61-5-cuttlefish

You can get v0.61.5 from the usual locations:

 * Git at git://github.com/ceph/ceph.git
 * Tarball at http://ceph.com/download/ceph-0.61.5.tar.gz
 * For Debian/Ubuntu packages, see http://ceph.com/docs/master/install/debian
 * For RPMs, see http://ceph.com/docs/master/install/rpm
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] all oas crush on start

[Gregory Farnum]

In the monitor log you sent along, the monitor was crashing on a
setcrushmap command. Where in this sequence of events did that happen?

On Wed, Jul 17, 2013 at 5:07 PM, Vladislav Gorbunov  wrote:
> That's what I did:
>
> cluster state HEALTH_OK
>
> 1. load crush map from cluster:
> https://dl.dropboxusercontent.com/u/2296931/ceph/crushmap1.txt
> 2. modify crush map for adding pool and ruleset iscsi with 2
> datacenters, upload crush map to cluster:
> https://dl.dropboxusercontent.com/u/2296931/ceph/crushmap2.txt
>
> 3. add host gstore1
>
> ceph-deploy osd create gstore1:/dev/sdh:/dev/sdb1
> ceph-deploy osd create gstore1:/dev/sdj:/dev/sdc1
> ceph-deploy osd create gstore1:/dev/sdk:/dev/sdc2
>
> 4. move this osds to datacenter datacenter-cod:
> ceph osd crush set 82 0 root=iscsi datacenter=datacenter-cod host=gstore1
> ceph osd crush set 83 0 root=iscsi datacenter=datacenter-cod host=gstore1
> ceph osd crush set 84 0 root=iscsi datacenter=datacenter-cod host=gstore1
>
> 5. cluster state HEALTH_OK, reweight new osds:
> ceph osd crush reweight osd.82 2.73
> ceph osd crush reweight osd.83 2.73
> ceph osd crush reweight osd.84 2.73
>
> 6. exclude osd.57 (in default pool) from cluster:
> ceph osd crush reweight osd.57 0
> cluster state HEALTH_WARN
>
> 7. add new node gstore2 same as gstore1
> ceph-deploy -v osd create gstore2:/dev/sdh:/dev/sdb1
> ceph osd crush set 94 2.73 root=iscsi datacenter=datacenter-rcod host=gstore2

Where are you getting these numbers 82-84 and 92-94 from? They don't
appear in any any of the maps you've sent along...


> 8. exclude osd.56 (in default pool) from cluster:
> ceph osd crush reweight osd.57 0
>
>
> 9. add new osd to gstore2
> ceph-deploy osd create gstore2:/dev/sdl:/dev/sdd1
> ceph-deploy osd create gstore2:/dev/sdm:/dev/sdd2
> …
> ceph-deploy osd create gstore2:/dev/sds:/dev/sdg2
>
> 10. rename pool iscsi in default crush pool :
> ceph osd pool rename iscsi iscsi-old
>
> 11. create new pool iscsi:
> ceph osd pool create iscsi 2048 2048
>
> 12. set ruleset iscsi to new pool iscsi
> ceph osd pool set iscsi crush_ruleset 3
>
> All OSDS downed with Segmentation fault

Okay, so you switched to actually start using the new rule and the
OSDs broke. It's possible there's a hole in our crush map testing that
would let this through.

> 13. failback ruleset 0 for pool iscsi
> ceph osd pool set iscsi crush_ruleset 0
>
> delete ruleset iscsi, upload crushmap to cluster
> https://dl.dropboxusercontent.com/u/2296931/ceph/crushmap14-new.txt
>
> OSD still Segmentation fault

Yeah, once you've put a bad map into the system then you can't fix it
by putting in a good one — all the OSDs need to evaluate the past maps
on startups, which includes the bad one, which makes them crash again.
:(

Can you provide us a tarball of one of your monitor directories? We'd
like to do some forensics on it to identify the scenario precisely and
prevent it from happening again.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Enjoy High Speed Maxis fibre Internet 10mbps only at RM148/month!

[Maxis Agent (Shanny)]

Title: MAXIS HOME AND BUSINESS FIBRE INTERNET







  


  
 
  
  
MAXIS HOME AND BUSINESS FIBRE INTERNET
  
  
Fastest Fibre Broadband in Malaysia with lowest price
  
  
10Mbps, 20Mbps, & 30Mbps from only RM148 / month
  
  

  
  
Pay Less & Enjoy High Speed Internet with Fibre Technology, Subscribe to Maxis Fibre Now!
  
  

  
  

For more enquiry: 
  1) Please visit http://shanny.maxispoint.com or 
  2) Reply this email or email to sha...@maxispoint.com, or
  3) Please contact Shanny (016-2585109), you may sms 
  We will contact you when we receive your enquiry 
  
  

  
  
Maxis Home Fibre Package (Home)
  
  


  Package
  Home Fibre 10Mbps
  Home Fibre 20Mbps
  Home Fibre 30Mbps


  Download Speed
  10Mbps
  20Mbps
  30Mbps


  Monthly Fee
  RM148 / month
  RM198 / month
  RM248 / month


  Contract
  24 Months


  Quota
  Unlimited


  Installation
  Free
Additional charges will be incurred should you request for non-standard installation (more than 15 metres)


  Free Home Voice Package
  Free RM30 call credit to any local and international numbers


  Free calls to Maxis fixed lines
  Unlimited


  Free devices
  Wi-Fi Router


  
  
 
  
  
Maxis Business Fibre Package (Business) - 20% Discount
  
  


  Package
  Business Fibre 4Mbps
  Business Fibre 8Mbps
  Business Fibre 16Mbps
  Business Fibre 32Mbps


  Download Speed
  4Mbps
  8Mbps
  16Mbps
  32Mbps


   Monthly Fee
  Normal Price
RM168/month
  Normal Price
RM208/month
  Normal Price
RM308/month
  Normal Price
RM488/month


  20% Discounted Price
RM134.40/month*
  20% Discounted Price
RM166.40/month*
  20% Discounted Price
RM246.40/month*
  20% Discounted Price
RM390.40/month*


  Contract
  24 Months


  Quota
  Unlimited


  Installation
  Free
Additional charges will be incurred should you request for non-standard installation (more than 15 metres)


  Voice Call Rates
  4 sen (local); 10 sen (Maxis Mobile); 12 sen (STD or OLO)


  Buy-over-contract from other ISP
  Maximum RM500 rebate** in customer bill, and based on customer ISP termination receipt. Customer will need to pay off the penalty first. Documents needed to ensure legitimate are ISP Penalty Receipt.


  
  

*Enjoy 20% discount for the first 24 months, normal price will be charged from 25th month onwards.(Only for Business Fibre Package)
  
  

**Maxis provide Buy-Over-Contract service from other internet service provider (ISP) up to maximum RM500. (Only for Business Fibre Package)
  

  
 
  
  

If you are interested to know more: 
  1) Please visit http://shanny.maxispoint.com or 
  2) Reply this email or email to sha...@maxispoint.com, or
  3) Please contact Shanny (016-2585109), you may sms 
  We will contact you when we receive your enquiry 
  
  
 
  

  


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Problems with tgt with ceph support

[Toni F. [ackstorm]]

Hi Dan,

When i say multipath i want to say multipath with round robin ;)

On 18/07/13 17:39, Toni F. [ackstorm] wrote:

Hi Dan, thanks for your reply

Yes, as you has said:

"RBD Device /dev/rbd/rbd/iscsi-image-part1 exported with tgt"  is 
kernel block device and "the TGT-RBD connector" is stgt/tgtd.


My tests was been done exporting it with tgt without multipath. It 
seems that are no very big difference in this tests. The real problem 
is not that.


I want to use tgt to connect the storage to an ESX but it presents a 
few problems.


With kernel block device, i can't use multipath (it turns inestable 
and turns a corrupted data)
With stgt/tgtd rbd implementation i can use multipath but the 
performance is very poor.


To imaginate the difference, Deploy a VM with 1st (3min) but with 2nd 
option (83min).


I hope your understand me. I'm newbie with this concepts.

A lot of thanks

On 16/07/13 04:23, Dan Mick wrote:
Apologies; I don't really understand the results.  They're labeled 
"RBD Device /dev/rbd/rbd/iscsi-image-part1 exported with tgt" and
"TGT-RBD connector".  Is the former nothing to do with tgt (i.e., 
just the kernel block device), and the latter is stgt/tgtd?


Do you interpret them to say the first run gave avg 258MB/s, and the 
second run 198MB/s?


On 07/15/2013 02:41 AM, Toni F. [ackstorm] wrote:

Here's my results.

The performance test was seq write

On 15/07/13 10:12, Toni F. [ackstorm] wrote:

I'm going to do a performance test with fio to see the difference.

Regards

On 12/07/13 18:15, Dan Mick wrote:


Ceph performance is a very very complicated subject. How does that
compare to other access methods?  Say, rbd import/export for an easy
test?

On Jul 12, 2013 8:22 AM, "Toni F. [ackstorm]"
mailto:toni.fuen...@ackstorm.es>> wrote:

It works, but the performance is very poor. 100MB/s or less

Which are your performance experience?

Regards

On 12/07/13 13:56, Toni F. [ackstorm] wrote:

It works!

Thanks for all

On 12/07/13 11:23, Toni F. [ackstorm] wrote:

Yes! it seems that i wasn't compiled the rbd support.

System:
State: ready
debug: off
LLDs:
iscsi: ready
Backing stores:
bsg
sg
null
ssc
aio
rdwr (bsoflags sync:direct)
Device types:
disk
cd/dvd
osd
controller
changer
tape
passthrough
iSNS:
iSNS=Off
iSNSServerIP=
iSNSServerPort=3205
iSNSAccessControl=Off

I'm going to recompile it
Thanks a lot!

On 11/07/13 07:45, Dan Mick wrote:



On 07/10/2013 04:12 AM, Toni F. [ackstorm] wrote:

Hi all,

I have installed the v0.37 of tgt.

To test this feature i follow the

http://ceph.com/dev-notes/adding-support-for-rbd-to-stgt/

guide

When i launch the command:

tgtadm --lld iscsi --mode logicalunit --op new
--tid 1 --lun 0
--backing-store iscsi-image --bstype rbd

fails

First i think that lun cannot be 0 because lun 0
is used by the
controller (previous command)


This worked when I first wrote the backend, but tgt
may have changed; I'll investigate and change the
blog entry if so.  Thanks.


If i launch the correct command with lun 1 i have
this error:

tgtadm: invalid request

In syslog:

Jul 10 12:54:03 datastore-lnx001 tgtd:
device_mgmt(245) sz:28
params:path=iscsi-image,bstype=rbd
Jul 10 12:54:03 datastore-lnx001 tgtd:
tgt_device_create(532) failed to
find bstype, rbd

What's wrong? not supported?



Where did you get your tgtd?  Was it built with rbd
support (CEPH_RBD defined
in the environment for make)?

sudo ./tgtadm --lld iscsi --op show --mode system

should tell you.

How did you set up access to ceph.conf?








--

Toni Fuentes Rico
toni.fuen...@ackstorm.es 
Administración de Sistemas

Oficina central: 902 888 345

ACK STORM, S.L.
ISO 9001:2008 (Cert.nº. 536932)
http://ackstorm.es

Este mensaje electrónico contiene información de ACK STORM, S.L.
que es privada y confidencial, siendo para el uso exclusivo de la

Re: [ceph-users] ceph -w warning "I don't have pgid 0.2c8"?

[Samuel Just]

What is the output of ceph pg dump | grep 'stale\|creating' ?

On Wed, Jul 17, 2013 at 7:56 PM, Ta Ba Tuan  wrote:
> zombie pgs might occured when i remove some data pools.
> but, with pgs in stale state, i can't delete it?
>
> I found this guide, but I don't understand it.
> http://ceph.com/docs/next/dev/osd_internals/pg_removal/
>
> Thanks!
> --tuantaba
>
>
> On 07/18/2013 09:22 AM, Ta Ba Tuan wrote:
>
> I'm using Ceph-0.61.4,
> I removed each osds (2TB) on data hosts and re-create with disks (4TB).
> When converting finish, Ceph warns that have 4 pgs in stale state and
> warning: i don't have pgid 
> after, I created 4 pgs by command: ceph pg force_create_pg 
>
> Now (after the long time), Ceph still warning: "pgmap v57451: 22944 pgs: 4
> creating, 22940 active+clean;"
>
> I don't know how to remove those pgs?.
>  Please guiding this error help me!
>
> Thank you!
> --tuantaba
> TA BA TUAN
>
>
> On 07/18/2013 01:16 AM, Samuel Just wrote:
>
> What version are you running?  How did you move the osds from 2TB to 4TB?
> -Sam
>
> On Wed, Jul 17, 2013 at 12:59 AM, Ta Ba Tuan  wrote:
>
> Hi everyone,
>
> I converted every osds from 2TB to 4TB, and when moving complete, show log
> Ceph realtime"ceph -w":
> displays error: "I don't have pgid 0.2c8"
>
> after then, I run:  "ceph pg force_create_pg 0.2c8"
> Ceph warning: pgmap v55175: 22944 pgs: 1 creating, 22940 active+clean, 3
> stale+active+degraded
>
> then, I can't read/write data to mounted CephFS on Client-side ==> notify on
> client side: "Operation not permitted"
>
> now, "ceph -w" still notify "22944 pgs: 1 creating, 22940 active+clean, 3
> stale+active+degraded"
> and I don't understand something occuring?
> Please, help me!!!
>
> Thanks to everyone.
>
> --tuantaba
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph -w warning "I don't have pgid 0.2c8"?

[Ta Ba Tuan]

Hi Samuel,

Output logs from : ceph pg dump | grep 'stale\|creating'

0.f4f 0 0 0 0 0 0 0 stale+creating 2013-07-17 16:35:06.882419 0'0 0'0 [] 
[68,12] 0│'0 0.00 0'0 0.00 │
2.f4d 0 0 0 0 0 0 0 stale+creating 2013-07-17 16:35:22.826552 0'0 0'0 [] 
[68,12] 0│

'0 0.00 0'0 0.00 │
0.2c8 0 0 0 0 0 0 0 stale+creating 2013-07-17 14:30:54.280454 0'0 0'0 [] 
[68,5] 0│

'0 0.00 0'0 0.00 │
2.2c6 0 0 0 0 0 0 0 stale+creating 2013-07-17 16:35:28.445878 0'0 0'0 [] 
[68,5] 0│

'0 0.00 0'0 0.00

Thanks!

On 07/19/2013 01:16 AM, Samuel Just wrote:

ceph pg dump | grep 'stale\|creating'


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Libvirt, quemu, ceph write cache settings

[Josh Durgin]

On 07/17/2013 11:39 PM, Maciej Gałkiewicz wrote:

I have created VM with KVM 1.1.2 and all I had was rbd_cache configured
in ceph.conf. Cache option in libvirt set to "none":


   
   
   
 
   
   
   f81d6108-d8c9-4e06-94ef-02b1943a873d
 
 
   
   
   
 
   
   
   9ab3e9b3-e153-447c-ab1d-2f8f9bae095c
 

Config settings received from admin socket show that cache is enabled. I
thought that without configuring libvirt with cache options it is not
possible to force kvm to use it. Can you explain it a little bit why it
works or claims to work?


Setting rbd_cache=true in ceph.conf will make librbd turn on the cache
regardless of qemu. Setting qemu to cache=none tells qemu that it
doesn't need to send flush requests to the underlying storage, so it
does not do so. This means librbd is caching data, but qemu isn't
telling it to persist that data when the guest requests it. This is
the same as qemu's cache=unsafe mode, which makes it easy to get a
corrupt fs if the guest isn't shut down cleanly.

There's a ceph option to make this safer -
rbd_cache_writethrough_until_flush. If this and rbd_cache are true,
librbd will operate with the cache in writethrough mode until it is
sure that the guest using it is capable of sending flushes (i.e. qemu
has cache=writeback). Perhaps we should enable this by default so
people are less likely to accidentally use an unsafe configuration.

Josh
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph stop service

[w...@aixit.com]

Hello All,

stupid question.


service ceph stop mon

or

/etc/init.d/ceph stop mon

doesn't work.

how i can stop some osds or mons ?


Thanks much.

--
AIXIT GmbH - Witalij Poljatchek
(T) +49 69 203 4709-13 - (F) +49 69 203 470 979
w...@aixit.com  -http://www.aixit.com

AIXIT GmbH

Strahlenbergerstr. 14
63067 Offenbach am Main
(T) +49 69 203 470 913

Amtsgericht Offenbach, HRB 43953
Geschäftsführer: Friedhelm Heyer, Holger Grauer

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph stop service

[Jens Kristian Søgaard]

Hi,


service ceph stop mon
doesn't work.
how i can stop some osds or mons ?


Try for example:

  service ceph stop mon.a

or

  service ceph stop osd.1

replacing "a" and "1" with the id, you want to stop.

--
Jens Kristian Søgaard, Mermaid Consulting ApS,
j...@mermaidconsulting.dk,
http://www.mermaidconsulting.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Libvirt, quemu, ceph write cache settings

[Maciej Gałkiewicz]

On 18 Jul 2013 20:25, "Josh Durgin"  wrote:
> Setting rbd_cache=true in ceph.conf will make librbd turn on the cache
> regardless of qemu. Setting qemu to cache=none tells qemu that it
> doesn't need to send flush requests to the underlying storage, so it
> does not do so. This means librbd is caching data, but qemu isn't
> telling it to persist that data when the guest requests it. This is
> the same as qemu's cache=unsafe mode, which makes it easy to get a
> corrupt fs if the guest isn't shut down cleanly.
>
> There's a ceph option to make this safer -
> rbd_cache_writethrough_until_flush. If this and rbd_cache are true,
> librbd will operate with the cache in writethrough mode until it is
> sure that the guest using it is capable of sending flushes (i.e. qemu
> has cache=writeback). Perhaps we should enable this by default so
> people are less likely to accidentally use an unsafe configuration.

Ok. Now it make sense. So the last question is how to make sure that qemu
actually operates with cache=writeback with rbd?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph stop service

[Gregory Farnum]

On Thu, Jul 18, 2013 at 11:31 AM, Jens Kristian Søgaard
 wrote:
> Hi,
>
>> service ceph stop mon
>>
>> doesn't work.
>> how i can stop some osds or mons ?
>
>
> Try for example:
>
>   service ceph stop mon.a
>
> or
>
>   service ceph stop osd.1
>
> replacing "a" and "1" with the id, you want to stop.

That should be right if you're using sysvinit. If you're on Ubuntu
then you're using Upstart instead:
"sudo stop ceph-mon id=" or "sudo stop ceph-osd id="
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph stop service

[Uwe Grohnwaldt]

Hi,

first ceph mon dump:
dumped monmap epoch 2
epoch 2
fsid b8c3e27a-5f9a-4367-9b73-4451360c747c
last_changed 2013-07-09 05:19:07.776307
created 0.00
0: 10.0.104.31:6789/0 mon.ceph01
1: 10.0.104.32:6789/0 mon.ceph02
2: 10.0.104.33:6789/0 mon.ceph03


second: /etc/ceph.conf
[global]
fsid = b8c3e27a-5f9a-4367-9b73-4451360c747c
mon_initial_members = infernum-dp-test-storage01, ceph02, ceph03
mon_host = 10.0.104.31,10.0.104.32,10.0.104.33
auth_supported = none
osd_journal_size = 1024
filestore_xattr_use_omap = true


we tried to stop ceph-mon with:
#service ceph stop mon.1
/etc/init.d/ceph: mon.1 not found (/etc/ceph/ceph.conf defines , /var/lib/ceph 
defines )


after this we added to /etc/ceph.conf:
[mon.0]
  host = ceph01
[mon.1]
  host = ceph02
[mon.2]
  host = ceph03


and tried again:
=== mon.1 ===
Stopping Ceph mon.1 on ceph02...done


and still get: root@ceph02:~# ceph -w
   health HEALTH_OK
   monmap e2: 3 mons at 
{ceph02=10.0.104.32:6789/0,ceph03=10.0.104.33:6789/0,ceph01=10.0.104.31:6789/0},
 election epoch 26, quorum 0,1,2 ceph02,ceph03,ceph01


even with stop ceph-mon id=1:
root@ceph02:~# stop ceph-mon id=1
stop: Unknown instance: ceph/1

Mit freundlichen Grüßen / Best Regards,
--
Uwe Grohnwaldt

- Original Message -
> From: "Jens Kristian Søgaard" 
> To: w...@aixit.com
> Cc: "ceph-users" 
> Sent: Donnerstag, 18. Juli 2013 20:31:00
> Subject: Re: [ceph-users] ceph stop service
> 
> Hi,
> 
> > service ceph stop mon
> > doesn't work.
> > how i can stop some osds or mons ?
> 
> Try for example:
> 
>service ceph stop mon.a
> 
> or
> 
>service ceph stop osd.1
> 
> replacing "a" and "1" with the id, you want to stop.
> 
> --
> Jens Kristian Søgaard, Mermaid Consulting ApS,
> j...@mermaidconsulting.dk,
> http://www.mermaidconsulting.com/
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mon down for 3 hours after clocksource glitch

[Dan van der Ster]

On Thu, Jul 18, 2013 at 6:27 PM, Sage Weil  wrote:
> Hi Dan,
>
> On Thu, 18 Jul 2013, Dan van der Ster wrote:
>> Hi,
>> Last night our cluster became unhealthy for 3 hours after one of the mons (a
>> qemu-kvm VM) had this glitch:
>>
>> Jul 18 00:12:43 andy03 kernel: Clocksource tsc unstable (delta =
>> -60129537028 ns).  Enable clocksource failover by adding
>> clocksource_failover kernel parameter.
>>
>> shortly afterwords the mon.2 log said:
>>
>> 2013-07-18 00:13:01.147770 7feca7a5d700  1 mon.2@1(probing).data_health(52)
>> service_dispatch not in quorum -- drop message
>> 2013-07-18 00:13:01.148622 7feca7a5d700  1 mon.2@1(synchronizing sync(
>> requester state start )) e3 sync_obtain_latest_monmap
>> 2013-07-18 00:13:01.148786 7feca7a5d700  1 mon.2@1(synchronizing sync(
>> requester state start )) e3 sync_obtain_latest_monmap obtained monmap e3
>> 2013-07-18 00:14:01.208569 7feca845e700  0 mon.2@1(synchronizing sync(
>> requester state chunks )).data_health(52) update_stats avail 59% total
>> 8547036 used 2993036 avail 5119824
>> 2013-07-18 00:15:01.298686 7feca845e700  0 mon.2@1(synchronizing sync(
>> requester state chunks )).data_health(52) update_stats avail 58% total
>> 8547036 used 3074968 avail 5037892
>> 2013-07-18 00:16:08.455941 7feca845e700  0 mon.2@1(synchronizing sync(
>> requester state chunks )).data_health(52) update_stats avail 58% total
>> 8547036 used 3147732 avail 4965128
>> ?
>>
>> and that continued for over three hours until:
>>
>> 2013-07-18 03:32:33.991232 7f334ecd0700  0 mon.2@1(synchronizing sync(
>> requester state chunks )).data_health(0) update_stats avail 56% total
>> 8547036 used 3260064 avail 4852796
>> 2013-07-18 03:33:34.314538 7f334ecd0700  0 mon.2@1(synchronizing sync(
>> requester state chunks )).data_health(0) update_stats avail 56% total
>> 8547036 used 3294832 avail 4818028
>> 2013-07-18 03:34:05.285568 7f334e2cf700  0 log [INF] : mon.2 calling new
>> monitor election
>> 2013-07-18 03:34:05.285747 7f334e2cf700  1 mon.2@1(electing).elector(52)
>> init, last seen epoch 52
>>
>> In the meantime I tried restarting each ceph-mon daemon, but that didn't
>> speed up the recovery.
>>
>> We are talking with our OpenStack team to understand if they can provide a
>> more stable clocksource, but we wanted to better understand why Ceph was so
>> sensitive to this glitch. It is good that mon.2 managed to recover
>> eventually, but does anyone have an idea why it took 3 hours??!!
>
> A couple of possibilities.  There are some sync improvements in 0.61.5
> (just out) that can prevent sync loops, which might explain things (hard
> to say from the log fragments above).  Were there sync start messages in
> that 3 hour interval, or just the update_stats messages?

Yes, many of those too. The whole set of relevant logs is here:
http://pastebin.com/hCcu8Wm9
The weird bit at  01:13:12 is me telneting to the daemon to check the
network, so just ignore that. Otherwise, the daemon was restarted
twice (once after a server rebooted) but I left it after ~1:33.

> If the clock skew peristed, the mons won't be able to hold a quorum, but
> that would happen after the election.  A 60s skew is definitely enough to
> cause problems...

I can't say for certain if the skew persisted or not -- but I had the
impression the clocks were ok shortly after the glitch. After the
machine rebooted ntp corrected the time by a bit -- not 60s:

Jul 18 01:40:44 andy03 ntpd[4918]: time reset -0.775446 s

If the ceph-mon log is useful or not to explain the long outage, that
would be good to know (especially if it is fixed in 0.61.5). In any
case, we're moving our mons to physical boxes soon, so the clock
problem will hopefully go away.

Thanks, Dan

>
> sage
>
>
>> thx, Dan
>>
>> --
>> Dan van der Ster
>> CERN IT-DSS
>>
>>
>>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mon down for 3 hours after clocksource glitch

[Sage Weil]

On Thu, 18 Jul 2013, Dan van der Ster wrote:
> On Thu, Jul 18, 2013 at 6:27 PM, Sage Weil  wrote:
> > Hi Dan,
> >
> > On Thu, 18 Jul 2013, Dan van der Ster wrote:
> >> Hi,
> >> Last night our cluster became unhealthy for 3 hours after one of the mons 
> >> (a
> >> qemu-kvm VM) had this glitch:
> >>
> >> Jul 18 00:12:43 andy03 kernel: Clocksource tsc unstable (delta =
> >> -60129537028 ns).  Enable clocksource failover by adding
> >> clocksource_failover kernel parameter.
> >>
> >> shortly afterwords the mon.2 log said:
> >>
> >> 2013-07-18 00:13:01.147770 7feca7a5d700  1 mon.2@1(probing).data_health(52)
> >> service_dispatch not in quorum -- drop message
> >> 2013-07-18 00:13:01.148622 7feca7a5d700  1 mon.2@1(synchronizing sync(
> >> requester state start )) e3 sync_obtain_latest_monmap
> >> 2013-07-18 00:13:01.148786 7feca7a5d700  1 mon.2@1(synchronizing sync(
> >> requester state start )) e3 sync_obtain_latest_monmap obtained monmap e3
> >> 2013-07-18 00:14:01.208569 7feca845e700  0 mon.2@1(synchronizing sync(
> >> requester state chunks )).data_health(52) update_stats avail 59% total
> >> 8547036 used 2993036 avail 5119824
> >> 2013-07-18 00:15:01.298686 7feca845e700  0 mon.2@1(synchronizing sync(
> >> requester state chunks )).data_health(52) update_stats avail 58% total
> >> 8547036 used 3074968 avail 5037892
> >> 2013-07-18 00:16:08.455941 7feca845e700  0 mon.2@1(synchronizing sync(
> >> requester state chunks )).data_health(52) update_stats avail 58% total
> >> 8547036 used 3147732 avail 4965128
> >> ?
> >>
> >> and that continued for over three hours until:
> >>
> >> 2013-07-18 03:32:33.991232 7f334ecd0700  0 mon.2@1(synchronizing sync(
> >> requester state chunks )).data_health(0) update_stats avail 56% total
> >> 8547036 used 3260064 avail 4852796
> >> 2013-07-18 03:33:34.314538 7f334ecd0700  0 mon.2@1(synchronizing sync(
> >> requester state chunks )).data_health(0) update_stats avail 56% total
> >> 8547036 used 3294832 avail 4818028
> >> 2013-07-18 03:34:05.285568 7f334e2cf700  0 log [INF] : mon.2 calling new
> >> monitor election
> >> 2013-07-18 03:34:05.285747 7f334e2cf700  1 mon.2@1(electing).elector(52)
> >> init, last seen epoch 52
> >>
> >> In the meantime I tried restarting each ceph-mon daemon, but that didn't
> >> speed up the recovery.
> >>
> >> We are talking with our OpenStack team to understand if they can provide a
> >> more stable clocksource, but we wanted to better understand why Ceph was so
> >> sensitive to this glitch. It is good that mon.2 managed to recover
> >> eventually, but does anyone have an idea why it took 3 hours??!!
> >
> > A couple of possibilities.  There are some sync improvements in 0.61.5
> > (just out) that can prevent sync loops, which might explain things (hard
> > to say from the log fragments above).  Were there sync start messages in
> > that 3 hour interval, or just the update_stats messages?
> 
> Yes, many of those too. The whole set of relevant logs is here:
> http://pastebin.com/hCcu8Wm9
> The weird bit at  01:13:12 is me telneting to the daemon to check the
> network, so just ignore that. Otherwise, the daemon was restarted
> twice (once after a server rebooted) but I left it after ~1:33.

Okay, this sounds exactly like the problem we just fixed in v0.61.5.  The 
trigger is that sync takes a while (largish mon data directory) and an 
intervening election confuses it enough that it restarts right after it 
finishes. 

> > If the clock skew peristed, the mons won't be able to hold a quorum, but
> > that would happen after the election.  A 60s skew is definitely enough to
> > cause problems...
> 
> I can't say for certain if the skew persisted or not -- but I had the
> impression the clocks were ok shortly after the glitch. After the
> machine rebooted ntp corrected the time by a bit -- not 60s:
> 
> Jul 18 01:40:44 andy03 ntpd[4918]: time reset -0.775446 s
> 
> If the ceph-mon log is useful or not to explain the long outage, that
> would be good to know (especially if it is fixed in 0.61.5). In any
> case, we're moving our mons to physical boxes soon, so the clock
> problem will hopefully go away.

Sounds good.  In this case it looks like sync was to blame.  :)

sage
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mon down for 3 hours after clocksource glitch

[Dan van der Ster]

On Thu, Jul 18, 2013 at 9:29 PM, Sage Weil  wrote:
> this sounds exactly like the problem we just fixed in v0.61.5.

Glad to hear that.
Thanks for the quick help :)

dan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph & hbase:

[ker can]

how is your hbase performance on ceph compared to hdfs  - was there some
special knobs that you needed to turn ?

I'm running a few hbase tests with the yahoo cloud serving benchmark -
ycsb  with ceph  & hdfs and the results were very surprising considering
that the hadoop + ceph results were not all that bad.  For the test setup,
I have about 50 million rows distributed over two nodes. Ran a single ycsb
client  with 128 threads reading all that doing 100% reads.

the hbase+hdfs throughput results were 38x better. Looking at the system
stats I don't see anything special going on - no significant cpu
utilization. Practically no disk reads which means its all cached
somewhere. The 10gig network is no where close to saturating. Plenty of
memory also.

Any thoughts on what might be going on ?


For hbase + hdfs:
-
[OVERALL], RunTime(ms), 720023.0
[OVERALL], Throughput(ops/sec), 16373.007528926159
[READ], Operations, 11788942
[READ], AverageLatency(us), 7814.152212217178
[READ], MinLatency(us), 166
[READ], MaxLatency(us), 596026
[READ], 95thPercentileLatency(ms), 19
[READ], 99thPercentileLatency(ms), 23
[READ], Return=0, 11788942

For hbase + ceph:
--
[OVERALL], RunTime(ms), 720681.0
[OVERALL], Throughput(ops/sec), 420.35102909609105
[READ], Operations, 302939
[READ], AverageLatency(us), 304294.1931676014
[READ], MinLatency(us), 312
[READ], MaxLatency(us), 1539757
[READ], 95thPercentileLatency(ms), 676
[READ], 99thPercentileLatency(ms), 931
[READ], Return=0, 302939

Thx



On Wed, Jul 17, 2013 at 6:27 PM, Mike Bryant  wrote:

> Yup, that was me.
> We have hbase working here.
> You'll want to disable localized reads, as per bug #5388. That bug
> will cause your regionservers to crash fairly often when doing
> compaction.
> You'll also want to restart each of the regionservers and masters
> often (We're doing it once a day) to mitigate the effects of bug
> #5039, being that your data pool is growing much faster than you might
> expect, and it being much larger than the visible filesize in cephfs.
>
> With those workarounds in place we're running a stable install of
> openTSDB on top of hbase.
>
> Mike
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] optimizing recovery throughput

[Dan van der Ster]

Hi,
We added some new OSDs today and since we've recently written many
many (small/tiny) objects to a test pool, backfilling those new disks
is going to take something like 24hrs. I'm therefore curious if we can
speed up the recovery at all or if the default settings in cuttlefish
already bring us to the limit.

Here is a line from the mons earlier today just after the OSDs were started:

2013-07-18 21:05:38.017063 mon.0 128.142.142.156:6789/0 27124 : [INF]
pgmap v111472: 9464 pgs: 8115 active+clean, 862
active+remapped+wait_backfill, 3 active+recovery_wait, 484
active+remapped+backfilling; 15994 GB data, 55536 GB used, 1
380 TB / 1434 TB avail; 12220476/151694232 degraded (8.056%);
recovering 254 o/s, 93913KB/s

There are 48 new OSDs, so the defaults of 10 max backfills roughly
corresponds with the 484 backfilling.

I started injecting higher backfilling options, specifically
   osd max backfills = 20
   osd recovery max active = 10

and that gives me something like this:

2013-07-18 21:22:56.546094 mon.0 128.142.142.156:6789/0 27984 : [INF]
pgmap v112308: 9464 pgs: 8129 active+clean, 398
active+remapped+wait_backfill, 3 active+recovery_wait, 933
active+remapped+backfilling, 1 active+clean+scrubbing; 15994
 GB data, 55567 GB used, 1380 TB / 1434 TB avail; 11982626/151538728
degraded (7.907%);  recovering 299 o/s, 114MB/s

but immediately I start to see slow requests piling up. Trying with
the different combinations, I found that it's the "max active = 10"
setting that leads to the slow requests. With a 20/5 setting, there
are no slow requests, but the recovery rate doesn't increase anyway.

So I'm wondering if you all agree that this indicates that the 10/5
setting for backfill/max active is already the limit for our cluster,
at least with this current set of test objects we have? Or am I
missing another option that should be tweaked to get more recovery
throughput?

Thanks in advance,
Dan

--
Dan van der Ster
CERN IT-DSS
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Libvirt, quemu, ceph write cache settings

[Josh Durgin]

On 07/18/2013 11:32 AM, Maciej Gałkiewicz wrote:

On 18 Jul 2013 20:25, "Josh Durgin" mailto:josh.dur...@inktank.com>> wrote:
 > Setting rbd_cache=true in ceph.conf will make librbd turn on the cache
 > regardless of qemu. Setting qemu to cache=none tells qemu that it
 > doesn't need to send flush requests to the underlying storage, so it
 > does not do so. This means librbd is caching data, but qemu isn't
 > telling it to persist that data when the guest requests it. This is
 > the same as qemu's cache=unsafe mode, which makes it easy to get a
 > corrupt fs if the guest isn't shut down cleanly.
 >
 > There's a ceph option to make this safer -
 > rbd_cache_writethrough_until_flush. If this and rbd_cache are true,
 > librbd will operate with the cache in writethrough mode until it is
 > sure that the guest using it is capable of sending flushes (i.e. qemu
 > has cache=writeback). Perhaps we should enable this by default so
 > people are less likely to accidentally use an unsafe configuration.

Ok. Now it make sense. So the last question is how to make sure that
qemu actually operates with cache=writeback with rbd?



If the setting is in the qemu command line, it'll send flushes,
and you can verify that librbd is seeing them by doing a 'perf dump'
on the admin socket and looking at the aio_flush count there.

This makes me notice that the synchronous flush perf counter went
missing, so it'll always read 0 [1].

Josh

[1] http://tracker.ceph.com/issues/5668
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to remove stale pgs?

[Gregory Farnum]

On Thu, Jul 18, 2013 at 3:53 AM, Ta Ba Tuan  wrote:
> Hi all,
>
> I have 4 (stale+inactive) pgs, how to delete those pgs?
>
> pgmap v59722: 21944 pgs: 4 stale, 12827 active+clean, 9113 active+degraded;
> 45689 MB data, 1006 GB used, 293 TB / 294 TB avail;
>
> I found on google a long time, still can't resolve it.
> Please, help me!

This depends on why they're stale+inactive. Can you pastebin the
output of "ceph pg dump" and provide the link?

Have you lost any OSDs?
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw create user using API

[Gregory Farnum]

On Thu, Jul 18, 2013 at 12:50 AM, Alvaro Izquierdo Jimeno
 wrote:
> Hi,
>
>
>
> Reading the URL http://ceph.com/docs/next/radosgw/adminops/#create-user ,
> I’m trying to create a new user with:
>
>
>
> curl -v -X PUT  -d '{"uid": "alvaro", "display-name": "alvaro"}
> http://myradosgw/admin/user?format=json
>
>
>
> and a 403 response is received L
>
>
>
> So, Do I need a token? A token of who? Do I need a superuser with a special
> feature? Or I have  to setup something special in radosgw part into
> ceph.conf?

I'm not sure what your exact problem here, but yes, you need to be a
"system" user to use the RGW admin API. I believe you should be able
to find how to do that if you search the docs.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] all oas crush on start

[Vladislav Gorbunov]

>In the monitor log you sent along, the monitor was crashing on a
setcrushmap command. Where in this sequence of events did that happen?
It's happened after I try to upload different crushmap, much later step 13.

>Where are you getting these numbers 82-84 and 92-94 from? They don't
appear in any any of the maps you've sent along...
Sorry, this is crushmap after OSDs was broken:
https://dl.dropboxusercontent.com/u/2296931/ceph/crushmap14-2.txt

>Can you provide us a tarball of one of your monitor directories?
https://dl.dropboxusercontent.com/u/2296931/ceph/ceph-mon.1.tar.bz2

2013/7/19 Gregory Farnum :
> In the monitor log you sent along, the monitor was crashing on a
> setcrushmap command. Where in this sequence of events did that happen?
>
> On Wed, Jul 17, 2013 at 5:07 PM, Vladislav Gorbunov  wrote:
>> That's what I did:
>>
>> cluster state HEALTH_OK
>>
>> 1. load crush map from cluster:
>> https://dl.dropboxusercontent.com/u/2296931/ceph/crushmap1.txt
>> 2. modify crush map for adding pool and ruleset iscsi with 2
>> datacenters, upload crush map to cluster:
>> https://dl.dropboxusercontent.com/u/2296931/ceph/crushmap2.txt
>>
>> 3. add host gstore1
>>
>> ceph-deploy osd create gstore1:/dev/sdh:/dev/sdb1
>> ceph-deploy osd create gstore1:/dev/sdj:/dev/sdc1
>> ceph-deploy osd create gstore1:/dev/sdk:/dev/sdc2
>>
>> 4. move this osds to datacenter datacenter-cod:
>> ceph osd crush set 82 0 root=iscsi datacenter=datacenter-cod host=gstore1
>> ceph osd crush set 83 0 root=iscsi datacenter=datacenter-cod host=gstore1
>> ceph osd crush set 84 0 root=iscsi datacenter=datacenter-cod host=gstore1
>>
>> 5. cluster state HEALTH_OK, reweight new osds:
>> ceph osd crush reweight osd.82 2.73
>> ceph osd crush reweight osd.83 2.73
>> ceph osd crush reweight osd.84 2.73
>>
>> 6. exclude osd.57 (in default pool) from cluster:
>> ceph osd crush reweight osd.57 0
>> cluster state HEALTH_WARN
>>
>> 7. add new node gstore2 same as gstore1
>> ceph-deploy -v osd create gstore2:/dev/sdh:/dev/sdb1
>> ceph osd crush set 94 2.73 root=iscsi datacenter=datacenter-rcod host=gstore2
>
> Where are you getting these numbers 82-84 and 92-94 from? They don't
> appear in any any of the maps you've sent along...
>
>
>> 8. exclude osd.56 (in default pool) from cluster:
>> ceph osd crush reweight osd.57 0
>>
>>
>> 9. add new osd to gstore2
>> ceph-deploy osd create gstore2:/dev/sdl:/dev/sdd1
>> ceph-deploy osd create gstore2:/dev/sdm:/dev/sdd2
>> …
>> ceph-deploy osd create gstore2:/dev/sds:/dev/sdg2
>>
>> 10. rename pool iscsi in default crush pool :
>> ceph osd pool rename iscsi iscsi-old
>>
>> 11. create new pool iscsi:
>> ceph osd pool create iscsi 2048 2048
>>
>> 12. set ruleset iscsi to new pool iscsi
>> ceph osd pool set iscsi crush_ruleset 3
>>
>> All OSDS downed with Segmentation fault
>
> Okay, so you switched to actually start using the new rule and the
> OSDs broke. It's possible there's a hole in our crush map testing that
> would let this through.
>
>> 13. failback ruleset 0 for pool iscsi
>> ceph osd pool set iscsi crush_ruleset 0
>>
>> delete ruleset iscsi, upload crushmap to cluster
>> https://dl.dropboxusercontent.com/u/2296931/ceph/crushmap14-new.txt
>>
>> OSD still Segmentation fault
>
> Yeah, once you've put a bad map into the system then you can't fix it
> by putting in a good one — all the OSDs need to evaluate the past maps
> on startups, which includes the bad one, which makes them crash again.
> :(
>
> Can you provide us a tarball of one of your monitor directories? We'd
> like to do some forensics on it to identify the scenario precisely and
> prevent it from happening again.
> -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw create user using API

[Yehuda Sadeh]

On Thu, Jul 18, 2013 at 12:50 AM, Alvaro Izquierdo Jimeno
 wrote:
> Hi,
>
>
>
> Reading the URL http://ceph.com/docs/next/radosgw/adminops/#create-user ,
> I’m trying to create a new user with:
>
>
>
> curl -v -X PUT  -d '{"uid": "alvaro", "display-name": "alvaro"}
> http://myradosgw/admin/user?format=json

To start with, you're not signing the request, so you're under the
'anonymous' user context. There's a s3curl script that can do it for
you.

>
>
>
> and a 403 response is received L
>
>
>
> So, Do I need a token? A token of who? Do I need a superuser with a special
> feature? Or I have  to setup something special in radosgw part into
> ceph.conf?

You need to have a user that have a specific caps set for it. In order
to create users you need the 'users' cap set to 'write'.

Yehuda
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to remove stale pgs?

[Ta Ba Tuan]

Hi Greg,

I don't lost any OSDs,

The first, Ceph had 4 pgs (0.f4f, 2.f4d, 0.2c8, 2.2c6) in stale state.
then, I created those pgs by following commands:

   ceph pg force_create_pg 0.f4f
   ceph pg force_create_pg 2.f4d
   ceph pg force_create_pg 0.2c8
   ceph pg force_create_pg 2.2c6

Now, after two days, Ceph still notify  above pgs in 'creating' state:

root@ceph-mon-01:~# ceph pg dump | grep 'stale\|creating'
0.f4f   0   0   0   0   0   0   0 stale+creating 
2013-07-17 16:35:06.882419  0'0 0'0 []  [68,12] 0'0 
0.000'0 0.00
2.f4d   0   0   0   0   0   0   0 
stale+creating  2013-07-17 16:35:22.826552  0'0 0'0 []  
[68,12] 0'0 0.000'0 0.00
0.2c8   0   0   0   0   0   0   0 
stale+creating  2013-07-17 14:30:54.280454  0'0 0'0 []  
[68,5]  0'0 0.000'0 0.00
2.2c6   0   0   0   0   0   0   0 
stale+creating  2013-07-17 16:35:28.445878  0'0 0'0 []  
[68,5]  0'0 0.000'0 0.00


How to delete above pgs, Greg?

Thank Greg so much.
--tuantaba



On 07/19/2013 05:01 AM, Gregory Farnum wrote:

On Thu, Jul 18, 2013 at 3:53 AM, Ta Ba Tuan  wrote:

Hi all,

I have 4 (stale+inactive) pgs, how to delete those pgs?

pgmap v59722: 21944 pgs: 4 stale, 12827 active+clean, 9113 active+degraded;
45689 MB data, 1006 GB used, 293 TB / 294 TB avail;

I found on google a long time, still can't resolve it.
Please, help me!

This depends on why they're stale+inactive. Can you pastebin the
output of "ceph pg dump" and provide the link?

Have you lost any OSDs?
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

ceph-users@lists.ceph.com

[zhan]

BAG（FOB&CIF).doc
Description: MS-Word document
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to remove stale pgs?

[Gregory Farnum]

On Thu, Jul 18, 2013 at 6:41 PM, Ta Ba Tuan  wrote:
> Hi Greg,
>
> I don't lost any OSDs,
>
> The first, Ceph had 4 pgs (0.f4f, 2.f4d, 0.2c8, 2.2c6) in stale state.
> then, I created those pgs by following commands:
>
> ceph pg force_create_pg 0.f4f
> ceph pg force_create_pg 2.f4d
> ceph pg force_create_pg 0.2c8
> ceph pg force_create_pg 2.2c6
>
> Now, after two days, Ceph still notify  above pgs in 'creating' state:

Were these PGs ever up and healthy, or is it a fresh cluster?
I notice that all of these PGs have osd 68 as primary; have you tried
restarting it? There are a couple of peering state bugs that people
have been turning up recently that can be resolved by restarting.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com

>
> root@ceph-mon-01:~# ceph pg dump | grep 'stale\|creating'
> 0.f4f   0   0   0   0   0   0   0
> stale+creating  2013-07-17 16:35:06.882419  0'0 0'0 []
> [68,12] 0'0  0.000'0 0.00
> 2.f4d   0   0   0   0   0   0   0
> stale+creating  2013-07-17 16:35:22.826552  0'0 0'0 []
> [68,12] 0'0  0.000'0 0.00
> 0.2c8   0   0   0   0   0   0   0
> stale+creating  2013-07-17 14:30:54.280454  0'0 0'0 []
> [68,5]  0'0  0.000'0 0.00
> 2.2c6   0   0   0   0   0   0   0
> stale+creating  2013-07-17 16:35:28.445878  0'0 0'0 []
> [68,5]  0'0  0.000'0 0.00
>
> How to delete above pgs, Greg?
>
> Thank Greg so much.
> --tuantaba
>
>
>
>
> On 07/19/2013 05:01 AM, Gregory Farnum wrote:
>
> On Thu, Jul 18, 2013 at 3:53 AM, Ta Ba Tuan  wrote:
>
> Hi all,
>
> I have 4 (stale+inactive) pgs, how to delete those pgs?
>
> pgmap v59722: 21944 pgs: 4 stale, 12827 active+clean, 9113 active+degraded;
> 45689 MB data, 1006 GB used, 293 TB / 294 TB avail;
>
> I found on google a long time, still can't resolve it.
> Please, help me!
>
> This depends on why they're stale+inactive. Can you pastebin the
> output of "ceph pg dump" and provide the link?
>
> Have you lost any OSDs?
> -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] xfs on ceph RBD resizing

[Jeffrey 'jf' Lim]

hey folks, I was hoping to be able to use xfs on top of RBD for a
deployment of mine. And was hoping for the resize of the RBD
(expansion, actually, would be my use case) in the future to be as
simple as a "resize on the fly", followed by an 'xfs_growfs'.'

I just found a recent post, though
(http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-June/002425.html),
that seems to indicate that what I have in mind may not be possible?
is this true? Do I have to unmount the system (or worse, do a reboot?)
before I can do an effective resize?

-jf

--
He who settles on the idea of the intelligent man as a static entity
only shows himself to be a fool.

"Every nonfree program has a lord, a master --
and if you use the program, he is your master."
--Richard Stallman
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v0.61.5 Cuttlefish update released

[Sage Weil]

A note on upgrading:

One of the fixes in 0.61.5 is with a 32bit vs 64bit bug with the feature 
bits.  We did not realize it before, but the fix will prevent 0.61.4 (or 
earlier) from forming a quorum with 0.61.5. This is similar to the upgrade 
from bobtail (and the future upgrade to dumpling). As such, we recommend 
you upgrade all monitors at once to avoid the potential for discruption in 
service.

I'm adding a note to the release notes.

Thanks!
sage


On Thu, 18 Jul 2013, Sage Weil wrote:

> We've prepared another update for the Cuttlefish v0.61.x series. This 
> release primarily contains monitor stability improvements, although there 
> are also some important fixes for ceph-osd for large clusters and a few 
> important CephFS fixes. We recommend that all v0.61.x users upgrade.
> 
>  * mon: misc sync improvements (faster, more reliable, better tuning)
>  * mon: enable leveldb cache by default (big performance improvement)
>  * mon: new scrub feature (primarily for diagnostic, testing purposes)
>  * mon: fix occasional leveldb assertion on startup
>  * mon: prevent reads until initial state is committed
>  * mon: improved logic for trimming old osdmaps
>  * mon: fix pick_addresses bug when expanding mon cluster
>  * mon: several small paxos fixes, improvements
>  * mon: fix bug osdmap trim behavior
>  * osd: fix several bugs with PG stat reporting
>  * osd: limit number of maps shared with peers (which could cause domino 
> failures)
>  * rgw: fix radosgw-admin buckets list (for all buckets)
>  * mds: fix occasional client failure to reconnect
>  * mds: fix bad list traversal after unlink
>  * mds: fix underwater dentry cleanup (occasional crash after mds restart)
>  * libcephfs, ceph-fuse: fix occasional hangs on umount
>  * libcephfs, ceph-fuse: fix old bug with O_LAZY vs O_NOATIME confusion
>  * ceph-disk: more robust journal device detection on RHEL/CentOS
>  * ceph-disk: better, simpler locking
>  * ceph-disk: do not inadvertantely mount over existing osd mounts
>  * ceph-disk: better handling for unusual device names
>  * sysvinit, upstart: handle symlinks in /var/lib/ceph/*
> 
> Please also refer to the complete release notes:
> 
>http://ceph.com/docs/master/release-notes/#v0-61-5-cuttlefish
> 
> You can get v0.61.5 from the usual locations:
> 
>  * Git at git://github.com/ceph/ceph.git
>  * Tarball at http://ceph.com/download/ceph-0.61.5.tar.gz
>  * For Debian/Ubuntu packages, see http://ceph.com/docs/master/install/debian
>  * For RPMs, see http://ceph.com/docs/master/install/rpm
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v0.61.5 Cuttlefish update released

[Stefan Priebe - Profihost AG]

All mons do not work anymore:

=== mon.a ===
Starting Ceph mon.a on ccad...
[21207]: (33) Numerical argument out of domain
failed: 'ulimit -n 8192;  /usr/bin/ceph-mon -i a --pid-file
/var/run/ceph/mon.a.pid -c /etc/ceph/ceph.conf '

Stefan

Am 19.07.2013 07:59, schrieb Sage Weil:
> A note on upgrading:
> 
> One of the fixes in 0.61.5 is with a 32bit vs 64bit bug with the feature 
> bits.  We did not realize it before, but the fix will prevent 0.61.4 (or 
> earlier) from forming a quorum with 0.61.5. This is similar to the upgrade 
> from bobtail (and the future upgrade to dumpling). As such, we recommend 
> you upgrade all monitors at once to avoid the potential for discruption in 
> service.
> 
> I'm adding a note to the release notes.
> 
> Thanks!
> sage
> 
> 
> On Thu, 18 Jul 2013, Sage Weil wrote:
> 
>> We've prepared another update for the Cuttlefish v0.61.x series. This 
>> release primarily contains monitor stability improvements, although there 
>> are also some important fixes for ceph-osd for large clusters and a few 
>> important CephFS fixes. We recommend that all v0.61.x users upgrade.
>>
>>  * mon: misc sync improvements (faster, more reliable, better tuning)
>>  * mon: enable leveldb cache by default (big performance improvement)
>>  * mon: new scrub feature (primarily for diagnostic, testing purposes)
>>  * mon: fix occasional leveldb assertion on startup
>>  * mon: prevent reads until initial state is committed
>>  * mon: improved logic for trimming old osdmaps
>>  * mon: fix pick_addresses bug when expanding mon cluster
>>  * mon: several small paxos fixes, improvements
>>  * mon: fix bug osdmap trim behavior
>>  * osd: fix several bugs with PG stat reporting
>>  * osd: limit number of maps shared with peers (which could cause domino 
>> failures)
>>  * rgw: fix radosgw-admin buckets list (for all buckets)
>>  * mds: fix occasional client failure to reconnect
>>  * mds: fix bad list traversal after unlink
>>  * mds: fix underwater dentry cleanup (occasional crash after mds restart)
>>  * libcephfs, ceph-fuse: fix occasional hangs on umount
>>  * libcephfs, ceph-fuse: fix old bug with O_LAZY vs O_NOATIME confusion
>>  * ceph-disk: more robust journal device detection on RHEL/CentOS
>>  * ceph-disk: better, simpler locking
>>  * ceph-disk: do not inadvertantely mount over existing osd mounts
>>  * ceph-disk: better handling for unusual device names
>>  * sysvinit, upstart: handle symlinks in /var/lib/ceph/*
>>
>> Please also refer to the complete release notes:
>>
>>http://ceph.com/docs/master/release-notes/#v0-61-5-cuttlefish
>>
>> You can get v0.61.5 from the usual locations:
>>
>>  * Git at git://github.com/ceph/ceph.git
>>  * Tarball at http://ceph.com/download/ceph-0.61.5.tar.gz
>>  * For Debian/Ubuntu packages, see http://ceph.com/docs/master/install/debian
>>  * For RPMs, see http://ceph.com/docs/master/install/rpm
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com