Re: [ceph-users] v0.80.4 Firefly released

[Andrija Panic]

Hi Sage,

can anyone validate, if there is still "bug" inside RPMs that does
automatic CEPH service restart after updating packages ?

We are instructed to first update/restart MONs, and after that OSD - but
that is impossible if we have MON+OSDs on same host...since the ceph is
automaticaly restarted with YUM/RPM, but NOT automaticaly restarted on
Ubuntu/Debian (as reported by some other list memeber...)

Thanks


On 16 July 2014 01:45, Sage Weil  wrote:

> This Firefly point release fixes an potential data corruption problem
> when ceph-osd daemons run on top of XFS and service Firefly librbd
> clients.  A recently added allocation hint that RBD utilizes triggers
> an XFS bug on some kernels (Linux 3.2, and likely others) that leads
> to data corruption and deep-scrub errors (and inconsistent PGs).  This
> release avoids the situation by disabling the allocation hint until we
> can validate which kernels are affected and/or are known to be safe to
> use the hint on.
>
> We recommend that all v0.80.x Firefly users urgently upgrade,
> especially if they are using RBD.
>
> Notable Changes
> ---
>
> * osd: disable XFS extsize hint by default (#8830, Samuel Just)
> * rgw: fix extra data pool default name (Yehuda Sadeh)
>
> For more detailed information, see:
>
>   http://ceph.com/docs/master/_downloads/v0.80.4.txt
>
> Getting Ceph
> 
>
> * Git at git://github.com/ceph/ceph.git
> * Tarball at http://ceph.com/download/ceph-0.80.4.tar.gz
> * For packages, see http://ceph.com/docs/master/install/get-packages
> * For ceph-deploy, see
> http://ceph.com/docs/master/install/install-ceph-deploy
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 

Andrija Panić
--
  http://admintweets.com
--
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-fuse couldn't be connect.

[Jaemyoun Lee]

The result is same.

# ceph-fuse --debug-ms 1 --debug-client 10 -m 192.168.122.106:6789 /mnt
ceph-fuse[3296] :  starting ceph client

And the log file is

# cat /var/log/ceph/ceph-client.admin.log
2014-07-16 17:08:13.146032 7f9a212f87c0  0 ceph version 0.80.1
(a38fe1169b6d2ac98b427334c12d7cf81f809b74), process ceph-fuse, pid 3294
2014-07-16 17:08:13.156429 7f9a212f87c0  1 -- :/0 messenger.start
2014-07-16 17:08:13.157537 7f9a212f87c0  1 -- :/3296 -->
192.168.122.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- ?+0
0x7f9a23c0e6c0 con 0x7f9a23c0dd30
2014-07-16 17:08:13.158198 7f9a212f6700  1 -- 192.168.122.166:0/3296
learned my addr 192.168.122.166:0/3296
2014-07-16 17:08:13.158505 7f9a167fc700 10 client.-1 ms_handle_connect on
192.168.122.106:6789/0
2014-07-16 17:08:13.159083 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
mon.0 192.168.122.106:6789/0 1  mon_map v1  193+0+0 (4132823754 0
0) 0x7f9a0ab0 con 0x7f9a23c0dd30
2014-07-16 17:08:13.159182 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
mon.0 192.168.122.106:6789/0 2  auth_reply(proto 2 0 (0) Success) v1
 33+0+0 (1915318666 0 0) 0x7f9a0f60 con 0x7f9a23c0dd30
2014-07-16 17:08:13.159375 7f9a167fc700  1 -- 192.168.122.166:0/3296 -->
192.168.122.106:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0
0x7f9a0c0013a0 con 0x7f9a23c0dd30
2014-07-16 17:08:13.159845 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
mon.0 192.168.122.106:6789/0 3  auth_reply(proto 2 0 (0) Success) v1
 206+0+0 (2967970554 0 0) 0x7f9a0f60 con 0x7f9a23c0dd30
2014-07-16 17:08:13.159976 7f9a167fc700  1 -- 192.168.122.166:0/3296 -->
192.168.122.106:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0
0x7f9a0c001ec0 con 0x7f9a23c0dd30
2014-07-16 17:08:13.160810 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
mon.0 192.168.122.106:6789/0 4  auth_reply(proto 2 0 (0) Success) v1
 409+0+0 (3799435439 0 0) 0x7f9a11d0 con 0x7f9a23c0dd30
2014-07-16 17:08:13.160945 7f9a167fc700  1 -- 192.168.122.166:0/3296 -->
192.168.122.106:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0
0x7f9a23c102c0 con 0x7f9a23c0dd30
2014-07-16 17:08:13.160979 7f9a167fc700  1 -- 192.168.122.166:0/3296 -->
192.168.122.106:6789/0 -- mon_subscribe({mdsmap=0+,osdmap=0}) v2 -- ?+0
0x7f9a23c10630 con 0x7f9a23c0dd30
2014-07-16 17:08:13.161033 7f9a212f87c0  2 client.4705 mounted: have osdmap
0 and mdsmap 0
2014-07-16 17:08:13.161056 7f9a212f87c0 10 client.4705 did not get mds
through better means, so chose random mds -1
2014-07-16 17:08:13.161059 7f9a212f87c0 10 client.4705  target mds.-1 not
active, waiting for new mdsmap
2014-07-16 17:08:13.161668 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
mon.0 192.168.122.106:6789/0 5  osd_map(45..45 src has 1..45) v3 
3907+0+0 (2386867192 0 0) 0x7f9a2060 con 0x7f9a23c0dd30
2014-07-16 17:08:13.161843 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
mon.0 192.168.122.106:6789/0 6  mdsmap(e 1) v1  396+0+0 (394292161
0 0) 0x7f9a2500 con 0x7f9a23c0dd30
2014-07-16 17:08:13.161861 7f9a167fc700  1 client.4705 handle_mds_map epoch
1
2014-07-16 17:08:13.161884 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
mon.0 192.168.122.106:6789/0 7  osd_map(45..45 src has 1..45) v3 
3907+0+0 (2386867192 0 0) 0x7f9a37a0 con 0x7f9a23c0dd30
2014-07-16 17:08:13.161900 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
mon.0 192.168.122.106:6789/0 8  mon_subscribe_ack(300s) v1  20+0+0
(4226112827 0 0) 0x7f9a3c40 con 0x7f9a23c0dd30
2014-07-16 17:08:13.161932 7f9a212f87c0 10 client.4705 did not get mds
through better means, so chose random mds -1
2014-07-16 17:08:13.161942 7f9a212f87c0 10 client.4705  target mds.-1 not
active, waiting for new mdsmap
2014-07-16 17:08:14.161453 7f9a177fe700 10 client.4705 renew_caps()
2014-07-16 17:08:34.166977 7f9a177fe700 10 client.4705 renew_caps()
2014-07-16 17:08:54.171234 7f9a177fe700 10 client.4705 renew_caps()
2014-07-16 17:09:14.174106 7f9a177fe700 10 client.4705 renew_caps()
2014-07-16 17:09:34.177062 7f9a177fe700 10 client.4705 renew_caps()
2014-07-16 17:09:54.179365 7f9a177fe700 10 client.4705 renew_caps()
2014-07-16 17:10:14.181731 7f9a177fe700 10 client.4705 renew_caps()
2014-07-16 17:10:34.184270 7f9a177fe700 10 client.4705 renew_caps()
2014-07-16 17:10:46.161158 7f9a15ffb700  1 -- 192.168.122.166:0/3296 -->
192.168.122.106:6789/0 -- mon_subscribe({mdsmap=2+,monmap=2+}) v2 -- ?+0
0x7f99f8002c50 con 0x7f9a23c0dd30
2014-07-16 17:10:46.161770 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
mon.0 192.168.122.106:6789/0 9  mon_subscribe_ack(300s) v1  20+0+0
(4226112827 0 0) 0x7f9a3c40 con 0x7f9a23c0dd30
2014-07-16 17:10:54.186908 7f9a177fe700 10 client.4705 renew_caps()
2014-07-16 17:11:14.189613 7f9a177fe700 10 client.4705 renew_caps()
2014-07-16 17:11:34.192055 7f9a177fe700 10 client.4705 renew_caps()
2014-07-16 17:11:54.194663 7f9a177fe700 10 client.4705 renew_caps()
2014-07-16 17:12:14.196991 7f9a177fe700 10 client.4705 renew_caps()
2014-07-16 17:12:34.199710 7f9a177fe700 10 client.4705

Re: [ceph-users] v0.80.4 Firefly released

[James Harper]

Can you offer some comments on what the impact is likely to be to the data in 
an affected cluster? Should all data now be treated with suspicion and restored 
back to before the firefly upgrade?

James

> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Sage Weil
> Sent: Wednesday, 16 July 2014 9:46 AM
> To: ceph-de...@vger.kernel.org; ceph-us...@ceph.com
> Subject: [ceph-users] v0.80.4 Firefly released
> 
> This Firefly point release fixes an potential data corruption problem
> when ceph-osd daemons run on top of XFS and service Firefly librbd
> clients.  A recently added allocation hint that RBD utilizes triggers
> an XFS bug on some kernels (Linux 3.2, and likely others) that leads
> to data corruption and deep-scrub errors (and inconsistent PGs).  This
> release avoids the situation by disabling the allocation hint until we
> can validate which kernels are affected and/or are known to be safe to
> use the hint on.
> 
> We recommend that all v0.80.x Firefly users urgently upgrade,
> especially if they are using RBD.
> 
> Notable Changes
> ---
> 
> * osd: disable XFS extsize hint by default (#8830, Samuel Just)
> * rgw: fix extra data pool default name (Yehuda Sadeh)
> 
> For more detailed information, see:
> 
>   http://ceph.com/docs/master/_downloads/v0.80.4.txt
> 
> Getting Ceph
> 
> 
> * Git at git://github.com/ceph/ceph.git
> * Tarball at http://ceph.com/download/ceph-0.80.4.tar.gz
> * For packages, see http://ceph.com/docs/master/install/get-packages
> * For ceph-deploy, see http://ceph.com/docs/master/install/install-ceph-
> deploy
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v0.80.4 Firefly released

[Pavel V. Kaygorodov]

Hi!

I'm trying to install ceph on Debian wheezy (from deb http://ceph.com/debian/ 
wheezy main) and getting following error:

# apt-get update && apt-get dist-upgrade -y && apt-get install -y ceph

...

The following packages have unmet dependencies:
 ceph : Depends: ceph-common (>= 0.78-500) but it is not going to be installed
Depends: libboost-system1.49.0 (>= 1.49.0-1) but it is not installable
Depends: libboost-thread1.49.0 (>= 1.49.0-1) but it is not installable
Recommends: btrfs-tools but it is not going to be installed
Recommends: ceph-mds but it is not going to be installed
Recommends: librados2 but it is not going to be installed
Recommends: librbd1 but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

Pavel.



16 июля 2014 г., в 3:45, Sage Weil  написал(а):

> This Firefly point release fixes an potential data corruption problem
> when ceph-osd daemons run on top of XFS and service Firefly librbd
> clients.  A recently added allocation hint that RBD utilizes triggers
> an XFS bug on some kernels (Linux 3.2, and likely others) that leads
> to data corruption and deep-scrub errors (and inconsistent PGs).  This
> release avoids the situation by disabling the allocation hint until we
> can validate which kernels are affected and/or are known to be safe to
> use the hint on.
> 
> We recommend that all v0.80.x Firefly users urgently upgrade,
> especially if they are using RBD.
> 
> Notable Changes
> ---
> 
> * osd: disable XFS extsize hint by default (#8830, Samuel Just)
> * rgw: fix extra data pool default name (Yehuda Sadeh)
> 
> For more detailed information, see:
> 
>  http://ceph.com/docs/master/_downloads/v0.80.4.txt
> 
> Getting Ceph
> 
> 
> * Git at git://github.com/ceph/ceph.git
> * Tarball at http://ceph.com/download/ceph-0.80.4.tar.gz
> * For packages, see http://ceph.com/docs/master/install/get-packages
> * For ceph-deploy, see http://ceph.com/docs/master/install/install-ceph-deploy
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v0.80.4 Firefly released

[Sylvain Munaut]

On Wed, Jul 16, 2014 at 10:50 AM, James Harper  wrote:
> Can you offer some comments on what the impact is likely to be to the data in 
> an affected cluster? Should all data now be treated with suspicion and 
> restored back to before the firefly upgrade?

Yes, I'd definitely like to know that too ... I upgraded a couple
weeks ago to firefly and I run precise with a 3.2 kernel on XFS. I
didn't see any deep scrubs error however.

Also does this affect RBD only or can also affect radosgw ?

And finally, with 0.80.4, radosgw still crashes on multi-part upload
as I reported in another thread.

Cheers,

   Sylvain
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v0.80.4 Firefly released

[James Harper]

> Hi!
> 
> I'm trying to install ceph on Debian wheezy (from deb
> http://ceph.com/debian/ wheezy main) and getting following error:
> 
> # apt-get update && apt-get dist-upgrade -y && apt-get install -y ceph
> 
> ...
> 
> The following packages have unmet dependencies:
>  ceph : Depends: ceph-common (>= 0.78-500) but it is not going to be
> installed
> Depends: libboost-system1.49.0 (>= 1.49.0-1) but it is not installable
> Depends: libboost-thread1.49.0 (>= 1.49.0-1) but it is not installable
> Recommends: btrfs-tools but it is not going to be installed
> Recommends: ceph-mds but it is not going to be installed
> Recommends: librados2 but it is not going to be installed
> Recommends: librbd1 but it is not going to be installed
> E: Unable to correct problems, you have held broken packages.
> 

I've upgraded just now. Are you running with Jessie but using wheezy from 
ceph.com (the only option for ceph.com)?

If so, download and manually install libboost-system1.49.0 and 
libboost-thread1.49.0 from packages.debian.org. It baffled me for a minute - 
1.55.0 is >= 1.49.0, but libboost-system1.49.0 is a completely different 
package to libboost-system1.55.0 so they can't be compared.

Debian has 0.80 in Jessie but it is too old. It's going to be a bit of a pain 
for a while.

Ceph maintainers: Could we get ceph packages built for Jessie?

James

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] [radosgw] Creating buckets with different owner

[Patrycja Szabłowska]

Hi,

Is it possible to set owner of a bucket or an object to someone else?
I've got a user who was created with flag --system and is able to
create buckets and objects.
I've created a bucket using boto and I've got FULL CONTROL over it:
http://acs.amazonaws.com/groups/global/AllUsers = READ, M.
Tester (owner) = FULL_CONTROL>

but trying to set owner to someone else gives me this:

boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
AccessDenied


So I wonder - is it even possible to change owner of a bucket or
create a bucket for a different owner than myself?



Thanks,
Patrycja Szabłowska
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-fuse couldn't be connect.

[Gregory Farnum]

Your MDS isn't running or isn't active.
-Greg

On Wednesday, July 16, 2014, Jaemyoun Lee  wrote:

>
> The result is same.
>
> # ceph-fuse --debug-ms 1 --debug-client 10 -m 192.168.122.106:6789 /mnt
> ceph-fuse[3296] :  starting ceph client
>
> And the log file is
>
> # cat /var/log/ceph/ceph-client.admin.log
> 2014-07-16 17:08:13.146032 7f9a212f87c0  0 ceph version 0.80.1
> (a38fe1169b6d2ac98b427334c12d7cf81f809b74), process ceph-fuse, pid 3294
> 2014-07-16 17:08:13.156429 7f9a212f87c0  1 -- :/0 messenger.start
> 2014-07-16 17:08:13.157537 7f9a212f87c0  1 -- :/3296 -->
> 192.168.122.106:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- ?+0
> 0x7f9a23c0e6c0 con 0x7f9a23c0dd30
> 2014-07-16 17:08:13.158198 7f9a212f6700  1 -- 192.168.122.166:0/3296
> learned my addr 192.168.122.166:0/3296
> 2014-07-16 17:08:13.158505 7f9a167fc700 10 client.-1 ms_handle_connect on
> 192.168.122.106:6789/0
> 2014-07-16 17:08:13.159083 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
> mon.0 192.168.122.106:6789/0 1  mon_map v1  193+0+0 (4132823754 0
> 0) 0x7f9a0ab0 con 0x7f9a23c0dd30
> 2014-07-16 17:08:13.159182 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
> mon.0 192.168.122.106:6789/0 2  auth_reply(proto 2 0 (0) Success) v1
>  33+0+0 (1915318666 0 0) 0x7f9a0f60 con 0x7f9a23c0dd30
> 2014-07-16 17:08:13.159375 7f9a167fc700  1 -- 192.168.122.166:0/3296 -->
> 192.168.122.106:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0
> 0x7f9a0c0013a0 con 0x7f9a23c0dd30
> 2014-07-16 17:08:13.159845 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
> mon.0 192.168.122.106:6789/0 3  auth_reply(proto 2 0 (0) Success) v1
>  206+0+0 (2967970554 0 0) 0x7f9a0f60 con 0x7f9a23c0dd30
> 2014-07-16 17:08:13.159976 7f9a167fc700  1 -- 192.168.122.166:0/3296 -->
> 192.168.122.106:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0
> 0x7f9a0c001ec0 con 0x7f9a23c0dd30
> 2014-07-16 17:08:13.160810 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
> mon.0 192.168.122.106:6789/0 4  auth_reply(proto 2 0 (0) Success) v1
>  409+0+0 (3799435439 0 0) 0x7f9a11d0 con 0x7f9a23c0dd30
> 2014-07-16 17:08:13.160945 7f9a167fc700  1 -- 192.168.122.166:0/3296 -->
> 192.168.122.106:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0
> 0x7f9a23c102c0 con 0x7f9a23c0dd30
> 2014-07-16 17:08:13.160979 7f9a167fc700  1 -- 192.168.122.166:0/3296 -->
> 192.168.122.106:6789/0 -- mon_subscribe({mdsmap=0+,osdmap=0}) v2 -- ?+0
> 0x7f9a23c10630 con 0x7f9a23c0dd30
> 2014-07-16 17:08:13.161033 7f9a212f87c0  2 client.4705 mounted: have
> osdmap 0 and mdsmap 0
> 2014-07-16 17:08:13.161056 7f9a212f87c0 10 client.4705 did not get mds
> through better means, so chose random mds -1
> 2014-07-16 17:08:13.161059 7f9a212f87c0 10 client.4705  target mds.-1 not
> active, waiting for new mdsmap
> 2014-07-16 17:08:13.161668 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
> mon.0 192.168.122.106:6789/0 5  osd_map(45..45 src has 1..45) v3 
> 3907+0+0 (2386867192 0 0) 0x7f9a2060 con 0x7f9a23c0dd30
> 2014-07-16 17:08:13.161843 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
> mon.0 192.168.122.106:6789/0 6  mdsmap(e 1) v1  396+0+0
> (394292161 0 0) 0x7f9a2500 con 0x7f9a23c0dd30
> 2014-07-16 17:08:13.161861 7f9a167fc700  1 client.4705 handle_mds_map
> epoch 1
> 2014-07-16 17:08:13.161884 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
> mon.0 192.168.122.106:6789/0 7  osd_map(45..45 src has 1..45) v3 
> 3907+0+0 (2386867192 0 0) 0x7f9a37a0 con 0x7f9a23c0dd30
> 2014-07-16 17:08:13.161900 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
> mon.0 192.168.122.106:6789/0 8  mon_subscribe_ack(300s) v1 
> 20+0+0 (4226112827 0 0) 0x7f9a3c40 con 0x7f9a23c0dd30
> 2014-07-16 17:08:13.161932 7f9a212f87c0 10 client.4705 did not get mds
> through better means, so chose random mds -1
> 2014-07-16 17:08:13.161942 7f9a212f87c0 10 client.4705  target mds.-1 not
> active, waiting for new mdsmap
> 2014-07-16 17:08:14.161453 7f9a177fe700 10 client.4705 renew_caps()
> 2014-07-16 17:08:34.166977 7f9a177fe700 10 client.4705 renew_caps()
> 2014-07-16 17:08:54.171234 7f9a177fe700 10 client.4705 renew_caps()
> 2014-07-16 17:09:14.174106 7f9a177fe700 10 client.4705 renew_caps()
> 2014-07-16 17:09:34.177062 7f9a177fe700 10 client.4705 renew_caps()
> 2014-07-16 17:09:54.179365 7f9a177fe700 10 client.4705 renew_caps()
> 2014-07-16 17:10:14.181731 7f9a177fe700 10 client.4705 renew_caps()
> 2014-07-16 17:10:34.184270 7f9a177fe700 10 client.4705 renew_caps()
> 2014-07-16 17:10:46.161158 7f9a15ffb700  1 -- 192.168.122.166:0/3296 -->
> 192.168.122.106:6789/0 -- mon_subscribe({mdsmap=2+,monmap=2+}) v2 -- ?+0
> 0x7f99f8002c50 con 0x7f9a23c0dd30
> 2014-07-16 17:10:46.161770 7f9a167fc700  1 -- 192.168.122.166:0/3296 <==
> mon.0 192.168.122.106:6789/0 9  mon_subscribe_ack(300s) v1 
> 20+0+0 (4226112827 0 0) 0x7f9a3c40 con 0x7f9a23c0dd30
> 2014-07-16 17:10:54.186908 7f9a177fe700 10 client.4705 renew_caps()
> 2014-07-16 17:11:14.189613 7f9a177fe700 10 client.4705 renew_caps

Re: [ceph-users] [ceph-calamari] Issue during ceph-deploy osd activate

[John Spray]

Hi Shubhendu,

ceph-deploy is not part of Calamari, the ceph-users list is a better
place to get help with that.  I have CC'd the list here.

It will help if you can specify the series of ceph-deploy commands you
ran before the failing one as well.

Thanks,
John

On Wed, Jul 16, 2014 at 9:29 AM, Shubhendu Tripathi  wrote:
> Hi,
>
> I am trying to setup a ceph cluster. But while activating the OSDs the admin
> gets timeout while running the command "sudo ceph --cluster=ceph osd stat
> --format=json" on remote node.
>
> If I try to execute the same command on one of the nodes of the cluster, I
> get the below errors -
>
> [ceph@dhcp43-15 ~]$ sudo ceph-disk-activate --mark-init sysvinit --mount
> /var/local/osd0
> INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=mycephcluster
> --show-config-value=fsid
> INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph
> --show-config-value=fsid
> INFO:ceph-disk:Running command: /usr/bin/ceph --cluster ceph --name
> client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd
> create --concise 450d3d27-95ce-4925-890a-908b35fc9255
> 2014-07-16 13:47:28.073034 7fce166fb700  0 -- 10.70.43.15:0/1031798 >>
> 10.70.43.20:6789/0 pipe(0x7fce08000c00 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7fce08000e70).fault
>
>
> It keeps trying for 300 sec and then gives up with error -
>
> 2014-07-16 13:02:48.077898 7f71b81dd700  0 monclient(hunting): authenticate
> timed out after 300
> 2014-07-16 13:02:48.078033 7f71b81dd700  0 librados: client.bootstrap-osd
> authentication error (110) Connection timed out
> Error connecting to cluster: TimedOut
> ceph-disk: Error: ceph osd create failed: Command '/usr/bin/ceph' returned
> non-zero exit status 1:
>
> Kindly help resolving the same.
>
> Regards,
> Shubhendu
>
>
> ___
> ceph-calamari mailing list
> ceph-calam...@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-calamari-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph osd crush tunables optimal AND add new OSD at the same time

[Quenten Grasso]

Hi Sage, Andrija & List

I have seen the tuneables issue on our cluster when I upgraded to firefly.

I ended up going back to legacy settings after about an hour as my cluster is 
of 55 3TB OSD’s over 5 nodes and it decided it needed to move around 32% of our 
data, which after an hour all of our vm’s were frozen and I had to revert the 
change back to legacy settings and wait about the same time again until our 
cluster had recovered and reboot our vms. (wasn’t really expecting that one 
from the patch notes)

Also our CPU usage went through the roof as well on our nodes, do you per 
chance have your metadata servers co-located on your osd nodes as we do?  I’ve 
been thinking about trying to move these to dedicated nodes as it may resolve 
our issues.

Regards,
Quenten

From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
Andrija Panic
Sent: Tuesday, 15 July 2014 8:38 PM
To: Sage Weil
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] ceph osd crush tunables optimal AND add new OSD at 
the same time

Hi Sage,

since this problem is tunables-related, do we need to expect same behavior or 
not  when we do regular data rebalancing caused by adding new/removing OSD? I 
guess not, but would like your confirmation.
I'm already on optimal tunables, but I'm afraid to test this by i.e. shuting 
down 1 OSD.

Thanks,
Andrija

On 14 July 2014 18:18, Sage Weil mailto:sw...@redhat.com>> 
wrote:
I've added some additional notes/warnings to the upgrade and release
notes:

 https://github.com/ceph/ceph/commit/fc597e5e3473d7db6548405ce347ca7732832451

If there is somewhere else where you think a warning flag would be useful,
let me know!

Generally speaking, we want to be able to cope with huge data rebalances
without interrupting service.  It's an ongoing process of improving the
recovery vs client prioritization, though, and removing sources of
overhead related to rebalancing... and it's clearly not perfect yet. :/

sage


On Sun, 13 Jul 2014, Andrija Panic wrote:

> Hi,
> after seting ceph upgrade (0.72.2 to 0.80.3) I have issued "ceph osd crush
> tunables optimal" and after only few minutes I have added 2 more OSDs to the
> CEPH cluster...
>
> So these 2 changes were more or a less done at the same time - rebalancing
> because of tunables optimal, and rebalancing because of adding new OSD...
>
> Result - all VMs living on CEPH storage have gone mad, no disk access
> efectively, blocked so to speak.
>
> Since this rebalancing took 5h-6h, I had bunch of VMs down for that long...
>
> Did I do wrong by causing "2 rebalancing" to happen at the same time ?
> Is this behaviour normal, to cause great load on all VMs because they are
> unable to access CEPH storage efectively ?
>
> Thanks for any input...
> --
>
> Andrija Pani?
>
>



--

Andrija Panić
--
  http://admintweets.com
--
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph osd crush tunables optimal AND add new OSD at the same time

[Andrei Mikhailovsky]

Quenten, 

We've got two monitors sitting on the osd servers and one on a different 
server. 

Andrei 

-- 
Andrei Mikhailovsky 
Director 
Arhont Information Security 

Web: http://www.arhont.com 
http://www.wi-foo.com 
Tel: +44 (0)870 4431337 
Fax: +44 (0)208 429 3111 
PGP: Key ID - 0x2B3438DE 
PGP: Server - keyserver.pgp.com 

DISCLAIMER 

The information contained in this email is intended only for the use of the 
person(s) to whom it is addressed and may be confidential or contain legally 
privileged information. If you are not the intended recipient you are hereby 
notified that any perusal, use, distribution, copying or disclosure is strictly 
prohibited. If you have received this email in error please immediately advise 
us by return email at and...@arhont.com and delete and purge the email and any 
attachments without making a copy. 


- Original Message -

From: "Quenten Grasso"  
To: "Andrija Panic" , "Sage Weil"  
Cc: ceph-users@lists.ceph.com 
Sent: Wednesday, 16 July, 2014 1:20:19 PM 
Subject: Re: [ceph-users] ceph osd crush tunables optimal AND add new OSD at 
the same time 



Hi Sage, Andrija & List 



I have seen the tuneables issue on our cluster when I upgraded to firefly. 



I ended up going back to legacy settings after about an hour as my cluster is 
of 55 3TB OSD’s over 5 nodes and it decided it needed to move around 32% of our 
data, which after an hour all of our vm’s were frozen and I had to revert the 
change back to legacy settings and wait about the same time again until our 
cluster had recovered and reboot our vms. (wasn’t really expecting that one 
from the patch notes) 



Also our CPU usage went through the roof as well on our nodes, do you per 
chance have your metadata servers co-located on your osd nodes as we do? I’ve 
been thinking about trying to move these to dedicated nodes as it may resolve 
our issues. 



Regards, 

Quenten 



From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
Andrija Panic 
Sent: Tuesday, 15 July 2014 8:38 PM 
To: Sage Weil 
Cc: ceph-users@lists.ceph.com 
Subject: Re: [ceph-users] ceph osd crush tunables optimal AND add new OSD at 
the same time 




Hi Sage, 





since this problem is tunables-related, do we need to expect same behavior or 
not when we do regular data rebalancing caused by adding new/removing OSD? I 
guess not, but would like your confirmation. 


I'm already on optimal tunables, but I'm afraid to test this by i.e. shuting 
down 1 OSD. 





Thanks, 
Andrija 





On 14 July 2014 18:18, Sage Weil < sw...@redhat.com > wrote: 



I've added some additional notes/warnings to the upgrade and release 
notes: 

https://github.com/ceph/ceph/commit/fc597e5e3473d7db6548405ce347ca7732832451 

If there is somewhere else where you think a warning flag would be useful, 
let me know! 

Generally speaking, we want to be able to cope with huge data rebalances 
without interrupting service. It's an ongoing process of improving the 
recovery vs client prioritization, though, and removing sources of 
overhead related to rebalancing... and it's clearly not perfect yet. :/ 

sage 




On Sun, 13 Jul 2014, Andrija Panic wrote: 

> Hi, 
> after seting ceph upgrade (0.72.2 to 0.80.3) I have issued "ceph osd crush 
> tunables optimal" and after only few minutes I have added 2 more OSDs to the 
> CEPH cluster... 
> 
> So these 2 changes were more or a less done at the same time - rebalancing 
> because of tunables optimal, and rebalancing because of adding new OSD... 
> 
> Result - all VMs living on CEPH storage have gone mad, no disk access 
> efectively, blocked so to speak. 
> 
> Since this rebalancing took 5h-6h, I had bunch of VMs down for that long... 
> 
> Did I do wrong by causing "2 rebalancing" to happen at the same time ? 
> Is this behaviour normal, to cause great load on all VMs because they are 
> unable to access CEPH storage efectively ? 
> 
> Thanks for any input... 
> -- 
> 


> Andrija Pani? 
> 
> 











-- 





Andrija Panić 


-- 


http://admintweets.com 


-- 

___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph osd crush tunables optimal AND add new OSD at the same time

[Andrija Panic]

For me, 3 nodes, 1MON+ 2x2TB OSDs on each node... no mds used...
I went through pain of waiting for data rebalancing and now I'm on
"optimal" tunables...
Cheers


On 16 July 2014 14:29, Andrei Mikhailovsky  wrote:

> Quenten,
>
> We've got two monitors sitting on the osd servers and one on a different
> server.
>
> Andrei
>
> --
> Andrei Mikhailovsky
> Director
> Arhont Information Security
>
> Web: http://www.arhont.com
> http://www.wi-foo.com
> Tel: +44 (0)870 4431337
> Fax: +44 (0)208 429 3111
> PGP: Key ID - 0x2B3438DE
> PGP: Server - keyserver.pgp.com
>
> DISCLAIMER
>
> The information contained in this email is intended only for the use of
> the person(s) to whom it is addressed and may be confidential or contain
> legally privileged information. If you are not the intended recipient you
> are hereby notified that any perusal, use, distribution, copying or
> disclosure is strictly prohibited. If you have received this email in error
> please immediately advise us by return email at and...@arhont.com and
> delete and purge the email and any attachments without making a copy.
>
>
> --
> *From: *"Quenten Grasso" 
> *To: *"Andrija Panic" , "Sage Weil" <
> sw...@redhat.com>
> *Cc: *ceph-users@lists.ceph.com
> *Sent: *Wednesday, 16 July, 2014 1:20:19 PM
>
> *Subject: *Re: [ceph-users] ceph osd crush tunables optimal AND add new
> OSD at the same time
>
> Hi Sage, Andrija & List
>
>
>
> I have seen the tuneables issue on our cluster when I upgraded to firefly.
>
>
>
> I ended up going back to legacy settings after about an hour as my cluster
> is of 55 3TB OSD’s over 5 nodes and it decided it needed to move around 32%
> of our data, which after an hour all of our vm’s were frozen and I had to
> revert the change back to legacy settings and wait about the same time
> again until our cluster had recovered and reboot our vms. (wasn’t really
> expecting that one from the patch notes)
>
>
>
> Also our CPU usage went through the roof as well on our nodes, do you per
> chance have your metadata servers co-located on your osd nodes as we do?
>  I’ve been thinking about trying to move these to dedicated nodes as it may
> resolve our issues.
>
>
>
> Regards,
>
> Quenten
>
>
>
> *From:* ceph-users [mailto:ceph-users-boun...@lists.ceph.com] *On Behalf
> Of *Andrija Panic
> *Sent:* Tuesday, 15 July 2014 8:38 PM
> *To:* Sage Weil
> *Cc:* ceph-users@lists.ceph.com
> *Subject:* Re: [ceph-users] ceph osd crush tunables optimal AND add new
> OSD at the same time
>
>
>
> Hi Sage,
>
>
>
> since this problem is tunables-related, do we need to expect same behavior
> or not  when we do regular data rebalancing caused by adding new/removing
> OSD? I guess not, but would like your confirmation.
>
> I'm already on optimal tunables, but I'm afraid to test this by i.e.
> shuting down 1 OSD.
>
>
>
> Thanks,
> Andrija
>
>
>
> On 14 July 2014 18:18, Sage Weil  wrote:
>
> I've added some additional notes/warnings to the upgrade and release
> notes:
>
>
> https://github.com/ceph/ceph/commit/fc597e5e3473d7db6548405ce347ca7732832451
>
> If there is somewhere else where you think a warning flag would be useful,
> let me know!
>
> Generally speaking, we want to be able to cope with huge data rebalances
> without interrupting service.  It's an ongoing process of improving the
> recovery vs client prioritization, though, and removing sources of
> overhead related to rebalancing... and it's clearly not perfect yet. :/
>
> sage
>
>
>
> On Sun, 13 Jul 2014, Andrija Panic wrote:
>
> > Hi,
> > after seting ceph upgrade (0.72.2 to 0.80.3) I have issued "ceph osd
> crush
> > tunables optimal" and after only few minutes I have added 2 more OSDs to
> the
> > CEPH cluster...
> >
> > So these 2 changes were more or a less done at the same time -
> rebalancing
> > because of tunables optimal, and rebalancing because of adding new OSD...
> >
> > Result - all VMs living on CEPH storage have gone mad, no disk access
> > efectively, blocked so to speak.
> >
> > Since this rebalancing took 5h-6h, I had bunch of VMs down for that
> long...
> >
> > Did I do wrong by causing "2 rebalancing" to happen at the same time ?
> > Is this behaviour normal, to cause great load on all VMs because they are
> > unable to access CEPH storage efectively ?
> >
> > Thanks for any input...
> > --
> >
>
> > Andrija Pani?
> >
> >
>
>
>
>
>
> --
>
>
>
> Andrija Panić
>
> --
>
>   http://admintweets.com
>
> --
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>


-- 

Andrija Panić
--
  http://admintweets.com
--
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph osd crush tunables optimal AND add new OSD at the same time

[Danny Luhde-Thompson]

With 34 x 4TB OSDs over 4 hosts, I had 30% objects moved - about half full
and took around 12 hours.  Except now I can't use the kclient any more -
wish I'd read that first.


On 16 July 2014 13:36, Andrija Panic  wrote:

> For me, 3 nodes, 1MON+ 2x2TB OSDs on each node... no mds used...
> I went through pain of waiting for data rebalancing and now I'm on
> "optimal" tunables...
> Cheers
>
>
> On 16 July 2014 14:29, Andrei Mikhailovsky  wrote:
>
>> Quenten,
>>
>> We've got two monitors sitting on the osd servers and one on a different
>> server.
>>
>> Andrei
>>
>> --
>> Andrei Mikhailovsky
>> Director
>> Arhont Information Security
>>
>> Web: http://www.arhont.com
>> http://www.wi-foo.com
>> Tel: +44 (0)870 4431337
>> Fax: +44 (0)208 429 3111
>> PGP: Key ID - 0x2B3438DE
>> PGP: Server - keyserver.pgp.com
>>
>> DISCLAIMER
>>
>> The information contained in this email is intended only for the use of
>> the person(s) to whom it is addressed and may be confidential or contain
>> legally privileged information. If you are not the intended recipient you
>> are hereby notified that any perusal, use, distribution, copying or
>> disclosure is strictly prohibited. If you have received this email in error
>> please immediately advise us by return email at and...@arhont.com and
>> delete and purge the email and any attachments without making a copy.
>>
>>
>> --
>> *From: *"Quenten Grasso" 
>> *To: *"Andrija Panic" , "Sage Weil" <
>> sw...@redhat.com>
>> *Cc: *ceph-users@lists.ceph.com
>> *Sent: *Wednesday, 16 July, 2014 1:20:19 PM
>>
>> *Subject: *Re: [ceph-users] ceph osd crush tunables optimal AND add new
>> OSD at the same time
>>
>> Hi Sage, Andrija & List
>>
>>
>>
>> I have seen the tuneables issue on our cluster when I upgraded to firefly.
>>
>>
>>
>> I ended up going back to legacy settings after about an hour as my
>> cluster is of 55 3TB OSD’s over 5 nodes and it decided it needed to move
>> around 32% of our data, which after an hour all of our vm’s were frozen and
>> I had to revert the change back to legacy settings and wait about the same
>> time again until our cluster had recovered and reboot our vms. (wasn’t
>> really expecting that one from the patch notes)
>>
>>
>>
>> Also our CPU usage went through the roof as well on our nodes, do you per
>> chance have your metadata servers co-located on your osd nodes as we do?
>>  I’ve been thinking about trying to move these to dedicated nodes as it may
>> resolve our issues.
>>
>>
>>
>> Regards,
>>
>> Quenten
>>
>>
>>
>> *From:* ceph-users [mailto:ceph-users-boun...@lists.ceph.com] *On Behalf
>> Of *Andrija Panic
>> *Sent:* Tuesday, 15 July 2014 8:38 PM
>> *To:* Sage Weil
>> *Cc:* ceph-users@lists.ceph.com
>> *Subject:* Re: [ceph-users] ceph osd crush tunables optimal AND add new
>> OSD at the same time
>>
>>
>>
>> Hi Sage,
>>
>>
>>
>> since this problem is tunables-related, do we need to expect same
>> behavior or not  when we do regular data rebalancing caused by adding
>> new/removing OSD? I guess not, but would like your confirmation.
>>
>> I'm already on optimal tunables, but I'm afraid to test this by i.e.
>> shuting down 1 OSD.
>>
>>
>>
>> Thanks,
>> Andrija
>>
>>
>>
>> On 14 July 2014 18:18, Sage Weil  wrote:
>>
>> I've added some additional notes/warnings to the upgrade and release
>> notes:
>>
>>
>> https://github.com/ceph/ceph/commit/fc597e5e3473d7db6548405ce347ca7732832451
>>
>> If there is somewhere else where you think a warning flag would be useful,
>> let me know!
>>
>> Generally speaking, we want to be able to cope with huge data rebalances
>> without interrupting service.  It's an ongoing process of improving the
>> recovery vs client prioritization, though, and removing sources of
>> overhead related to rebalancing... and it's clearly not perfect yet. :/
>>
>> sage
>>
>>
>>
>> On Sun, 13 Jul 2014, Andrija Panic wrote:
>>
>> > Hi,
>> > after seting ceph upgrade (0.72.2 to 0.80.3) I have issued "ceph osd
>> crush
>> > tunables optimal" and after only few minutes I have added 2 more OSDs
>> to the
>> > CEPH cluster...
>> >
>> > So these 2 changes were more or a less done at the same time -
>> rebalancing
>> > because of tunables optimal, and rebalancing because of adding new
>> OSD...
>> >
>> > Result - all VMs living on CEPH storage have gone mad, no disk access
>> > efectively, blocked so to speak.
>> >
>> > Since this rebalancing took 5h-6h, I had bunch of VMs down for that
>> long...
>> >
>> > Did I do wrong by causing "2 rebalancing" to happen at the same time ?
>> > Is this behaviour normal, to cause great load on all VMs because they
>> are
>> > unable to access CEPH storage efectively ?
>> >
>> > Thanks for any input...
>> > --
>> >
>>
>> > Andrija Pani?
>> >
>> >
>>
>>
>>
>>
>>
>> --
>>
>>
>>
>> Andrija Panić
>>
>> --
>>
>>   http://admintweets.com
>>
>> --
>>
>> ___
>> ceph-users mailing list

[ceph-users] Failed monitors

[george.ryall]

On Friday I managed to run a command I probably shouldn't and knock half our 
OSDs offline. By setting the noout and nodown flags and bringing up the OSDS on 
the boxes that don't also have mons running on them I got most of the cluster 
back up by today (it took me a while to discover the nodown flag). However 
along the way I had to restart the mon service a few times and  in two cases 
the monitors didn't seem to be allowed to re-join the cluster and I reinstalled 
the monitor service on them manually. Then this morning I am getting the error 
message I associate with the mons being down whenever I try and run commands on 
the cluster. However, restarting the mon service on the three machines acting 
as monitors does not appear to help.

The message I get is:
2014-07-16 13:33:11.389331 7f6ba845b700  0 -- 130.246.179.122:0/1015725 >> 
130.246.179.181:6789/0 pipe(0x7f6b98005f20 sd=4 :0 s=1 pgs=0 cs=0 l=1 
c=0x7f6b980097d0).fault

What else can I try to bring the cluster back? What logs would it be useful for 
me to look at? Have I missed something?


George Ryall

Scientific Computing | STFC Rutherford Appleton Laboratory | Harwell Oxford | 
Didcot | OX11 0QX
(01235 44) 5021


-- 
Scanned by iCritical.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Some OSD and MDS crash

[Pierre BLONDEAU]

Hi,

After the repair process, i have :
1926 active+clean
   2 active+clean+inconsistent

This two PGs seem to be on the same osd ( #34 ):
# ceph pg dump | grep inconsistent
dumped all in format plain
0.2e4   0   0   0   8388660 4   4 
active+clean+inconsistent   2014-07-16 11:39:43.819631  9463'4 
438411:133968   [34,4]  34  [34,4]  34  9463'4  2014-07-16 
04:52:54.417333  9463'4  2014-07-11 09:29:22.041717
0.1ed   5   0   0   0   8388623 10  10 
active+clean+inconsistent   2014-07-16 11:39:45.820142  9712'10 
438411:144792   [34,2]  34  [34,2]  34  9712'10 2014-07-16 
09:12:44.742488  9712'10 2014-07-10 21:57:11.345241


It's can explain why my MDS won't to start ? If i remove ( or shutdown ) 
this OSD, it's can solved my problem ?


Regards.

Le 10/07/2014 11:51, Pierre BLONDEAU a écrit :

Hi,

Great.

All my OSD restart :
osdmap e438044: 36 osds: 36 up, 36 in

All PG page are active and some in recovery :
1604040/49575206 objects degraded (3.236%)
  1780 active+clean
  17 active+degraded+remapped+backfilling
  61 active+degraded+remapped+wait_backfill
  11 active+clean+scrubbing+deep
  34 active+remapped+backfilling
  21 active+remapped+wait_backfill
  4 active+clean+replay

But all mds crash. Logs are here :
https://blondeau.users.greyc.fr/cephlog/legacy/

In any case, thank you very much for your help.

Pierre

Le 09/07/2014 19:34, Joao Eduardo Luis a écrit :

On 07/09/2014 02:22 PM, Pierre BLONDEAU wrote:

Hi,

There is any chance to restore my data ?


Okay, I talked to Sam and here's what you could try before anything else:

- Make sure you have everything running on the same version.
- unset the the chooseleaf_vary_r flag -- this can be accomplished by
setting tunables to legacy.
- have the osds join in the cluster
- you should then either upgrade to firefly (if you haven't done so by
now) or wait for the point-release before you move on to setting
tunables to optimal again.

Let us know how it goes.

   -Joao




Regards
Pierre

Le 07/07/2014 15:42, Pierre BLONDEAU a écrit :

No chance to have those logs and even less in debug mode. I do this
change 3 weeks ago.

I put all my log here if it's can help :
https://blondeau.users.greyc.fr/cephlog/all/

I have a chance to recover my +/- 20TB of data ?

Regards

Le 03/07/2014 21:48, Joao Luis a écrit :

Do those logs have a higher debugging level than the default? If not
nevermind as they will not have enough information. If they do
however,
we'd be interested in the portion around the moment you set the
tunables. Say, before the upgrade and a bit after you set the tunable.
If you want to be finer grained, then ideally it would be the moment
where those maps were created, but you'd have to grep the logs for
that.

Or drop the logs somewhere and I'll take a look.

   -Joao

On Jul 3, 2014 5:48 PM, "Pierre BLONDEAU" mailto:pierre.blond...@unicaen.fr>> wrote:

Le 03/07/2014 13:49, Joao Eduardo Luis a écrit :

On 07/03/2014 12:15 AM, Pierre BLONDEAU wrote:

Le 03/07/2014 00:55, Samuel Just a écrit :

Ah,

~/logs » for i in 20 23; do ../ceph/src/osdmaptool
--export-crush
/tmp/crush$i osd-$i*; ../ceph/src/crushtool -d
/tmp/crush$i >
/tmp/crush$i.d; done; diff /tmp/crush20.d
/tmp/crush23.d
../ceph/src/osdmaptool: osdmap file
'osd-20_osdmap.13258__0___4E62BB79__none'
../ceph/src/osdmaptool: exported crush map to
/tmp/crush20
../ceph/src/osdmaptool: osdmap file
'osd-23_osdmap.13258__0___4E62BB79__none'
../ceph/src/osdmaptool: exported crush map to
/tmp/crush23
6d5
< tunable chooseleaf_vary_r 1

  Looks like the chooseleaf_vary_r tunable somehow
ended
up divergent?


The only thing that comes to mind that could cause this is
if we
changed
the leader's in-memory map, proposed it, it failed, and only
the
leader
got to write the map to disk somehow.  This happened once on a
totally
different issue (although I can't pinpoint right now which).

In such a scenario, the leader would serve the incorrect
osdmap to
whoever asked osdmaps from it, the remaining quorum would
serve the
correct osdmaps to all the others.  This could cause this
divergence. Or
it could be something else.

Are there logs for the monitors for the timeframe this may
have
happened
in?


Which exactly timeframe you want ? I have 7 days of logs, I should
have informations about the upgrade from firefly to 0.82.
Which mon's log do you want ? Three ?

Regards

-Joao


Pierre: do you recall how and when that got set?


I am not sure to understand, but if I good remember af

Re: [ceph-users] 403-Forbidden error using radosgw

[lakshmi k s]

Resending my earlier message. 

On Tuesday, July 15, 2014 10:58 PM, lakshmi k s  wrote:
 


Hello Ceph Users - 


My Ceph setup consists of 1 admin
node, 3 OSDs, I radosgw and 1 client. One of OSD node also hosts monitor node.
Ceph Health is OK and I have verified the radosgw runtime. I have created S3
and Swift users using radosgw-admin. But when I try to make any S3 or Swift
calls, everything falls apart. For example - 
 
Python script - 
import boto
import boto.s3.connection
access_key = '123'
secret_key = '456'
 
conn = boto.connect_s3(
   
aws_access_key_id = access_key,
   
aws_secret_access_key = secret_key,
   
host = 'ceph-gateway.ex.com',
   
is_secure=False,
   
calling_format = boto.s3.connection.OrdinaryCallingFormat(),
   
)
for bucket in conn.get_all_buckets():
   
print "{name}\t{created}".format(
   
name = bucket.name,
   
created = bucket.creation_date,
   
)
 
Client error- 
Traceback (most recent call last):
  File "dconnect.py",
line 18, in 
    for bucket in
conn.get_all_buckets():
  File
"/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line 387,
in get_all_buckets
    response.status,
response.reason, body)
boto.exception.S3ResponseError:
S3ResponseError: 403 Forbidden
AccessDenied
 
Radosgw log 

2014-07-15 22:48:15.769125
7fbb85fdb700  1 == starting new
request req=0x7fbbe910b290 =
2014-07-15 22:48:15.769443
7fbb85fdb700  2 req 17:0.000334::GET
http://ceph-gateway.ex.com/::initializing
2014-07-15 22:48:15.769998
7fbb85fdb700 10 s->object= s->bucket=
2014-07-15 22:48:15.770199
7fbb85fdb700  2 req 17:0.001084:s3:GET
http://ceph-gateway.ex.com/::getting op
2014-07-15 22:48:15.770345
7fbb85fdb700  2 req 17:0.001231:s3:GET
http://ceph-gateway.ex.com/:list_buckets:authorizing
2014-07-15 22:48:15.770846
7fbb85fdb700 20 get_obj_state: rctx=0x7fbbc800f750
obj=.users:I420IKX56ZP09BTN4CML state=0x7fbbc8007c08 s->prefetch_data=0
2014-07-15 22:48:15.771314
7fbb85fdb700 10 cache get: name=.users+I420IKX56ZP09BTN4CML : hit
2014-07-15 22:48:15.771442
7fbb85fdb700 20 get_obj_state: s->obj_tag was set empty
2014-07-15 22:48:15.771537
7fbb85fdb700 10 cache get: name=.users+I420IKX56ZP09BTN4CML : hit
2014-07-15 22:48:15.773278
7fbb85fdb700 20 get_obj_state: rctx=0x7fbbc800f750 obj=.users.uid:lakshmi
state=0x7fbbc8008208 s->prefetch_data=0
2014-07-15 22:48:15.773288
7fbb85fdb700 10 cache get: name=.users.uid+lakshmi : hit
2014-07-15 22:48:15.773293
7fbb85fdb700 20 get_obj_state: s->obj_tag was set empty
2014-07-15 22:48:15.773297
7fbb85fdb700 10 cache get: name=.users.uid+lakshmi : hit
2014-07-15 22:48:15.774247
7fbb85fdb700 10 get_canon_resource(): dest=http://ceph-gateway.ex.com/
2014-07-15 22:48:15.774326
7fbb85fdb700 10 auth_hdr:
GET
 
 
Wed, 16 Jul 2014 05:48:48 GMT
http://ceph-gateway.ex.com/
2014-07-15 22:48:15.775425
7fbb85fdb700 15 calculated digest=k80Z0p3KlwX4TtrZa0Ws0IWCpVU=
2014-07-15 22:48:15.775498
7fbb85fdb700 15 auth_sign=aAd2u8uD1x/FwLAojm+vceWaITY=
2014-07-15 22:48:15.775536
7fbb85fdb700 15 compare=-10
2014-07-15 22:48:15.775603
7fbb85fdb700 10 failed to authorize request
2014-07-15 22:48:15.776202
7fbb85fdb700  2 req 17:0.007071:s3:GET
http://ceph-gateway.ex.com/:list_buckets:http status=403
2014-07-15 22:48:15.776325
7fbb85fdb700  1 == req done
req=0x7fbbe910b290 http_status=403 ==
2014-07-15 22:48:15.776435
7fbb85fdb700 20 process_request() returned -1
 


Using Swift-Client - 
swift --debug -V 1.0 -A http://ceph-gateway.ex.com/auth/1.0 -U ganapati:swift -K
"GIn60fmdvnEh5tSiRziixcO5wVxZjg9eoYmtX3hJ" list
INFO:urllib3.connectionpool:Starting
new HTTP connection (1): ceph-gateway.ex.com
DEBUG:urllib3.connectionpool:Setting
read timeout to 
DEBUG:urllib3.connectionpool:"GET
/auth/1.0 HTTP/1.1" 403 23
('lks: response %s', )
INFO:swiftclient:REQ: curl -i
http://ceph-gateway.ex.com/auth/1.0 -X GET
INFO:swiftclient:RESP STATUS: 403
Forbidden
INFO:swiftclient:RESP HEADERS:
[('date', 'Wed, 16 Jul 2014 05:45:22 GMT'), ('accept-ranges', 'bytes'),
('content-type', 'application/json'), ('content-length', '23'), ('server',
'Apache/2.4.7 (Ubuntu)')]
INFO:swiftclient:RESP BODY:
{"Code":"AccessDenied"}
ERROR:swiftclient:Auth GET failed:
http://ceph-gateway.ex.com/auth/1.0 403 Forbidden
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py",
line 1187, in _retry
    self.url, self.token = self.get_auth()
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py",
line 1161, in get_auth
    insecure=self.insecure)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py",
line 324, in get_auth
    insecure=insecure)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py",
line 247, in get_auth_1_0
    http_reason=resp.reason)
ClientException: Auth GET failed:
http://ceph-gateway.ex.com/auth/1.0 403 Forbidden
 
Radosgw log - 
2014-0

Re: [ceph-users] 403-Forbidden error using radosgw

[Vincenzo Pii]

You may try to debug your issue by using curl requests.

If you use your Swift credentials, a request of this format should give you
a 20X return code (probably 204):

curl -v -i http:///auth -X GET -H "X-Auth-User:
testuser:swiftuser" -H "X-Auth-Key:
ksYDp8dul80Ta1PeDkFFyLem1FlrtvnyzYiaqvh8"

If this works, you at least know that your auth mechanism is working.

2014-07-16 8:33 GMT+02:00 Wido den Hollander :

> On 07/16/2014 07:58 AM, lakshmi k s wrote:
> > Hello Ceph Users -
> >
> > My Ceph setup consists of 1 admin node, 3 OSDs, I radosgw and 1 client.
> > One of OSD node also hosts monitor node. Ceph Health is OK and I have
> > verified the radosgw runtime. I have created S3 and Swift users using
> > radosgw-admin. But when I try to make any S3 or Swift calls, everything
> > falls apart. For example -
> > Python script -
> > import boto
> > import boto.s3.connection
> > access_key = '123'
> > secret_key = '456'
>
> Are you sure the access and secret key are correct? See my lines a bit
> below.
>
> > conn = boto.connect_s3(
> > aws_access_key_id = access_key,
> > aws_secret_access_key = secret_key,
> > host = 'ceph-gateway.ex.com',
> > is_secure=False,
> > calling_format = boto.s3.connection.OrdinaryCallingFormat(),
> > )
> > for bucket in conn.get_all_buckets():
> > print "{name}\t{created}".format(
> > name = bucket.name,
> > created = bucket.creation_date,
> > )
> > Client error-
> > Traceback (most recent call last):
> >File "dconnect.py", line 18, in 
> >  for bucket in conn.get_all_buckets():
> >File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line
> > 387, in get_all_buckets
> >  response.status, response.reason, body)
> > boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
> >  > encoding="UTF-8"?>AccessDenied
> > Radosgw log
> > 2014-07-15 22:48:15.769125 7fbb85fdb7001 == starting new request
> > req=0x7fbbe910b290 =
> > 2014-07-15 22:48:15.769443 7fbb85fdb7002 req 17:0.000334::GET
> > http://ceph-gateway.ex.com/::initializing
> > 2014-07-15 22:48:15.769998 7fbb85fdb700 10 s->object=
> s->bucket=
> > 2014-07-15 22:48:15.770199 7fbb85fdb7002 req 17:0.001084:s3:GET
> > http://ceph-gateway.ex.com/::getting op
> > 2014-07-15 22:48:15.770345 7fbb85fdb7002 req 17:0.001231:s3:GET
> > http://ceph-gateway.ex.com/:list_buckets:authorizing
> > 2014-07-15 22:48:15.770846 7fbb85fdb700 20 get_obj_state:
> > rctx=0x7fbbc800f750 obj=.users:I420IKX56ZP09BTN4CML state=0x7fbbc8007c08
> > s->prefetch_data=0
> > 2014-07-15 22:48:15.771314 7fbb85fdb700 10 cache get:
> > name=.users+I420IKX56ZP09BTN4CML : hit
> > 2014-07-15 22:48:15.771442 7fbb85fdb700 20 get_obj_state: s->obj_tag was
> > set empty
> > 2014-07-15 22:48:15.771537 7fbb85fdb700 10 cache get:
> > name=.users+I420IKX56ZP09BTN4CML : hit
> > 2014-07-15 22:48:15.773278 7fbb85fdb700 20 get_obj_state:
> > rctx=0x7fbbc800f750 obj=.users.uid:lakshmi state=0x7fbbc8008208
> > s->prefetch_data=0
> > 2014-07-15 22:48:15.773288 7fbb85fdb700 10 cache get:
> > name=.users.uid+lakshmi : hit
> > 2014-07-15 22:48:15.773293 7fbb85fdb700 20 get_obj_state: s->obj_tag was
> > set empty
> > 2014-07-15 22:48:15.773297 7fbb85fdb700 10 cache get:
> > name=.users.uid+lakshmi : hit
> > 2014-07-15 22:48:15.774247 7fbb85fdb700 10 get_canon_resource():
> > dest=http://ceph-gateway.ex.com/
> > 2014-07-15 22:48:15.774326 7fbb85fdb700 10 auth_hdr:
> > GET
> > Wed, 16 Jul 2014 05:48:48 GMT
> > http://ceph-gateway.ex.com/
> > 2014-07-15 22:48:15.775425 7fbb85fdb700 15 calculated
> > digest=k80Z0p3KlwX4TtrZa0Ws0IWCpVU=
> > 2014-07-15 22:48:15.775498 7fbb85fdb700 15
> > auth_sign=aAd2u8uD1x/FwLAojm+vceWaITY=
> > 2014-07-15 22:48:15.775536 7fbb85fdb700 15 compare=-10
> > 2014-07-15 22:48:15.775603 7fbb85fdb700 10 failed to authorize request
>
> That tells you that the gateway calculated a different signature then
> your client did. So something with the access and secret key is wrong.
>
> Wido
>
> > 2014-07-15 22:48:15.776202 7fbb85fdb7002 req 17:0.007071:s3:GET
> > http://ceph-gateway.ex.com/:list_buckets:http status=403
> > 2014-07-15 22:48:15.776325 7fbb85fdb7001 == req done
> > req=0x7fbbe910b290 http_status=403 ==
> > 2014-07-15 22:48:15.776435 7fbb85fdb700 20 process_request() returned -1
> >
> >
> 
> > Using Swift-Client -
> > swift --debug -V 1.0 -A http://ceph-gateway.ex.com/auth/1.0 -U
> > ganapati:swift -K "GIn60fmdvnEh5tSiRziixcO5wVxZjg9eoYmtX3hJ" list
> > INFO:urllib3.connectionpool:Starting new HTTP connection (1):
> > ceph-gateway.ex.com
> > DEBUG:urllib3.connectionpool:Setting read timeout to  > 0x7f3e1cf38090>
> > DEBUG:urllib3.connectionpool:"GET /auth/1.0 HTTP/1.1" 403 23
> > ('lks: response %s', )
> > INFO:swiftclient:REQ: curl -i http://ceph-gateway.ex.com/auth/1.0 -X GET
> > INFO:swiftclient:RESP STATUS: 403 Forbidden
> > INFO:swiftclient:RESP HEADERS: [('date', 'Wed, 16 Jul 2014 05:45:22

Re: [ceph-users] v0.80.4 Firefly released

[Travis Rhoden]

Hi Andrija,

I'm running a cluster with both CentOS and Ubuntu machines in it.  I just
did some upgrades to 0.80.4, and I can confirm that doing "yum update ceph"
on the CentOS machine did result in having all OSDs on that machine
restarted automatically.  I actually did not know that would happen, as the
CentOS machines were new additions (first update since deploying them with
0.80.1), and I'm used the Ubuntu behavior where I can update the package
first, then reboot things at will.

So yeah, that still happens with RPM.  :/


On Wed, Jul 16, 2014 at 3:55 AM, Andrija Panic 
wrote:

> Hi Sage,
>
> can anyone validate, if there is still "bug" inside RPMs that does
> automatic CEPH service restart after updating packages ?
>
> We are instructed to first update/restart MONs, and after that OSD - but
> that is impossible if we have MON+OSDs on same host...since the ceph is
> automaticaly restarted with YUM/RPM, but NOT automaticaly restarted on
> Ubuntu/Debian (as reported by some other list memeber...)
>
> Thanks
>
>
> On 16 July 2014 01:45, Sage Weil  wrote:
>
>> This Firefly point release fixes an potential data corruption problem
>> when ceph-osd daemons run on top of XFS and service Firefly librbd
>> clients.  A recently added allocation hint that RBD utilizes triggers
>> an XFS bug on some kernels (Linux 3.2, and likely others) that leads
>> to data corruption and deep-scrub errors (and inconsistent PGs).  This
>> release avoids the situation by disabling the allocation hint until we
>> can validate which kernels are affected and/or are known to be safe to
>> use the hint on.
>>
>> We recommend that all v0.80.x Firefly users urgently upgrade,
>> especially if they are using RBD.
>>
>> Notable Changes
>> ---
>>
>> * osd: disable XFS extsize hint by default (#8830, Samuel Just)
>> * rgw: fix extra data pool default name (Yehuda Sadeh)
>>
>> For more detailed information, see:
>>
>>   http://ceph.com/docs/master/_downloads/v0.80.4.txt
>>
>> Getting Ceph
>> 
>>
>> * Git at git://github.com/ceph/ceph.git
>> * Tarball at http://ceph.com/download/ceph-0.80.4.tar.gz
>> * For packages, see http://ceph.com/docs/master/install/get-packages
>> * For ceph-deploy, see
>> http://ceph.com/docs/master/install/install-ceph-deploy
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
>
>
> --
>
> Andrija Panić
> --
>   http://admintweets.com
> --
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 403-Forbidden error using radosgw

[lakshmi k s]

Thanks for the response. Curl yields the following - 

ceph-gateway@ceph-gateway:~$ curl -v -i http://ceph-gateway/auth -X GET -H 
"X-Auth-User:ganapati:swift" -H 
"X-Auth-Key:GIn60fmdvnEh5tSiRziixcO5wVxZjg9eoYmtX3hJ"

Hostname was NOT found in DNS cache
Trying 127.0.1.1...
Connected to ceph-gateway (127.0.1.1) port 80 (#0)
GET /auth HTTP/1.1
User-Agent: curl/7.35.0
Host: ceph-gateway
Accept: */*
X-Auth-User:ganapati:swift
X-Auth-Key:GIn60fmdvnEh5tSiRziixcO5wVxZjg9eoYmtX3hJ

HTTP/1.1 403 Forbidden
HTTP/1.1 403 Forbidden
Date: Wed, 16 Jul 2014 14:24:11 GMT
Date: Wed, 16 Jul 2014 14:24:11 GMT
Server Apache/2.4.7 (Ubuntu) is not blacklisted
Server: Apache/2.4.7 (Ubuntu)
Server: Apache/2.4.7 (Ubuntu)
Accept-Ranges: bytes
Accept-Ranges: bytes
Content-Length: 23
Content-Length: 23
Content-Type: application/json
Content-Type: application/json

<
* Connection #0 to host ceph-gateway left intact
{"Code":"AccessDenied"}ceph-gateway@ceph-gateway:~$




On Wednesday, July 16, 2014 7:06 AM, Vincenzo Pii  wrote:
 


You may try to debug your issue by using curl requests.


If you use your Swift credentials, a request of this format should give you a 
20X return code (probably 204):

curl -v -i http:///auth -X GET -H "X-Auth-User: testuser:swiftuser" 
-H "X-Auth-Key: ksYDp8dul80Ta1PeDkFFyLem1FlrtvnyzYiaqvh8"


If this works, you at least know that your auth mechanism is working.


2014-07-16 8:33 GMT+02:00 Wido den Hollander :

On 07/16/2014 07:58 AM, lakshmi k s wrote:
>> Hello Ceph Users -
>>
>> My Ceph setup consists of 1 admin node, 3 OSDs, I radosgw and 1 client.
>> One of OSD node also hosts monitor node. Ceph Health is OK and I have
>> verified the radosgw runtime. I have created S3 and Swift users using
>> radosgw-admin. But when I try to make any S3 or Swift calls, everything
>> falls apart. For example -
>> Python script -
>> import boto
>> import boto.s3.connection
>> access_key = '123'
>> secret_key = '456'
>
>Are you sure the access and secret key are correct? See my lines a bit
>below.
>
>> conn = boto.connect_s3(
>> aws_access_key_id = access_key,
>> aws_secret_access_key = secret_key,
>> host = 'ceph-gateway.ex.com',
>> is_secure=False,
>> calling_format = boto.s3.connection.OrdinaryCallingFormat(),
>> )
>> for bucket in conn.get_all_buckets():
>> print "{name}\t{created}".format(
>> name = bucket.name,
>> created = bucket.creation_date,
>> )
>> Client error-
>> Traceback (most recent call last):
>>    File "dconnect.py", line 18, in 
>>      for bucket in conn.get_all_buckets():
>>    File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line
>> 387, in get_all_buckets
>>      response.status, response.reason, body)
>> boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
>> > encoding="UTF-8"?>AccessDenied
>> Radosgw log
>> 2014-07-15 22:48:15.769125 7fbb85fdb7001 == starting new request
>> req=0x7fbbe910b290 =
>> 2014-07-15 22:48:15.769443 7fbb85fdb7002 req 17:0.000334::GET
>> http://ceph-gateway.ex.com/::initializing
>> 2014-07-15 22:48:15.769998 7fbb85fdb700 10 s->object= s->bucket=
>> 2014-07-15 22:48:15.770199 7fbb85fdb7002 req 17:0.001084:s3:GET
>> http://ceph-gateway.ex.com/::getting op
>> 2014-07-15 22:48:15.770345 7fbb85fdb7002 req 17:0.001231:s3:GET
>> http://ceph-gateway.ex.com/:list_buckets:authorizing
>> 2014-07-15 22:48:15.770846 7fbb85fdb700 20 get_obj_state:
>> rctx=0x7fbbc800f750 obj=.users:I420IKX56ZP09BTN4CML state=0x7fbbc8007c08
>> s->prefetch_data=0
>> 2014-07-15 22:48:15.771314 7fbb85fdb700 10 cache get:
>> name=.users+I420IKX56ZP09BTN4CML : hit
>> 2014-07-15 22:48:15.771442 7fbb85fdb700 20 get_obj_state: s->obj_tag was
>> set empty
>> 2014-07-15 22:48:15.771537 7fbb85fdb700 10 cache get:
>> name=.users+I420IKX56ZP09BTN4CML : hit
>> 2014-07-15 22:48:15.773278 7fbb85fdb700 20 get_obj_state:
>> rctx=0x7fbbc800f750 obj=.users.uid:lakshmi state=0x7fbbc8008208
>> s->prefetch_data=0
>> 2014-07-15 22:48:15.773288 7fbb85fdb700 10 cache get:
>> name=.users.uid+lakshmi : hit
>> 2014-07-15 22:48:15.773293 7fbb85fdb700 20 get_obj_state: s->obj_tag was
>> set empty
>> 2014-07-15 22:48:15.773297 7fbb85fdb700 10 cache get:
>> name=.users.uid+lakshmi : hit
>> 2014-07-15 22:48:15.774247 7fbb85fdb700 10 get_canon_resource():
>> dest=http://ceph-gateway.ex.com/
>> 2014-07-15 22:48:15.774326 7fbb85fdb700 10 auth_hdr:
>> GET
>> Wed, 16 Jul 2014 05:48:48 GMT
>> http://ceph-gateway.ex.com/
>> 2014-07-15 22:48:15.775425 7fbb85fdb700 15 calculated
>> digest=k80Z0p3KlwX4TtrZa0Ws0IWCpVU=
>> 2014-07-15 22:48:15.775498 7fbb85fdb700 15
>> auth_sign=aAd2u8uD1x/FwLAojm+vceWaITY=
>> 2014-07-15 22:48:15.775536 7fbb85fdb700 15 compare=-10
>> 2014-07-15 22:48:15.775603 7fbb85fdb700 10 failed to authorize request
>
>That tells you that the gateway calculated a different signature then
>your client did. So something with the access and secret key is wrong.
>
>Wido
>
>> 2014-07-15 22:48:15.776202 7fbb85fdb7002 req 17:0.007071:s3:GET
>> http://ceph-gateway.ex.com/:list_buckets:http status=403
>>

Re: [ceph-users] Failed monitors

[george.ryall]

This now appears to have partially fixed itself. I am now able to run commands 
on the cluster, though one of the monitors is down. I still have no idea what 
was going on.


George

From: george.ry...@stfc.ac.uk [mailto:george.ry...@stfc.ac.uk]
Sent: 16 July 2014 13:59
To: ceph-users@lists.ceph.com
Subject: [ceph-users] Failed monitors

On Friday I managed to run a command I probably shouldn't and knock half our 
OSDs offline. By setting the noout and nodown flags and bringing up the OSDS on 
the boxes that don't also have mons running on them I got most of the cluster 
back up by today (it took me a while to discover the nodown flag). However 
along the way I had to restart the mon service a few times and  in two cases 
the monitors didn't seem to be allowed to re-join the cluster and I reinstalled 
the monitor service on them manually. Then this morning I am getting the error 
message I associate with the mons being down whenever I try and run commands on 
the cluster. However, restarting the mon service on the three machines acting 
as monitors does not appear to help.

The message I get is:
2014-07-16 13:33:11.389331 7f6ba845b700  0 -- 130.246.179.122:0/1015725 >> 
130.246.179.181:6789/0 pipe(0x7f6b98005f20 sd=4 :0 s=1 pgs=0 cs=0 l=1 
c=0x7f6b980097d0).fault

What else can I try to bring the cluster back? What logs would it be useful for 
me to look at? Have I missed something?


George Ryall

Scientific Computing | STFC Rutherford Appleton Laboratory | Harwell Oxford | 
Didcot | OX11 0QX
(01235 44) 5021



--
Scanned by iCritical.


-- 
Scanned by iCritical.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] how to scale out RADOSgw?

[Riccardo Murri]

Hello,

I am new to Ceph; the group I'm working in is currently evaluating it
for our new large-scale storage.

How scalable is the S3-like object storage server RADOSgw?  What kind
of architecture should one use to ensure we can scale RADOSgw out if
the need arises?  (We plan to host a few PBs on the Ceph cluster, and
a large part of it could be served by RADOSgw.)

Since RADOSgw is a FastCGI module, can one scale it by just adding
more HTTP servers behind a load-balancer?

Thanks for any explanation and suggestion!

Riccardo

--
Riccardo Murri
http://www.gc3.uzh.ch/people/rm

S3IT: Services and Support for Science IT
University of Zurich
Winterthurerstrasse 190, CH-8057 Zürich (Switzerland)
Tel: +41 44 635 4222
Fax: +41 44 635 6888
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] HW recommendations for OSD journals?

[Riccardo Murri]

Hello,

I am new to Ceph; the group I'm working in is currently evaluating it
for our new large-scale storage.

Is there any recommendation for the OSD journals?  E.g., does it make
sense to keep them on SSDs?  Would it make sense to host the journal
on a RAID-1 array for added safety? (IOW: what happens if the journal
device fails and the journal is lost?)

Thanks for any explanation and suggestion!

Riccardo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] how to scale out RADOSgw?

[Wido den Hollander]

> Op 16 jul. 2014 om 16:54 heeft "Riccardo Murri"  het 
> volgende geschreven:
> 
> Hello,
> 
> I am new to Ceph; the group I'm working in is currently evaluating it
> for our new large-scale storage.
> 
> How scalable is the S3-like object storage server RADOSgw?  What kind
> of architecture should one use to ensure we can scale RADOSgw out if
> the need arises?  (We plan to host a few PBs on the Ceph cluster, and
> a large part of it could be served by RADOSgw.)
> 
> Since RADOSgw is a FastCGI module, can one scale it by just adding
> more HTTP servers behind a load-balancer?
> 

Yes, that's the way it works. Simply add more machines when you need to.

The amount of data is not the problem, it's the amount of requests ofcourse.

Wido

> Thanks for any explanation and suggestion!
> 
> Riccardo
> 
> --
> Riccardo Murri
> http://www.gc3.uzh.ch/people/rm
> 
> S3IT: Services and Support for Science IT
> University of Zurich
> Winterthurerstrasse 190, CH-8057 Zürich (Switzerland)
> Tel: +41 44 635 4222
> Fax: +41 44 635 6888
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 403-Forbidden error using radosgw

[Vincenzo Pii]

Maybe some of the user data is not correct...

If you try

radosgw-admin user info --uid=ganapati

is the subuser there?
The key that you must use should be under "swift_keys".

Otherwise, be sure that the user is created with

radosgw-admin key create --subuser=username:subusername --key-type=swift
--gen-secret


2014-07-16 16:31 GMT+02:00 lakshmi k s :

>  Thanks for the response. Curl yields the following -
>
>  ceph-gateway@ceph-gateway:~$ curl -v -i http://ceph-gateway/auth -X GET
> -H "X-Auth-User:ganapati:swift" -H
> "X-Auth-Key:GIn60fmdvnEh5tSiRziixcO5wVxZjg9eoYmtX3hJ"
>  Hostname was NOT found in DNS cache
> Trying 127.0.1.1...
> Connected to ceph-gateway (127.0.1.1) port 80 (#0)
> GET /auth HTTP/1.1
> User-Agent: curl/7.35.0
> Host: ceph-gateway
> Accept: */*
> X-Auth-User:ganapati:swift
> X-Auth-Key:GIn60fmdvnEh5tSiRziixcO5wVxZjg9eoYmtX3hJ
>
>  HTTP/1.1 403 Forbidden
> HTTP/1.1 403 Forbidden
> Date: Wed, 16 Jul 2014 14:24:11 GMT
> Date: Wed, 16 Jul 2014 14:24:11 GMT
> Server Apache/2.4.7 (Ubuntu) is not blacklisted
> Server: Apache/2.4.7 (Ubuntu)
> Server: Apache/2.4.7 (Ubuntu)
> Accept-Ranges: bytes
> Accept-Ranges: bytes
> Content-Length: 23
> Content-Length: 23
> Content-Type: application/json
> Content-Type: application/json
>
>  <
> * Connection #0 to host ceph-gateway left intact
> {"Code":"AccessDenied"}ceph-gateway@ceph-gateway:~$
>
>
>
>
>   On Wednesday, July 16, 2014 7:06 AM, Vincenzo Pii  wrote:
>
>
>   You may try to debug your issue by using curl requests.
>
> If you use your Swift credentials, a request of this format should give
> you a 20X return code (probably 204):
>
>  curl -v -i http:///auth -X GET -H "X-Auth-User:
> testuser:swiftuser" -H "X-Auth-Key:
> ksYDp8dul80Ta1PeDkFFyLem1FlrtvnyzYiaqvh8"
>
>  If this works, you at least know that your auth mechanism is working.
>
> 2014-07-16 8:33 GMT+02:00 Wido den Hollander :
>
> On 07/16/2014 07:58 AM, lakshmi k s wrote:
> > Hello Ceph Users -
> >
> > My Ceph setup consists of 1 admin node, 3 OSDs, I radosgw and 1 client.
> > One of OSD node also hosts monitor node. Ceph Health is OK and I have
> > verified the radosgw runtime. I have created S3 and Swift users using
> > radosgw-admin. But when I try to make any S3 or Swift calls, everything
> > falls apart. For example -
> > Python script -
> > import boto
> > import boto.s3.connection
> > access_key = '123'
> > secret_key = '456'
>
> Are you sure the access and secret key are correct? See my lines a bit
> below.
>
> > conn = boto.connect_s3(
> > aws_access_key_id = access_key,
> > aws_secret_access_key = secret_key,
> > host = 'ceph-gateway.ex.com',
> > is_secure=False,
> > calling_format = boto.s3.connection.OrdinaryCallingFormat(),
> > )
> > for bucket in conn.get_all_buckets():
> > print "{name}\t{created}".format(
> > name = bucket.name,
> > created = bucket.creation_date,
> > )
> > Client error-
> > Traceback (most recent call last):
> >File "dconnect.py", line 18, in 
> >  for bucket in conn.get_all_buckets():
> >File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line
> > 387, in get_all_buckets
> >  response.status, response.reason, body)
> > boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
> >  > encoding="UTF-8"?>AccessDenied
> > Radosgw log
> > 2014-07-15 22:48:15.769125 7fbb85fdb7001 == starting new request
> > req=0x7fbbe910b290 =
> > 2014-07-15 22:48:15.769443 7fbb85fdb7002 req 17:0.000334::GET
> > http://ceph-gateway.ex.com/::initializing
> > 2014-07-15 22:48:15.769998 7fbb85fdb700 10 s->object=
> s->bucket=
> > 2014-07-15 22:48:15.770199 7fbb85fdb7002 req 17:0.001084:s3:GET
> > http://ceph-gateway.ex.com/::getting op
> > 2014-07-15 22:48:15.770345 7fbb85fdb7002 req 17:0.001231:s3:GET
> > http://ceph-gateway.ex.com/:list_buckets:authorizing
> > 2014-07-15 22:48:15.770846 7fbb85fdb700 20 get_obj_state:
> > rctx=0x7fbbc800f750 obj=.users:I420IKX56ZP09BTN4CML state=0x7fbbc8007c08
> > s->prefetch_data=0
> > 2014-07-15 22:48:15.771314 7fbb85fdb700 10 cache get:
> > name=.users+I420IKX56ZP09BTN4CML : hit
> > 2014-07-15 22:48:15.771442 7fbb85fdb700 20 get_obj_state: s->obj_tag was
> > set empty
> > 2014-07-15 22:48:15.771537 7fbb85fdb700 10 cache get:
> > name=.users+I420IKX56ZP09BTN4CML : hit
> > 2014-07-15 22:48:15.773278 7fbb85fdb700 20 get_obj_state:
> > rctx=0x7fbbc800f750 obj=.users.uid:lakshmi state=0x7fbbc8008208
> > s->prefetch_data=0
> > 2014-07-15 22:48:15.773288 7fbb85fdb700 10 cache get:
> > name=.users.uid+lakshmi : hit
> > 2014-07-15 22:48:15.773293 7fbb85fdb700 20 get_obj_state: s->obj_tag was
> > set empty
> > 2014-07-15 22:48:15.773297 7fbb85fdb700 10 cache get:
> > name=.users.uid+lakshmi : hit
> > 2014-07-15 22:48:15.774247 7fbb85fdb700 10 get_canon_resource():
> > dest=http://ceph-gateway.ex.com/
> > 2014-07-15 22:48:15.774326 7fbb85fdb700 10 auth_hdr:
> > GET
> > Wed, 16 Jul 2014 05:48:48 GMT
> > http://ceph-gateway.ex.com/
> > 2014-07-15 22:48:15.775425 7fbb85fdb700 15 calculated
> > digest=k80Z0

Re: [ceph-users] HW recommendations for OSD journals?

[Wido den Hollander]

> Op 16 jul. 2014 om 16:58 heeft "Riccardo Murri"  het 
> volgende geschreven:
> 
> Hello,
> 
> I am new to Ceph; the group I'm working in is currently evaluating it
> for our new large-scale storage.
> 
> Is there any recommendation for the OSD journals?  E.g., does it make
> sense to keep them on SSDs?  

Yes, that is what is usually done.


> Would it make sense to host the journal
> on a RAID-1 array for added safety? 

No, since both SSDs will wear out at the same pace. Better to spread out the 
OSDs over two journals.

> (IOW: what happens if the journal
> device fails and the journal is lost?)

That OSD has to be considered as lost then. Let recovery take over and replace 
the OSD.

Wido

> Thanks for any explanation and suggestion!
> 
> Riccardo
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 403-Forbidden error using radosgw

[lakshmi k s]

Below is the output of radosgw admin user info. Am I missing something here. 
Appreciate your help.

ceph-gateway@ceph-gateway:~$ radosgw-admin user info --uid=ganapati
{ "user_id": "ganapati",
  "display_name": "I",
  "email": "",
  "suspended": 0,
  "max_buckets": 1000,
  "auid": 0,
  "subusers": [
        { "id": "ganapati:swift",
          "permissions": "full-control"}],
  "keys": [
        { "user": "ganapati",
          "access_key": "123",
          "secret_key": "456"},
        { "user": "ganapati:swift",
          "access_key": "Q39BTCD9D0MKN546RNDO",
          "secret_key": ""}],
  "swift_keys": [
        { "user": "ganapati:swift",
          "secret_key": "GIn60fmdvnEh5tSiRziixcO5wVxZjg9eoYmtX3hJ"}],
  "caps": [
        { "type": "metadata",
          "perm": "*"},
        { "type": "usage",
          "perm": "*"},
        { "type": "users",
          "perm": "*"},
        { "type": "zone",
          "perm": "*"}],
  "op_mask": "read, write, delete",
  "default_placement": "",
  "placement_tags": [],
  "bucket_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "user_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "temp_url_keys": []}



On Wednesday, July 16, 2014 8:23 AM, Vincenzo Pii  wrote:
 


Maybe some of the user data is not correct...

If you try

    radosgw-admin user info --uid=ganapati


is the subuser there?
The key that you must use should be under "swift_keys".

Otherwise, be sure that the user is created with

radosgw-admin key create --subuser=username:subusername --key-type=swift 
--gen-secret




2014-07-16 16:31 GMT+02:00 lakshmi k s :

Thanks for the response. Curl yields the following - 
>
>
>ceph-gateway@ceph-gateway:~$ curl -v -i http://ceph-gateway/auth -X GET -H 
>"X-Auth-User:ganapati:swift" -H 
>"X-Auth-Key:GIn60fmdvnEh5tSiRziixcO5wVxZjg9eoYmtX3hJ"
>
>Hostname was NOT found in DNS cache
>Trying 127.0.1.1...
>Connected to ceph-gateway (127.0.1.1) port 80 (#0)
>GET /auth HTTP/1.1
>User-Agent: curl/7.35.0
>Host: ceph-gateway
>Accept: */*
>X-Auth-User:ganapati:swift
>X-Auth-Key:GIn60fmdvnEh5tSiRziixcO5wVxZjg9eoYmtX3hJ
>
>
>HTTP/1.1 403 Forbidden
>HTTP/1.1 403 Forbidden
>Date: Wed, 16 Jul 2014 14:24:11 GMT
>Date: Wed, 16 Jul 2014 14:24:11 GMT
>Server Apache/2.4.7 (Ubuntu) is not blacklisted
>Server: Apache/2.4.7 (Ubuntu)
>Server: Apache/2.4.7 (Ubuntu)
>Accept-Ranges: bytes
>Accept-Ranges: bytes
>Content-Length: 23
>Content-Length: 23
>Content-Type: application/json
>Content-Type: application/json
>
>
><
>* Connection #0 to host ceph-gateway left intact
>{"Code":"AccessDenied"}ceph-gateway@ceph-gateway:~$
>
>
>
>
>
>
>
>On Wednesday, July 16, 2014 7:06 AM, Vincenzo Pii  wrote:
>
>
>
>You may try to debug your issue by using curl requests.
>
>
>If you use your Swift credentials, a request of this format should give you a 
>20X return code (probably 204):
>
>
>curl -v -i http:///auth -X GET -H "X-Auth-User: testuser:swiftuser" 
>-H "X-Auth-Key: ksYDp8dul80Ta1PeDkFFyLem1FlrtvnyzYiaqvh8"
>
>
>
>If this works, you at least know that your auth mechanism is working.
>
>
>2014-07-16 8:33 GMT+02:00 Wido den Hollander :
>
>On 07/16/2014 07:58 AM, lakshmi k s wrote:
>>> Hello Ceph Users -
>>>
>>> My Ceph setup consists of 1 admin node, 3 OSDs, I radosgw and 1 client.
>>> One of OSD node also hosts monitor node. Ceph Health is OK and I have
>>> verified the radosgw runtime. I have created S3 and Swift users using
>>> radosgw-admin. But when I try to make any S3 or Swift calls, everything
>>> falls apart. For example -
>>> Python script -
>>> import boto
>>> import boto.s3.connection
>>> access_key = '123'
>>> secret_key = '456'
>>
>>Are you sure the access and secret key are correct? See my lines a bit
>>below.
>>
>>> conn = boto.connect_s3(
>>> aws_access_key_id = access_key,
>>> aws_secret_access_key = secret_key,
>>> host = 'ceph-gateway.ex.com',
>>> is_secure=False,
>>> calling_format = boto.s3.connection.OrdinaryCallingFormat(),
>>> )
>>> for bucket in conn.get_all_buckets():
>>> print "{name}\t{created}".format(
>>> name = bucket.name,
>>> created = bucket.creation_date,
>>> )
>>> Client error-
>>> Traceback (most recent call last):
>>>    File "dconnect.py", line 18, in 
>>>      for bucket in conn.get_all_buckets():
>>>    File "/usr/lib/python2.7/dist-packages/boto/s3/connection.py", line
>>> 387, in get_all_buckets
>>>      response.status, response.reason, body)
>>> boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
>>> >> encoding="UTF-8"?>AccessDenied
>>> Radosgw log
>>> 2014-07-15 22:48:15.769125 7fbb85fdb7001 == starting new request
>>> req=0x7fbbe910b290 =
>>> 2014-07-15 22:48:15.769443 7fbb85fdb7002 req 17:0.000334::GET
>>> http://ceph-gateway.ex.com/::initializing
>>> 2014-07-15 22:48:15.769998 7fbb85fdb700 10 s->object= s->bucket=
>>> 2014-07-15 22:48:15.770199 7fbb85fdb7002 req 17:0.001084:s3:GET
>>> http://ceph-gateway.ex.com/::getting op
>>> 2014-07-15 22:48:15.770345 7fbb85fdb7002

Re: [ceph-users] HW recommendations for OSD journals?

[Mark Nelson]

On 07/16/2014 09:58 AM, Riccardo Murri wrote:

Hello,

I am new to Ceph; the group I'm working in is currently evaluating it
for our new large-scale storage.

Is there any recommendation for the OSD journals?  E.g., does it make
sense to keep them on SSDs?  Would it make sense to host the journal
on a RAID-1 array for added safety? (IOW: what happens if the journal
device fails and the journal is lost?)

Thanks for any explanation and suggestion!


Hi,

There are a couple of common configurations that make sense imho:

1) Leave journals on the same disks as the data (best to have them in 
their own partition).  This is a fairly safe option since the OSDs only 
have a single disk they rely on (IE minimize potential failures).  It 
can be slow, but it depends on the controller you use and possibly the 
IO scheduler.  Often times a controller with writeback cache seems to 
help avoid seek contention during writes, but you will currently lose 
about half your disk throughput to journal writes during sequential 
write IO.


2) Put journals on SSDs.  In this scenario you want to match your per 
journal SSD speed and disk speed.  IE if you have an SSD that can do 
400MB/s and disks that can do ~125MB/s of sequential writes, you 
probably want to put somewhere around 3-5 journals on the SSD depending 
on how much sequential write throughput matters to you.  OSDs are now 
dependant on both the spinning disk and the SSD not to fail, and one SSD 
failure will take down multiple OSDs.  You gain speed though and may not 
need more expensive controllers with WB cache (though they may still be 
useful to protect against power failure).


Some folks have used raid-1 LUNs for the journals and it works fine, but 
I'm not really a fan of it, especially with SSDs.  You are causing 
double the writes to the SSDs, and SSDs tend to fail in clumps based on 
the number of writes.  If the choice is between 6 journals per SSD 
RAID-1 or 3 journals per SSD JBOD, I'd choose the later.  I'd want to 
keep my overall OSD count high though to minimize the fallout from 3 
OSDs going down at once.


Arguably if you do the RAID1, can swap failed SSDs quickly, and 
anticipate that the remaining SSD is likely going to die soon after the 
first, maybe the RAID1 is worth it.  The disadvantages seem pretty steep 
to me though.


Mark



Riccardo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [ceph-calamari] Issue during ceph-deploy osd activate

[Dan Mick]

There is a log kept in ceph.log of every ceph-deploy command.
On Jul 16, 2014 5:21 AM, "John Spray"  wrote:

> Hi Shubhendu,
>
> ceph-deploy is not part of Calamari, the ceph-users list is a better
> place to get help with that.  I have CC'd the list here.
>
> It will help if you can specify the series of ceph-deploy commands you
> ran before the failing one as well.
>
> Thanks,
> John
>
> On Wed, Jul 16, 2014 at 9:29 AM, Shubhendu Tripathi 
> wrote:
> > Hi,
> >
> > I am trying to setup a ceph cluster. But while activating the OSDs the
> admin
> > gets timeout while running the command "sudo ceph --cluster=ceph osd stat
> > --format=json" on remote node.
> >
> > If I try to execute the same command on one of the nodes of the cluster,
> I
> > get the below errors -
> >
> > [ceph@dhcp43-15 ~]$ sudo ceph-disk-activate --mark-init sysvinit --mount
> > /var/local/osd0
> > INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=mycephcluster
> > --show-config-value=fsid
> > INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph
> > --show-config-value=fsid
> > INFO:ceph-disk:Running command: /usr/bin/ceph --cluster ceph --name
> > client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
> osd
> > create --concise 450d3d27-95ce-4925-890a-908b35fc9255
> > 2014-07-16 13:47:28.073034 7fce166fb700  0 -- 10.70.43.15:0/1031798 >>
> > 10.70.43.20:6789/0 pipe(0x7fce08000c00 sd=4 :0 s=1 pgs=0 cs=0 l=1
> > c=0x7fce08000e70).fault
> >
> >
> > It keeps trying for 300 sec and then gives up with error -
> >
> > 2014-07-16 13:02:48.077898 7f71b81dd700  0 monclient(hunting):
> authenticate
> > timed out after 300
> > 2014-07-16 13:02:48.078033 7f71b81dd700  0 librados: client.bootstrap-osd
> > authentication error (110) Connection timed out
> > Error connecting to cluster: TimedOut
> > ceph-disk: Error: ceph osd create failed: Command '/usr/bin/ceph'
> returned
> > non-zero exit status 1:
> >
> > Kindly help resolving the same.
> >
> > Regards,
> > Shubhendu
> >
> >
> > ___
> > ceph-calamari mailing list
> > ceph-calam...@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-calamari-ceph.com
> ___
> ceph-calamari mailing list
> ceph-calam...@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-calamari-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph-fuse remount

[Scottix]

I wanted to update ceph-fuse to a new version and I would like to have
it seamless.
I thought I could do a remount to update the running version but came to a fail.
Here is the error I got.

# mount /mnt/ceph/ -o remount
2014-07-16 09:08:57.690464 7f669be1a760 -1 asok(0x1285eb0)
AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen:
failed to bind the UNIX domain socket to
'/var/run/ceph/ceph-client.admin.asok': (17) File exists
ceph-fuse[10474]: starting ceph client
fuse: mountpoint is not empty
fuse: if you are sure this is safe, use the 'nonempty' mount option
ceph-fuse[10474]: fuse failed to initialize
2014-07-16 09:08:57.784900 7f669be1a760 -1
fuse_mount(mountpoint=/mnt/ceph) failed.
ceph-fuse[10461]: mount failed: (5) Input/output error

Or is there a better way to do this?

-- 
Follow Me: @Scottix
http://about.me/scottix
scot...@gmail.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v0.80.4 Firefly released

[Sage Weil]

On Wed, 16 Jul 2014, Travis Rhoden wrote:
> Hi Andrija,
> 
> I'm running a cluster with both CentOS and Ubuntu machines in it.  I just
> did some upgrades to 0.80.4, and I can confirm that doing "yum update ceph"
> on the CentOS machine did result in having all OSDs on that machine
> restarted automatically.  I actually did not know that would happen, as the
> CentOS machines were new additions (first update since deploying them with
> 0.80.1), and I'm used the Ubuntu behavior where I can update the package
> first, then reboot things at will.
> 
> So yeah, that still happens with RPM.  :/

We'll make sure this is fixed in 0.80.5!

Thanks-
sage


> 
> 
> On Wed, Jul 16, 2014 at 3:55 AM, Andrija Panic 
> wrote:
>   Hi Sage,
> can anyone validate, if there is still "bug" inside RPMs that does
> automatic CEPH service restart after updating packages ?
> 
> We are instructed to first update/restart MONs, and after that OSD -
> but that is impossible if we have MON+OSDs on same host...since the
> ceph is automaticaly restarted with YUM/RPM, but NOT automaticaly
> restarted on Ubuntu/Debian (as reported by some other list memeber...)
> 
> Thanks
> 
> 
> On 16 July 2014 01:45, Sage Weil  wrote:
>   This Firefly point release fixes an potential data
>   corruption problem
>   when ceph-osd daemons run on top of XFS and service
>   Firefly librbd
>   clients.  A recently added allocation hint that RBD
>   utilizes triggers
>   an XFS bug on some kernels (Linux 3.2, and likely others)
>   that leads
>   to data corruption and deep-scrub errors (and inconsistent
>   PGs).  This
>   release avoids the situation by disabling the allocation
>   hint until we
>   can validate which kernels are affected and/or are known
>   to be safe to
>   use the hint on.
> 
>   We recommend that all v0.80.x Firefly users urgently
>   upgrade,
>   especially if they are using RBD.
> 
>   Notable Changes
>   ---
> 
>   * osd: disable XFS extsize hint by default (#8830, Samuel
>   Just)
>   * rgw: fix extra data pool default name (Yehuda Sadeh)
> 
>   For more detailed information, see:
> 
>     http://ceph.com/docs/master/_downloads/v0.80.4.txt
> 
>   Getting Ceph
>   
> 
>   * Git at git://github.com/ceph/ceph.git
>   * Tarball at http://ceph.com/download/ceph-0.80.4.tar.gz
>   * For packages, see
>   http://ceph.com/docs/master/install/get-packages
>   * For ceph-deploy, see
>   http://ceph.com/docs/master/install/install-ceph-deploy
> 
>   ___
>   ceph-users mailing list
>   ceph-users@lists.ceph.com
>   http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> 
> 
> --
> 
> Andrija Pani?
> --
>   http://admintweets.com
> --
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> 
> ___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v0.80.4 Firefly released

[Gregory Farnum]

On Wed, Jul 16, 2014 at 1:50 AM, James Harper  wrote:
> Can you offer some comments on what the impact is likely to be to the data in 
> an affected cluster? Should all data now be treated with suspicion and 
> restored back to before the firefly upgrade?

I am under the impression that it's not actually a data corruption
bug, so much as a "reveal garbage data in sparse files" bug. So far
everybody who's run into this has run a repair and not seen any
issues, at least. Backups *should* be unnecessary, but we're still
nailing down *exactly* what the bug is, so it's possible our
suspicions are wrong or my information is out of date.

On Wed, Jul 16, 2014 at 2:49 AM, Sylvain Munaut
 wrote:
> On Wed, Jul 16, 2014 at 10:50 AM, James Harper  
> wrote:
>> Can you offer some comments on what the impact is likely to be to the data 
>> in an affected cluster? Should all data now be treated with suspicion and 
>> restored back to before the firefly upgrade?
>
> Yes, I'd definitely like to know that too ... I upgraded a couple
> weeks ago to firefly and I run precise with a 3.2 kernel on XFS. I
> didn't see any deep scrubs error however.
>
> Also does this affect RBD only or can also affect radosgw ?

RBD-only, and if you haven't seen/don't get deep-scrub errors you
won't have any issues.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph-fuse remount

[Gregory Farnum]

On Wed, Jul 16, 2014 at 9:20 AM, Scottix  wrote:
> I wanted to update ceph-fuse to a new version and I would like to have
> it seamless.
> I thought I could do a remount to update the running version but came to a 
> fail.
> Here is the error I got.
>
> # mount /mnt/ceph/ -o remount
> 2014-07-16 09:08:57.690464 7f669be1a760 -1 asok(0x1285eb0)
> AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen:
> failed to bind the UNIX domain socket to
> '/var/run/ceph/ceph-client.admin.asok': (17) File exists
> ceph-fuse[10474]: starting ceph client
> fuse: mountpoint is not empty
> fuse: if you are sure this is safe, use the 'nonempty' mount option
> ceph-fuse[10474]: fuse failed to initialize
> 2014-07-16 09:08:57.784900 7f669be1a760 -1
> fuse_mount(mountpoint=/mnt/ceph) failed.
> ceph-fuse[10461]: mount failed: (5) Input/output error
>
> Or is there a better way to do this?

I don't think that FUSE supports remounting, and the Ceph client
implementation definitely doesn't. Sorry!
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Some OSD and MDS crash

[Gregory Farnum]

On Wed, Jul 16, 2014 at 6:21 AM, Pierre BLONDEAU
 wrote:
> Hi,
>
> After the repair process, i have :
> 1926 active+clean
>2 active+clean+inconsistent
>
> This two PGs seem to be on the same osd ( #34 ):
> # ceph pg dump | grep inconsistent
> dumped all in format plain
> 0.2e4   0   0   0   8388660 4   4
> active+clean+inconsistent   2014-07-16 11:39:43.819631  9463'4
> 438411:133968   [34,4]  34  [34,4]  34  9463'4  2014-07-16
> 04:52:54.417333  9463'4  2014-07-11 09:29:22.041717
> 0.1ed   5   0   0   0   8388623 10  10
> active+clean+inconsistent   2014-07-16 11:39:45.820142  9712'10
> 438411:144792   [34,2]  34  [34,2]  34  9712'10 2014-07-16
> 09:12:44.742488  9712'10 2014-07-10 21:57:11.345241
>
> It's can explain why my MDS won't to start ? If i remove ( or shutdown )
> this OSD, it's can solved my problem ?

You want to figure out why they're inconsistent (if they're still
going inconsistent, or maybe just need to be repaired), but this
shouldn't be causing your MDS troubles.
Can you dump the MDS journal and put it somewhere accessible? (You can
use ceph-post-file to upload it.) John has been trying to reproduce
this crash but hasn't succeeded yet.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] PERC H710 raid card

[Robert Fantini]

I've 2 dell systems with PERC H710 raid cards. Those are very good end
cards , but do not support jbod .

They support raid 0, 1, 5, 6, 10, 50, 60 .

lspci shows them as:  LSI Logic / Symbios Logic MegaRAID SAS 2208
[Thunderbolt] (rev 05)

The firmware Dell uses on the card does not support jbod.

My question is how can this be best used for Ceph? Or should it not be used?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] PERC H710 raid card

[Paul Santos]

In my test cluster in systems with similar RAID cards, I create single-disk 
RAID-0 volumes.

That does the trick.

 

Paul

 

From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Robert 
Fantini
Sent: Wednesday, July 16, 2014 1:55 PM
To: ceph-users@lists.ceph.com
Subject: [ceph-users] PERC H710 raid card

 

I've 2 dell systems with PERC H710 raid cards. Those are very good end cards , 
but do not support jbod . 

They support raid 0, 1, 5, 6, 10, 50, 60 .

lspci shows them as:  LSI Logic / Symbios Logic MegaRAID SAS 2208 [Thunderbolt] 
(rev 05)

The firmware Dell uses on the card does not support jbod. 

My question is how can this be best used for Ceph? Or should it not be used?




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] PERC H710 raid card

[Shain Miley]

Robert,
We use those cards here in our Dell R-720 servers.

We just ended up creating a bunch of single disk RAID-0 units, since 
there was no jbod option available.


Shain


On 07/16/2014 04:55 PM, Robert Fantini wrote:
I've 2 dell systems with PERC H710 raid cards. Those are very good end 
cards , but do not support jbod .


They support raid 0, 1, 5, 6, 10, 50, 60 .

lspci shows them as:  LSI Logic / Symbios Logic MegaRAID SAS 2208 
[Thunderbolt] (rev 05)


The firmware Dell uses on the card does not support jbod.

My question is how can this be best used for Ceph? Or should it not be 
used?






___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


--
Shain Miley | Manager of Systems and Infrastructure, Digital Media | 
smi...@npr.org | 202.513.3649
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] PERC H710 raid card

[Andrey Korolyov]

On Thu, Jul 17, 2014 at 12:55 AM, Robert Fantini
 wrote:
> I've 2 dell systems with PERC H710 raid cards. Those are very good end cards
> , but do not support jbod .
>
> They support raid 0, 1, 5, 6, 10, 50, 60 .
>
> lspci shows them as:  LSI Logic / Symbios Logic MegaRAID SAS 2208
> [Thunderbolt] (rev 05)
>
> The firmware Dell uses on the card does not support jbod.
>
> My question is how can this be best used for Ceph? Or should it not be used?
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>

Hi,

probably single-disk R0 configuration with writeback cache is a best
possible option.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] PERC H710 raid card

[Robert Fantini]

thank you very much for the responses!


On Wed, Jul 16, 2014 at 5:02 PM, Paul Santos  wrote:

> In my test cluster in systems with similar RAID cards, I create
> single-disk RAID-0 volumes.
>
> That does the trick.
>
>
>
> Paul
>
>
>
> *From:* ceph-users [mailto:ceph-users-boun...@lists.ceph.com] *On Behalf
> Of *Robert Fantini
> *Sent:* Wednesday, July 16, 2014 1:55 PM
> *To:* ceph-users@lists.ceph.com
> *Subject:* [ceph-users] PERC H710 raid card
>
>
>
> I've 2 dell systems with PERC H710 raid cards. Those are very good end
> cards , but do not support jbod .
>
> They support raid 0, 1, 5, 6, 10, 50, 60 .
>
> lspci shows them as:  LSI Logic / Symbios Logic MegaRAID SAS 2208
> [Thunderbolt] (rev 05)
>
> The firmware Dell uses on the card does not support jbod.
>
> My question is how can this be best used for Ceph? Or should it not be
> used?
>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] [Nova] [RBD] Copy-on-write cloning for RBD-backed disks

[Dmitry Borodaenko]

I've got a bit of good news and bad news about the state of landing
the rbd-ephemeral-clone patch series for Nova in Juno.

The good news is that the first patch in the series
(https://review.openstack.org/91722 fixing a data loss inducing bug
with live migrations of instances with RBD backed ephemeral drives)
was merged yesterday.

The bad news is that after 2 months of sitting in review queue and
only getting its first a +1 from a core reviewer on the spec approval
freeze day, the spec for the blueprint rbd-clone-image-handler
(https://review.openstack.org/91486) wasn't approved in time. Because
of that, today the blueprint was rejected along with the rest of the
commits in the series, even though the code itself was reviewed and
approved a number of times.

Our last chance to avoid putting this work on hold for yet another
OpenStack release cycle is to petition for a spec freeze exception in
the next Nova team meeting:
https://wiki.openstack.org/wiki/Meetings/Nova

If you're using Ceph RBD as backend for ephemeral disks in Nova and
are interested this patch series, please speak up. Since the biggest
concern raised about this spec so far has been lack of CI coverage,
please let us know if you're already using this patch series with
Juno, Icehouse, or Havana.

I've put together an etherpad with a summary of where things are with
this patch series and how we got here:
https://etherpad.openstack.org/p/nova-ephemeral-rbd-clone-status

Previous thread about this patch series on ceph-users ML:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-March/028097.html

-- 
Dmitry Borodaenko
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Some OSD and MDS crash

[Pierre BLONDEAU]

Le 16/07/2014 22:40, Gregory Farnum a écrit :

On Wed, Jul 16, 2014 at 6:21 AM, Pierre BLONDEAU
 wrote:

Hi,

After the repair process, i have :
1926 active+clean
2 active+clean+inconsistent

This two PGs seem to be on the same osd ( #34 ):
# ceph pg dump | grep inconsistent
dumped all in format plain
0.2e4   0   0   0   8388660 4   4
active+clean+inconsistent   2014-07-16 11:39:43.819631  9463'4
438411:133968   [34,4]  34  [34,4]  34  9463'4  2014-07-16
04:52:54.417333  9463'4  2014-07-11 09:29:22.041717
0.1ed   5   0   0   0   8388623 10  10
active+clean+inconsistent   2014-07-16 11:39:45.820142  9712'10
438411:144792   [34,2]  34  [34,2]  34  9712'10 2014-07-16
09:12:44.742488  9712'10 2014-07-10 21:57:11.345241

It's can explain why my MDS won't to start ? If i remove ( or shutdown )
this OSD, it's can solved my problem ?


You want to figure out why they're inconsistent (if they're still
going inconsistent, or maybe just need to be repaired), but this
shouldn't be causing your MDS troubles.
Can you dump the MDS journal and put it somewhere accessible? (You can
use ceph-post-file to upload it.) John has been trying to reproduce
this crash but hasn't succeeded yet.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com


Hi,

I try to do :
cephfs-journal-tool journal export ceph-journal.bin 2> 
cephfs-journal-tool.log


But the program crash. I upload log file : 
e069c6ac-3cb4-4a52-8950-da7c600e2b01


There is a mistake in 
http://ceph.com/docs/master/cephfs/cephfs-journal-tool/ in "Example: 
journal inspect". The good syntax seems to be :

# cephfs-journal-tool  journal inspect
2014-07-17 00:54:14.155382 7ff89d239780 -1 Header is invalid 
(inconsistent offsets)

Overall journal integrity: DAMAGED
Header could not be decoded

Regards

--
--
Pierre BLONDEAU
Administrateur Systèmes & réseaux
Université de Caen
Laboratoire GREYC, Département d'informatique

tel : 02 31 56 75 42
bureau  : Campus 2, Science 3, 406
--



smime.p7s
Description: Signature cryptographique S/MIME
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] HW recommendations for OSD journals?

[Craig Lewis]

The good SSDs will report how much of their estimated life has been used.
 It's not in the SMART spec though, so different manufacturers do it
differently (or not at all).


I've got Intel DC S3700s, and the SMART attributes include:
233 Media_Wearout_Indicator 0x0032   100   100   000Old_age   Always
-   0

I'm planning to monitor those value, and replace the SSD when "gets old".
 I don't know exactly what that means yet, but I'll figure it out.  It's
easy to replace SSDs before they fail, without losing the whole OSD.

With my write volume, I might just make it a quarterly manual task instead
of adding it to my monitoring tool.  TBD.



Of course, this won't prevent other types of SSD failure.  I've lost both
SSDs in a RAID1 when I triggered an Intel firmware bug.  I've lost both
SSDs in a RAID1 when the colo lost power (older SSDs without super caps).

The only way I can think of that would make RAID1 SSDs safer than a single
SSD is if you use two SSDs from different manufacturers.

Ceph's mantra is "failure is constant".  I'm not going to RAID my journal
devices.  I will use SSDs with power loss protection though.  I can handle
one or two SSDs dropping out at a time.  I can't handle a large percentage
of them dropping out at the same time.




On Wed, Jul 16, 2014 at 8:28 AM, Mark Nelson 
wrote:

> On 07/16/2014 09:58 AM, Riccardo Murri wrote:
>
>> Hello,
>>
>> I am new to Ceph; the group I'm working in is currently evaluating it
>> for our new large-scale storage.
>>
>> Is there any recommendation for the OSD journals?  E.g., does it make
>> sense to keep them on SSDs?  Would it make sense to host the journal
>> on a RAID-1 array for added safety? (IOW: what happens if the journal
>> device fails and the journal is lost?)
>>
>> Thanks for any explanation and suggestion!
>>
>
> Hi,
>
> There are a couple of common configurations that make sense imho:
>
> 1) Leave journals on the same disks as the data (best to have them in
> their own partition).  This is a fairly safe option since the OSDs only
> have a single disk they rely on (IE minimize potential failures).  It can
> be slow, but it depends on the controller you use and possibly the IO
> scheduler.  Often times a controller with writeback cache seems to help
> avoid seek contention during writes, but you will currently lose about half
> your disk throughput to journal writes during sequential write IO.
>
> 2) Put journals on SSDs.  In this scenario you want to match your per
> journal SSD speed and disk speed.  IE if you have an SSD that can do
> 400MB/s and disks that can do ~125MB/s of sequential writes, you probably
> want to put somewhere around 3-5 journals on the SSD depending on how much
> sequential write throughput matters to you.  OSDs are now dependant on both
> the spinning disk and the SSD not to fail, and one SSD failure will take
> down multiple OSDs.  You gain speed though and may not need more expensive
> controllers with WB cache (though they may still be useful to protect
> against power failure).
>
> Some folks have used raid-1 LUNs for the journals and it works fine, but
> I'm not really a fan of it, especially with SSDs.  You are causing double
> the writes to the SSDs, and SSDs tend to fail in clumps based on the number
> of writes.  If the choice is between 6 journals per SSD RAID-1 or 3
> journals per SSD JBOD, I'd choose the later.  I'd want to keep my overall
> OSD count high though to minimize the fallout from 3 OSDs going down at
> once.
>
> Arguably if you do the RAID1, can swap failed SSDs quickly, and anticipate
> that the remaining SSD is likely going to die soon after the first, maybe
> the RAID1 is worth it.  The disadvantages seem pretty steep to me though.
>
> Mark
>
>
>
>> Riccardo
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph osd crush tunables optimal AND add new OSD at the same time

[Craig Lewis]

One of the things I've learned is that many small changes to the cluster
are better than one large change.  Adding 20% more OSDs?  Don't add them
all at once, trickle them in over time.  Increasing pg_num & pgp_num from
128 to 1024?  Go in steps, not one leap.

I try to avoid operations that will touch more than 20% of the disks
simultaneously.  When I had journals on HDD, I tried to avoid going over
10% of the disks.


Is there a way to execute `ceph osd crush tunables optimal` in a way that
takes smaller steps?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph osd crush tunables optimal AND add new OSD at the same time

[Gregory Farnum]

On Wed, Jul 16, 2014 at 4:45 PM, Craig Lewis  wrote:
> One of the things I've learned is that many small changes to the cluster are
> better than one large change.  Adding 20% more OSDs?  Don't add them all at
> once, trickle them in over time.  Increasing pg_num & pgp_num from 128 to
> 1024?  Go in steps, not one leap.
>
> I try to avoid operations that will touch more than 20% of the disks
> simultaneously.  When I had journals on HDD, I tried to avoid going over 10%
> of the disks.
>
>
> Is there a way to execute `ceph osd crush tunables optimal` in a way that
> takes smaller steps?

Unfortunately not; the crush tunables are changes to the core
placement algorithms at work.

If I blue-sky it I suppose we could do something like ship multiple
CRUSH configurations and the hash ranges within which you use each
one, and then incrementally move the dividing lines from 0 over to
1...but that'd be a big change to get protocol support (*none* of the
currently-deployed software could work with a cluster in such a
hypothetical state) and test everything with it, and is not on
anybody's roadmap. I wouldn't expect anybody to be changing CRUSH
tunables on a non-toy cluster; there's a reason we had the config
option for whether you get warnings or not. :)
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph osd crush tunables optimal AND add new OSD at the same time

[Sage Weil]

On Wed, 16 Jul 2014, Gregory Farnum wrote:
> On Wed, Jul 16, 2014 at 4:45 PM, Craig Lewis  
> wrote:
> > One of the things I've learned is that many small changes to the cluster are
> > better than one large change.  Adding 20% more OSDs?  Don't add them all at
> > once, trickle them in over time.  Increasing pg_num & pgp_num from 128 to
> > 1024?  Go in steps, not one leap.
> >
> > I try to avoid operations that will touch more than 20% of the disks
> > simultaneously.  When I had journals on HDD, I tried to avoid going over 10%
> > of the disks.
> >
> >
> > Is there a way to execute `ceph osd crush tunables optimal` in a way that
> > takes smaller steps?
> 
> Unfortunately not; the crush tunables are changes to the core
> placement algorithms at work.

Well, there is one way, but it is only somewhat effective.  If you 
decompile the crush maps for bobtail vs firefly the actual difference is

 tunable chooseleaf_vary_r 1

and this is written such that a value of 1 is the optimal 'new' way, 0 is 
the legacy old way, but values > 1 are less-painful steps between the two 
(though mostly closer to the firefly value of 1).  So, you could set

 tunable chooseleaf_vary_r 4

wait for it to settle, and then do

 tunable chooseleaf_vary_r 3

...and so forth down to 1.  I did some limited testing of the data 
movement involved and noted it here:

 https://github.com/ceph/ceph/commit/37f840b499da1d39f74bfb057cf2b92ef4e84dc6

In my test case, going from 0 to 4 was about 1/10th as bad as going 
straight from 0 to 1, but the final step from 2 to 1 is still about 1/2 as 
bad.  I'm not sure if that means it's not worth the trouble of not just 
jumping straight to the firefly tunables, or whether it means legacy users 
should just set (and leave) this at 2 or 3 or 4 and get almost all the 
benefit without the rebalance pain.

sage
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph osd crush tunables optimal AND add new OSD at the same time

[Craig Lewis]

Thanks, that's worth a try.  Half as bad might make all the difference.

I have the luxury of a federated setup, and I can test on the secondary
cluster fairly safely.  If the change doesn't cause replication timeouts,
it's probably ok to deploy on the primary.

I'll go to CRUSH_TUNABLES2 manually by making the changes in
http://ceph.com/docs/master/rados/operations/crush-map/#tunables, one at a
time.  Then do chooseleaf_vary_r => 4, and see what happens.


I won't get a chance to try for at least a couple weeks, probably longer.




On Wed, Jul 16, 2014 at 5:06 PM, Sage Weil  wrote:

> On Wed, 16 Jul 2014, Gregory Farnum wrote:
> > On Wed, Jul 16, 2014 at 4:45 PM, Craig Lewis 
> wrote:
> > > One of the things I've learned is that many small changes to the
> cluster are
> > > better than one large change.  Adding 20% more OSDs?  Don't add them
> all at
> > > once, trickle them in over time.  Increasing pg_num & pgp_num from 128
> to
> > > 1024?  Go in steps, not one leap.
> > >
> > > I try to avoid operations that will touch more than 20% of the disks
> > > simultaneously.  When I had journals on HDD, I tried to avoid going
> over 10%
> > > of the disks.
> > >
> > >
> > > Is there a way to execute `ceph osd crush tunables optimal` in a way
> that
> > > takes smaller steps?
> >
> > Unfortunately not; the crush tunables are changes to the core
> > placement algorithms at work.
>
> Well, there is one way, but it is only somewhat effective.  If you
> decompile the crush maps for bobtail vs firefly the actual difference is
>
>  tunable chooseleaf_vary_r 1
>
> and this is written such that a value of 1 is the optimal 'new' way, 0 is
> the legacy old way, but values > 1 are less-painful steps between the two
> (though mostly closer to the firefly value of 1).  So, you could set
>
>  tunable chooseleaf_vary_r 4
>
> wait for it to settle, and then do
>
>  tunable chooseleaf_vary_r 3
>
> ...and so forth down to 1.  I did some limited testing of the data
> movement involved and noted it here:
>
>
> https://github.com/ceph/ceph/commit/37f840b499da1d39f74bfb057cf2b92ef4e84dc6
>
> In my test case, going from 0 to 4 was about 1/10th as bad as going
> straight from 0 to 1, but the final step from 2 to 1 is still about 1/2 as
> bad.  I'm not sure if that means it's not worth the trouble of not just
> jumping straight to the firefly tunables, or whether it means legacy users
> should just set (and leave) this at 2 or 3 or 4 and get almost all the
> benefit without the rebalance pain.
>
> sage
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v0.80.4 Firefly released

[Cheng Wei-Chung]

hi all:

This issue will meet deep-scrub error 100% with RBD user?

I mean it is possible just occur sometimes.

I use firefly v0.80.1 with kernel 3.8.0-37 and did not see any deep-scrub
error.

Then I still need to upgrade to 0.80.4 or I didn't get this issue?

Thanks!!


2014-07-17 1:19 GMT+08:00 Gregory Farnum :

> On Wed, Jul 16, 2014 at 1:50 AM, James Harper 
> wrote:
> > Can you offer some comments on what the impact is likely to be to the
> data in an affected cluster? Should all data now be treated with suspicion
> and restored back to before the firefly upgrade?
>
> I am under the impression that it's not actually a data corruption
> bug, so much as a "reveal garbage data in sparse files" bug. So far
> everybody who's run into this has run a repair and not seen any
> issues, at least. Backups *should* be unnecessary, but we're still
> nailing down *exactly* what the bug is, so it's possible our
> suspicions are wrong or my information is out of date.
>
> On Wed, Jul 16, 2014 at 2:49 AM, Sylvain Munaut
>  wrote:
> > On Wed, Jul 16, 2014 at 10:50 AM, James Harper 
> wrote:
> >> Can you offer some comments on what the impact is likely to be to the
> data in an affected cluster? Should all data now be treated with suspicion
> and restored back to before the firefly upgrade?
> >
> > Yes, I'd definitely like to know that too ... I upgraded a couple
> > weeks ago to firefly and I run precise with a 3.2 kernel on XFS. I
> > didn't see any deep scrubs error however.
> >
> > Also does this affect RBD only or can also affect radosgw ?
>
> RBD-only, and if you haven't seen/don't get deep-scrub errors you
> won't have any issues.
> -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph osd crush tunables optimal AND add new OSD at the same time

[Quenten Grasso]

Hi Sage & List

I understand this is probably a hard question to answer.

I mentioned previously our cluster is co-located MON’s on OSD servers, which 
are R515’s w/ 1 x AMD 6 Core processor & 11 3TB OSD’s w/ dual 10GBE.

When our cluster is doing these busy operations and IO has stopped as in my 
case, I mentioned earlier running/setting tuneable to optimal or heavy recovery
operations is there a way to ensure our IO doesn’t get completely 
blocked/stopped/frozen in our vms?

Could it be as simple as putting all 3 of our mon servers on baremetal  
w/ssd’s? (I recall reading somewhere that a mon disk was doing several thousand 
IOPS during a recovery operation)

I assume putting just one on baremetal won’t help because our mon’s will only 
ever be as fast as our slowest mon server?

Thanks,
Quenten
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] logrotate

[Sahana Lokeshappa]

Hi Sage,

Even I am facing the problem.
ls -l /var/log/ceph/
total 54280
-rw-r--r-- 1 root root0 Jul 17 06:39 ceph-osd.0.log
-rw-r--r-- 1 root root 19603037 Jul 16 19:01 ceph-osd.0.log.1.gz
-rw-r--r-- 1 root root0 Jul 17 06:39 ceph-osd.1.log
-rw-r--r-- 1 root root 18008247 Jul 16 19:01 ceph-osd.1.log.1.gz
-rw-r--r-- 1 root root0 Jul 17 06:39 ceph-osd.2.log
-rw-r--r-- 1 root root 17969054 Jul 16 19:01 ceph-osd.2.log.1.gz

Due to this , I lost logs, until I restarted the osds.

thanks
Sahana Lokeshappa
Test Development Engineer I

3rd Floor, Bagmane Laurel, Bagmane Tech Park
C V Raman nagar, Bangalore 560093
T: +918042422283
sahana.lokesha...@sandisk.com

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Uwe 
Grohnwaldt
Sent: Sunday, July 13, 2014 7:10 AM
To: ceph-us...@ceph.com
Subject: Re: [ceph-users] logrotate

Hi,

we are observing the same problem. After logrotate the new logfile is empty.
The old logfiles are marked as deleted in lsof. At the moment we are restarting 
osds on a regular basis.

Uwe

> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf
> Of James Eckersall
> Sent: Freitag, 11. Juli 2014 17:06
> To: Sage Weil
> Cc: ceph-us...@ceph.com
> Subject: Re: [ceph-users] logrotate
>
> Hi Sage,
>
> Many thanks for the info.
> I have inherited this cluster, but I believe it may have been created
> with mkcephfs rather than ceph-deploy.
>
> I'll touch the done files and see what happens.  Looking at the logic
> in the logrotate script I'm sure this will resolve the problem.
>
> Thanks
>
> J
>
>
> On 11 July 2014 15:04, Sage Weil   > wrote:
>
>
>   On Fri, 11 Jul 2014, James Eckersall wrote:
>   > Upon further investigation, it looks like this part of the ceph
> logrotate
>   > script is causing me the problem:
>   >
>   > if [ -e "/var/lib/ceph/$daemon/$f/done" ] && [ -e
>   > "/var/lib/ceph/$daemon/$f/upstart" ] && [ ! -e
>   > "/var/lib/ceph/$daemon/$f/sysvinit" ]; then
>   >
>   > I don't have a "done" file in the mounted directory for any of my
> osd's.  My
>   > mon's all have the done file and logrotate is working fine for those.
>
>
>   Was this cluster created a while ago with mkcephfs?
>
>
>   > So my question is, what is the purpose of the "done" file and
> should I just
>   > create one for each of my osd's ?
>
>
>   It's used by the newer ceph-disk stuff to indicate whether the OSD
>   directory is propertly 'prepared' and whether the startup stuff
> should pay
>   attention.
>
>   If these are active OSDs, yeah, just touch 'done'.  (Don't touch
> sysvinit,
>   though, if you are enumerating the daemons in ceph.conf with host =
> foo
>   lines.)
>
>   sage
>
>
>
>   >
>   >
>   >
>   > On 10 July 2014 11:10, James Eckersall   > wrote:
>   >   Hi,
>   > I've just upgraded a ceph cluster from Ubuntu 12.04 with 0.73.1 to
>   > Ubuntu 14.04 with 0.80.1.
>   >
>   > I've noticed that the log rotation doesn't appear to work correctly.
>   > The OSD's are just not logging to the current ceph-osd-X.log file.
>   > If I restart the OSD's, they start logging, but then overnight, they
>   > stop logging when the logs are rotated.
>   >
>   > Has anyone else noticed a problem with this?
>   >
>   >
>   >
>   >
>




PLEASE NOTE: The information contained in this electronic mail message is 
intended only for the use of the designated recipient(s) named above. If the 
reader of this message is not the intended recipient, you are hereby notified 
that you have received this message in error and that any review, 
dissemination, distribution, or copying of this message is strictly prohibited. 
If you have received this communication in error, please notify the sender by 
telephone or e-mail (as shown above) immediately and destroy any and all copies 
of this message in your possession (whether hard copies or electronically 
stored copies).

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] PERC H710 raid card

[Dennis Kramer (DT)]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi,

What do you recommend in case of a disk failure in this kind of
configuration? Are you bringing down the host when you replace the
disk and re-create the raid-0 for the replaced disk? I reckon that
linux doesn't automatically get the disk replacement either...

Dennis

On 07/16/2014 11:02 PM, Shain Miley wrote:
> Robert, We use those cards here in our Dell R-720 servers.
> 
> We just ended up creating a bunch of single disk RAID-0 units,
> since there was no jbod option available.
> 
> Shain
> 
> 
> On 07/16/2014 04:55 PM, Robert Fantini wrote:
>> I've 2 dell systems with PERC H710 raid cards. Those are very
>> good end cards , but do not support jbod .
>> 
>> They support raid 0, 1, 5, 6, 10, 50, 60 .
>> 
>> lspci shows them as:  LSI Logic / Symbios Logic MegaRAID SAS 2208
>>  [Thunderbolt] (rev 05)
>> 
>> The firmware Dell uses on the card does not support jbod.
>> 
>> My question is how can this be best used for Ceph? Or should it
>> not be used?
>> 
>> 
>> 
>> 
>> 
>> ___ ceph-users
>> mailing list ceph-users@lists.ceph.com 
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> -- Shain Miley | Manager of Systems and Infrastructure, Digital
> Media | smi...@npr.org | 202.513.3649
> 


-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.15 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlPHZ1MACgkQiJDTKUBxIRusogCeJ+jnADW/KBoQAxnDSz62yT3P
FNoAnin3A52AqiA+KlFJQoc5bdQRoyYe
=/MPE
-END PGP SIGNATURE-
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [Nova] [RBD] Copy-on-write cloning for RBD-backed disks

[Dennis Kramer (DT)]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi Dmitry,

I've been using Ubuntu 14.04LTS + Icehouse /w CEPH as a storage
backend for glance, cinder and nova (kvm/libvirt). I *really* would
love to see this patch cycle in Juno. It's been a real performance
issue because of the unnecessary re-copy from-and-to CEPH when using
the default "boot from image"-option. It seems that the your fix would
be the solution to all. IMHO this is one of the most important
features when using CEPH RBD as a backend for Openstack Nova.

Can you point me in the right direction in how to apply this patch of
yours on a default Ubuntu14.04LTS + Icehouse installation? I'm using
the default ubuntu packages since Icehouse lives in core and I'm not
sure how to apply the patch series. I would love to test and review it.

With regards,

Dennis

On 07/16/2014 11:18 PM, Dmitry Borodaenko wrote:
> I've got a bit of good news and bad news about the state of
> landing the rbd-ephemeral-clone patch series for Nova in Juno.
> 
> The good news is that the first patch in the series 
> (https://review.openstack.org/91722 fixing a data loss inducing
> bug with live migrations of instances with RBD backed ephemeral
> drives) was merged yesterday.
> 
> The bad news is that after 2 months of sitting in review queue and 
> only getting its first a +1 from a core reviewer on the spec
> approval freeze day, the spec for the blueprint
> rbd-clone-image-handler (https://review.openstack.org/91486) wasn't
> approved in time. Because of that, today the blueprint was rejected
> along with the rest of the commits in the series, even though the
> code itself was reviewed and approved a number of times.
> 
> Our last chance to avoid putting this work on hold for yet another 
> OpenStack release cycle is to petition for a spec freeze exception
> in the next Nova team meeting: 
> https://wiki.openstack.org/wiki/Meetings/Nova
> 
> If you're using Ceph RBD as backend for ephemeral disks in Nova
> and are interested this patch series, please speak up. Since the
> biggest concern raised about this spec so far has been lack of CI
> coverage, please let us know if you're already using this patch
> series with Juno, Icehouse, or Havana.
> 
> I've put together an etherpad with a summary of where things are
> with this patch series and how we got here: 
> https://etherpad.openstack.org/p/nova-ephemeral-rbd-clone-status
> 
> Previous thread about this patch series on ceph-users ML: 
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-March/028097.html
>
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.15 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlPHa6kACgkQiJDTKUBxIRtpOwCeNjTlYlyypOsaGeI/+HRxZ6nt
Y2kAoNLckOlSaEfw+dwSBacXP3JGkcAj
=0Ez1
-END PGP SIGNATURE-
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Failed monitors

[Kenneth Waegeman]

- Message from george.ry...@stfc.ac.uk -
   Date: Wed, 16 Jul 2014 14:45:35 +
   From: george.ry...@stfc.ac.uk
Subject: Re: [ceph-users] Failed monitors
 To: ceph-users@lists.ceph.com


This now appears to have partially fixed itself. I am now able to  
run commands on the cluster, though one of the monitors is down. I  
still have no idea what was going on.


Hi George,

What do the logs /var/log/ceph/ceph-mon.*.log say?

Kenneth



George

From: george.ry...@stfc.ac.uk [mailto:george.ry...@stfc.ac.uk]
Sent: 16 July 2014 13:59
To: ceph-users@lists.ceph.com
Subject: [ceph-users] Failed monitors

On Friday I managed to run a command I probably shouldn't and knock  
half our OSDs offline. By setting the noout and nodown flags and  
bringing up the OSDS on the boxes that don't also have mons running  
on them I got most of the cluster back up by today (it took me a  
while to discover the nodown flag). However along the way I had to  
restart the mon service a few times and  in two cases the monitors  
didn't seem to be allowed to re-join the cluster and I reinstalled  
the monitor service on them manually. Then this morning I am getting  
the error message I associate with the mons being down whenever I  
try and run commands on the cluster. However, restarting the mon  
service on the three machines acting as monitors does not appear to  
help.


The message I get is:
2014-07-16 13:33:11.389331 7f6ba845b700  0 --  
130.246.179.122:0/1015725 >> 130.246.179.181:6789/0  
pipe(0x7f6b98005f20 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f6b980097d0).fault


What else can I try to bring the cluster back? What logs would it be  
useful for me to look at? Have I missed something?



George Ryall

Scientific Computing | STFC Rutherford Appleton Laboratory | Harwell  
Oxford | Didcot | OX11 0QX

(01235 44) 5021



--
Scanned by iCritical.


--
Scanned by iCritical.



- End message from george.ry...@stfc.ac.uk -

--

Met vriendelijke groeten,
Kenneth Waegeman

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] HW recommendations for OSD journals?

[Robert van Leeuwen]

> The good SSDs will report how much of their estimated life has been used.
> It's not in the SMART spec though, so different manufacturers do it 
> differently (or not at all).

> I'm planning to monitor those value, and replace the SSD when "gets old".
> I don't know exactly what that means yet, but I'll figure it out.
> It's easy to replace SSDs before they fail, without losing the whole OSD.

We have a smallish cluster (3 nodes, 30TB RAW storage) with 6 Intel dc S3500 
120GB disks for journals.
We have between 300-600Mbit of continuous incoming data and lose a 2-3 percent 
of lifetime per week.
I would highly recommend to monitor this if you are not doing this already ;)

Buying bigger SSDs will help because the writes are spread across more cells.
So a 240GB drive should last 2x a 120GB drive.

Cheers,
Robert van Leeuwen


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [Nova] [RBD] Copy-on-write cloning for RBD-backed disks

[Somhegyi Benjamin]

Hi Dmitry,

Will you please share with us how things went on the meeting?

Many thanks,
Benjamin



> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Dmitry Borodaenko
> Sent: Wednesday, July 16, 2014 11:18 PM
> To: ceph-users@lists.ceph.com
> Cc: OpenStack Development Mailing List (not for usage questions)
> Subject: [ceph-users] [Nova] [RBD] Copy-on-write cloning for RBD-backed
> disks
> 
> I've got a bit of good news and bad news about the state of landing the
> rbd-ephemeral-clone patch series for Nova in Juno.
> 
> The good news is that the first patch in the series
> (https://review.openstack.org/91722 fixing a data loss inducing bug with
> live migrations of instances with RBD backed ephemeral drives) was
> merged yesterday.
> 
> The bad news is that after 2 months of sitting in review queue and only
> getting its first a +1 from a core reviewer on the spec approval freeze
> day, the spec for the blueprint rbd-clone-image-handler
> (https://review.openstack.org/91486) wasn't approved in time. Because of
> that, today the blueprint was rejected along with the rest of the
> commits in the series, even though the code itself was reviewed and
> approved a number of times.
> 
> Our last chance to avoid putting this work on hold for yet another
> OpenStack release cycle is to petition for a spec freeze exception in
> the next Nova team meeting:
> https://wiki.openstack.org/wiki/Meetings/Nova
> 
> If you're using Ceph RBD as backend for ephemeral disks in Nova and are
> interested this patch series, please speak up. Since the biggest concern
> raised about this spec so far has been lack of CI coverage, please let
> us know if you're already using this patch series with Juno, Icehouse,
> or Havana.
> 
> I've put together an etherpad with a summary of where things are with
> this patch series and how we got here:
> https://etherpad.openstack.org/p/nova-ephemeral-rbd-clone-status
> 
> Previous thread about this patch series on ceph-users ML:
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-
> March/028097.html
> 
> --
> Dmitry Borodaenko
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] HW recommendations for OSD journals?

[Christian Balzer]

On Thu, 17 Jul 2014 06:38:42 + Robert van Leeuwen wrote:

> 
> > The good SSDs will report how much of their estimated life has been
> > used. It's not in the SMART spec though, so different manufacturers do
> > it differently (or not at all).
> 
> > I'm planning to monitor those value, and replace the SSD when "gets
> > old". I don't know exactly what that means yet, but I'll figure it out.
> > It's easy to replace SSDs before they fail, without losing the whole
> > OSD.
> 
> We have a smallish cluster (3 nodes, 30TB RAW storage) with 6 Intel dc
> S3500 120GB disks for journals. We have between 300-600Mbit of
> continuous incoming data and lose a 2-3 percent of lifetime per week. I
> would highly recommend to monitor this if you are not doing this
> already ;)
> 
> Buying bigger SSDs will help because the writes are spread across more
> cells. So a 240GB drive should last 2x a 120GB drive.
> 
> 
At that rate a DC3700 (instead of a bigger drive) will definitely be more
attractive when it comes to $/TBW. 

A 240GB DC3500 is rated for  140TBW and will cost about $240.
A 200GB DC3700 is rated for 3650TBW and will cost about $380.

Christian
-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Fusion Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com