date:20140521

Re: [ceph-users] Access denied error for list users

2014-05-21 Thread alain.dechorgnat

There is no detail with GET /admin/metadata/user, only ids.

For PHP, have a look at http://ceph.com/docs/master/radosgw/s3/php/

Alain

De : Shanil S [mailto:xielessha...@gmail.com] 
Envoyé : mercredi 21 mai 2014 05:48
À : DECHORGNAT Alain IMT/OLPS
Objet : Re: [ceph-users] Access denied error for list users

Hi Alain,

Thanks..
I used the GET /admin/metadata/user to fetch the user list but it only shows 
the usernames in the list. I would like to show the other details too like 
bucket number,id etc. Can i use the same GET /admin/metadata/user to get all 
these details ? Also, is there any easy way to generate the access token 
authorization header using php ?

On Tue, May 20, 2014 at 7:36 PM,  wrote:
Hi,

GET /admin/user with no parameter doesn't work.

You must use GET /admin/metadata/user to fetch the user list (with metadata 
capabity).

Alain


De : ceph-users [mailto:ceph-users-boun...@lists.ceph.com] De la part de Shanil 
S
Envoyé : mardi 20 mai 2014 07:13
À : ceph-users@lists.ceph.com; w...@42on.com; s...@inktank.com; Yehuda Sadeh
Objet : [ceph-users] Access denied error for list users

Hi,

I am trying to create and list all users by using the functions 
http://ceph.com/docs/master/radosgw/adminops/ and i successfully created the 
access tokens but i am getting an access denied and 403 for listing users 
function. The GET /{admin}/user is used for getting the complete users list, 
but its not listing and getting the error. The user which called this function 
has the complete permission and i am adding the permission of this user

{ "type": "admin",
  "perm": "*"},
    { "type": "buckets",
  "perm": "*"},
    { "type": "caps",
  "perm": "*"},
    { "type": "metadata",
  "perm": "*"},
    { "type": "usage",
  "perm": "*"},
    { "type": "users",
  "perm": "*"}],
  "op_mask": "read, write, delete",
  "default_placement": "",
  "placement_tags": [],
  "bucket_quota": { "enabled": false,
  "max_size_kb": -1,
  "max_objects": -1}}


This is in the log file which executed the list user function

-

GET

application/x-www-form-urlencoded
Tue, 20 May 2014 05:06:57 GMT
/admin/user/
2014-05-20 13:06:59.506233 7f0497fa7700 15 calculated 
digest=Z8FgXRLk+ah5MUThpP9IBJrMnrA=
2014-05-20 13:06:59.506236 7f0497fa7700 15 
auth_sign=Z8FgXRLk+ah5MUThpP9IBJrMnrA=
2014-05-20 13:06:59.506237 7f0497fa7700 15 compare=0
2014-05-20 13:06:59.506240 7f0497fa7700  2 req 98:0.000308::GET 
/admin/user/:get_user_info:reading permissions
2014-05-20 13:06:59.506244 7f0497fa7700  2 req 98:0.000311::GET 
/admin/user/:get_user_info:init op
2014-05-20 13:06:59.506247 7f0497fa7700  2 req 98:0.000314::GET 
/admin/user/:get_user_info:verifying op mask
2014-05-20 13:06:59.506249 7f0497fa7700 20 required_mask= 0 user.op_mask=7
2014-05-20 13:06:59.506251 7f0497fa7700  2 req 98:0.000319::GET 
/admin/user/:get_user_info:verifying op permissions
2014-05-20 13:06:59.506254 7f0497fa7700  2 req 98:0.000322::GET 
/admin/user/:get_user_info:verifying op params
2014-05-20 13:06:59.506257 7f0497fa7700  2 req 98:0.000324::GET 
/admin/user/:get_user_info:executing
2014-05-20 13:06:59.506291 7f0497fa7700  2 req 98:0.000359::GET 
/admin/user/:get_user_info:http status=403
2014-05-20 13:06:59.506294 7f0497fa7700  1 == req done req=0x7f04c800d7f0 
http_status=403 ==
2014-05-20 13:06:59.506302 7f0497fa7700 20 process_request() returned -13

-

Could you please check what is the issue ?
I am using the ceph version : ceph version 0.80.1

_

Ce message et ses pieces jointes peuvent contenir des informations 
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce 
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou 
falsifie. Merci.

This message and its attachments may contain confidential or privileged 
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete 
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been 
modified, changed or falsified.
Thank you.


_

Ce message et ses pieces jointes peuvent contenir des informations 
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses,

Re: [ceph-users] Data still in OSD directories after removing

2014-05-21 Thread Olivier Bonvalet

Hi,

I have a lot of space wasted by this problem (about 10GB per OSD, just
for this RBD image).
If OSDs can't detect orphans files, should I manually detect them, then
remove them ?

This command can do the job, at least for this image prefix :
find /var/lib/ceph/osd/ -name 'rb.0.14bfb5a.238e1f29.*' -delete

Thanks for any advice,
Olivier

PS : not sure if this kind of problem is for the user or dev mailing
list.

Le mardi 20 mai 2014 à 11:32 +0200, Olivier Bonvalet a écrit :
> Hi,
> 
> short : I removed a 1TB RBD image, but I still see files about it on
> OSD.
> 
> 
> long :
> 1) I did : "rbd snap purge $pool/$img"
>but since it overload the cluster, I stopped it (CTRL+C)
> 2) latter, "rbd snap purge $pool/$img"
> 3) then, "rbd rm $pool/$img"
> 
> now, on the disk I can found files of this v1 RBD image (prefix was
> rb.0.14bfb5a.238e1f29) :
> 
> # find /var/lib/ceph/osd/ceph-64/ -name 'rb.0.14bfb5a.238e1f29.*'
> /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_3/rb.0.14bfb5a.238e1f29.00021431__snapdir_C96635C1__9
> /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_3/rb.0.14bfb5a.238e1f29.5622__a252_32F435C1__9
> /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_3/rb.0.14bfb5a.238e1f29.00021431__a252_C96635C1__9
> /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_3/rb.0.14bfb5a.238e1f29.5622__snapdir_32F435C1__9
> /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_9/rb.0.14bfb5a.238e1f29.00011e08__a172_594495C1__9
> /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_9/rb.0.14bfb5a.238e1f29.00011e08__snapdir_594495C1__9
> /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_A/rb.0.14bfb5a.238e1f29.00021620__a252_779FA5C1__9
> ...
> 
> 
> So, is there a way to force OSD to detect if files are orphans, then
> remove them ?
> 
> Thanks,
> Olivier
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Expanding pg's of an erasure coded pool

2014-05-21 Thread Kenneth Waegeman

Thanks! I increased the max processes parameter for all daemons quite  
a lot (until ulimit -u 3802720)


These are the limits for the daemons now..
[root@ ~]# cat /proc/17006/limits
Limit Soft Limit   Hard Limit   Units
Max cpu time  unlimitedunlimitedseconds
Max file size unlimitedunlimitedbytes
Max data size unlimitedunlimitedbytes
Max stack size10485760 unlimitedbytes
Max core file sizeunlimitedunlimitedbytes
Max resident set  unlimitedunlimitedbytes
Max processes 3802720  3802720  processes
Max open files3276832768files
Max locked memory 6553665536bytes
Max address space unlimitedunlimitedbytes
Max file locksunlimitedunlimitedlocks
Max pending signals   9506895068signals
Max msgqueue size 819200   819200   bytes
Max nice priority 00
Max realtime priority 00
Max realtime timeout  unlimitedunlimitedus

But this didn't help. Are there other parameters I should change?

I also got an 'bash: fork: Cannot allocate memory' error once when  
running a command after starting the ceph services. It shouldn't be a  
memory shortage issue itself because when monitoring the failure there  
is still enough (cached) available..



- Message from Gregory Farnum  -
   Date: Tue, 20 May 2014 10:33:30 -0700
   From: Gregory Farnum 
Subject: Re: [ceph-users] Expanding pg's of an erasure coded pool
 To: Kenneth Waegeman 
 Cc: ceph-users 



This failure means the messenger subsystem is trying to create a
thread and is getting an error code back — probably due to a process
or system thread limit that you can turn up with ulimit.

This is happening because a replicated PG primary needs a connection
to only its replicas (generally 1 or 2 connections), but with an
erasure-coded PG the primary requires a connection to m+n-1 replicas
(everybody who's in the erasure-coding set, including itself). Right
now our messenger requires a thread for each connection, so kerblam.
(And it actually requires a couple such connections because we have
separate heartbeat, cluster data, and client data systems.)
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com


On Tue, May 20, 2014 at 3:43 AM, Kenneth Waegeman
 wrote:

Hi,

On a setup of 400 OSDs (20 nodes, with 20 OSDs per node), I first tried to
create a erasure coded pool with 4096 pgs, but this crashed the cluster.
I then started with 1024 pgs, expanding to 2048 (pg_num and pgp_num), when I
then try to expand to 4096 (not even quite enough) the cluster crashes
again. ( Do we need less of pg's with erasure coding?)

The crash starts with individual OSDs crashing, eventually bringing down the
mons (until there is no more quorum or too few osds)

Out of the logs:


   -16> 2014-05-20 10:31:55.545590 7fd42f34d700  5 -- op tracker -- , seq:
14301, time: 2014-05-20 10:31:55.545590, event: started, request:
pg_query(0.974 epoch 3315) v3
   -15> 2014-05-20 10:31:55.545776 7fd42f34d700  1 --
130.246.178.141:6836/10446 --> 130.246.179.191:6826/21854 -- pg_notify(0.974
epoch 3326) v5 -- ?+0 0xc8b4ec0 con 0x9
026b40
   -14> 2014-05-20 10:31:55.545807 7fd42f34d700  5 -- op tracker -- , seq:
14301, time: 2014-05-20 10:31:55.545807, event: done, request:
pg_query(0.974 epoch 3315) v3
   -13> 2014-05-20 10:31:55.559661 7fd3fdb0f700  1 --
130.246.178.141:6837/10446 >> :/0 pipe(0xce0c380 sd=468 :6837 s=0 pgs=0 cs=0
l=0 c=0x1255f0c0).accept sd=468 130.246.179.191:60618/0
   -12> 2014-05-20 10:31:55.564034 7fd3bf72f700  1 --
130.246.178.141:6838/10446 >> :/0 pipe(0xe3f2300 sd=596 :6838 s=0 pgs=0 cs=0
l=0 c=0x129b5ee0).accept sd=596 130.246.179.191:43913/0
   -11> 2014-05-20 10:31:55.627776 7fd42df4b700  1 --
130.246.178.141:0/10446 <== osd.170 130.246.179.191:6827/21854 3 
osd_ping(ping_reply e3316 stamp 2014-05-20 10:31:52.994368) v2  47+0+0
(855262282 0 0) 0xb6863c0 con 0x1255b9c0
   -10> 2014-05-20 10:31:55.629425 7fd42df4b700  1 --
130.246.178.141:0/10446 <== osd.170 130.246.179.191:6827/21854 4 
osd_ping(ping_reply e3316 stamp 2014-05-20 10:31:53.509621) v2  47+0+0
(2581193378 0 0) 0x93d6c80 con 0x1255b9c0
-9> 2014-05-20 10:31:55.631270 7fd42f34d700  1 --
130.246.178.141:6836/10446 <== osd.169 130.246.179.191:6841/25473 2 
pg_query(7.3ffs6 epoch 3326) v3  144+0+0 (221596234 0 0) 0x10b994a0 con
0x9383860
-8> 2014-05-20 10:31:55.631308 7fd42f34d700  5 -- op tracker -- , seq:
14302, time: 2014-05-20 10:31:55.631130, event: header_read, request:
pg_query(7.3ffs6 epoch 3326

Re: [ceph-users] Access denied error for list users

2014-05-21 Thread Shanil S

Hi Alian,

Thanks for your reply.

Do you mean we can't list out all users with complete user details using
GET /admin/metadata/user or using GET /admin/user?

Yes, i checked http://ceph.com/docs/master/radosgw/s3/php/ and it contains
only the bucket operations and not any admin operations like list
users,create users modify user etc. Is there any other php for this ? if
so, i can use directly that api for admin operations



On Wed, May 21, 2014 at 1:33 PM,  wrote:

> There is no detail with GET /admin/metadata/user, only ids.
>
> For PHP, have a look at http://ceph.com/docs/master/radosgw/s3/php/
>
> Alain
>
> De : Shanil S [mailto:xielessha...@gmail.com]
> Envoyé : mercredi 21 mai 2014 05:48
> À : DECHORGNAT Alain IMT/OLPS
> Objet : Re: [ceph-users] Access denied error for list users
>
> Hi Alain,
>
> Thanks..
> I used the GET /admin/metadata/user to fetch the user list but it only
> shows the usernames in the list. I would like to show the other details too
> like bucket number,id etc. Can i use the same GET /admin/metadata/user to
> get all these details ? Also, is there any easy way to generate the access
> token authorization header using php ?
>
> On Tue, May 20, 2014 at 7:36 PM,  wrote:
> Hi,
>
> GET /admin/user with no parameter doesn't work.
>
> You must use GET /admin/metadata/user to fetch the user list (with
> metadata capabity).
>
> Alain
>
>
> De : ceph-users [mailto:ceph-users-boun...@lists.ceph.com] De la part de
> Shanil S
> Envoyé : mardi 20 mai 2014 07:13
> À : ceph-users@lists.ceph.com; w...@42on.com; s...@inktank.com; Yehuda
> Sadeh
> Objet : [ceph-users] Access denied error for list users
>
> Hi,
>
> I am trying to create and list all users by using the functions
> http://ceph.com/docs/master/radosgw/adminops/ and i successfully created
> the access tokens but i am getting an access denied and 403 for listing
> users function. The GET /{admin}/user is used for getting the complete
> users list, but its not listing and getting the error. The user which
> called this function has the complete permission and i am adding the
> permission of this user
>
> { "type": "admin",
>   "perm": "*"},
> { "type": "buckets",
>   "perm": "*"},
> { "type": "caps",
>   "perm": "*"},
> { "type": "metadata",
>   "perm": "*"},
> { "type": "usage",
>   "perm": "*"},
> { "type": "users",
>   "perm": "*"}],
>   "op_mask": "read, write, delete",
>   "default_placement": "",
>   "placement_tags": [],
>   "bucket_quota": { "enabled": false,
>   "max_size_kb": -1,
>   "max_objects": -1}}
>
>
> This is in the log file which executed the list user function
>
>
> -
>
> GET
>
> application/x-www-form-urlencoded
> Tue, 20 May 2014 05:06:57 GMT
> /admin/user/
> 2014-05-20 13:06:59.506233 7f0497fa7700 15 calculated
> digest=Z8FgXRLk+ah5MUThpP9IBJrMnrA=
> 2014-05-20 13:06:59.506236 7f0497fa7700 15
> auth_sign=Z8FgXRLk+ah5MUThpP9IBJrMnrA=
> 2014-05-20 13:06:59.506237 7f0497fa7700 15 compare=0
> 2014-05-20 13:06:59.506240 7f0497fa7700  2 req 98:0.000308::GET
> /admin/user/:get_user_info:reading permissions
> 2014-05-20 13:06:59.506244 7f0497fa7700  2 req 98:0.000311::GET
> /admin/user/:get_user_info:init op
> 2014-05-20 13:06:59.506247 7f0497fa7700  2 req 98:0.000314::GET
> /admin/user/:get_user_info:verifying op mask
> 2014-05-20 13:06:59.506249 7f0497fa7700 20 required_mask= 0 user.op_mask=7
> 2014-05-20 13:06:59.506251 7f0497fa7700  2 req 98:0.000319::GET
> /admin/user/:get_user_info:verifying op permissions
> 2014-05-20 13:06:59.506254 7f0497fa7700  2 req 98:0.000322::GET
> /admin/user/:get_user_info:verifying op params
> 2014-05-20 13:06:59.506257 7f0497fa7700  2 req 98:0.000324::GET
> /admin/user/:get_user_info:executing
> 2014-05-20 13:06:59.506291 7f0497fa7700  2 req 98:0.000359::GET
> /admin/user/:get_user_info:http status=403
> 2014-05-20 13:06:59.506294 7f0497fa7700  1 == req done
> req=0x7f04c800d7f0 http_status=403 ==
> 2014-05-20 13:06:59.506302 7f0497fa7700 20 process_request() returned -13
>
>
> -
>
> Could you please check what is the issue ?
> I am using the ceph version : ceph version 0.80.1
>
>
> _
>
> Ce message et ses pieces jointes peuvent contenir des informations
> confidentielles ou privilegiees et ne doivent donc
> pas etre diffuses, exploites ou copies sans autorisation. Si vous avez
> recu ce message par erreur, veuillez le signaler
> a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
> electroniques etant susceptibles d'alteration,
> Orange decline toute responsabilite si ce message a ete altere, deforme ou
> falsifie. Merci.
>
> This message and

Re: [ceph-users] 70+ OSD are DOWN and not coming up

2014-05-21 Thread Karan Singh

Hello Sage

nodown, noout set on cluster

# ceph status
cluster 009d3518-e60d-4f74-a26d-c08c1976263c
 health HEALTH_WARN 1133 pgs degraded; 44 pgs incomplete; 42 pgs stale; 45 
pgs stuck inactive; 42 pgs stuck stale; 2602 pgs stuck unclean; recovery 
206/2199 objects degraded (9.368%); 40/165 in osds are down; nodown,noout 
flag(s) set
 monmap e4: 4 mons at 
{storage0101-ib=192.168.100.101:6789/0,storage0110-ib=192.168.100.110:6789/0,storage0114-ib=192.168.100.114:6789/0,storage0115-ib=192.168.100.115:6789/0},
 election epoch 18, quorum 0,1,2,3 
storage0101-ib,storage0110-ib,storage0114-ib,storage0115-ib
 osdmap e358031: 165 osds: 125 up, 165 in
flags nodown,noout
  pgmap v604305: 4544 pgs, 6 pools, 4309 MB data, 733 objects
3582 GB used, 357 TB / 361 TB avail
206/2199 objects degraded (9.368%)
   1 inactive
   5 stale+active+degraded+remapped
1931 active+clean
   2 stale+incomplete
  21 stale+active+remapped
 380 active+degraded+remapped
  38 incomplete
1403 active+remapped
   2 stale+active+degraded
   1 stale+remapped+incomplete
 746 active+degraded
  11 stale+active+clean
   3 remapped+incomplete


 Here is my ceph.conf  http://pastebin.com/KZdgPJm7  (debus osd , ms set )
I tried restarting all OSD services of  node-13 , services came up after 
several attempts of “service ceph restart”   http://pastebin.com/yMk86YHh
For Node : 14
All services are up

[root@storage0114-ib ~]# service ceph status
=== osd.142 ===
osd.142: running {"version":"0.80-475-g9e80c29"}
=== osd.36 ===
osd.36: running {"version":"0.80-475-g9e80c29"}
=== osd.83 ===
osd.83: running {"version":"0.80-475-g9e80c29"}
=== osd.107 ===
osd.107: running {"version":"0.80-475-g9e80c29"}
=== osd.47 ===
osd.47: running {"version":"0.80-475-g9e80c29"}
=== osd.130 ===
osd.130: running {"version":"0.80-475-g9e80c29"}
=== osd.155 ===
osd.155: running {"version":"0.80-475-g9e80c29"}
=== osd.60 ===
osd.60: running {"version":"0.80-475-g9e80c29"}
=== osd.118 ===
osd.118: running {"version":"0.80-475-g9e80c29"}
=== osd.98 ===
osd.98: running {"version":"0.80-475-g9e80c29"}
=== osd.70 ===
osd.70: running {"version":"0.80-475-g9e80c29"}
=== mon.storage0114-ib ===
mon.storage0114-ib: running {"version":"0.80-475-g9e80c29"}
[root@storage0114-ib ~]#

— But ceph osd tree says , osd.118  is down

-10 29.93   host storage0114-ib
36  2.63osd.36  up  1
47  2.73osd.47  up  1
60  2.73osd.60  up  1
70  2.73osd.70  up  1
83  2.73osd.83  up  1
98  2.73osd.98  up  1
107 2.73osd.107 up  1
118 2.73osd.118 down1
130 2.73osd.130 up  1
142 2.73osd.142 up  1
155 2.73osd.155 up  1

— I restarted osd.118 service and it was successful , But still its showing as 
down in ceph osd tree . I waited for 30 minutes to get it stable but still not 
showing UP in ceph osd tree. 
Moreover its generating HUGE logs http://pastebin.com/mDYnjAni



The problem now is if i manually visit every host and check for “service ceph 
status “ all services are running on all 15 hosts. But this is not getting 
reflected to ceph osd tree and ceph -s and they continue to show as OSD DOWN.

My irc id is ksingh , let me know by email once you are available on IRC (my 
time zone is Finland +2)



- Karan Singh - 

On 20 May 2014, at 18:18, Sage Weil  wrote:

> On Tue, 20 May 2014, Karan Singh wrote:
>> Hello Cephers , need your suggestion for troubleshooting.
>> 
>> My cluster is terribly struggling , 70+ osd are down out of 165
>> 
>> Problem ?>OSD are getting marked out of cluster and are down. The cluster is
>> degraded. On checking logs of failed OSD we are getting wired entries that
>> are continuously getting generated.
> 
> Tracking this at http://tracker.ceph.com/issues/8387
> 
> The most recent bits you posted in the ticket don't quite make sense: the 
> OSD is trying to connect to an address for an OSD that is currently marked 
> down.  I suspect this is just timing between when the logs were captured 
> and when teh ceph osd dump was captured.  To get a complete pictures, 
> please:
> 
> 1) add
> 
> debug osd = 20
> debug ms = 1
> 
> in [osd] and restart all osds
> 
> 2) ceph osd set nodown
> 
> (to prevent flapping)
> 
> 3) find some OSD that is showing these messages
> 
> 4) capture a 'ceph osd dump' output.
> 
> Also happy to debug this interactively over IRC; that will likely be 
> faster!
> 
> Thanks-
> sage
> 
> 
> 
>> 
>> Osd Debug logs ::  http://pastebin.com/agTKh6zB
>> 
>> 
>> 1. 2014-05-20 10:19

[ceph-users] Ceph Firefly on Centos 6.5 cannot deploy osd

2014-05-21 Thread 10 minus

Hi,

I have just started to dabble with ceph - went thru the docs
http://ceph.com/howto/deploying-ceph-with-ceph-deploy/


I have a 3 node setup with 2 nodes for OSD

I use ceph-deploy mechanism.

The ceph init scripts expects that cluster.conf  to be ceph.conf . If I
give any other name the init scripts dont work. So for test purpose Im
using  ceph.conf


--ceph.conf--
[global]
auth_service_required = cephx
filestore_xattr_use_omap = true
auth_client_required = cephx
auth_cluster_required = cephx
mon_host = 172.18.1.31,172.18.1.32,172.18.1.33
mon_initial_members = cc01, cc02, cc03
fsid = b58e50f1-13a3-4b14-9cff-32b6edd851c9
--snip--

I managed to get mon deployed but ceph -s returns health error

--snip--
 ceph -s
cluster b58e50f1-13a3-4b14-9cff-32b6edd851c9
 health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean; no
osds
 monmap e1: 3 mons at {cc01=
172.18.1.31:6789/0,cc02=172.18.1.32:6789/0,cc03=172.18.1.33:6789/0},
election epoch 4, quorum 0,1,2 cc01,cc02,cc03
 osdmap e1: 0 osds: 0 up, 0 in
  pgmap v2: 192 pgs, 3 pools, 0 bytes data, 0 objects
0 kB used, 0 kB / 0 kB avail
 192 creating
--snip--

I tried creating two osds. Well they fail too probably has to do with
health error message.

 --snip--
 ceph-deploy osd create cc01:/dev/sdb cc02:/dev/sdb
[ceph_deploy.conf][DEBUG ] found configuration file at:
/root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.2): /usr/bin/ceph-deploy osd create
cc01:/dev/sdb cc02:/dev/sdb
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks cc01:/dev/sdb:
cc02:/dev/sdb:
[cc01][DEBUG ] connected to host: cc01
[cc01][DEBUG ] detect platform information from remote host
[cc01][DEBUG ] detect machine type
[ceph_deploy.osd][INFO  ] Distro info: CentOS 6.5 Final
[ceph_deploy.osd][DEBUG ] Deploying osd to cc01
[cc01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[cc01][INFO  ] Running command: udevadm trigger --subsystem-match=block
--action=add
[ceph_deploy.osd][DEBUG ] Preparing host cc01 disk /dev/sdb journal None
activate True
[cc01][INFO  ] Running command: ceph-disk-prepare --fs-type xfs --cluster
ceph -- /dev/sdb
[cc01][WARNIN] INFO:ceph-disk:Will colocate journal with data on /dev/sdb
[cc01][WARNIN] Could not create partition 2 from 10485761 to 10485760
[cc01][WARNIN] Error encountered; not saving changes.
[cc01][WARNIN] ceph-disk: Error: Command '['/usr/sbin/sgdisk',
'--new=2:0:5120M', '--change-name=2:ceph journal',
'--partition-guid=2:d882631c-0069-4238-86df-9762ad478daa',
'--typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106', '--mbrtogpt', '--',
'/dev/sdb']' returned non-zero exit status 4
[cc01][DEBUG ] Setting name!
[cc01][DEBUG ] partNum is 1
[cc01][DEBUG ] REALLY setting name!
[cc01][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy.osd][ERROR ] Failed to execute command: ceph-disk-prepare
--fs-type xfs --cluster ceph -- /dev/sdb
[cc02][DEBUG ] connected to host: cc02
[cc02][DEBUG ] detect platform information from remote host
[cc02][DEBUG ] detect machine type
[ceph_deploy.osd][INFO  ] Distro info: CentOS 6.5 Final
[ceph_deploy.osd][DEBUG ] Deploying osd to cc02
[cc02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[cc02][INFO  ] Running command: udevadm trigger --subsystem-match=block
--action=add
[ceph_deploy.osd][DEBUG ] Preparing host cc02 disk /dev/sdb journal None
activate True
[cc02][INFO  ] Running command: ceph-disk-prepare --fs-type xfs --cluster
ceph -- /dev/sdb
[cc02][WARNIN] INFO:ceph-disk:Will colocate journal with data on /dev/sdb
[cc02][WARNIN] Could not create partition 2 from 10485761 to 10485760
[cc02][WARNIN] Error encountered; not saving changes.
[cc02][WARNIN] ceph-disk: Error: Command '['/usr/sbin/sgdisk',
'--new=2:0:5120M', '--change-name=2:ceph journal',
'--partition-guid=2:486c9081-a73c-4906-b97a-c03458feba26',
'--typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106', '--mbrtogpt', '--',
'/dev/sdb']' returned non-zero exit status 4
[cc02][DEBUG ] Found valid GPT with corrupt MBR; using GPT and will write
new
[cc02][DEBUG ] protective MBR on save.
[cc02][DEBUG ] Setting name!
[cc02][DEBUG ] partNum is 1
[cc02][DEBUG ] REALLY setting name!
[cc02][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy.osd][ERROR ] Failed to execute command: ceph-disk-prepare
--fs-type xfs --cluster ceph -- /dev/sdb
[ceph_deploy][ERROR ] GenericError: Failed to create 2 OSDs
--snip--

Any pointers to fix the issue.

Cheers
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Firefly on Centos 6.5 cannot deploy osd

2014-05-21 Thread ceph


Hi,

When you just create a cluster, with no OSD, the HEALTH_ERR is "normal"
It mean that your storage is damaged, but you don't care since you've no 
storage at this point


About your OSDs, I think you should create partition on your disks (a 
single partition, properly aligned etc), instead of using the whole raw disk


I'm using these steps:
- create a partition
- mkfs.xfs
- mkdir & mount
- ceph-deploy osd prepare host:/path/to/mounted-fs

Dunno if it's the right way, seems to work so far



On 21/05/2014 16:05, 10 minus wrote:

Hi,

I have just started to dabble with ceph - went thru the docs
http://ceph.com/howto/deploying-ceph-with-ceph-deploy/


I have a 3 node setup with 2 nodes for OSD

I use ceph-deploy mechanism.

The ceph init scripts expects that cluster.conf  to be ceph.conf . If I
give any other name the init scripts dont work. So for test purpose Im
using  ceph.conf


--ceph.conf--
[global]
auth_service_required = cephx
filestore_xattr_use_omap = true
auth_client_required = cephx
auth_cluster_required = cephx
mon_host = 172.18.1.31,172.18.1.32,172.18.1.33
mon_initial_members = cc01, cc02, cc03
fsid = b58e50f1-13a3-4b14-9cff-32b6edd851c9
--snip--

I managed to get mon deployed but ceph -s returns health error

--snip--
  ceph -s
 cluster b58e50f1-13a3-4b14-9cff-32b6edd851c9
  health HEALTH_ERR 192 pgs stuck inactive; 192 pgs stuck unclean;
no osds
  monmap e1: 3 mons at
{cc01=172.18.1.31:6789/0,cc02=172.18.1.32:6789/0,cc03=172.18.1.33:6789/0
},
election epoch 4, quorum 0,1,2 cc01,cc02,cc03
  osdmap e1: 0 osds: 0 up, 0 in
   pgmap v2: 192 pgs, 3 pools, 0 bytes data, 0 objects
 0 kB used, 0 kB / 0 kB avail
  192 creating
--snip--

I tried creating two osds. Well they fail too probably has to do with
health error message.

  --snip--
  ceph-deploy osd create cc01:/dev/sdb cc02:/dev/sdb
[ceph_deploy.conf][DEBUG ] found configuration file at:
/root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.2): /usr/bin/ceph-deploy osd
create cc01:/dev/sdb cc02:/dev/sdb
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks cc01:/dev/sdb:
cc02:/dev/sdb:
[cc01][DEBUG ] connected to host: cc01
[cc01][DEBUG ] detect platform information from remote host
[cc01][DEBUG ] detect machine type
[ceph_deploy.osd][INFO  ] Distro info: CentOS 6.5 Final
[ceph_deploy.osd][DEBUG ] Deploying osd to cc01
[cc01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[cc01][INFO  ] Running command: udevadm trigger --subsystem-match=block
--action=add
[ceph_deploy.osd][DEBUG ] Preparing host cc01 disk /dev/sdb journal None
activate True
[cc01][INFO  ] Running command: ceph-disk-prepare --fs-type xfs
--cluster ceph -- /dev/sdb
[cc01][WARNIN] INFO:ceph-disk:Will colocate journal with data on /dev/sdb
[cc01][WARNIN] Could not create partition 2 from 10485761 to 10485760
[cc01][WARNIN] Error encountered; not saving changes.
[cc01][WARNIN] ceph-disk: Error: Command '['/usr/sbin/sgdisk',
'--new=2:0:5120M', '--change-name=2:ceph journal',
'--partition-guid=2:d882631c-0069-4238-86df-9762ad478daa',
'--typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106', '--mbrtogpt', '--',
'/dev/sdb']' returned non-zero exit status 4
[cc01][DEBUG ] Setting name!
[cc01][DEBUG ] partNum is 1
[cc01][DEBUG ] REALLY setting name!
[cc01][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy.osd][ERROR ] Failed to execute command: ceph-disk-prepare
--fs-type xfs --cluster ceph -- /dev/sdb
[cc02][DEBUG ] connected to host: cc02
[cc02][DEBUG ] detect platform information from remote host
[cc02][DEBUG ] detect machine type
[ceph_deploy.osd][INFO  ] Distro info: CentOS 6.5 Final
[ceph_deploy.osd][DEBUG ] Deploying osd to cc02
[cc02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[cc02][INFO  ] Running command: udevadm trigger --subsystem-match=block
--action=add
[ceph_deploy.osd][DEBUG ] Preparing host cc02 disk /dev/sdb journal None
activate True
[cc02][INFO  ] Running command: ceph-disk-prepare --fs-type xfs
--cluster ceph -- /dev/sdb
[cc02][WARNIN] INFO:ceph-disk:Will colocate journal with data on /dev/sdb
[cc02][WARNIN] Could not create partition 2 from 10485761 to 10485760
[cc02][WARNIN] Error encountered; not saving changes.
[cc02][WARNIN] ceph-disk: Error: Command '['/usr/sbin/sgdisk',
'--new=2:0:5120M', '--change-name=2:ceph journal',
'--partition-guid=2:486c9081-a73c-4906-b97a-c03458feba26',
'--typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106', '--mbrtogpt', '--',
'/dev/sdb']' returned non-zero exit status 4
[cc02][DEBUG ] Found valid GPT with corrupt MBR; using GPT and will
write new
[cc02][DEBUG ] protective MBR on save.
[cc02][DEBUG ] Setting name!
[cc02][DEBUG ] partNum is 1
[cc02][DEBUG ] REALLY setting name!
[cc02][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy.osd][ERROR ] Failed to execute command: ceph-disk-prepare
--fs-type xfs --cluster ceph -

[ceph-users] How to find the disk partitions attached to a OSD

2014-05-21 Thread Sharmila Govind

Hi,
 I am new to Ceph. I have a storage node with 2 OSDs. Iam trying to figure
out to which pyhsical device/partition each of the OSDs are attached to. Is
there are command that can be executed in the storage node to find out the
same.

Thanks in Advance,
Sharmila
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to find the disk partitions attached to a OSD

2014-05-21 Thread Mike Dawson


Perhaps:

# mount | grep ceph

- Mike Dawson


On 5/21/2014 11:00 AM, Sharmila Govind wrote:

Hi,
  I am new to Ceph. I have a storage node with 2 OSDs. Iam trying to
figure out to which pyhsical device/partition each of the OSDs are
attached to. Is there are command that can be executed in the storage
node to find out the same.

Thanks in Advance,
Sharmila


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to find the disk partitions attached to a OSD

2014-05-21 Thread Sharmila Govind

Hi Mike,
Thanks for your quick response. When I try mount on the storage node this
is what I get:

*root@cephnode4:~# mount*
*/dev/sda1 on / type ext4 (rw,errors=remount-ro)*
*proc on /proc type proc (rw,noexec,nosuid,nodev)*
*sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)*
*none on /sys/fs/fuse/connections type fusectl (rw)*
*none on /sys/kernel/debug type debugfs (rw)*
*none on /sys/kernel/security type securityfs (rw)*
*udev on /dev type devtmpfs (rw,mode=0755)*
*devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)*
*tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)*
*none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)*
*none on /run/shm type tmpfs (rw,nosuid,nodev)*
*/dev/sdb on /mnt/CephStorage1 type ext4 (rw)*
*/dev/sdc on /mnt/CephStorage2 type ext4 (rw)*
*/dev/sda7 on /mnt/Storage type ext4 (rw)*
*/dev/sda2 on /boot type ext4 (rw)*
*/dev/sda5 on /home type ext4 (rw)*
*/dev/sda6 on /mnt/CephStorage type ext4 (rw)*



Is there anything wrong in the setup I have? I dont have any 'ceph' related
mounts.

Thanks,
Sharmila



On Wed, May 21, 2014 at 8:34 PM, Mike Dawson wrote:

> Perhaps:
>
> # mount | grep ceph
>
> - Mike Dawson
>
>
>
> On 5/21/2014 11:00 AM, Sharmila Govind wrote:
>
>> Hi,
>>   I am new to Ceph. I have a storage node with 2 OSDs. Iam trying to
>> figure out to which pyhsical device/partition each of the OSDs are
>> attached to. Is there are command that can be executed in the storage
>> node to find out the same.
>>
>> Thanks in Advance,
>> Sharmila
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Data still in OSD directories after removing

2014-05-21 Thread Sage Weil

Hi Olivier,

On Wed, 21 May 2014, Olivier Bonvalet wrote:
> Hi,
> 
> I have a lot of space wasted by this problem (about 10GB per OSD, just
> for this RBD image).
> If OSDs can't detect orphans files, should I manually detect them, then
> remove them ?
> 
> This command can do the job, at least for this image prefix :
> find /var/lib/ceph/osd/ -name 'rb.0.14bfb5a.238e1f29.*' -delete

You should definitely not do this!  :)

You're certain that that is the correct prefix for the rbd image you 
removed?  Do you see the objects lists when you do 'rados -p rbd ls - | 
grep '?

If the objects really are orphaned, teh way to clean them up is via 'rados 
-p rbd rm '.  I'd like to get to the bottom of how they ended 
up that way first, though!

sage

> 
> Thanks for any advice,
> Olivier
> 
> PS : not sure if this kind of problem is for the user or dev mailing
> list.
> 
> Le mardi 20 mai 2014 ? 11:32 +0200, Olivier Bonvalet a ?crit :
> > Hi,
> > 
> > short : I removed a 1TB RBD image, but I still see files about it on
> > OSD.
> > 
> > 
> > long :
> > 1) I did : "rbd snap purge $pool/$img"
> >but since it overload the cluster, I stopped it (CTRL+C)
> > 2) latter, "rbd snap purge $pool/$img"
> > 3) then, "rbd rm $pool/$img"
> > 
> > now, on the disk I can found files of this v1 RBD image (prefix was
> > rb.0.14bfb5a.238e1f29) :
> > 
> > # find /var/lib/ceph/osd/ceph-64/ -name 'rb.0.14bfb5a.238e1f29.*'
> > /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_3/rb.0.14bfb5a.238e1f29.00021431__snapdir_C96635C1__9
> > /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_3/rb.0.14bfb5a.238e1f29.5622__a252_32F435C1__9
> > /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_3/rb.0.14bfb5a.238e1f29.00021431__a252_C96635C1__9
> > /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_3/rb.0.14bfb5a.238e1f29.5622__snapdir_32F435C1__9
> > /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_9/rb.0.14bfb5a.238e1f29.00011e08__a172_594495C1__9
> > /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_9/rb.0.14bfb5a.238e1f29.00011e08__snapdir_594495C1__9
> > /var/lib/ceph/osd/ceph-64/current/9.5c1_head/DIR_1/DIR_C/DIR_5/DIR_A/rb.0.14bfb5a.238e1f29.00021620__a252_779FA5C1__9
> > ...
> > 
> > 
> > So, is there a way to force OSD to detect if files are orphans, then
> > remove them ?
> > 
> > Thanks,
> > Olivier
> > 
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to find the disk partitions attached to a OSD

2014-05-21 Thread Mike Dawson

Looks like you may not have any OSDs properly setup and mounted. It 
should look more like:


user@host:~# mount | grep ceph
/dev/sdb1 on /var/lib/ceph/osd/ceph-0 type xfs (rw,noatime,inode64)
/dev/sdc1 on /var/lib/ceph/osd/ceph-1 type xfs (rw,noatime,inode64)
/dev/sdd1 on /var/lib/ceph/osd/ceph-2 type xfs (rw,noatime,inode64)

Confirm the OSD in your ceph cluster with:

user@host:~# ceph osd tree

- Mike


On 5/21/2014 11:15 AM, Sharmila Govind wrote:

Hi Mike,
Thanks for your quick response. When I try mount on the storage node
this is what I get:

*root@cephnode4:~# mount*
*/dev/sda1 on / type ext4 (rw,errors=remount-ro)*
*proc on /proc type proc (rw,noexec,nosuid,nodev)*
*sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)*
*none on /sys/fs/fuse/connections type fusectl (rw)*
*none on /sys/kernel/debug type debugfs (rw)*
*none on /sys/kernel/security type securityfs (rw)*
*udev on /dev type devtmpfs (rw,mode=0755)*
*devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)*
*tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)*
*none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)*
*none on /run/shm type tmpfs (rw,nosuid,nodev)*
*/dev/sdb on /mnt/CephStorage1 type ext4 (rw)*
*/dev/sdc on /mnt/CephStorage2 type ext4 (rw)*
*/dev/sda7 on /mnt/Storage type ext4 (rw)*
*/dev/sda2 on /boot type ext4 (rw)*
*/dev/sda5 on /home type ext4 (rw)*
*/dev/sda6 on /mnt/CephStorage type ext4 (rw)*



Is there anything wrong in the setup I have? I dont have any 'ceph'
related mounts.

Thanks,
Sharmila



On Wed, May 21, 2014 at 8:34 PM, Mike Dawson mailto:mike.daw...@cloudapt.com>> wrote:

Perhaps:

# mount | grep ceph

- Mike Dawson



On 5/21/2014 11:00 AM, Sharmila Govind wrote:

Hi,
   I am new to Ceph. I have a storage node with 2 OSDs. Iam
trying to
figure out to which pyhsical device/partition each of the OSDs are
attached to. Is there are command that can be executed in the
storage
node to find out the same.

Thanks in Advance,
Sharmila


_
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/__listinfo.cgi/ceph-users-ceph.__com




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Problem with radosgw and some file name characters

2014-05-21 Thread Yehuda Sadeh

On Tue, May 20, 2014 at 4:13 AM, Andrei Mikhailovsky  wrote:
> Anyone have any idea how to fix the problem with getting 403 when trying to
> upload files with none standard characters? I am sure I am not the only one
> with these requirements.

It might be the specific client that you're using and the way it signs
the requests. Can you try a different S3 client, see whether it works
or not? Are you by any chance going through some kind of a load
balancer that rewrites the urls?

Yehuda

> 
> From: "Andrei Mikhailovsky" 
> To: "Yehuda Sadeh" 
> Cc: ceph-users@lists.ceph.com
> Sent: Monday, 19 May, 2014 12:38:29 PM
>
> Subject: Re: [ceph-users] Problem with radosgw and some file name characters
>
> Yehuda,
>
> Never mind my last post, i've found the issue with the rule that you've
> suggested. my fastcgi script is called differently, so that's why i was
> getting the 404.
>
> I've tried your rewrite rule and I am still having the same issues. The same
> characters are failing with the rule you've suggested.
>
>
> Any idea how to fix the issue?
>
> Cheers
>
> Andrei
> 
> From: "Andrei Mikhailovsky" 
> To: "Yehuda Sadeh" 
> Cc: ceph-users@lists.ceph.com
> Sent: Monday, 19 May, 2014 9:30:03 AM
> Subject: Re: [ceph-users] Problem with radosgw and some file name characters
>
> Yehuda,
>
> I've tried the rewrite rule that you've suggested, but it is not working for
> me. I get 404 when trying to access the service.
>
> RewriteRule  ^/(.*) /s3gw.3.fcgi?%{QUERY_STRING}
> [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
>
> Any idea what is wrong with this rule?
>
> Cheers
>
> Andrei
>
> 
>
>
> From: "Yehuda Sadeh" 
> To: "Andrei Mikhailovsky" 
> Cc: ceph-users@lists.ceph.com
> Sent: Friday, 16 May, 2014 5:44:52 PM
> Subject: Re: [ceph-users] Problem with radosgw and some file name characters
>
> Was talking about this. There is a different and simpler rule that we
> use nowadays, for some reason it's not well documented:
>
> RewriteRule  ^/(.*) /s3gw.3.fcgi?%{QUERY_STRING}
> [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
>
> I still need to see a more verbose log to make a better educated guess.
>
> Yehuda
>
> On Thu, May 15, 2014 at 3:01 PM, Andrei Mikhailovsky 
> wrote:
>>
>> Yehuda,
>>
>> what do you mean by the rewrite rule? is this for Apache? I've used the
>> ceph
>> documentation to create it. My rule is:
>>
>>
>> RewriteRule ^/([a-zA-Z0-9-_.]*)([/]?.*)
>> /s3gw.fcgi?page=$1¶ms=$2&%{QUERY_STRING}
>> [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
>>
>> Or are you talking about something else?
>>
>> Cheers
>>
>> Andrei
>> 
>> From: "Yehuda Sadeh" 
>> To: "Andrei Mikhailovsky" 
>> Cc: ceph-users@lists.ceph.com
>> Sent: Thursday, 15 May, 2014 4:05:06 PM
>> Subject: Re: [ceph-users] Problem with radosgw and some file name
>> characters
>>
>>
>> Your rewrite rule might be off a bit. Can you provide log with 'debug rgw
>> =
>> 20'?
>>
>> Yehuda
>>
>> On Thu, May 15, 2014 at 8:02 AM, Andrei Mikhailovsky 
>> wrote:
>>> Hello guys,
>>>
>>>
>>> I am trying to figure out what is the problem here.
>>>
>>>
>>> Currently running Ubuntu 12.04 with latest updates and radosgw version
>>> 0.72.2-1precise. My ceph.conf file is pretty standard from the radosgw
>>> howto.
>>>
>>>
>>>
>>> I am testing radosgw as a backup solution to S3 compatible clients. I am
>>> planning to copy a large number of files/folders and I am having issues
>>> with
>>> a large number of files. The client reports the following error on some
>>> files:
>>>
>>>
>>> 
>>>
>>> 
>>>
>>> AccessDenied
>>>
>>> 
>>>
>>>
>>> Looking on the server backup I only see the following errors in the
>>> radosgw.log file:
>>>
>>> 2014-05-13 23:50:35.786181 7f09467dc700  1 == starting new request
>>> req=0x245d7e0 =
>>> 2014-05-13 23:50:35.786470 7f09467dc700  1 == req done req=0x245d7e0
>>> http_status=403 ==
>>>
>>>
>>> So, i've done  a small file set comprising of test files including the
>>> following names:
>>>
>>> Testing and Testing.txt
>>> Testing ^ Testing.txt
>>> Testing = Testing.txt
>>> Testing _ Testing.txt
>>> Testing - Testing.txt
>>> Testing ; Testing.txt
>>> Testing ! Testing.txt
>>> Testing ? Testing.txt
>>> Testing ( Testing.txt
>>> Testing ) Testing.txt
>>> Testing @ Testing.txt
>>> Testing $ Testing.txt
>>> Testing * Testing.txt
>>> Testing & Testing.txt
>>> Testing # Testing.txt
>>> Testing % Testing.txt
>>> Testing + Testing.txt
>>>
>>> From the above list the files with the following characters are giving me
>>> Access Denied / 403 error:
>>>
>>> =;()@$*&+
>>>
>>> The rest of the files are successfully uploaded.
>>>
>>> Does anyone know what is required to fix the problem?
>>>
>>> Many thanks
>>>
>>> Andrei
>>>
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>
>
>
> __

Re: [ceph-users] How to find the disk partitions attached to a OSD

2014-05-21 Thread Sage Weil

You might also try

 ceph-disk list

sage


On Wed, 21 May 2014, Mike Dawson wrote:

> Looks like you may not have any OSDs properly setup and mounted. It should
> look more like:
> 
> user@host:~# mount | grep ceph
> /dev/sdb1 on /var/lib/ceph/osd/ceph-0 type xfs (rw,noatime,inode64)
> /dev/sdc1 on /var/lib/ceph/osd/ceph-1 type xfs (rw,noatime,inode64)
> /dev/sdd1 on /var/lib/ceph/osd/ceph-2 type xfs (rw,noatime,inode64)
> 
> Confirm the OSD in your ceph cluster with:
> 
> user@host:~# ceph osd tree
> 
> - Mike
> 
> 
> On 5/21/2014 11:15 AM, Sharmila Govind wrote:
> > Hi Mike,
> > Thanks for your quick response. When I try mount on the storage node
> > this is what I get:
> > 
> > *root@cephnode4:~# mount*
> > */dev/sda1 on / type ext4 (rw,errors=remount-ro)*
> > *proc on /proc type proc (rw,noexec,nosuid,nodev)*
> > *sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)*
> > *none on /sys/fs/fuse/connections type fusectl (rw)*
> > *none on /sys/kernel/debug type debugfs (rw)*
> > *none on /sys/kernel/security type securityfs (rw)*
> > *udev on /dev type devtmpfs (rw,mode=0755)*
> > *devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)*
> > *tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)*
> > *none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)*
> > *none on /run/shm type tmpfs (rw,nosuid,nodev)*
> > */dev/sdb on /mnt/CephStorage1 type ext4 (rw)*
> > */dev/sdc on /mnt/CephStorage2 type ext4 (rw)*
> > */dev/sda7 on /mnt/Storage type ext4 (rw)*
> > */dev/sda2 on /boot type ext4 (rw)*
> > */dev/sda5 on /home type ext4 (rw)*
> > */dev/sda6 on /mnt/CephStorage type ext4 (rw)*
> > 
> > 
> > 
> > Is there anything wrong in the setup I have? I dont have any 'ceph'
> > related mounts.
> > 
> > Thanks,
> > Sharmila
> > 
> > 
> > 
> > On Wed, May 21, 2014 at 8:34 PM, Mike Dawson  > > wrote:
> > 
> > Perhaps:
> > 
> > # mount | grep ceph
> > 
> > - Mike Dawson
> > 
> > 
> > 
> > On 5/21/2014 11:00 AM, Sharmila Govind wrote:
> > 
> > Hi,
> >I am new to Ceph. I have a storage node with 2 OSDs. Iam
> > trying to
> > figure out to which pyhsical device/partition each of the OSDs are
> > attached to. Is there are command that can be executed in the
> > storage
> > node to find out the same.
> > 
> > Thanks in Advance,
> > Sharmila
> > 
> > 
> > _
> > ceph-users mailing list
> > ceph-users@lists.ceph.com 
> > http://lists.ceph.com/__listinfo.cgi/ceph-users-ceph.__com
> > 
> > 
> > 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Inter-region data replication through radosgw

2014-05-21 Thread Fabrizio G. Ventola

Hi everybody,

I'm reading the doc regarding the replication through radosgw. It
talks just about inter-region METAdata replication, nothing about data
replication.

My question is, it's possible to have (everything) geo-replicated
through radosgw? Actually we have 2 ceph cluster (geo-dislocated)
instances and we wanna exploit the radosgw to make replicas across our
two clusters.

It's possible to read/write on both replicas (one placed on primary
region and one on the secondary one) done through radosgw? I'm
wondering because on the doc it's suggested to write just on a master
zone, avoiding to write on secondary zones. It's the same for
primary/secondary regions?


Cheers,
Fabrizio
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to find the disk partitions attached to a OSD

2014-05-21 Thread Jimmy Lu

This would give you pretty good understanding where the mounts and
/dev/sd* are.

[jlu@gfsnode1 osd]$ ceph-disk list; pwd; ls -lai
/dev/sda :
 /dev/sda1 other, mounted on /boot
 /dev/sda2 other
/dev/sdb other, unknown, mounted on /ceph/osd120
/dev/sdc other, unknown, mounted on /ceph/osd121
/dev/sdd other, unknown, mounted on /ceph/osd122
/dev/sde other, unknown, mounted on /ceph/osd123
/dev/sdf other, unknown, mounted on /ceph/osd124
/dev/sdg other, unknown, mounted on /ceph/osd125
/dev/sdh other, unknown, mounted on /ceph/osd126
/dev/sdi other, unknown, mounted on /ceph/osd127
/dev/sdj other, unknown, mounted on /ceph/osd128
/dev/sdk other, unknown, mounted on /ceph/osd129
/dev/sdl other, unknown, mounted on /ceph/osd130
/var/lib/ceph/osd
total 8
 497 drwxr-xr-x 2 root root 4096 May 15 15:28 .
 492 drwxr-xr-x 8 root root 4096 May 15 15:13 ..
 725 lrwxrwxrwx 1 root root   12 May 15 15:21 ceph-33 -> /ceph/osd120
 782 lrwxrwxrwx 1 root root   12 May 15 15:22 ceph-34 -> /ceph/osd121
1081 lrwxrwxrwx 1 root root   12 May 15 15:23 ceph-35 -> /ceph/osd122
1095 lrwxrwxrwx 1 root root   12 May 15 15:24 ceph-36 -> /ceph/osd123
1129 lrwxrwxrwx 1 root root   12 May 15 15:24 ceph-37 -> /ceph/osd124
1137 lrwxrwxrwx 1 root root   12 May 15 15:25 ceph-38 -> /ceph/osd125
1143 lrwxrwxrwx 1 root root   12 May 15 15:25 ceph-39 -> /ceph/osd126
1185 lrwxrwxrwx 1 root root   12 May 15 15:26 ceph-40 -> /ceph/osd127
1242 lrwxrwxrwx 1 root root   12 May 15 15:27 ceph-41 -> /ceph/osd128
1398 lrwxrwxrwx 1 root root   12 May 15 15:27 ceph-42 -> /ceph/osd129
1437 lrwxrwxrwx 1 root root   12 May 15 15:28 ceph-43 -> /ceph/osd130
[jlu@gfsnode1 osd]$

-Jimmy


On 5/21/14, 8:33 AM, "Sage Weil"  wrote:

>You might also try
>
> ceph-disk list
>
>sage
>
>
>On Wed, 21 May 2014, Mike Dawson wrote:
>
>> Looks like you may not have any OSDs properly setup and mounted. It
>>should
>> look more like:
>> 
>> user@host:~# mount | grep ceph
>> /dev/sdb1 on /var/lib/ceph/osd/ceph-0 type xfs (rw,noatime,inode64)
>> /dev/sdc1 on /var/lib/ceph/osd/ceph-1 type xfs (rw,noatime,inode64)
>> /dev/sdd1 on /var/lib/ceph/osd/ceph-2 type xfs (rw,noatime,inode64)
>> 
>> Confirm the OSD in your ceph cluster with:
>> 
>> user@host:~# ceph osd tree
>> 
>> - Mike
>> 
>> 
>> On 5/21/2014 11:15 AM, Sharmila Govind wrote:
>> > Hi Mike,
>> > Thanks for your quick response. When I try mount on the storage node
>> > this is what I get:
>> > 
>> > *root@cephnode4:~# mount*
>> > */dev/sda1 on / type ext4 (rw,errors=remount-ro)*
>> > *proc on /proc type proc (rw,noexec,nosuid,nodev)*
>> > *sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)*
>> > *none on /sys/fs/fuse/connections type fusectl (rw)*
>> > *none on /sys/kernel/debug type debugfs (rw)*
>> > *none on /sys/kernel/security type securityfs (rw)*
>> > *udev on /dev type devtmpfs (rw,mode=0755)*
>> > *devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)*
>> > *tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)*
>> > *none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)*
>> > *none on /run/shm type tmpfs (rw,nosuid,nodev)*
>> > */dev/sdb on /mnt/CephStorage1 type ext4 (rw)*
>> > */dev/sdc on /mnt/CephStorage2 type ext4 (rw)*
>> > */dev/sda7 on /mnt/Storage type ext4 (rw)*
>> > */dev/sda2 on /boot type ext4 (rw)*
>> > */dev/sda5 on /home type ext4 (rw)*
>> > */dev/sda6 on /mnt/CephStorage type ext4 (rw)*
>> > 
>> > 
>> > 
>> > Is there anything wrong in the setup I have? I dont have any 'ceph'
>> > related mounts.
>> > 
>> > Thanks,
>> > Sharmila
>> > 
>> > 
>> > 
>> > On Wed, May 21, 2014 at 8:34 PM, Mike Dawson > > > wrote:
>> > 
>> > Perhaps:
>> > 
>> > # mount | grep ceph
>> > 
>> > - Mike Dawson
>> > 
>> > 
>> > 
>> > On 5/21/2014 11:00 AM, Sharmila Govind wrote:
>> > 
>> > Hi,
>> >I am new to Ceph. I have a storage node with 2 OSDs. Iam
>> > trying to
>> > figure out to which pyhsical device/partition each of the
>>OSDs are
>> > attached to. Is there are command that can be executed in the
>> > storage
>> > node to find out the same.
>> > 
>> > Thanks in Advance,
>> > Sharmila
>> > 
>> > 
>> > _
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com 
>> > http://lists.ceph.com/__listinfo.cgi/ceph-users-ceph.__com
>> > 
>> > 
>> > 
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> 
>> 
>___
>ceph-users mailing list
>ceph-users@lists.ceph.com
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listi

[ceph-users] CephFS MDS Setup

2014-05-21 Thread Scottix

I am setting a CephFS cluster and wondering about MDS setup.
I know you are still hesitant to put the stable label on it but I have
a few questions what would be an adequate setup.

I know active active is not developed yet so that is pretty much out
of the question right now.

What about active standby? How reliable is the standby? or should a
single active mds be sufficient?

Thanks

-- 
Follow Me: @Scottix
http://about.me/scottix
scot...@gmail.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS MDS Setup

2014-05-21 Thread Wido den Hollander


On 05/21/2014 09:04 PM, Scottix wrote:

I am setting a CephFS cluster and wondering about MDS setup.
I know you are still hesitant to put the stable label on it but I have
a few questions what would be an adequate setup.

I know active active is not developed yet so that is pretty much out
of the question right now.

What about active standby? How reliable is the standby? or should a
single active mds be sufficient?



Active/Standby is fairly stable, but I wouldn't recommend putting it 
into production right now.


The general advice is always to run a recent Ceph version and a recent 
kernel as well. Like 3.13 in Ubuntu 14.04


But the best advice: Test your use-case extensively! The more feedback, 
the better.



Thanks




--
Wido den Hollander
42on B.V.
Ceph trainer and consultant

Phone: +31 (0)20 700 9902
Skype: contact42on
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] v0.67.9 Dumpling released

2014-05-21 Thread Sage Weil

This Dumpling point release fixes several minor bugs. The most prevalent 
in the field is one that occasionally prevents OSDs from starting on 
recently created clusters.

We recommand that all v0.67.x Dumpling users upgrade at their convenience.

Notable Changes
---

* ceph-fuse, libcephfs: client admin socket command to kick and inspect 
  MDS sessions (#8021, Zheng Yan)
* monclient: fix failure detection during mon handshake (#8278, Sage Weil)
* mon: set tid on no-op PGStatsAck messages (#8280, Sage Weil)
* msgr: fix a rare bug with connection negotiation between OSDs (Guang 
  Yang)
* osd: allow snap trim throttling with simple delay (#6278, Sage Weil)
* osd: check for splitting when processing recover/backfill reservations 
  (#6565, Samuel Just)
* osd: fix backfill position tracking (#8162, Samuel Just)
* osd: fix bug in backfill stats (Samuel Just)
* osd: fix bug preventing OSD startup for infant clusters (#8162, Greg 
  Farnum)
* osd: fix rare PG resurrection race causing an incomplete PG (#7740, 
  Samuel Just)
* osd: only complete replicas count toward min_size (#7805, Samuel Just)
* rgw: allow setting ACLs with empty owner (#6892, Yehuda Sadeh)
* rgw: send user manifest header field (#8170, Yehuda Sadeh)

For more detailed information, see the complete changelog at

  http://ceph.com/docs/master/_downloads/v0.67.9.txt

Getting Ceph


* Git at git://github.com/ceph/ceph.git
* Tarball at http://ceph.com/download/ceph-0.67.9.tar.gz
* For packages, see http://ceph.com/docs/master/install/get-packages
* For ceph-deploy, see http://ceph.com/docs/master/install/install-ceph-deploy

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] RBD cache pool - not cleaning up

2014-05-21 Thread Michael


Hi All,

Experimenting with cache pools for RBD, created two pools, slowdata-hot 
backed by slowdata-cold. Set up max data to be stored in hot to be 
100GB, data to be moved to cold above 40% hot usage. Created a 100GB RBD 
image, mounted it tested reading/writing, then dumped in 80GB of data. 
All looked to be going well. Left everything to settle down and then 
unmounted the RBD and deleted it. Afterwards:


pool name   category KB  objects clones 
degraded  unfound   rdrd KB wrwr KB
slowdata-cold   -   5583598013817 0
0   0277413 16818 67715123
slowdata-hot- 1223610 0
0   0  1863785112250386   150951 138481712


Deleting the RBD image cleaned out the hot pool's data but left the cold 
pool full of data and both of them full of objects.

Anyone else trying this out?

-Michael
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RBD cache pool - not cleaning up

2014-05-21 Thread Sage Weil

On Wed, 21 May 2014, Michael wrote:
> Hi All,
> 
> Experimenting with cache pools for RBD, created two pools, slowdata-hot backed
> by slowdata-cold. Set up max data to be stored in hot to be 100GB, data to be
> moved to cold above 40% hot usage. Created a 100GB RBD image, mounted it
> tested reading/writing, then dumped in 80GB of data. All looked to be going
> well. Left everything to settle down and then unmounted the RBD and deleted
> it. Afterwards:
> 
> pool name   category KB  objects clones degraded
> unfound   rdrd KB wrwr KB
> slowdata-cold   -   5583598013817 00
> 0277413 16818 67715123
> slowdata-hot- 1223610 00
> 0  1863785112250386   150951 138481712
> 
> Deleting the RBD image cleaned out the hot pool's data but left the cold 
> pool full of data and both of them full of objects. Anyone else trying 
> this out?

This is normal.  The cache pool contains a sort of "whiteout" that 
indicates the object is deleted, but it isn't cold enough yet to have 
flushed that change to the base pool.  As you write more data you will 
find the backend pool's objects for that image will eventually disappear.

sage
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RBD cache pool - not cleaning up

2014-05-21 Thread Michael

Thanks Sage, the cache system's look pretty great so far. Combined with 
erasure coding it's really adding a lot of options.


-Michael

On 21/05/2014 21:54, Sage Weil wrote:

On Wed, 21 May 2014, Michael wrote:

Hi All,

Experimenting with cache pools for RBD, created two pools, slowdata-hot backed
by slowdata-cold. Set up max data to be stored in hot to be 100GB, data to be
moved to cold above 40% hot usage. Created a 100GB RBD image, mounted it
tested reading/writing, then dumped in 80GB of data. All looked to be going
well. Left everything to settle down and then unmounted the RBD and deleted
it. Afterwards:

pool name   category KB  objects clones degraded
unfound   rdrd KB wrwr KB
slowdata-cold   -   5583598013817 00
0277413 16818 67715123
slowdata-hot- 1223610 00
0  1863785112250386   150951 138481712

Deleting the RBD image cleaned out the hot pool's data but left the cold
pool full of data and both of them full of objects. Anyone else trying
this out?

This is normal.  The cache pool contains a sort of "whiteout" that
indicates the object is deleted, but it isn't cold enough yet to have
flushed that change to the base pool.  As you write more data you will
find the backend pool's objects for that image will eventually disappear.

sage


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Data still in OSD directories after removing

2014-05-21 Thread Olivier Bonvalet

Le mercredi 21 mai 2014 à 08:20 -0700, Sage Weil a écrit :
> 
> You should definitely not do this!  :)

Of course ;)

> 
> You're certain that that is the correct prefix for the rbd image you 
> removed?  Do you see the objects lists when you do 'rados -p rbd ls - | 
> grep '?

I'm pretty sure yes : since I didn't see a lot of space freed by the
"rbd snap purge" command, I looked at the RBD prefix before to do the
"rbd rm" (it's not the first time I see that problem, but previous time
without the RBD prefix I was not able to check).

So : 
- "rados -p sas3copies ls - | grep rb.0.14bfb5a.238e1f29" return nothing
at all
- # rados stat -p sas3copies rb.0.14bfb5a.238e1f29.0002f026
 error stat-ing sas3copies/rb.0.14bfb5a.238e1f29.0002f026: No such
file or directory
- # rados stat -p sas3copies rb.0.14bfb5a.238e1f29.
 error stat-ing sas3copies/rb.0.14bfb5a.238e1f29.: No such
file or directory
- # ls -al 
/var/lib/ceph/osd/ceph-67/current/9.1fe_head/DIR_E/DIR_F/DIR_1/DIR_7/rb.0.14bfb5a.238e1f29.0002f026__a252_E68871FE__9
-rw-r--r-- 1 root root 4194304 oct.   8  2013 
/var/lib/ceph/osd/ceph-67/current/9.1fe_head/DIR_E/DIR_F/DIR_1/DIR_7/rb.0.14bfb5a.238e1f29.0002f026__a252_E68871FE__9


> If the objects really are orphaned, teh way to clean them up is via 'rados 
> -p rbd rm '.  I'd like to get to the bottom of how they ended 
> up that way first, though!

I suppose the problem came from me, by doing CTRL+C while "rbd snap
purge $IMG".
"rados rm -p sas3copies rb.0.14bfb5a.238e1f29.0002f026" don't remove
thoses files, and just answer with a "No such file or directory".

Thanks,
Olivier



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Quota Management in CEPH

2014-05-21 Thread Vilobh Meshram

Hi All,

I want to understand on how do CEPH users go about Quota Management when CEPH 
is used with Openstack.

  1.  Is it recommended to use a common pool say “volumes” for creating volumes 
which is shared by all tenants ? In this case a common keyring 
ceph.common.keyring will be shared across all the tenants/common volume pool.
  2.  Or is it recommended to use a pool for each tenant say “volume1 pool for 
tenant1” , “volume2 pool for tenant2" ?  In this case we will have a keyring 
per volume pool/ tenant I.e. Keyring 1 for volume/tenant1 and so on.

Considering both of these cases how do we guarantee that we enforce a quota for 
each user inside a tenant say a quota of say 5 volumes to be created by each 
user.

Thanks,
Vilobh


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Expanding pg's of an erasure coded pool

2014-05-21 Thread Gregory Farnum

On Wed, May 21, 2014 at 3:52 AM, Kenneth Waegeman
 wrote:
> Thanks! I increased the max processes parameter for all daemons quite a lot
> (until ulimit -u 3802720)
>
> These are the limits for the daemons now..
> [root@ ~]# cat /proc/17006/limits
> Limit Soft Limit   Hard Limit   Units
> Max cpu time  unlimitedunlimitedseconds
> Max file size unlimitedunlimitedbytes
> Max data size unlimitedunlimitedbytes
> Max stack size10485760 unlimitedbytes
> Max core file sizeunlimitedunlimitedbytes
> Max resident set  unlimitedunlimitedbytes
> Max processes 3802720  3802720
> processes
> Max open files3276832768files
> Max locked memory 6553665536bytes
> Max address space unlimitedunlimitedbytes
> Max file locksunlimitedunlimitedlocks
> Max pending signals   9506895068signals
> Max msgqueue size 819200   819200   bytes
> Max nice priority 00
> Max realtime priority 00
> Max realtime timeout  unlimitedunlimitedus
>
> But this didn't help. Are there other parameters I should change?

Hrm, is it exactly the same stack trace? You might need to bump the
open files limit as well, although I'd be surprised. :/
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Inter-region data replication through radosgw

2014-05-21 Thread Craig Lewis


On 5/21/14 09:02 , Fabrizio G. Ventola wrote:

Hi everybody,

I'm reading the doc regarding the replication through radosgw. It
talks just about inter-region METAdata replication, nothing about data
replication.

My question is, it's possible to have (everything) geo-replicated
through radosgw? Actually we have 2 ceph cluster (geo-dislocated)
instances and we wanna exploit the radosgw to make replicas across our
two clusters.

It's possible to read/write on both replicas (one placed on primary
region and one on the secondary one) done through radosgw? I'm
wondering because on the doc it's suggested to write just on a master
zone, avoiding to write on secondary zones. It's the same for
primary/secondary regions?


Cheers,
Fabrizio
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


The federated setup will replicate both data and metadata.  You can do 
just metadata if you want, but it's not the default.


You can have all of the RadosGW data geo-replicated.  Raw Rados isn't 
possible yet, and rbd under development.


You can read from both the master and slave, but you don't want to write 
to a slave.  The master and slave have different URLs, so it's up to you 
to use the appropriate URL.


You can run multiple zones in each cluster, as long as each zone has 
it's own URL.  If you do this, you might want to share 
apache/radosgw/osd across all the zone, or dedicated them to specific 
zones.  It's entirely possible multiple zones in one cluster share 
everything, or just the monitors.


If you really want both clusters to handle writes, this is how you'd do it:
ClusterWest1 contains us-west-1 (master), and us-west-2 (slave for 
us-east-2).
ClusterEast1 constains us-east-1 (slave for us-west-1), and us-east-2 
(master).
If users and buckets need to be globally unique across all zones, setup 
metadata (not data) replication between the two zones.

Write to us-west-1 or us-east-2, up to you.


This replication setup to make more sense when you have more than 3+ 
data centers, and you set them up in a ring.



Does that help?

--

*Craig Lewis*
Senior Systems Engineer
Office +1.714.602.1309
Email cle...@centraldesktop.com 

*Central Desktop. Work together in ways you never thought possible.*
Connect with us Website   | Twitter 
  | Facebook 
  | LinkedIn 
  | Blog 



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Data still in OSD directories after removing

2014-05-21 Thread Josh Durgin


On 05/21/2014 03:03 PM, Olivier Bonvalet wrote:

Le mercredi 21 mai 2014 à 08:20 -0700, Sage Weil a écrit :

You're certain that that is the correct prefix for the rbd image you
removed?  Do you see the objects lists when you do 'rados -p rbd ls - |
grep '?


I'm pretty sure yes : since I didn't see a lot of space freed by the
"rbd snap purge" command, I looked at the RBD prefix before to do the
"rbd rm" (it's not the first time I see that problem, but previous time
without the RBD prefix I was not able to check).

So :
- "rados -p sas3copies ls - | grep rb.0.14bfb5a.238e1f29" return nothing
at all
- # rados stat -p sas3copies rb.0.14bfb5a.238e1f29.0002f026
  error stat-ing sas3copies/rb.0.14bfb5a.238e1f29.0002f026: No such
file or directory
- # rados stat -p sas3copies rb.0.14bfb5a.238e1f29.
  error stat-ing sas3copies/rb.0.14bfb5a.238e1f29.: No such
file or directory
- # ls -al 
/var/lib/ceph/osd/ceph-67/current/9.1fe_head/DIR_E/DIR_F/DIR_1/DIR_7/rb.0.14bfb5a.238e1f29.0002f026__a252_E68871FE__9
-rw-r--r-- 1 root root 4194304 oct.   8  2013 
/var/lib/ceph/osd/ceph-67/current/9.1fe_head/DIR_E/DIR_F/DIR_1/DIR_7/rb.0.14bfb5a.238e1f29.0002f026__a252_E68871FE__9



If the objects really are orphaned, teh way to clean them up is via 'rados
-p rbd rm '.  I'd like to get to the bottom of how they ended
up that way first, though!


I suppose the problem came from me, by doing CTRL+C while "rbd snap
purge $IMG".
"rados rm -p sas3copies rb.0.14bfb5a.238e1f29.0002f026" don't remove
thoses files, and just answer with a "No such file or directory".


Those files are all for snapshots, which are removed by the osds
asynchronously in a process called 'snap trimming'. There's no
way to directly remove them via rados.

Since you stopped 'rbd snap purge' partway through, it may
have removed the reference to the snapshot before removing
the snapshot itself.

You can get a list of snapshot ids for the remaining objects
via the 'rados listsnaps' command, and use
rados_ioctx_selfmanaged_snap_remove() (no convenient wrapper
unfortunately) on each of those snapshot ids to be sure they are all
scheduled for asynchronous deletion.

Josh
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 70+ OSD are DOWN and not coming up

2014-05-21 Thread Craig Lewis


On 5/20/14 08:18 , Sage Weil wrote:

On Tue, 20 May 2014, Karan Singh wrote:

Hello Cephers , need your suggestion for troubleshooting.

My cluster is terribly struggling , 70+ osd are down out of 165

Problem ?>OSD are getting marked out of cluster and are down. The cluster is
degraded. On checking logs of failed OSD we are getting wired entries that
are continuously getting generated.


Also happy to debug this interactively over IRC; that will likely be
faster!

Thanks-
sage




If you do this over IRC, can you please post a summary to the mailling 
list?


I believe I'm having this issue as well.


--

*Craig Lewis*
Senior Systems Engineer
Office +1.714.602.1309
Email cle...@centraldesktop.com 

*Central Desktop. Work together in ways you never thought possible.*
Connect with us Website   | Twitter 
  | Facebook 
  | LinkedIn 
  | Blog 



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Quota Management in CEPH

2014-05-21 Thread Josh Durgin

On 05/21/2014 03:29 PM, Vilobh Meshram wrote:

Hi All,

I want to understand on how do CEPH users go about Quota Management when
CEPH is used with Openstack.

1. Is it recommended to use a common pool say “volumes” for creating
volumes which is shared by all tenants ? In this case a common
keyring ceph.common.keyring will be shared across all the
tenants/common volume pool.

Yes, using a common pool is recommended. More pools take up more cpu and
memory on the osds, since placement groups (shards of pools) are the
unit of recovery. Having a pool per tenant would be a scaling issue.

There is a further level of division in rados called a 'namespace',
which can provide finer-grained cephx security within a pool, but
rbd does not support it yet, and as it stands it would not be useful
for quotas [1].

2. Or is it recommended to use a pool for each tenant say “volume1 pool
for tenant1” , “volume2 pool for tenant2" ? In this case we will
have a keyring per volume pool/ tenant I.e. Keyring 1 for
volume/tenant1 and so on.

Considering both of these cases how do we guarantee that we enforce a
quota for each user inside a tenant say a quota of say 5 volumes to be
created by each user.

When using OpenStack, Cinder does the quota management for volumes based
on its database, and can limit total space, number of volumes and
number of snapshots [2]. RBD is entirely unaware of OpenStack tenants.

Josh

[1] http://wiki.ceph.com/Planning/Sideboard/rbd%3A_namespace_support
[2]
http://docs.openstack.org/user-guide-admin/content/cli_set_quotas.html#cli_set_block_storage_quotas

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rbd watchers

2014-05-21 Thread Mandell Degerness

The times I have seen this message, it has always been because there
are snapshots of the image that haven't been deleted yet. You can see
the snapshots with "rbd snap list ".

On Tue, May 20, 2014 at 4:26 AM, James Eckersall
 wrote:
> Hi,
>
>
>
> I'm having some trouble with an rbd image.  I want to rename the current rbd
> and create a new rbd with the same name.
>
> I renamed the rbd with rbd mv, but it was still mapped on another node, so
> rbd mv gave me an error that it was unable to remove the source.
>
>
> I then unmapped the original rbd and tried to remove it.
>
>
> Despite it being unmapped, the cluster still believes that there is a
> watcher on the rbd:
>
>
> root@ceph-admin:~# rados -p poolname listwatchers rbdname.rbd
>
> watcher=x.x.x.x:0/2329830975 client.26367 cookie=48
>
> root@ceph-admin:~# rbd rm -p poolname rbdname
>
> Removing image: 99% complete...failed.2014-05-20 11:50:15.023823
> 7fa6372e4780 -1 librbd: error removing header: (16) Device or resource busy
>
>
> rbd: error: image still has watchers
>
> This means the image is still open or the client using it crashed. Try again
> after closing/unmapping it or waiting 30s for the crashed client to timeout.
>
>
>
> I've already rebooted the node that the cluster claims is a watcher and
> confirmed it definitely is not mapped.
>
> I'm 99.9% sure that there are no nodes actually using this rbd.
>
>
> Does anyone know how I can get rid of it?
>
>
> Currently running ceph 0.73-1 on Ubuntu 12.04.
>
>
> Thanks
>
>
> J
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Questions about zone and disater recovery

2014-05-21 Thread wsnote

Hi，everyone！
I have 2 ceph clusters, one master zone, another secondary zone.
Now I have some question.
1. Can ceph have two or more secondary zones?


2. Can the role of master zone and secondary zone transform mutual?
I mean I can change the secondary zone to be master and the master zone to 
secondary.


3. How to deal with the situation when the master zone is down?
Now the secondary zone forbids all the operations of files, such as create 
objects, delete objects.
When the master zone is down, users can't do anything to the files except read 
objects from the secondary zone.
It's a bad user experience. Additionly, it will have a bad influence on the 
confidence of the users.
I know the limit of secondary zone is out of consideration for the consistency 
of data. However, is there another way to improve some experience?
I think:
There can be a config that allow the files operations of the secondary zone.If 
the master zone is down, the admin can enable it, then the users can do files 
opeartions as usually. The secondary record all the files operations of the 
files. When the master zone gets right, the admin can sync files to the master 
zone manually.


Thanks!







___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 70+ OSD are DOWN and not coming up

2014-05-21 Thread Sage Weil

On Wed, 21 May 2014, Craig Lewis wrote:
> If you do this over IRC, can you please post a summary to the mailling
> list? 
> 
> I believe I'm having this issue as well.

In the other case, we found that some of the OSDs were behind processing 
maps (by several thousand epochs).  The trick here to give them a chance 
to catch up is

 ceph osd set noup
 ceph osd set nodown
 ceph osd set noout

and wait for them to stop spinning on the CPU.  You can check which map 
each OSD is on with

 ceph daemon osd.NNN status

to see which epoch they are on and compare that to

 ceph osd stat

Once they are within 100 or less epochs,

 ceph osd unset noup

and let them all start up.

We haven't determined whether the original problem was caused by this or 
the other way around; we'll see once they are all caught up.

sage___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Inter-region data replication through radosgw

2014-05-21 Thread wsnote

Hi, Lewis!
With your way, there will be a contradition because of the limit of secondary 
zone.
In secondary zone, one can't do any files operations.
Let me give some example.I define the symbols first.


The instances of cluster 1:
M1: master zone of cluster 1
S2: Slave zone for M2 of cluster2, the files of cluster2 will be synced from M2 
to S2
I13: the third instance of cluster 1(M1 and S2 are both the instances too.)


The instances of cluster 2:
M2: master zone of cluster 2
S1: Slave zone for M1 of cluster 1, the files of cluster1 will be synced from 
M1 to S1
I23: the third instance of cluster 1(M2 and S1 are both the instances too.)


cluster 1:  M1  S2  I13
cluster 2:  M2  S1  I23


Questions:
1. If I upload objects form I13 of cluster 1, is it synced to cluster 2 from M1?
2. In cluster 1, can I do some operations for the files synced from cluster2 
through M1 or I13?
3. If I upload an object in cluster 1, the matadata will be synced to cluster 2 
before file data.When  matadata of it has been synced but filedata not, cluster 
1 is down, that is to say the object hasnot been synced yet.Then I upload the 
same object in cluster 2. Can it succeed?
I think it will fail. cluster 2 has the matadata of object and will consider 
the object is in cluster 2, and this object is synced from cluster 1, so I have 
no permission to operate it.
Do I right? 


Because of the limit of files operations in slave zone, I think there will be 
some contradition.


Looking forward to your reply.
Thanks!







At 2014-05-22 07:12:17,"Craig Lewis"  wrote:

On 5/21/14 09:02 , Fabrizio G. Ventola wrote:

Hi everybody,

I'm reading the doc regarding the replication through radosgw. It
talks just about inter-region METAdata replication, nothing about data
replication.

My question is, it's possible to have (everything) geo-replicated
through radosgw? Actually we have 2 ceph cluster (geo-dislocated)
instances and we wanna exploit the radosgw to make replicas across our
two clusters.

It's possible to read/write on both replicas (one placed on primary
region and one on the secondary one) done through radosgw? I'm
wondering because on the doc it's suggested to write just on a master
zone, avoiding to write on secondary zones. It's the same for
primary/secondary regions?


Cheers,
Fabrizio
___
ceph-users mailing list
ceph-us...@lists.ceph.comhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

The federated setup will replicate both data and metadata.  You can do just 
metadata if you want, but it's not the default.

You can have all of the RadosGW data geo-replicated.  Raw Rados isn't possible 
yet, and rbd under development.

You can read from both the master and slave, but you don't want to write to a 
slave.  The master and slave have different URLs, so it's up to you to use the 
appropriate URL.

You can run multiple zones in each cluster, as long as each zone has it's own 
URL.  If you do this, you might want to share apache/radosgw/osd across all the 
zone, or dedicated them to specific zones.  It's entirely possible multiple 
zones in one cluster share everything, or just the monitors.

If you really want both clusters to handle writes, this is how you'd do it:
ClusterWest1 contains us-west-1 (master), and us-west-2 (slave for us-east-2).
ClusterEast1 constains us-east-1 (slave for us-west-1), and us-east-2 (master).
If users and buckets need to be globally unique across all zones, setup 
metadata (not data) replication between the two zones.
Write to us-west-1 or us-east-2, up to you.


This replication setup to make more sense when you have more than 3+ data 
centers, and you set them up in a ring.


Does that help?


--


Craig Lewis
Senior Systems Engineer
Office +1.714.602.1309
Email cle...@centraldesktop.com

Central Desktop. Work together in ways you never thought possible.
Connect with us   Website  |  Twitter  |  Facebook  |  LinkedIn  |  Blog
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Access denied error for list users

Re: [ceph-users] Data still in OSD directories after removing

Re: [ceph-users] Expanding pg's of an erasure coded pool

Re: [ceph-users] Access denied error for list users

Re: [ceph-users] 70+ OSD are DOWN and not coming up

[ceph-users] Ceph Firefly on Centos 6.5 cannot deploy osd

Re: [ceph-users] Ceph Firefly on Centos 6.5 cannot deploy osd

[ceph-users] How to find the disk partitions attached to a OSD

Re: [ceph-users] How to find the disk partitions attached to a OSD

Re: [ceph-users] How to find the disk partitions attached to a OSD

Re: [ceph-users] Data still in OSD directories after removing

Re: [ceph-users] How to find the disk partitions attached to a OSD

Re: [ceph-users] Problem with radosgw and some file name characters

Re: [ceph-users] How to find the disk partitions attached to a OSD

[ceph-users] Inter-region data replication through radosgw

Re: [ceph-users] How to find the disk partitions attached to a OSD

[ceph-users] CephFS MDS Setup

Re: [ceph-users] CephFS MDS Setup

[ceph-users] v0.67.9 Dumpling released

[ceph-users] RBD cache pool - not cleaning up

Re: [ceph-users] RBD cache pool - not cleaning up

Re: [ceph-users] RBD cache pool - not cleaning up

Re: [ceph-users] Data still in OSD directories after removing

[ceph-users] Quota Management in CEPH

Re: [ceph-users] Expanding pg's of an erasure coded pool

Re: [ceph-users] Inter-region data replication through radosgw

Re: [ceph-users] Data still in OSD directories after removing

Re: [ceph-users] 70+ OSD are DOWN and not coming up

Re: [ceph-users] Quota Management in CEPH

Re: [ceph-users] rbd watchers

[ceph-users] Questions about zone and disater recovery

Re: [ceph-users] 70+ OSD are DOWN and not coming up

Re: [ceph-users] Inter-region data replication through radosgw

33 matches

Site Navigation

Mail list logo

Footer information