[ceph-users] SSD journal overload?

[Indra Pramana]

Hi,

On one of our test clusters, I have a node with 4 OSDs with SAS / non-SSD
drives (sdb, sdc, sdd, sde) and 2 SSD drives (sdf and sdg) for journals to
serve the 4 OSDs (2 each).

Model: ATA ST100FM0012 (scsi)
Disk /dev/sdf: 100GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start   End SizeFile system  Name  Flags
 1  1049kB  10.7GB  10.7GB   ceph journal
 2  10.7GB  21.5GB  10.7GB   ceph journal


Model: ATA ST100FM0012 (scsi)
Disk /dev/sdg: 100GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start   End SizeFile system  Name  Flags
 1  1049kB  10.7GB  10.7GB   ceph journal
 2  10.7GB  21.5GB  10.7GB   ceph journal

When I did a rados bench test, I noted that the two SSD drives are always
overloaded with I/O requests, thus the performance is very bad. A ceph tell
osd.X bench test will only give around 25 MB/s of throughput and iostat
shows the two SSD journal drives are overloaded:


avg-cpu:  %user   %nice %system %iowait  %steal   %idle
   2.410.001.653.000.00   92.93

Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sda   0.00 0.000.000.00 0.00 0.00
0.00 0.000.000.000.00   0.00   0.00
sdc   0.00 5.673.33  166.0022.67 14923.50
176.5325.16  148.589.60  151.37   2.39  40.40
sdb   0.00 0.002.000.3313.33 2.17
13.29 0.016.294.67   16.00   5.14   1.20
sdd   0.00 0.002.00   20.3310.67  1526.33
137.64 0.125.49   10.005.05   4.72  10.53
sde   0.00 5.004.67   67.6734.67  5837.00
162.35 8.92  124.06   14.00  131.65   2.49  18.00
sdg   0.00 0.000.00   54.67 0.00 25805.33
944.1036.41  655.880.00  655.88  18.29 *100.00*
sdf   0.00 0.000.00   53.67 0.00 25252.00
941.0735.61  636.070.00  636.07  18.63 *100.00*

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
   2.010.001.282.150.00   94.56

Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sda   0.00 0.000.000.00 0.00 0.00
0.00 0.000.000.000.00   0.00   0.00
sdc   0.00 0.002.67   14.3313.33 4.17
2.06 0.032.128.500.93   1.33   2.27
sdb   0.00 0.002.00   12.3310.67  4130.00
577.77 0.096.334.006.70   4.09   5.87
sdd   0.00 0.002.33   36.6718.67 12425.17
638.15 3.77   96.589.14  102.15   3.93  15.33
sde   0.00 0.331.67  104.33 9.33 11484.00
216.8611.96  161.61   33.60  163.65   2.93  31.07
sdg   0.00 0.000.00   54.33 0.00 25278.67
930.5033.55  644.540.00  644.54  18.38  *99.87*
sdf   0.00 0.000.00   58.33 0.00 25493.33
874.0622.67  422.260.00  422.26  17.10  *99.73*

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
   2.190.001.603.950.00   92.26

Device: rrqm/s   wrqm/s r/s w/srkB/swkB/s avgrq-sz
avgqu-sz   await r_await w_await  svctm  %util
sda   0.00 0.000.000.00 0.00 0.00
0.00 0.000.000.000.00   0.00   0.00
sdc   0.00 0.002.679.0017.33  1431.00
248.29 0.076.178.005.63   5.03   5.87
sdb   0.00 0.331.67   88.33 8.00 30435.33
676.5217.16  139.64   85.60  140.66   3.30  29.73
sdd   0.00 0.002.33   17.3313.33  3040.17
310.53 0.115.427.435.15   4.47   8.80
sde   0.00 0.002.677.6714.67  2767.00
538.39 0.088.008.507.83   5.16   5.33
sdg   0.00 0.000.00   60.00 0.00 24841.33
828.0421.26  332.270.00  332.27  16.51  99.07
sdf   0.00 0.000.00   56.33 0.00 24365.33
865.0425.00  449.210.00  449.21  17.70  99.73


Anyone can advise what could be the problem? I am using 100 GB SSD drives
and journal size is only 10 GB, meaning both only occupied 20 GB of the
space for each disk. The server is using SATA 3 connectors.

I am not able to do a dd test on the SSDs since it's not mounted as
filesystem, but dd on the OSD (non-SSD) drives gives normal result.

I am using Ceph v0.67.7, latest stable version of Dumpling.

Any advice is appreciated.

Looking forward to your reply, thank you.

Cheers.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] FW: Ceph mds laggy and failed assert in function replay mds/journal.cc

[Mohd Bazli Ab Karim]

Dear ceph-users,

I am currently facing issue with my ceph mds server. Ceph-mds daemon does not 
want to bring up back.
I tried running it manually with ceph-mds -i mon01 -d but aborted and the log 
shows that it stucks at failed assert(session) line 1303 in mds/journal.cc.

Can someone shed some light in this issue?
I'm running ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60)

Let me know if I need to send log with debug enabled.

Regards,
Bazli


DISCLAIMER:


This e-mail (including any attachments) is for the addressee(s) only and may be 
confidential, especially as regards personal data. If you are not the intended 
recipient, please note that any dealing, review, distribution, printing, 
copying or use of this e-mail is strictly prohibited. If you have received this 
email in error, please notify the sender immediately and delete the original 
message (including any attachments).


MIMOS Berhad is a research and development institution under the purview of the 
Malaysian Ministry of Science, Technology and Innovation. Opinions, conclusions 
and other information in this e-mail that do not relate to the official 
business of MIMOS Berhad and/or its subsidiaries shall be understood as neither 
given nor endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS 
Berhad nor its subsidiaries accepts responsibility for the same. All liability 
arising from or in connection with computer viruses and/or corrupted e-mails is 
excluded to the fullest extent permitted by law.


--
-
-
DISCLAIMER:

This e-mail (including any attachments) is for the addressee(s)
only and may contain confidential information. If you are not the
intended recipient, please note that any dealing, review,
distribution, printing, copying or use of this e-mail is strictly
prohibited. If you have received this email in error, please notify
the sender  immediately and delete the original message.
MIMOS Berhad is a research and development institution under
the purview of the Malaysian Ministry of Science, Technology and
Innovation. Opinions, conclusions and other information in this e-
mail that do not relate to the official business of MIMOS Berhad
and/or its subsidiaries shall be understood as neither given nor
endorsed by MIMOS Berhad and/or its subsidiaries and neither
MIMOS Berhad nor its subsidiaries accepts responsibility for the
same. All liability arising from or in connection with computer
viruses and/or corrupted e-mails is excluded to the fullest extent
permitted by law.

2014-04-25 12:17:27.210367 7f3c30250780  0 ceph version 0.72.2 
(a913ded2ff138aefb8cb84d347d72164099cfd60), process ceph-mds, pid 5492
starting mds.mon01 at :/0
2014-04-25 12:17:27.441530 7f3c2b2d6700  1 mds.-1.0 handle_mds_map standby
2014-04-25 12:17:27.624820 7f3c2b2d6700  1 mds.0.12834 handle_mds_map i am now 
mds.0.12834
2014-04-25 12:17:27.624825 7f3c2b2d6700  1 mds.0.12834 handle_mds_map state 
change up:standby --> up:replay
2014-04-25 12:17:27.624830 7f3c2b2d6700  1 mds.0.12834 replay_start
2014-04-25 12:17:27.624836 7f3c2b2d6700  1 mds.0.12834  recovery set is 
2014-04-25 12:17:27.624837 7f3c2b2d6700  1 mds.0.12834  need osdmap epoch 
29082, have 29081
2014-04-25 12:17:27.624839 7f3c2b2d6700  1 mds.0.12834  waiting for osdmap 
29082 (which blacklists prior instance)
2014-04-25 12:17:30.138623 7f3c2b2d6700  0 mds.0.cache creating system inode 
with ino:100
2014-04-25 12:17:30.138890 7f3c2b2d6700  0 mds.0.cache creating system inode 
with ino:1
mds/journal.cc: In function 'void EMetaBlob::replay(MDS*, LogSegment*, 
MDSlaveUpdate*)' thread 7f3c26fbd700 time 2014-04-25 12:17:30.441635
mds/journal.cc: 1303: FAILED assert(session)
 ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60)
 1: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x7830) [0x5af890]
 2: (EUpdate::replay(MDS*)+0x3a) [0x5b67ea]
 3: (MDLog::_replay_thread()+0x678) [0x79dbb8]
 4: (MDLog::ReplayThread::entry()+0xd) [0x58bded]
 5: (()+0x7e9a) [0x7f3c2f675e9a]
 6: (clone()+0x6d) [0x7f3c2e56a3fd]
 NOTE: a copy of the executable, or `objdump -rdS ` is needed to 
interpret this.
2014-04-25 12:17:30.442489 7f3c26fbd700 -1 mds/journal.cc: In function 'void 
EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)' thread 7f3c26fbd700 time 
2014-04-25 12:17:30.441635
mds/journal.cc: 1303: FAILED assert(session)

 ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60)
 1: (EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)+0x7830) [0x5af890]
 2: (EUpdate::replay(MDS*)+0x3a) [0x5b67ea]
 3: (MDLog::_replay_thread()+0x678) [0x79dbb8]
 4: (MDLog::ReplayThread::entry()+0xd) [0x58bded]
 5: (()+0x7e9a) [0x7f3c2f675e9a]
 6: (clone()+0x6d) [0x7f3c2e56a3fd]
 NOTE: a copy of the executable, or `objdump -rdS ` is needed to 
interpret this.

--- begin dump of recent events ---
  -172> 2014-04-25 12:17:27.208884 7f3c30250780  5 asok(0x1b99000) 
register_command perfcounters_

[ceph-users] OpenStack Icehouse and ephemeral disks created from image

[Maciej Gałkiewicz]

Hi

After upgrading my OpenStack cluster to Icehouse I came across a very
suprising bug. It is no longer possible to create cinder volumes
(rbd-backed) from image (rbd-backed) by copy-on-write cloning:
https://blueprints.launchpad.net/nova/+spec/rbd-clone-image-handler

Both rbd volumes would be stored within the same ceph cluster. It worked
great in Havana. Do you have any idea how to workaround this? My instances
used to start within seconds and now it is a matter of minutes:/

-- 
Maciej Gałkiewicz
Shelly Cloud Sp. z o. o., Co-founder, Sysadmin
http://shellycloud.com/, mac...@shellycloud.com
KRS: 440358 REGON: 101504426
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Access denied error

[Punit Dambiwal]

Hi Yehuda,

Thanks for your help...that missing date error gone but still i am getting
the access denied error :-

-
2014-04-25 15:52:56.988025 7f00d37c6700  1 == starting new request
req=0x237a090 =
2014-04-25 15:52:56.988072 7f00d37c6700  2 req 24:0.46::GET
/admin/usage::initializing
2014-04-25 15:52:56.988077 7f00d37c6700 10 host=gateway.3linux.comrgw_dns_name=
gateway.3linux.com
2014-04-25 15:52:56.988102 7f00d37c6700 20 FCGI_ROLE=RESPONDER
2014-04-25 15:52:56.988103 7f00d37c6700 20 SCRIPT_URL=/admin/usage
2014-04-25 15:52:56.988104 7f00d37c6700 20 SCRIPT_URI=
http://gateway.3linux.com/admin/usage
2014-04-25 15:52:56.988105 7f00d37c6700 20 HTTP_AUTHORIZATION=AWS
KGXJJGKDM5G7G4CNKC7R:LC7S0twZdhtXA1XxthfMDsj5TgJpeKhZrloWa9WN
2014-04-25 15:52:56.988107 7f00d37c6700 20 HTTP_USER_AGENT=curl/7.22.0
(x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23
librtmp/2.3
2014-04-25 15:52:56.988108 7f00d37c6700 20 HTTP_ACCEPT=*/*
2014-04-25 15:52:56.988109 7f00d37c6700 20 HTTP_HOST=gateway.3linux.com
2014-04-25 15:52:56.988110 7f00d37c6700 20 HTTP_DATE=Fri, 25 April 2014
07:50:00 GMT
2014-04-25 15:52:56.988111 7f00d37c6700 20 CONTENT_LENGTH=0
2014-04-25 15:52:56.988112 7f00d37c6700 20 PATH=/usr/local/bin:/usr/bin:/bin
2014-04-25 15:52:56.988113 7f00d37c6700 20 SERVER_SIGNATURE=
2014-04-25 15:52:56.988114 7f00d37c6700 20 SERVER_SOFTWARE=Apache/2.2.22
(Ubuntu)
2014-04-25 15:52:56.988115 7f00d37c6700 20 SERVER_NAME=gateway.3linux.com
2014-04-25 15:52:56.988116 7f00d37c6700 20 SERVER_ADDR=117.18.79.110
2014-04-25 15:52:56.988117 7f00d37c6700 20 SERVER_PORT=80
2014-04-25 15:52:56.988117 7f00d37c6700 20 REMOTE_ADDR=122.166.115.191
2014-04-25 15:52:56.988118 7f00d37c6700 20 DOCUMENT_ROOT=/var/www
2014-04-25 15:52:56.988119 7f00d37c6700 20 SERVER_ADMIN=c...@3linux.com
2014-04-25 15:52:56.988120 7f00d37c6700 20
SCRIPT_FILENAME=/var/www/s3gw.fcgi
2014-04-25 15:52:56.988120 7f00d37c6700 20 REMOTE_PORT=28840
2014-04-25 15:52:56.988121 7f00d37c6700 20 GATEWAY_INTERFACE=CGI/1.1
2014-04-25 15:52:56.988122 7f00d37c6700 20 SERVER_PROTOCOL=HTTP/1.1
2014-04-25 15:52:56.988123 7f00d37c6700 20 REQUEST_METHOD=GET
2014-04-25 15:52:56.988123 7f00d37c6700 20
QUERY_STRING=page=admin¶ms=/usage&format=json
2014-04-25 15:52:56.988124 7f00d37c6700 20
REQUEST_URI=/admin/usage?format=json
2014-04-25 15:52:56.988125 7f00d37c6700 20 SCRIPT_NAME=/admin/usage
2014-04-25 15:52:56.988126 7f00d37c6700  2 req 24:0.000101::GET
/admin/usage::getting op
2014-04-25 15:52:56.988129 7f00d37c6700  2 req 24:0.000104::GET
/admin/usage:get_usage:authorizing
2014-04-25 15:52:56.988141 7f00d37c6700 20 get_obj_state:
rctx=0x7effbc004aa0 obj=.users:KGXJJGKDM5G7G4CNKC7R state=0x7effbc00e718
s->prefetch_data=0
2014-04-25 15:52:56.988148 7f00d37c6700 10 moving
.users+KGXJJGKDM5G7G4CNKC7R to cache LRU end
2014-04-25 15:52:56.988150 7f00d37c6700 10 cache get:
name=.users+KGXJJGKDM5G7G4CNKC7R : hit
2014-04-25 15:52:56.988155 7f00d37c6700 20 get_obj_state: s->obj_tag was
set empty
2014-04-25 15:52:56.988160 7f00d37c6700 10 moving
.users+KGXJJGKDM5G7G4CNKC7R to cache LRU end
2014-04-25 15:52:56.988161 7f00d37c6700 10 cache get:
name=.users+KGXJJGKDM5G7G4CNKC7R : hit
2014-04-25 15:52:56.988179 7f00d37c6700 20 get_obj_state:
rctx=0x7effbc001ce0 obj=.users.uid:admin state=0x7effbc00ec58
s->prefetch_data=0
2014-04-25 15:52:56.988185 7f00d37c6700 10 moving .users.uid+admin to cache
LRU end
2014-04-25 15:52:56.988186 7f00d37c6700 10 cache get: name=.users.uid+admin
: hit
2014-04-25 15:52:56.988190 7f00d37c6700 20 get_obj_state: s->obj_tag was
set empty
2014-04-25 15:52:56.988193 7f00d37c6700 10 moving .users.uid+admin to cache
LRU end
2014-04-25 15:52:56.988195 7f00d37c6700 10 cache get: name=.users.uid+admin
: hit
2014-04-25 15:52:56.988236 7f00d37c6700 10 get_canon_resource():
dest=/admin/usage
2014-04-25 15:52:56.988239 7f00d37c6700 10 auth_hdr:
GET


Fri, 25 April 2014 07:50:00 GMT
/admin/usage
2014-04-25 15:52:56.988325 7f00d37c6700 15 calculated
digest=nLKirQEEPeSS0Lzvr52NAB2phpA=
2014-04-25 15:52:56.988329 7f00d37c6700 15
auth_sign=LC7S0twZdhtXA1XxthfMDsj5TgJpeKhZrloWa9WN
2014-04-25 15:52:56.988330 7f00d37c6700 15 compare=-34
2014-04-25 15:52:56.988332 7f00d37c6700 10 failed to authorize request
2014-04-25 15:52:56.988351 7f00d37c6700  2 req 24:0.000325::GET
/admin/usage:get_usage:http status=403
2014-04-25 15:52:56.988426 7f00d37c6700  1 == req done req=0x237a090
http_status=403 ==
-



On Fri, Apr 25, 2014 at 12:13 AM, Yehuda Sadeh  wrote:

> On Thu, Apr 24, 2014 at 8:04 AM, Punit Dambiwal  wrote:
> >
> >
> > Hi Yehuda,
> >
> > I am getting the following from the radosgw logs :-
> >
> > ---
> > 2014-04-22 09:36:00.024618 7ff16ccf6700  1 == starting new request
> req=0x1ec7270 =
> > 2014-04-22 09:36:00.024719 7ff16ccf6700  2 req 15:0.000100::GET
> /admin/usage::initializing
> > 2014-04-22 09:36:00.024731 7ff16ccf6700 10 
> > host=gateway.3linux.com

Re: [ceph-users] UI of radosgw admin

[Ray Lv]

Hi Alain,

Thanks for your prompt answer. It looks cool. It seems to be a separated
project rather under ceph. Will it be incorporated into ceph? And what¹s
the schedule for remaining features such as user/bucket management?

Thanks,
Ray


On 4/22/14, 10:21 PM, "alain.dechorg...@orange.com"
 wrote:

>We are coding a such UI now in the Inkscope project. (follow us on github
>: https://github.com/inkscope/inkscope)
>First step is to manage users and buckets for S3.
>It will be available soon.
>
>Alain Dechorgnat
>Orange Labs
>
>
>
>-Message d'origine-
>De : ceph-users-boun...@lists.ceph.com
>[mailto:ceph-users-boun...@lists.ceph.com] De la part de Wido den
>Hollander
>Envoyé : mardi 22 avril 2014 16:05
>À : ceph-users@lists.ceph.com
>Objet : Re: [ceph-users] UI of radosgw admin
>
>On 04/22/2014 02:12 PM, Ray Lv wrote:
>> Hi there,
>>
>> Currently, the REST layer of CEPH has a bunch of admin APIs with
>> radosgw to manage user, bucket, key, etc. Is there an UI for managing
>> them based on the admin ops APIs right now or in the future? Any
>> interested parties on it?
>>
>
>Not that I'm aware of. Would be great to see though. Could make the life
>of admins easier.
>
>> Thanks,
>> Ray
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
>
>--
>Wido den Hollander
>42on B.V.
>Ceph trainer and consultant
>
>Phone: +31 (0)20 700 9902
>Skype: contact42on
>___
>ceph-users mailing list
>ceph-users@lists.ceph.com
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>__
>___
>
>Ce message et ses pieces jointes peuvent contenir des informations
>confidentielles ou privilegiees et ne doivent donc
>pas etre diffuses, exploites ou copies sans autorisation. Si vous avez
>recu ce message par erreur, veuillez le signaler
>a l'expediteur et le detruire ainsi que les pieces jointes. Les messages
>electroniques etant susceptibles d'alteration,
>Orange decline toute responsabilite si ce message a ete altere, deforme
>ou falsifie. Merci.
>
>This message and its attachments may contain confidential or privileged
>information that may be protected by law;
>they should not be distributed, used or copied without authorisation.
>If you have received this email in error, please notify the sender and
>delete this message and its attachments.
>As emails may be altered, Orange is not liable for messages that have
>been modified, changed or falsified.
>Thank you.
>
>___
>ceph-users mailing list
>ceph-users@lists.ceph.com
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Pool with empty name recreated

[mykr0t]

Hi, All.
Yesterday i managed to reproduce the bug on my test environment
with a fresh installation of dumpling release. I`ve attached the
link to archive with debug logs.
http://lamcdn.net/pool_with_empty_name_bug_logs.tar.gz
Test cluster contains only one bucket with name
"test" and one file in this bucket with name README and acl public-read.
Pool with empty name is created when RGW processes request with
non-existent bucket name. For example:
$ curl -kIL http://rgw.test.lo/test/README
HTTP/1.1 200 OK <- bucket exists, file exists
$ curl -kIL http://test.rgw.test.lo/README
HTTP/1.1 200 OK <- bucket exists, file exists
$ curl -kIL http://rgw.test.lo/test/README2
HTTP/1.1 403 OK <- bucket exists, file does not exists
$ curl -kIL http://test.rgw.test.lo/README2
HTTP/1.1 403 Forbidden <- bucket exists, file does not exists
$ curl -kIL http://rgw.test.lo/test2/README
HTTP/1.1 404 Not Found <- bucket does not exists, pool with empty name
is created
$ curl -kIL http://test2.rgw.test.lo/README
HTTP/1.1 404 Not Found <- bucket does not exists, pool with empty name
is created

If someone confirm this behaviour we can file a bug and request
backport.

-- 
Regards,
Mikhail


On Thu, 24 Apr 2014 10:33:00 -0700
Gregory Farnum  wrote:

> Yehuda says he's fixed several of these bugs in recent code, but if
> you're seeing it from a recent dev release, please file a bug!
> Likewise if you're on a named release and would like to see a
> backport. :) -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com
> 
> 
> On Thu, Apr 24, 2014 at 4:10 AM, Dan van der Ster
>  wrote:
> > Hi,
> > We also get the '' pool from rgw, which is clearly a bug somewhere.
> > But we recently learned that you can prevent it from being
> > recreated by removing the 'x' capability on the mon from your
> > client.radosgw.* users, for example:
> >
> > client.radosgw.cephrgw1
> > key: xxx
> > caps: [mon] allow r
> > caps: [osd] allow rwx
> >
> >
> > Cheers, Dan
> >
> >
> > myk...@gmail.com wrote:
> >>
> >> Hi,
> >>
> >> I cant delete pool with empty name:
> >>
> >> $ sudo rados rmpool "" "" --yes-i-really-really-mean-it
> >> successfully deleted pool
> >>
> >> but after a few seconds it is recreated automatically.
> >>
> >> $ sudo ceph osd dump | grep '^pool'
> >> pool 3 '.rgw' rep size 2 min_size 1 crush_ruleset 0 object_hash
> >> rjenkins pg_num 8 pgp_num 8 last_change 9 owner
> >> 18446744073709551615 pool 4 '.rgw.gc' rep size 2 min_size 1
> >> crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8
> >> last_change 10 owner 18446744073709551615 pool 5 '.rgw.control'
> >> rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num
> >> 8 pgp_num 8 last_change 11 owner 18446744073709551615 pool 6
> >> '.users.uid' rep size 2 min_size 1 crush_ruleset 0 object_hash
> >> rjenkins pg_num 8 pgp_num 8 last_change 13 owner 0 pool 7
> >> '.users.email' rep size 2 min_size 1 crush_ruleset 0 object_hash
> >> rjenkins pg_num 8 pgp_num 8 last_change 15 owner 0 pool 8 '.users'
> >> rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num
> >> 8 pgp_num 8 last_change 17 owner 0 pool 9 '.rgw.buckets' rep size
> >> 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 1024
> >> pgp_num 1024 last_change 38 owner 18446744073709551615 pool 10
> >> '.rgw.root' rep size 2 min_size 1 crush_ruleset 0 object_hash
> >> rjenkins pg_num 8 pgp_num 8 last_change 100 owner 0 pool 17 '' rep
> >> size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8
> >> pgp_num 8 last_change 3347 owner 0
> >>
> >> ceph version 0.67.7 (d7ab4244396b57aac8b7e80812115bbd079e6b73)
> >>
> >> How can i delete it forever?
> >>
> >
> > -- Dan van der Ster || Data & Storage Services || CERN IT
> > Department --
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Pool with empty name recreated

[Irek Fasikhov]

Hi.

radosgw-admin bucket list



2014-04-25 15:32 GMT+04:00 :

> Hi, All.
> Yesterday i managed to reproduce the bug on my test environment
> with a fresh installation of dumpling release. I`ve attached the
> link to archive with debug logs.
> http://lamcdn.net/pool_with_empty_name_bug_logs.tar.gz
> Test cluster contains only one bucket with name
> "test" and one file in this bucket with name README and acl public-read.
> Pool with empty name is created when RGW processes request with
> non-existent bucket name. For example:
> $ curl -kIL http://rgw.test.lo/test/README
> HTTP/1.1 200 OK <- bucket exists, file exists
> $ curl -kIL http://test.rgw.test.lo/README
> HTTP/1.1 200 OK <- bucket exists, file exists
> $ curl -kIL http://rgw.test.lo/test/README2
> HTTP/1.1 403 OK <- bucket exists, file does not exists
> $ curl -kIL http://test.rgw.test.lo/README2
> HTTP/1.1 403 Forbidden <- bucket exists, file does not exists
> $ curl -kIL http://rgw.test.lo/test2/README
> HTTP/1.1 404 Not Found <- bucket does not exists, pool with empty name
> is created
> $ curl -kIL http://test2.rgw.test.lo/README
> HTTP/1.1 404 Not Found <- bucket does not exists, pool with empty name
> is created
>
> If someone confirm this behaviour we can file a bug and request
> backport.
>
> --
> Regards,
> Mikhail
>
>
> On Thu, 24 Apr 2014 10:33:00 -0700
> Gregory Farnum  wrote:
>
> > Yehuda says he's fixed several of these bugs in recent code, but if
> > you're seeing it from a recent dev release, please file a bug!
> > Likewise if you're on a named release and would like to see a
> > backport. :) -Greg
> > Software Engineer #42 @ http://inktank.com | http://ceph.com
> >
> >
> > On Thu, Apr 24, 2014 at 4:10 AM, Dan van der Ster
> >  wrote:
> > > Hi,
> > > We also get the '' pool from rgw, which is clearly a bug somewhere.
> > > But we recently learned that you can prevent it from being
> > > recreated by removing the 'x' capability on the mon from your
> > > client.radosgw.* users, for example:
> > >
> > > client.radosgw.cephrgw1
> > > key: xxx
> > > caps: [mon] allow r
> > > caps: [osd] allow rwx
> > >
> > >
> > > Cheers, Dan
> > >
> > >
> > > myk...@gmail.com wrote:
> > >>
> > >> Hi,
> > >>
> > >> I cant delete pool with empty name:
> > >>
> > >> $ sudo rados rmpool "" "" --yes-i-really-really-mean-it
> > >> successfully deleted pool
> > >>
> > >> but after a few seconds it is recreated automatically.
> > >>
> > >> $ sudo ceph osd dump | grep '^pool'
> > >> pool 3 '.rgw' rep size 2 min_size 1 crush_ruleset 0 object_hash
> > >> rjenkins pg_num 8 pgp_num 8 last_change 9 owner
> > >> 18446744073709551615 pool 4 '.rgw.gc' rep size 2 min_size 1
> > >> crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8
> > >> last_change 10 owner 18446744073709551615 pool 5 '.rgw.control'
> > >> rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num
> > >> 8 pgp_num 8 last_change 11 owner 18446744073709551615 pool 6
> > >> '.users.uid' rep size 2 min_size 1 crush_ruleset 0 object_hash
> > >> rjenkins pg_num 8 pgp_num 8 last_change 13 owner 0 pool 7
> > >> '.users.email' rep size 2 min_size 1 crush_ruleset 0 object_hash
> > >> rjenkins pg_num 8 pgp_num 8 last_change 15 owner 0 pool 8 '.users'
> > >> rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num
> > >> 8 pgp_num 8 last_change 17 owner 0 pool 9 '.rgw.buckets' rep size
> > >> 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 1024
> > >> pgp_num 1024 last_change 38 owner 18446744073709551615 pool 10
> > >> '.rgw.root' rep size 2 min_size 1 crush_ruleset 0 object_hash
> > >> rjenkins pg_num 8 pgp_num 8 last_change 100 owner 0 pool 17 '' rep
> > >> size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8
> > >> pgp_num 8 last_change 3347 owner 0
> > >>
> > >> ceph version 0.67.7 (d7ab4244396b57aac8b7e80812115bbd079e6b73)
> > >>
> > >> How can i delete it forever?
> > >>
> > >
> > > -- Dan van der Ster || Data & Storage Services || CERN IT
> > > Department --
> > >
> > > ___
> > > ceph-users mailing list
> > > ceph-users@lists.ceph.com
> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
С уважением, Фасихов Ирек Нургаязович
Моб.: +79229045757
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Pool with empty name recreated

[mykr0t]

$ radosgw-admin bucket list
[
"test"]


-- 
Regards,
Mikhail


On Fri, 25 Apr 2014 15:48:23 +0400
Irek Fasikhov  wrote:

> Hi.
> 
> radosgw-admin bucket list
> 
> 
> 
> 2014-04-25 15:32 GMT+04:00 :
> 
> > Hi, All.
> > Yesterday i managed to reproduce the bug on my test environment
> > with a fresh installation of dumpling release. I`ve attached the
> > link to archive with debug logs.
> > http://lamcdn.net/pool_with_empty_name_bug_logs.tar.gz
> > Test cluster contains only one bucket with name
> > "test" and one file in this bucket with name README and acl
> > public-read. Pool with empty name is created when RGW processes
> > request with non-existent bucket name. For example:
> > $ curl -kIL http://rgw.test.lo/test/README
> > HTTP/1.1 200 OK <- bucket exists, file exists
> > $ curl -kIL http://test.rgw.test.lo/README
> > HTTP/1.1 200 OK <- bucket exists, file exists
> > $ curl -kIL http://rgw.test.lo/test/README2
> > HTTP/1.1 403 OK <- bucket exists, file does not exists
> > $ curl -kIL http://test.rgw.test.lo/README2
> > HTTP/1.1 403 Forbidden <- bucket exists, file does not exists
> > $ curl -kIL http://rgw.test.lo/test2/README
> > HTTP/1.1 404 Not Found <- bucket does not exists, pool with empty
> > name is created
> > $ curl -kIL http://test2.rgw.test.lo/README
> > HTTP/1.1 404 Not Found <- bucket does not exists, pool with empty
> > name is created
> >
> > If someone confirm this behaviour we can file a bug and request
> > backport.
> >
> > --
> > Regards,
> > Mikhail
> >
> >
> > On Thu, 24 Apr 2014 10:33:00 -0700
> > Gregory Farnum  wrote:
> >
> > > Yehuda says he's fixed several of these bugs in recent code, but
> > > if you're seeing it from a recent dev release, please file a bug!
> > > Likewise if you're on a named release and would like to see a
> > > backport. :) -Greg
> > > Software Engineer #42 @ http://inktank.com | http://ceph.com
> > >
> > >
> > > On Thu, Apr 24, 2014 at 4:10 AM, Dan van der Ster
> > >  wrote:
> > > > Hi,
> > > > We also get the '' pool from rgw, which is clearly a bug
> > > > somewhere. But we recently learned that you can prevent it from
> > > > being recreated by removing the 'x' capability on the mon from
> > > > your client.radosgw.* users, for example:
> > > >
> > > > client.radosgw.cephrgw1
> > > > key: xxx
> > > > caps: [mon] allow r
> > > > caps: [osd] allow rwx
> > > >
> > > >
> > > > Cheers, Dan
> > > >
> > > >
> > > > myk...@gmail.com wrote:
> > > >>
> > > >> Hi,
> > > >>
> > > >> I cant delete pool with empty name:
> > > >>
> > > >> $ sudo rados rmpool "" "" --yes-i-really-really-mean-it
> > > >> successfully deleted pool
> > > >>
> > > >> but after a few seconds it is recreated automatically.
> > > >>
> > > >> $ sudo ceph osd dump | grep '^pool'
> > > >> pool 3 '.rgw' rep size 2 min_size 1 crush_ruleset 0 object_hash
> > > >> rjenkins pg_num 8 pgp_num 8 last_change 9 owner
> > > >> 18446744073709551615 pool 4 '.rgw.gc' rep size 2 min_size 1
> > > >> crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8
> > > >> last_change 10 owner 18446744073709551615 pool 5 '.rgw.control'
> > > >> rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins
> > > >> pg_num 8 pgp_num 8 last_change 11 owner 18446744073709551615
> > > >> pool 6 '.users.uid' rep size 2 min_size 1 crush_ruleset 0
> > > >> object_hash rjenkins pg_num 8 pgp_num 8 last_change 13 owner 0
> > > >> pool 7 '.users.email' rep size 2 min_size 1 crush_ruleset 0
> > > >> object_hash rjenkins pg_num 8 pgp_num 8 last_change 15 owner 0
> > > >> pool 8 '.users' rep size 2 min_size 1 crush_ruleset 0
> > > >> object_hash rjenkins pg_num 8 pgp_num 8 last_change 17 owner 0
> > > >> pool 9 '.rgw.buckets' rep size 2 min_size 1 crush_ruleset 0
> > > >> object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 38
> > > >> owner 18446744073709551615 pool 10 '.rgw.root' rep size 2
> > > >> min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8
> > > >> pgp_num 8 last_change 100 owner 0 pool 17 '' rep size 2
> > > >> min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8
> > > >> pgp_num 8 last_change 3347 owner 0
> > > >>
> > > >> ceph version 0.67.7 (d7ab4244396b57aac8b7e80812115bbd079e6b73)
> > > >>
> > > >> How can i delete it forever?
> > > >>
> > > >
> > > > -- Dan van der Ster || Data & Storage Services || CERN IT
> > > > Department --
> > > >
> > > > ___
> > > > ceph-users mailing list
> > > > ceph-users@lists.ceph.com
> > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> 
> 
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Pool with empty name recreated

[Irek Fasikhov]

You correctly configured DNS records?


2014-04-25 16:24 GMT+04:00 :

> $ radosgw-admin bucket list
> [
> "test"]
>
>
> --
> Regards,
> Mikhail
>
>
> On Fri, 25 Apr 2014 15:48:23 +0400
> Irek Fasikhov  wrote:
>
> > Hi.
> >
> > radosgw-admin bucket list
> >
> >
> >
> > 2014-04-25 15:32 GMT+04:00 :
> >
> > > Hi, All.
> > > Yesterday i managed to reproduce the bug on my test environment
> > > with a fresh installation of dumpling release. I`ve attached the
> > > link to archive with debug logs.
> > > http://lamcdn.net/pool_with_empty_name_bug_logs.tar.gz
> > > Test cluster contains only one bucket with name
> > > "test" and one file in this bucket with name README and acl
> > > public-read. Pool with empty name is created when RGW processes
> > > request with non-existent bucket name. For example:
> > > $ curl -kIL http://rgw.test.lo/test/README
> > > HTTP/1.1 200 OK <- bucket exists, file exists
> > > $ curl -kIL http://test.rgw.test.lo/README
> > > HTTP/1.1 200 OK <- bucket exists, file exists
> > > $ curl -kIL http://rgw.test.lo/test/README2
> > > HTTP/1.1 403 OK <- bucket exists, file does not exists
> > > $ curl -kIL http://test.rgw.test.lo/README2
> > > HTTP/1.1 403 Forbidden <- bucket exists, file does not exists
> > > $ curl -kIL http://rgw.test.lo/test2/README
> > > HTTP/1.1 404 Not Found <- bucket does not exists, pool with empty
> > > name is created
> > > $ curl -kIL http://test2.rgw.test.lo/README
> > > HTTP/1.1 404 Not Found <- bucket does not exists, pool with empty
> > > name is created
> > >
> > > If someone confirm this behaviour we can file a bug and request
> > > backport.
> > >
> > > --
> > > Regards,
> > > Mikhail
> > >
> > >
> > > On Thu, 24 Apr 2014 10:33:00 -0700
> > > Gregory Farnum  wrote:
> > >
> > > > Yehuda says he's fixed several of these bugs in recent code, but
> > > > if you're seeing it from a recent dev release, please file a bug!
> > > > Likewise if you're on a named release and would like to see a
> > > > backport. :) -Greg
> > > > Software Engineer #42 @ http://inktank.com | http://ceph.com
> > > >
> > > >
> > > > On Thu, Apr 24, 2014 at 4:10 AM, Dan van der Ster
> > > >  wrote:
> > > > > Hi,
> > > > > We also get the '' pool from rgw, which is clearly a bug
> > > > > somewhere. But we recently learned that you can prevent it from
> > > > > being recreated by removing the 'x' capability on the mon from
> > > > > your client.radosgw.* users, for example:
> > > > >
> > > > > client.radosgw.cephrgw1
> > > > > key: xxx
> > > > > caps: [mon] allow r
> > > > > caps: [osd] allow rwx
> > > > >
> > > > >
> > > > > Cheers, Dan
> > > > >
> > > > >
> > > > > myk...@gmail.com wrote:
> > > > >>
> > > > >> Hi,
> > > > >>
> > > > >> I cant delete pool with empty name:
> > > > >>
> > > > >> $ sudo rados rmpool "" "" --yes-i-really-really-mean-it
> > > > >> successfully deleted pool
> > > > >>
> > > > >> but after a few seconds it is recreated automatically.
> > > > >>
> > > > >> $ sudo ceph osd dump | grep '^pool'
> > > > >> pool 3 '.rgw' rep size 2 min_size 1 crush_ruleset 0 object_hash
> > > > >> rjenkins pg_num 8 pgp_num 8 last_change 9 owner
> > > > >> 18446744073709551615 pool 4 '.rgw.gc' rep size 2 min_size 1
> > > > >> crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8
> > > > >> last_change 10 owner 18446744073709551615 pool 5 '.rgw.control'
> > > > >> rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins
> > > > >> pg_num 8 pgp_num 8 last_change 11 owner 18446744073709551615
> > > > >> pool 6 '.users.uid' rep size 2 min_size 1 crush_ruleset 0
> > > > >> object_hash rjenkins pg_num 8 pgp_num 8 last_change 13 owner 0
> > > > >> pool 7 '.users.email' rep size 2 min_size 1 crush_ruleset 0
> > > > >> object_hash rjenkins pg_num 8 pgp_num 8 last_change 15 owner 0
> > > > >> pool 8 '.users' rep size 2 min_size 1 crush_ruleset 0
> > > > >> object_hash rjenkins pg_num 8 pgp_num 8 last_change 17 owner 0
> > > > >> pool 9 '.rgw.buckets' rep size 2 min_size 1 crush_ruleset 0
> > > > >> object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 38
> > > > >> owner 18446744073709551615 pool 10 '.rgw.root' rep size 2
> > > > >> min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8
> > > > >> pgp_num 8 last_change 100 owner 0 pool 17 '' rep size 2
> > > > >> min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8
> > > > >> pgp_num 8 last_change 3347 owner 0
> > > > >>
> > > > >> ceph version 0.67.7 (d7ab4244396b57aac8b7e80812115bbd079e6b73)
> > > > >>
> > > > >> How can i delete it forever?
> > > > >>
> > > > >
> > > > > -- Dan van der Ster || Data & Storage Services || CERN IT
> > > > > Department --
> > > > >
> > > > > ___
> > > > > ceph-users mailing list
> > > > > ceph-users@lists.ceph.com
> > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > > ___
> > > ceph-users mailing list
> > > ceph-users@lists.ceph.com
> > > http://lists.ceph.com/listinfo.cgi/ceph-users

Re: [ceph-users] Pool with empty name recreated

[mykr0t]

I think yes, you can see request headers in the attached radosgw.log.
Can you try access you cluster with curl or wget with none-existent
bucket and file?
and then show ceph osd dump?

-- 
Regards,
Mikhail


On Fri, 25 Apr 2014 16:26:09 +0400
Irek Fasikhov  wrote:

> You correctly configured DNS records?
> 
> 
> 2014-04-25 16:24 GMT+04:00 :
> 
> > $ radosgw-admin bucket list
> > [
> > "test"]
> >
> >
> > --
> > Regards,
> > Mikhail
> >
> >
> > On Fri, 25 Apr 2014 15:48:23 +0400
> > Irek Fasikhov  wrote:
> >
> > > Hi.
> > >
> > > radosgw-admin bucket list
> > >
> > >
> > >
> > > 2014-04-25 15:32 GMT+04:00 :
> > >
> > > > Hi, All.
> > > > Yesterday i managed to reproduce the bug on my test environment
> > > > with a fresh installation of dumpling release. I`ve attached the
> > > > link to archive with debug logs.
> > > > http://lamcdn.net/pool_with_empty_name_bug_logs.tar.gz
> > > > Test cluster contains only one bucket with name
> > > > "test" and one file in this bucket with name README and acl
> > > > public-read. Pool with empty name is created when RGW processes
> > > > request with non-existent bucket name. For example:
> > > > $ curl -kIL http://rgw.test.lo/test/README
> > > > HTTP/1.1 200 OK <- bucket exists, file exists
> > > > $ curl -kIL http://test.rgw.test.lo/README
> > > > HTTP/1.1 200 OK <- bucket exists, file exists
> > > > $ curl -kIL http://rgw.test.lo/test/README2
> > > > HTTP/1.1 403 OK <- bucket exists, file does not exists
> > > > $ curl -kIL http://test.rgw.test.lo/README2
> > > > HTTP/1.1 403 Forbidden <- bucket exists, file does not exists
> > > > $ curl -kIL http://rgw.test.lo/test2/README
> > > > HTTP/1.1 404 Not Found <- bucket does not exists, pool with
> > > > empty name is created
> > > > $ curl -kIL http://test2.rgw.test.lo/README
> > > > HTTP/1.1 404 Not Found <- bucket does not exists, pool with
> > > > empty name is created
> > > >
> > > > If someone confirm this behaviour we can file a bug and request
> > > > backport.
> > > >
> > > > --
> > > > Regards,
> > > > Mikhail
> > > >
> > > >
> > > > On Thu, 24 Apr 2014 10:33:00 -0700
> > > > Gregory Farnum  wrote:
> > > >
> > > > > Yehuda says he's fixed several of these bugs in recent code,
> > > > > but if you're seeing it from a recent dev release, please
> > > > > file a bug! Likewise if you're on a named release and would
> > > > > like to see a backport. :) -Greg
> > > > > Software Engineer #42 @ http://inktank.com | http://ceph.com
> > > > >
> > > > >
> > > > > On Thu, Apr 24, 2014 at 4:10 AM, Dan van der Ster
> > > > >  wrote:
> > > > > > Hi,
> > > > > > We also get the '' pool from rgw, which is clearly a bug
> > > > > > somewhere. But we recently learned that you can prevent it
> > > > > > from being recreated by removing the 'x' capability on the
> > > > > > mon from your client.radosgw.* users, for example:
> > > > > >
> > > > > > client.radosgw.cephrgw1
> > > > > > key: xxx
> > > > > > caps: [mon] allow r
> > > > > > caps: [osd] allow rwx
> > > > > >
> > > > > >
> > > > > > Cheers, Dan
> > > > > >
> > > > > >
> > > > > > myk...@gmail.com wrote:
> > > > > >>
> > > > > >> Hi,
> > > > > >>
> > > > > >> I cant delete pool with empty name:
> > > > > >>
> > > > > >> $ sudo rados rmpool "" "" --yes-i-really-really-mean-it
> > > > > >> successfully deleted pool
> > > > > >>
> > > > > >> but after a few seconds it is recreated automatically.
> > > > > >>
> > > > > >> $ sudo ceph osd dump | grep '^pool'
> > > > > >> pool 3 '.rgw' rep size 2 min_size 1 crush_ruleset 0
> > > > > >> object_hash rjenkins pg_num 8 pgp_num 8 last_change 9 owner
> > > > > >> 18446744073709551615 pool 4 '.rgw.gc' rep size 2 min_size 1
> > > > > >> crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8
> > > > > >> last_change 10 owner 18446744073709551615 pool 5
> > > > > >> '.rgw.control' rep size 2 min_size 1 crush_ruleset 0
> > > > > >> object_hash rjenkins pg_num 8 pgp_num 8 last_change 11
> > > > > >> owner 18446744073709551615 pool 6 '.users.uid' rep size 2
> > > > > >> min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8
> > > > > >> pgp_num 8 last_change 13 owner 0 pool 7 '.users.email' rep
> > > > > >> size 2 min_size 1 crush_ruleset 0 object_hash rjenkins
> > > > > >> pg_num 8 pgp_num 8 last_change 15 owner 0 pool 8 '.users'
> > > > > >> rep size 2 min_size 1 crush_ruleset 0 object_hash rjenkins
> > > > > >> pg_num 8 pgp_num 8 last_change 17 owner 0 pool 9
> > > > > >> '.rgw.buckets' rep size 2 min_size 1 crush_ruleset 0
> > > > > >> object_hash rjenkins pg_num 1024 pgp_num 1024 last_change
> > > > > >> 38 owner 18446744073709551615 pool 10 '.rgw.root' rep size
> > > > > >> 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8
> > > > > >> pgp_num 8 last_change 100 owner 0 pool 17 '' rep size 2
> > > > > >> min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 8
> > > > > >> pgp_num 8 last_change 3347 owner 0
> > > > > >>
> > > > > >> ceph version 0.67.7
> > > > > >> (d7ab4244396b57aac8b7e80812115bbd079e6b73)
> > 

Re: [ceph-users] UI of radosgw admin

[alain.dechorgnat]

For yet, the inkscope project is independent of Ceph.
The S3 user management could be finished before the end of the next week.

Alain

-Message d'origine-
De : Ray Lv [mailto:ra...@yahoo-inc.com] 
Envoyé : vendredi 25 avril 2014 11:57
À : DECHORGNAT Alain IMT/OLPS
Cc : ceph-users@lists.ceph.com
Objet : Re: [ceph-users] UI of radosgw admin

Hi Alain,

Thanks for your prompt answer. It looks cool. It seems to be a separated 
project rather under ceph. Will it be incorporated into ceph? And what¹s the 
schedule for remaining features such as user/bucket management?

Thanks,
Ray


On 4/22/14, 10:21 PM, "alain.dechorg...@orange.com"
 wrote:

>We are coding a such UI now in the Inkscope project. (follow us on 
>github
>: https://github.com/inkscope/inkscope)
>First step is to manage users and buckets for S3.
>It will be available soon.
>
>Alain Dechorgnat
>Orange Labs
>
>
>
>-Message d'origine-
>De : ceph-users-boun...@lists.ceph.com
>[mailto:ceph-users-boun...@lists.ceph.com] De la part de Wido den 
>Hollander Envoyé : mardi 22 avril 2014 16:05 À : 
>ceph-users@lists.ceph.com Objet : Re: [ceph-users] UI of radosgw admin
>
>On 04/22/2014 02:12 PM, Ray Lv wrote:
>> Hi there,
>>
>> Currently, the REST layer of CEPH has a bunch of admin APIs with 
>> radosgw to manage user, bucket, key, etc. Is there an UI for managing 
>> them based on the admin ops APIs right now or in the future? Any 
>> interested parties on it?
>>
>
>Not that I'm aware of. Would be great to see though. Could make the 
>life of admins easier.
>
>> Thanks,
>> Ray
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
>
>--
>Wido den Hollander
>42on B.V.
>Ceph trainer and consultant
>
>Phone: +31 (0)20 700 9902
>Skype: contact42on
>___
>ceph-users mailing list
>ceph-users@lists.ceph.com
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>___
>___ ___
>
>Ce message et ses pieces jointes peuvent contenir des informations 
>confidentielles ou privilegiees et ne doivent donc pas etre diffuses, 
>exploites ou copies sans autorisation. Si vous avez recu ce message par 
>erreur, veuillez le signaler a l'expediteur et le detruire ainsi que 
>les pieces jointes. Les messages electroniques etant susceptibles 
>d'alteration, Orange decline toute responsabilite si ce message a ete 
>altere, deforme ou falsifie. Merci.
>
>This message and its attachments may contain confidential or privileged 
>information that may be protected by law; they should not be 
>distributed, used or copied without authorisation.
>If you have received this email in error, please notify the sender and 
>delete this message and its attachments.
>As emails may be altered, Orange is not liable for messages that have 
>been modified, changed or falsified.
>Thank you.
>
>___
>ceph-users mailing list
>ceph-users@lists.ceph.com
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


_

Ce message et ses pieces jointes peuvent contenir des informations 
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce 
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou 
falsifie. Merci.

This message and its attachments may contain confidential or privileged 
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete 
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been 
modified, changed or falsified.
Thank you.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [Ceph-rgw] pool assignment

[ghislain.chevalier]

Before configuring region and zone, I would like to known which tags can be 
updated in metadata bucket.instance?

Are there restrictions according to the capabilities applied to radosgw-admin?


-Message d'origine-
De : ceph-users-boun...@lists.ceph.com 
[mailto:ceph-users-boun...@lists.ceph.com] De la part de 
ghislain.cheval...@orange.com
Envoyé : mardi 15 avril 2014 18:21
À : Yehuda Sadeh
Cc : ceph-users@lists.ceph.com
Objet : Re: [ceph-users] [Ceph-rgw] pool assignment

Thanks

Sorry for answering late

I'm going to implement region, zone and placement targets in order to reach my 
goals.

Best regards

-Message d'origine-
De : Yehuda Sadeh [mailto:yeh...@inktank.com] 
Envoyé : vendredi 11 avril 2014 18:34
À : CHEVALIER Ghislain IMT/OLPS
Cc : ceph-users@lists.ceph.com
Objet : Re: [ceph-users] [Ceph-rgw] pool assigment

On Fri, Apr 11, 2014 at 3:20 AM,   wrote:
> Hi all,
>
> Context : CEPH dumpling on Ubuntu 12.04
>
> I would like to manage as accurately as possible the pools assigned to Rados
> gateway
> My goal is to apply specific SLA to applications which use http-driven
> storage.
> I 'd like to store the contents by associating a pool to a bucket, or a pool
> to a user(account)
>
> Today, all the contents are stored in .rgw.buckets
> I read in a post that if the pool .rgw. exists, contents will be
> stored in it, but it fails.

That's incorrect.

>
> I tried to modify the metadata associated to the bucket but it fails.
> radosgw-admin metadata get bucket: gives a set information
>
> { "key": "bucket:",
>   "ver": { "tag": " _fimhlBaK9aEKoX--jQKuvjT",
>   "ver": 1},
>   "mtime": 1394793288,
>   "data": { "bucket": { "name": "",
>   "pool": ".rgw. ",
>   "index_pool": ".rgw. .index",
>   "marker": "default.9841.1",
>   "bucket_id": "default.9841.1"},
>   "owner": "johndoe",
>   "creation_time": 1394793288,
>   "linked": "true",
>   "has_bucket_info": "false"}}
>
> which don't match with information given by radosgw-admin bucket
> -bucket= stats
> { "bucket": "",
>   "pool": ".rgw.buckets",
>   "index_pool": ".rgw.buckets.index",
>   "id": "default.9841.1",
>   "marker": "default.9841.1",
>   "owner": "johndoe",
>   "ver": 45,
>   "master_ver": 0,
>   "mtime": 1394793288,
>   "max_marker": "",
>   "usage": { "rgw.main": { "size_kb": 3005,
>   "size_kb_actual": 3016,
>   "num_objects": 3}}
>
> Is there a guideline I can rely on in order to reach my goal?


Look here:

http://comments.gmane.org/gmane.comp.file-systems.ceph.user/4992

The idea is to define different placement targets. You can change a
user's default placement target, and on bucket creation users can
define which placement target to use (if they're allowed).

Yehuda

_

Ce message et ses pieces jointes peuvent contenir des informations 
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce 
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou 
falsifie. Merci.

This message and its attachments may contain confidential or privileged 
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete 
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been 
modified, changed or falsified.
Thank you.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_

Ce message et ses pieces jointes peuvent contenir des informations 
confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce 
message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages 
electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou 
falsifie. Merci.

This message and its attachments may contain confidential or privileged 
information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete 
this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been 
modified, changed or falsified.
Thank you.

___
ceph-users mailing list
ceph-users@l

Re: [ceph-users] OpenStack Icehouse and ephemeral disks created from image

[Gregory Farnum]

If you had it working in Havana I think you must have been using a
customized code base; you can still do the same for Icehouse.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com


On Fri, Apr 25, 2014 at 12:55 AM, Maciej Gałkiewicz
 wrote:
> Hi
>
> After upgrading my OpenStack cluster to Icehouse I came across a very
> suprising bug. It is no longer possible to create cinder volumes
> (rbd-backed) from image (rbd-backed) by copy-on-write cloning:
> https://blueprints.launchpad.net/nova/+spec/rbd-clone-image-handler
>
> Both rbd volumes would be stored within the same ceph cluster. It worked
> great in Havana. Do you have any idea how to workaround this? My instances
> used to start within seconds and now it is a matter of minutes:/
>
> --
> Maciej Gałkiewicz
> Shelly Cloud Sp. z o. o., Co-founder, Sysadmin
> http://shellycloud.com/, mac...@shellycloud.com
> KRS: 440358 REGON: 101504426
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OpenStack Icehouse and ephemeral disks created from image

[Maciej Gałkiewicz]

On 25 April 2014 16:00, Gregory Farnum  wrote:

> If you had it working in Havana I think you must have been using a
> customized code base; you can still do the same for Icehouse.
> -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com


I was using a standard OpenStack version from Debian official repository.

This is how I was creating the volume:

# cinder create --image-id 1b84776e-25a0-441e-a74f-dc5d3bf5c103 5

The volume is created instantly.

# rbd info volume-abca46dc-0a69-43f0-b91a-412dbf30810f -p cinder_volumes
rbd image 'volume-abca46dc-0a69-43f0-b91a-412dbf30810f':
size 5120 MB in 640 objects
order 23 (8192 kB objects)
block_name_prefix: rbd_data.301adaa4c2e28e
format: 2
features: layering
parent: glance_images/1b84776e-25a0-441e-a74f-dc5d3bf5c103@snap
overlap: 4608 MB

Isn't it a copy-on-write clone?

-- 
Maciej Gałkiewicz
Shelly Cloud Sp. z o. o., Co-founder, Sysadmin
http://shellycloud.com/, mac...@shellycloud.com
KRS: 440358 REGON: 101504426
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OpenStack Icehouse and ephemeral disks created from image

[Sebastien Han]

This is a COW clone, but the BP you pointed doesn’t match the feature you 
described. This might explain Greg’s answer.
The BP refers to the libvirt_image_type functionality for Nova.

What do you get now when you try to create a volume from an image?

 
Sébastien Han 
Cloud Engineer 

"Always give 100%. Unless you're giving blood.” 

Phone: +33 (0)1 49 70 99 72 
Mail: sebastien@enovance.com 
Address : 11 bis, rue Roquépine - 75008 Paris
Web : www.enovance.com - Twitter : @enovance 

On 25 Apr 2014, at 16:34, Maciej Gałkiewicz  wrote:

> On 25 April 2014 16:00, Gregory Farnum  wrote:
> If you had it working in Havana I think you must have been using a
> customized code base; you can still do the same for Icehouse.
> -Greg
> Software Engineer #42 @ http://inktank.com | http://ceph.com
> 
> I was using a standard OpenStack version from Debian official repository.
> 
> This is how I was creating the volume:
> 
> # cinder create --image-id 1b84776e-25a0-441e-a74f-dc5d3bf5c103 5
> 
> The volume is created instantly.
> 
> # rbd info volume-abca46dc-0a69-43f0-b91a-412dbf30810f -p cinder_volumes
> rbd image 'volume-abca46dc-0a69-43f0-b91a-412dbf30810f':
> size 5120 MB in 640 objects
> order 23 (8192 kB objects)
> block_name_prefix: rbd_data.301adaa4c2e28e
> format: 2
> features: layering
> parent: glance_images/1b84776e-25a0-441e-a74f-dc5d3bf5c103@snap
> overlap: 4608 MB
> 
> Isn't it a copy-on-write clone?
> 
> -- 
> Maciej Gałkiewicz
> Shelly Cloud Sp. z o. o., Co-founder, Sysadmin
> http://shellycloud.com/, mac...@shellycloud.com
> KRS: 440358 REGON: 101504426
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OSD distribution unequally -- osd crashes

[Kenneth Waegeman]

- Message from Craig Lewis  -
   Date: Thu, 24 Apr 2014 11:20:08 -0700
   From: Craig Lewis 
Subject: Re: [ceph-users] OSD distribution unequally -- osd crashes
 To: Kenneth Waegeman 
 Cc: ceph-users@lists.ceph.com


Your OSDs shouldn't be crashing during a remap.  Although.. you  
might try to get that one OSDs below 96% first, it could have an  
effect.  If OSDs continue to crash after that, I'd start a new  
thread about the crashes.



If you have the option to delete some data and reload it it later,  
I'd do that now.  If you can't, or you can't delete enough, read up  
on mon_osd_full_ratio, mon_osd_nearfull_ratio,  
osd_failsafe_nearfull_ratio, and osd_backfill_full_ratio.  You can  
temporarily changes these values, but *be very careful*. Make sure  
you understand the docs before doing it.  Ceph is protecting you.   
You'll cause yourself a lot more pain if you let the disks get to  
100% full.


You can change those params on the fly with:
ceph tell osd.* injectargs '--mon_osd_nearfull_ratio 0.85'

Get the current values with:
ceph --admin-daemon /var/run/ceph/ceph-osd..asok config get  
mon_osd_nearfull_ratio


injectargs changes won't survive restart.  If you need them to last  
though a reboot or daemon restart, you'll want to add them to  
ceph.conf too.  I didn't, I just reran the injectargs whenever I  
manually restarted something.



Once you get IO working again, I'd knock down those specific OSDs  
before proceeding with more pg_num and pgp_num changes.  
reweight-by-utilization is  wrapper around:

ceph osd reweight  

You can get the current weights from
ceph osd dump | grep ^osd

Assuming all your OSDs are weighted 1.0, I'd drop the weight on all  
OSDs > 85% full to 0.95, and see what that looks like.  You might  
need a couple passes, because some of the migrated PGs will be  
migrated to other nearfull OSDs.  I dropped weights by 0.025 every  
pass, and I ended up with one OSD weighted 0.875.


While it's remapping, some will get into state backfill_toofull.  
Assuming your reweights are enough, they will clear, but it does  
slow things down.  If you get to the point that all of the backfill  
PGs are backfill_toofull, then make another pass on the OSD reweights.



Once they're all below 85%, you can start doing the pg_num and  
pgp_num expansions.  You may need to revisit the OSD weights during  
this process, and you'll probably want to reset all weights to 1.0  
when you're done.  Since you have OSDs crashing, you'll want to note  
that marking an OSD out then in will reset the weight to 1.0.  You  
might want to track your weights outside of Ceph, so you can  
manually reweight if that happens.



Lastly, you'll want monitor this.  If you had noticed when the first  
OSD hit 86%, things would be much less painful.



Thanks for the very useful information!

I started with deleting some data, and this helped for the stability  
of the OSDs. No OSD has crashed the past 6 hours (before it was  
multiple an hour).
There are indeed a lot of PGs in backfill_toofull . Now the osds  
stopped crashing it can actually recover and resume the backfilling!






*Craig Lewis*
Senior Systems Engineer
Office +1.714.602.1309
Email cle...@centraldesktop.com 

*Central Desktop. Work together in ways you never thought possible.*
Connect with us Website   | Twitter  
  | Facebook  
  | LinkedIn  
  | Blog  



On 4/24/14 02:19 , Kenneth Waegeman wrote:


- Message from Craig Lewis  -
  Date: Fri, 18 Apr 2014 14:59:25 -0700
  From: Craig Lewis 
Subject: Re: [ceph-users] OSD distribution unequally
To: ceph-users@lists.ceph.com


When you increase the number of PGs, don't just go to the max  
value.  Step into it.
You'll want to end up around 2048, so do 400 -> 512, wait for it  
to finish, -> 1024, wait, -> 2048.


Thanks, I changed it to 512, also doing the reweight-by-utilisation.
While doing this, some osds crashed a few times. I thought it was  
maybe because of the fact they were almost full.
When this was finishes, some pgs were still in active-remapped  
state. I read in another mail from the list to try to do  ceph osd  
crush tunables optimal. So that is running now. But after a few  
hours, again 17 out of 42 osds were crashed. (I don't know the  
crashing is connected with the reweight and the pgs stuck in  
active-remapped)

Out of log file:

2014-04-24 03:46:57.442110 7f4968a11700 -1 osd/PG.cc: In function  
'PG::RecoveryState::Crashed::Crashed(boost::statechart::statePG::RecoveryS
tate::RecoveryMachine, boost::mpl::listmpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,  
mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,  
mpl_::na>, (boost::statechart::histo

[ceph-users] Ceph mds laggy and failed to assert session in function mds/journal.cc line 1303

[Bazli Karim]

Dear Ceph-devel, ceph-users,



I am currently facing issue with my ceph mds server. Ceph-mds daemon does
not want to bring up back.

I tried running that manually with ceph-mds –i mon01 –d but it got aborted
and the log shows that it stucks at failed assert(session) line 1303 in
mds/journal.cc.



Can someone shed some light in this issue?

I am running ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60)



Let me know if I need to send more logs.



Regards,

Bazli
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OpenStack Icehouse and ephemeral disks created from image

[Maciej Gałkiewicz]

On 25 April 2014 16:37, Sebastien Han  wrote:

> This is a COW clone, but the BP you pointed doesn’t match the feature you
> described. This might explain Greg’s answer.
> The BP refers to the libvirt_image_type functionality for Nova.
>
> What do you get now when you try to create a volume from an image?
>

The image is downloaded and then imported to ceph. Detailed log:
https://gist.github.com/mgalkiewicz/e0558939e435cb6d5d28

-- 
Maciej Gałkiewicz
Shelly Cloud Sp. z o. o., Co-founder, Sysadmin
http://shellycloud.com/, mac...@shellycloud.com
KRS: 440358 REGON: 101504426
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OpenStack Icehouse and ephemeral disks created from image

[Sebastien Han]

I just tried, I have the same problem, it looks like a regression…
It’s weird because the code didn’t change that much during the Icehouse cycle.

I just reported the bug here: https://bugs.launchpad.net/cinder/+bug/1312819

 
Sébastien Han 
Cloud Engineer 

"Always give 100%. Unless you're giving blood.” 

Phone: +33 (0)1 49 70 99 72 
Mail: sebastien@enovance.com 
Address : 11 bis, rue Roquépine - 75008 Paris
Web : www.enovance.com - Twitter : @enovance 

On 25 Apr 2014, at 16:37, Sebastien Han  wrote:

> g



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [Bug]radosgw-agent can't sync files with Chinese filename

[Yehuda Sadeh]

On Thu, Apr 24, 2014 at 7:03 PM, wsnote  wrote:
> Hi, Yehuda.
> It doesn't matter.We have fixed it.
> The filename will be transcoded by url_encode and decoded by url_decode.
> There is a bug when decoding the filename.
> There is another bug when decoding the filename. when radosgw-agent fails
> decoding a filename, files sync will get stuck and other files will be not
> synced. This must be optimized. If one file synced fail, radosgw-agent must
> try to continuing sync other files.

Do you have an actual fix for it that you want to share? It's not
clear to me whether you assume it's the sync agent issue or the
gateway issue. My assumption is that the gateway fails to encode it
correctly when sending the cross zones copy request.


Yehuda

>
> At 2014-04-25 03:32:02,"Yehuda Sadeh"  wrote:
>
>>Hi,
>>
>>  sorry for the late response. I opened a ceph tracker issue for it
>> (#8202).
>>
>>Thanks,
>>Yehuda
>>
>>On Wed, Apr 16, 2014 at 1:00 AM, wsnote  wrote:
>>> OS: CentOS 6.5
>>> Version: Ceph 0.67.7 or 0.79
>>>
>>> Hello, everyone!
>>> I have configured federated gateways for several.
>>> Now I can sync files from master zone to slave zone.
>>> However I found that files with English filename could be  synced, but
>>> files
>>> with Chinese filename would not be synced.
>>> I compared the log between file with English filename and with Chinese
>>> filename in master zone's log.
>>> These two files are with name radosgw-agent.txt and
>>> radosgw-agent-测试用例-233.txt .
>>>
>>> radosgw-agent.txt in master zone's log
>>> 2014-04-16 15:10:20.883445 7fd1635be700  1 == starting new request
>>> req=0x12ea120 =
>>> 2014-04-16 15:10:20.883501 7fd1635be700  2 req 199:0.57::GET
>>> /ttt/rad%2Fradosgw-agent.txt::initializing
>>> 2014-04-16 15:10:20.883507 7fd1635be700 10 host=s3.ceph69.com
>>> rgw_dns_name=s3.ceph69.com
>>> 2014-04-16 15:10:20.883518 7fd1635be700 10 meta>> HTTP_X_AMZ_COPY_SOURCE
>>> 2014-04-16 15:10:20.883524 7fd1635be700 10 x>>
>>> x-amz-copy-source:ttt/rad/radosgw-agent.txt
>>> 2014-04-16 15:10:20.883560 7fd1635be700 10
>>> s->object=rad/radosgw-agent.txt
>>> s->bucket=ttt
>>> 2014-04-16 15:10:20.883572 7fd1635be700 20 FCGI_ROLE=RESPONDER
>>> 2014-04-16 15:10:20.883573 7fd1635be700 20
>>> SCRIPT_URL=/ttt/rad/radosgw-agent.txt
>>> 2014-04-16 15:10:20.883573 7fd1635be700 20
>>> SCRIPT_URI=http://s3.ceph69.com/ttt/rad/radosgw-agent.txt
>>> 2014-04-16 15:10:20.883574 7fd1635be700 20 HTTP_AUTHORIZATION=AWS
>>> 18GNN0DH1900H0L1LEBY:T23G/DqMa8KeIfJuv95XVRS4Hes=
>>> 2014-04-16 15:10:20.883575 7fd1635be700 20 SERVER_PORT_SECURE=443
>>> 2014-04-16 15:10:20.883575 7fd1635be700 20 HTTP_HOST=s3.ceph69.com
>>>
>>> radosgw-agent-测试用例-233.txt in master zone's log
>>> 2014-04-16 15:10:21.108608 7fd1739d8700  1 == starting new request
>>> req=0x126fe10 =
>>> 2014-04-16 15:10:21.108670 7fd1739d8700  2 req 200:0.63::GET
>>>
>>> /ttt/rad%2Fradosgw-agent-%FFE6%FFB5%FF8B%FFE8%FFAF%FF95%FFE7%FF94%FFA8%FFE4%FFBE%FF8B-233.txt::initializing
>>> 2014-04-16 15:10:21.108677 7fd1739d8700 10 host=s3.ceph69.com
>>> rgw_dns_name=s3.ceph69.com
>>> 2014-04-16 15:10:21.108687 7fd1739d8700 10 meta>> HTTP_X_AMZ_COPY_SOURCE
>>> 2014-04-16 15:10:21.108693 7fd1739d8700 10 x>>
>>>
>>> x-amz-copy-source:ttt/rad/radosgw-agent-%E6%B5%8B%E8%AF%95%E7%94%A8%E4%BE%8B-233.txt
>>> 2014-04-16 15:10:21.108714 7fd1739d8700 10
>>>
>>> s->object=rad/radosgw-agent-󆆆FE6󆆆FB5󆆆F8B󆆆FE8󆆆FAF󆆆F95󆆆FE7󆆆F94󆆆FA8󆆆FE4󆆆FBE󆆆F8B-233.txt
>>> s->bucket=ttt
>>> 2014-04-16 15:10:21.108738 7fd1739d8700  2 req 200:0.000131::GET
>>>
>>> /ttt/rad%2Fradosgw-agent-%FFE6%FFB5%FF8B%FFE8%FFAF%FF95%FFE7%FF94%FFA8%FFE4%FFBE%FF8B-233.txt::http
>>> status=400
>>> 2014-04-16 15:10:21.108921 7fd1739d8700  1 == req done req=0x126fe10
>>> http_status=400 ==
>>>
>>> The difference between them is where I bold.
>>> radosgw-agent.txt in master zone's log
>>> 2014-04-16 15:10:20.883572 7fd1635be700 20 FCGI_ROLE=RESPONDER
>>>
>>> radosgw-agent-测试用例-233.txt in master zone's log
>>> 2014-04-16 15:10:21.108738 7fd1739d8700  2 req 200:0.000131::GET
>>>
>>> /ttt/rad%2Fradosgw-agent-%FFE6%FFB5%FF8B%FFE8%FFAF%FF95%FFE7%FF94%FFA8%FFE4%FFBE%FF8B-233.txt::http
>>> status=400
>>>
>>> Therefor I don't know whether it's ceph's error or fastcgi's error.
>>> I attach the test files. Anyone can test it. Another two files are the
>>> full
>>> log about the test files.
>>>
>>> Thanks!
>>>
>>>
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>
>
>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Access denied error

[Yehuda Sadeh]

On Fri, Apr 25, 2014 at 1:03 AM, Punit Dambiwal  wrote:
> Hi Yehuda,
>
> Thanks for your help...that missing date error gone but still i am getting
> the access denied error :-
>
> -
> 2014-04-25 15:52:56.988025 7f00d37c6700  1 == starting new request
> req=0x237a090 =
> 2014-04-25 15:52:56.988072 7f00d37c6700  2 req 24:0.46::GET
> /admin/usage::initializing
> 2014-04-25 15:52:56.988077 7f00d37c6700 10 host=gateway.3linux.com
> rgw_dns_name=gateway.3linux.com
> 2014-04-25 15:52:56.988102 7f00d37c6700 20 FCGI_ROLE=RESPONDER
> 2014-04-25 15:52:56.988103 7f00d37c6700 20 SCRIPT_URL=/admin/usage
> 2014-04-25 15:52:56.988104 7f00d37c6700 20
> SCRIPT_URI=http://gateway.3linux.com/admin/usage
> 2014-04-25 15:52:56.988105 7f00d37c6700 20 HTTP_AUTHORIZATION=AWS
> KGXJJGKDM5G7G4CNKC7R:LC7S0twZdhtXA1XxthfMDsj5TgJpeKhZrloWa9WN
> 2014-04-25 15:52:56.988107 7f00d37c6700 20 HTTP_USER_AGENT=curl/7.22.0
> (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23
> librtmp/2.3
> 2014-04-25 15:52:56.988108 7f00d37c6700 20 HTTP_ACCEPT=*/*
> 2014-04-25 15:52:56.988109 7f00d37c6700 20 HTTP_HOST=gateway.3linux.com
> 2014-04-25 15:52:56.988110 7f00d37c6700 20 HTTP_DATE=Fri, 25 April 2014
> 07:50:00 GMT
> 2014-04-25 15:52:56.988111 7f00d37c6700 20 CONTENT_LENGTH=0
> 2014-04-25 15:52:56.988112 7f00d37c6700 20 PATH=/usr/local/bin:/usr/bin:/bin
> 2014-04-25 15:52:56.988113 7f00d37c6700 20 SERVER_SIGNATURE=
> 2014-04-25 15:52:56.988114 7f00d37c6700 20 SERVER_SOFTWARE=Apache/2.2.22
> (Ubuntu)
> 2014-04-25 15:52:56.988115 7f00d37c6700 20 SERVER_NAME=gateway.3linux.com
> 2014-04-25 15:52:56.988116 7f00d37c6700 20 SERVER_ADDR=117.18.79.110
> 2014-04-25 15:52:56.988117 7f00d37c6700 20 SERVER_PORT=80
> 2014-04-25 15:52:56.988117 7f00d37c6700 20 REMOTE_ADDR=122.166.115.191
> 2014-04-25 15:52:56.988118 7f00d37c6700 20 DOCUMENT_ROOT=/var/www
> 2014-04-25 15:52:56.988119 7f00d37c6700 20 SERVER_ADMIN=c...@3linux.com
> 2014-04-25 15:52:56.988120 7f00d37c6700 20
> SCRIPT_FILENAME=/var/www/s3gw.fcgi
> 2014-04-25 15:52:56.988120 7f00d37c6700 20 REMOTE_PORT=28840
> 2014-04-25 15:52:56.988121 7f00d37c6700 20 GATEWAY_INTERFACE=CGI/1.1
> 2014-04-25 15:52:56.988122 7f00d37c6700 20 SERVER_PROTOCOL=HTTP/1.1
> 2014-04-25 15:52:56.988123 7f00d37c6700 20 REQUEST_METHOD=GET
> 2014-04-25 15:52:56.988123 7f00d37c6700 20
> QUERY_STRING=page=admin¶ms=/usage&format=json
> 2014-04-25 15:52:56.988124 7f00d37c6700 20
> REQUEST_URI=/admin/usage?format=json
> 2014-04-25 15:52:56.988125 7f00d37c6700 20 SCRIPT_NAME=/admin/usage
> 2014-04-25 15:52:56.988126 7f00d37c6700  2 req 24:0.000101::GET
> /admin/usage::getting op
> 2014-04-25 15:52:56.988129 7f00d37c6700  2 req 24:0.000104::GET
> /admin/usage:get_usage:authorizing
> 2014-04-25 15:52:56.988141 7f00d37c6700 20 get_obj_state:
> rctx=0x7effbc004aa0 obj=.users:KGXJJGKDM5G7G4CNKC7R state=0x7effbc00e718
> s->prefetch_data=0
> 2014-04-25 15:52:56.988148 7f00d37c6700 10 moving
> .users+KGXJJGKDM5G7G4CNKC7R to cache LRU end
> 2014-04-25 15:52:56.988150 7f00d37c6700 10 cache get:
> name=.users+KGXJJGKDM5G7G4CNKC7R : hit
> 2014-04-25 15:52:56.988155 7f00d37c6700 20 get_obj_state: s->obj_tag was set
> empty
> 2014-04-25 15:52:56.988160 7f00d37c6700 10 moving
> .users+KGXJJGKDM5G7G4CNKC7R to cache LRU end
> 2014-04-25 15:52:56.988161 7f00d37c6700 10 cache get:
> name=.users+KGXJJGKDM5G7G4CNKC7R : hit
> 2014-04-25 15:52:56.988179 7f00d37c6700 20 get_obj_state:
> rctx=0x7effbc001ce0 obj=.users.uid:admin state=0x7effbc00ec58
> s->prefetch_data=0
> 2014-04-25 15:52:56.988185 7f00d37c6700 10 moving .users.uid+admin to cache
> LRU end
> 2014-04-25 15:52:56.988186 7f00d37c6700 10 cache get: name=.users.uid+admin
> : hit
> 2014-04-25 15:52:56.988190 7f00d37c6700 20 get_obj_state: s->obj_tag was set
> empty
> 2014-04-25 15:52:56.988193 7f00d37c6700 10 moving .users.uid+admin to cache
> LRU end
> 2014-04-25 15:52:56.988195 7f00d37c6700 10 cache get: name=.users.uid+admin
> : hit
> 2014-04-25 15:52:56.988236 7f00d37c6700 10 get_canon_resource():
> dest=/admin/usage
> 2014-04-25 15:52:56.988239 7f00d37c6700 10 auth_hdr:
> GET
>
>
> Fri, 25 April 2014 07:50:00 GMT
> /admin/usage
> 2014-04-25 15:52:56.988325 7f00d37c6700 15 calculated
> digest=nLKirQEEPeSS0Lzvr52NAB2phpA=
> 2014-04-25 15:52:56.988329 7f00d37c6700 15
> auth_sign=LC7S0twZdhtXA1XxthfMDsj5TgJpeKhZrloWa9WN
> 2014-04-25 15:52:56.988330 7f00d37c6700 15 compare=-34


Still signing issues. If you're manually constructing the auth header
you need to make it look like the above (copy pasted here):

> 2014-04-25 15:52:56.988239 7f00d37c6700 10 auth_hdr:
> GET
>
>
> Fri, 25 April 2014 07:50:00 GMT
> /admin/usage

Then you need to run hmac-sha1 on it, as described here:

http://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthentication.html

If you have any backslash in the key then you need to remove it, it's
just an escape character for representing slashes in json.

Yehuda
___
ceph-users 

Re: [ceph-users] Ceph mds laggy and failed assert in function replay mds/journal.cc

[Gregory Farnum]

Hmm, it looks like your on-disk SessionMap is horrendously out of
date. Did your cluster get full at some point?

In any case, we're working on tools to repair this now but they aren't
ready for use yet. Probably the only thing you could do is create an
empty sessionmap with a higher version than the ones the journal
refers to, but that might have other fallout effects...
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com


On Fri, Apr 25, 2014 at 2:57 AM, Mohd Bazli Ab Karim
 wrote:
> More logs. I ran ceph-mds  with debug-mds=20.
>
> -2> 2014-04-25 17:47:54.839672 7f0d6f3f0700 10 mds.0.journal EMetaBlob.replay 
> inotable tablev 4316124 <= table 4317932
> -1> 2014-04-25 17:47:54.839674 7f0d6f3f0700 10 mds.0.journal EMetaBlob.replay 
> sessionmap v8632368 -(1|2) == table 7239603 prealloc [141df86~1] used 
> 141db9e
>   0> 2014-04-25 17:47:54.840733 7f0d6f3f0700 -1 mds/journal.cc: In function 
> 'void EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)' thread 
> 7f0d6f3f0700 time 2014-04-25 17:47:54.839688 mds/journal.cc: 1303: FAILED 
> assert(session)
>
> Please look at the attachment for more details.
>
> Regards,
> Bazli
>
> From: Mohd Bazli Ab Karim
> Sent: Friday, April 25, 2014 12:26 PM
> To: 'ceph-de...@vger.kernel.org'; ceph-users@lists.ceph.com
> Subject: Ceph mds laggy and failed assert in function replay mds/journal.cc
>
> Dear Ceph-devel, ceph-users,
>
> I am currently facing issue with my ceph mds server. Ceph-mds daemon does not 
> want to bring up back.
> Tried running that manually with ceph-mds -i mon01 -d but it shows that it 
> stucks at failed assert(session) line 1303 in mds/journal.cc and aborted.
>
> Can someone shed some light in this issue.
> ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60)
>
> Let me know if I need to send log with debug enabled.
>
> Regards,
> Bazli
>
> 
> DISCLAIMER:
>
>
> This e-mail (including any attachments) is for the addressee(s) only and may 
> be confidential, especially as regards personal data. If you are not the 
> intended recipient, please note that any dealing, review, distribution, 
> printing, copying or use of this e-mail is strictly prohibited. If you have 
> received this email in error, please notify the sender immediately and delete 
> the original message (including any attachments).
>
>
> MIMOS Berhad is a research and development institution under the purview of 
> the Malaysian Ministry of Science, Technology and Innovation. Opinions, 
> conclusions and other information in this e-mail that do not relate to the 
> official business of MIMOS Berhad and/or its subsidiaries shall be understood 
> as neither given nor endorsed by MIMOS Berhad and/or its subsidiaries and 
> neither MIMOS Berhad nor its subsidiaries accepts responsibility for the 
> same. All liability arising from or in connection with computer viruses 
> and/or corrupted e-mails is excluded to the fullest extent permitted by law.
>
>
> --
> -
> -
> DISCLAIMER:
>
> This e-mail (including any attachments) is for the addressee(s)
> only and may contain confidential information. If you are not the
> intended recipient, please note that any dealing, review,
> distribution, printing, copying or use of this e-mail is strictly
> prohibited. If you have received this email in error, please notify
> the sender  immediately and delete the original message.
> MIMOS Berhad is a research and development institution under
> the purview of the Malaysian Ministry of Science, Technology and
> Innovation. Opinions, conclusions and other information in this e-
> mail that do not relate to the official business of MIMOS Berhad
> and/or its subsidiaries shall be understood as neither given nor
> endorsed by MIMOS Berhad and/or its subsidiaries and neither
> MIMOS Berhad nor its subsidiaries accepts responsibility for the
> same. All liability arising from or in connection with computer
> viruses and/or corrupted e-mails is excluded to the fullest extent
> permitted by law.
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] mon osd min down reports

[Craig Lewis]

I was reading about mon osd min down reports at 
http://ceph.com/docs/master/rados/configuration/mon-osd-interaction/, 
and I had a question.


Are mon osd min down reporters and mon osd min down reports both 
required to mark an OSD down, or just one?


For example, if I set
[global]
mon osd min down reporters = 9
  mon osd min down reports = 3

Does that mean that 9 OSDs have to report an OSD down AND one of them 
has to report it 3 times?  Or does it mean a single OSD reporting 
another OSD is down 3 times will still mark it down?


In the first case, reports = 3 seems like a good number.  In the 2nd 
case, it seems that reports should be (reporters * 3).



I'm thinking that in real clusters, reporters should be

 * larger than the largest number of OSDs in a single node
 * less than (Num OSDs / 2)
 * less than the smallest number of PGs per OSD

Those rules might conflict for 1-3 node clusters, but shouldn't cause 
problems for larger clusters.



Thoughts?

--

*Craig Lewis*
Senior Systems Engineer
Office +1.714.602.1309
Email cle...@centraldesktop.com 

*Central Desktop. Work together in ways you never thought possible.*
Connect with us Website   | Twitter 
  | Facebook 
  | LinkedIn 
  | Blog 



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] SSD journal overload?

[Craig Lewis]

I am not able to do a dd test on the SSDs since it's not mounted as 
filesystem, but dd on the OSD (non-SSD) drives gives normal result.


Since you have free space on the SSDs, you could add a 3rd 10G partition 
to one of the SSDs.  Then you could put a filesystem on that partition, 
or just dd the third partition directly with dd if=/dev/zero 
of=/dev/sdf3 bs=... count=...


Be careful not to make a typo. of=/dev/sdf or of=/dev/sdf2 would destroy 
your other journals.



What does the manufacturer claim for the SSD performance specs?

--
Craig
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Bit of trouble using the S3 api for adminops

[Drew Weaver]

Greetings, I got a ceph test cluster setup this week and I thought it would be 
neat if I could write a php script that let me start working with the adminops 
API.

I did some research to figure out how to correctly 'authorize' in the AWS 
fashion and wrote this little script.

http://host.com/";;
$url_string = "secretadmin/usage?format=json";
$the_url = "$url_prefix$url_string";
$AWSAccessKeyId = "accesskey";
$AWSSecretAccessKey = "secretaccesskey";
$Expires = time();
$HTTPVerb = "GET";
$ContentMD5 = "";
$ContentType = "";
$CanonicalizedAmzHeaders = "";
$CanonicalizedResource = $url_string;
$string_to_sign = $HTTPVerb . "\n" . $ContentMD5 . "\n" . $ContentType . "\n" . 
$Expires . "\n" . $CanonicalizedAmzHeaders . $CanonicalizedResource;
$sig = hash_hmac("sha1", $string_to_sign, $AWSSecretAccessKey);
$bsf = hex2b64($sig);
$ch = curl_init();
$head_array = Array("Authorization: AWS $AWSAccessKeyId:$bsf", "Host: 
host.com");
curl_setopt($ch, CURLOPT_HTTPHEADER, $head_array);
curl_setopt($ch, CURLOPT_URL,$the_url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$data = curl_exec($ch);
$err = curl_error($ch);
echo "making call to $the_url\n";
if ($err) {
die("$err");
} else {
die("$data");
}

Basically all I am trying to do is view the 'usage' in JSON format and no 
matter what I do it Returns AccessDenied

I did create a radosgw user and said user does have read for usage.

Can anyone give me a push in the right direction?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mon osd min down reports

[Gregory Farnum]

The monitor requires at least  number of reports, from a set of
OSDs whose size is at least . So with 9 reporters and 3 reports,
it would wait until 9 OSDs had reported an OSD down (basically ignoring the
reports setting, as it is smaller).
-Greg

On Friday, April 25, 2014, Craig Lewis  wrote:

>  I was reading about mon osd min down reports at
> http://ceph.com/docs/master/rados/configuration/mon-osd-interaction/, and
> I had a question.
>
> Are mon osd min down reporters and mon osd min down reports both required
> to mark an OSD down, or just one?
>
> For example, if I set
> [global]
>   mon osd min down reporters = 9
>   mon osd min down reports = 3
>
> Does that mean that 9 OSDs have to report an OSD down AND one of them has
> to report it 3 times?  Or does it mean a single OSD reporting another OSD
> is down 3 times will still mark it down?
>
> In the first case, reports = 3 seems like a good number.  In the 2nd case,
> it seems that reports should be (reporters * 3).
>
>
> I'm thinking that in real clusters, reporters should be
>
>- larger than the largest number of OSDs in a single node
>- less than (Num OSDs / 2)
>- less than the smallest number of PGs per OSD
>
> Those rules might conflict for 1-3 node clusters, but shouldn't cause
> problems for larger clusters.
>
> Thoughts?
>
>  --
>
>  *Craig Lewis*
>  Senior Systems Engineer
> Office +1.714.602.1309
> Email 
> cle...@centraldesktop.com
>
>  *Central Desktop. Work together in ways you never thought possible.*
>  Connect with us   Website   |  
> Twitter |
> Facebook   |  
> LinkedIn |
> Blog 
>


-- 
Software Engineer #42 @ http://inktank.com | http://ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] packages for Trusty

[Travis Rhoden]

Are there packages for Trusty being built yet?

I don't see it listed at http://ceph.com/debian-emperor/dists/

Thanks,

 - Travis
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] packages for Trusty

[Drew Weaver]

You can actually just install it using the Ubuntu packages. I did it yesterday 
on Trusty.

Thanks,
-Drew


From: ceph-users-boun...@lists.ceph.com 
[mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Travis Rhoden
Sent: Friday, April 25, 2014 3:06 PM
To: ceph-users
Subject: [ceph-users] packages for Trusty

Are there packages for Trusty being built yet?

I don't see it listed at http://ceph.com/debian-emperor/dists/

Thanks,
 - Travis
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] packages for Trusty

[Sebastien]

Well as far as I know trusty has 0.79 and will get firefly as soon as it's ready so I'm not sure if it's that urgent. Precise repo should work fine. 
My 2 cents

Sébastien Han
Cloud Engineer
"Always give 100%. Unless you're giving blood.”
Phone: +33 (0)1 49 70 99 72
Mail: sebastien@enovance.com
Address : 11 bis, rue Roquépine 75008 Paris
Web : www.enovance.com - Twitter : @enovance
On Fri, Apr 25, 2014 at 9:05 PM, Travis Rhoden  wrote:Are there packages for Trusty being built yet?I don't see it listed at http://ceph.com/debian-emperor/dists/Thanks, - Travis



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] packages for Trusty

[Cédric Lemarchand]

Yes, juste apt-get install ceph ;-)

Cheers

--
Cédric Lemarchand

> Le 25 avr. 2014 à 21:07, Drew Weaver  a écrit :
> 
> You can actually just install it using the Ubuntu packages. I did it 
> yesterday on Trusty.
>  
> Thanks,
> -Drew
>  
>  
> From: ceph-users-boun...@lists.ceph.com 
> [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Travis Rhoden
> Sent: Friday, April 25, 2014 3:06 PM
> To: ceph-users
> Subject: [ceph-users] packages for Trusty
>  
> Are there packages for Trusty being built yet?
> 
> I don't see it listed at http://ceph.com/debian-emperor/dists/
> 
> Thanks,
> 
>  - Travis
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] bandwidth with Ceph - v0.59 (Bobtail)

[Gregory Farnum]

Bobtail is really too old to draw any meaningful conclusions from; why
did you choose it?

That's not to say that performance on current code will be better
(though it very much might be), but the internal architecture has
changed in some ways that will be particularly important for the futex
profiling you did, and are probably important for these throughput
results as well.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com


On Fri, Apr 25, 2014 at 1:38 PM, Xing  wrote:
> Hi,
>
> I also did a few other experiments, trying to get what the maximum bandwidth 
> we can get from each data disk. The output is not encouraging: for disks that 
> can provide 150 MB/s block-level sequential read bandwidths, we can only get 
> about 90MB/s from each disk. Something that is particular interesting is that 
> the replica size also affects the bandwidth we could get from the cluster. It 
> seems that there is no such observation/conversations in the Ceph community 
> and I think it may be helpful to share my findings.
>
> The experiment was run with two d820 machines in Emulab at University of 
> Utah. One is used as the data node and the other is used as the client. They 
> are connected by a 10 GB/s Ethernet. The data node has 7 disks, one for OS 
> and the rest 6 for OSDs. For the rest 6 disks, we use one for journal and the 
> other for data. Thus in total we have 3 OSDs. The network bandwidth is 
> sufficient to support reading from 3 disks in full bandwidth.
>
> I varied the read-ahead size for the rbd block device (exp1), osd op threads 
> for each osd (exp2), varied the replica size (exp3), and object size (exp4). 
> The most interesting is varying the replica size. As I varied replica size 
> from 1, to 2 and to 3, the aggregated bandwidth dropped from 267 MB/s to 211 
> and 180. The reason for the drop I believe is as we increase the number of 
> replicas, we store more data into each OSD. then when we need to read it 
> back, we have to read from a larger range (more seeks). The fundamental 
> problem is likely because we are doing replication synchronously, and thus 
> layout object files in a raid 10 - near format, rather than the far format. 
> For the difference between the near format and far format for raid 10, you 
> could have a look at the link provided below.
>
> http://lxr.free-electrons.com/source/Documentation/device-mapper/dm-raid.txt
>
> For results about other experiments, you could download my slides at the link 
> provided below.
> http://www.cs.utah.edu/~xinglin/slides/ceph-bandiwdth-exp.pptx
>
>
> I do not know why Ceph only gets about 60% of the disk bandwidth. To do a 
> comparison, I ran tar to read every rbd object files to create a tarball and 
> see how much bandwidth I can get from this workload. Interestingly, the tar 
> workload actually gets a higher bandwidth (80% of block level bandwidth), 
> even though it is accessing the disk more randomly (tar reads each object 
> file in a dir sequentially while the object files were created in a different 
> order.). For more detail, please goto my blog to have a read.
> http://xinglin-system.blogspot.com/2014/04/ceph-lab-note-1-disk-read-bandwidth-in.html
>
> Here are a few questions.
> 1. What are the maximum bandwidth people can get from each disk? I found 
> Jiangang from Intel also reported 57% efficiency for disk bandwidth. He 
> suggested one reason: interference among so many sequential read workloads. I 
> agree but when I tried to run with one single workload, I still do not get a 
> higher efficiency.
> 2. If the efficiency is about 60%, what are the reasons that cause this? 
> Could it be because of the locks (futex as I mentioned in my previous email) 
> or anything else?
>
> Thanks very much for any feedback.
>
> Thanks,
> Xing
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] packages for Trusty

[Travis Rhoden]

Thanks guys.  I don't know why I didn't try that.  I guess just too much
habit of setting up the additional repo.  =)


On Fri, Apr 25, 2014 at 4:09 PM, Cédric Lemarchand  wrote:

> Yes, juste apt-get install ceph ;-)
>
> Cheers
>
> --
> Cédric Lemarchand
>
> Le 25 avr. 2014 à 21:07, Drew Weaver  a écrit :
>
>  You can actually just install it using the Ubuntu packages. I did it
> yesterday on Trusty.
>
>
>
> Thanks,
>
> -Drew
>
>
>
>
>
> *From:* ceph-users-boun...@lists.ceph.com [
> mailto:ceph-users-boun...@lists.ceph.com]
> *On Behalf Of *Travis Rhoden
> *Sent:* Friday, April 25, 2014 3:06 PM
> *To:* ceph-users
> *Subject:* [ceph-users] packages for Trusty
>
>
>
> Are there packages for Trusty being built yet?
>
> I don't see it listed at http://ceph.com/debian-emperor/dists/
>
> Thanks,
>
>  - Travis
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] packages for Trusty

[Craig Lewis]

Using the Emperor builds for Precise seems to work on Trusty.  I just 
put a hold on all of the ceph, rados, and apache packages before the 
release upgrade.


It makes me nervous though.  I haven't stressed it much, and I don't 
really want to roll it out to production.


I would like to see Emperor builds for Trusty, so I can get started 
rolling out Trusty independently of Firefly.  Changing one thing at a 
time is invaluable when bad things start happening.





*Craig Lewis*
Senior Systems Engineer
Office +1.714.602.1309
Email cle...@centraldesktop.com 

*Central Desktop. Work together in ways you never thought possible.*
Connect with us Website   | Twitter 
  | Facebook 
  | LinkedIn 
  | Blog 



On 4/25/14 12:10 , Sebastien wrote:


Well as far as I know trusty has 0.79 and will get firefly as soon as 
it's ready so I'm not sure if it's that urgent. Precise repo should 
work fine.


My 2 cents


Sébastien Han
Cloud Engineer

"Always give 100%. Unless you're giving blood."

Phone: +33 (0)1 49 70 99 72
Mail: sebastien@enovance.com
Address : 11 bis, rue Roquépine 75008 Paris
Web : www.enovance.com - Twitter : @enovance


On Fri, Apr 25, 2014 at 9:05 PM, Travis Rhoden > wrote:


Are there packages for Trusty being built yet?

I don't see it listed at http://ceph.com/debian-emperor/dists/

Thanks,

- Travis



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] bandwidth with Ceph - v0.59 (Bobtail)

[Mark Nelson]

For what it's worth, I've been able to achieve up to around 120MB/s with 
btrfs before things fragment.


Mark

On 04/25/2014 03:59 PM, Xing wrote:

Hi Gregory,

Thanks very much for your quick reply. When I started to look into Ceph, 
Bobtail was the latest stable release and that was why I picked that version 
and started to make a few modifications. I have not ported my changes to 0.79 
yet. The plan is if v-0.79 can provide a higher disk bandwidth efficiency, I 
will switch to 0.79. Unfortunately, that does not seem to be the case.

The futex trace was done with version 0.79, not 0.59. I did a profile in 0.59 
too. There are some improvements, such as the introduction of fd cache. But 
lots of futex calls are still there in v-0.79. I also measured the maximum 
bandwidth from each disk we can get in Version 0.79. It does not improve 
significantly: we can still only get 90~100 MB/s from each disk.

Thanks,
Xing


On Apr 25, 2014, at 2:42 PM, Gregory Farnum  wrote:


Bobtail is really too old to draw any meaningful conclusions from; why
did you choose it?

That's not to say that performance on current code will be better
(though it very much might be), but the internal architecture has
changed in some ways that will be particularly important for the futex
profiling you did, and are probably important for these throughput
results as well.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com


On Fri, Apr 25, 2014 at 1:38 PM, Xing  wrote:

Hi,

I also did a few other experiments, trying to get what the maximum bandwidth we 
can get from each data disk. The output is not encouraging: for disks that can 
provide 150 MB/s block-level sequential read bandwidths, we can only get about 
90MB/s from each disk. Something that is particular interesting is that the 
replica size also affects the bandwidth we could get from the cluster. It seems 
that there is no such observation/conversations in the Ceph community and I 
think it may be helpful to share my findings.

The experiment was run with two d820 machines in Emulab at University of Utah. 
One is used as the data node and the other is used as the client. They are 
connected by a 10 GB/s Ethernet. The data node has 7 disks, one for OS and the 
rest 6 for OSDs. For the rest 6 disks, we use one for journal and the other for 
data. Thus in total we have 3 OSDs. The network bandwidth is sufficient to 
support reading from 3 disks in full bandwidth.

I varied the read-ahead size for the rbd block device (exp1), osd op threads 
for each osd (exp2), varied the replica size (exp3), and object size (exp4). 
The most interesting is varying the replica size. As I varied replica size from 
1, to 2 and to 3, the aggregated bandwidth dropped from 267 MB/s to 211 and 
180. The reason for the drop I believe is as we increase the number of 
replicas, we store more data into each OSD. then when we need to read it back, 
we have to read from a larger range (more seeks). The fundamental problem is 
likely because we are doing replication synchronously, and thus layout object 
files in a raid 10 - near format, rather than the far format. For the 
difference between the near format and far format for raid 10, you could have a 
look at the link provided below.

http://lxr.free-electrons.com/source/Documentation/device-mapper/dm-raid.txt

For results about other experiments, you could download my slides at the link 
provided below.
http://www.cs.utah.edu/~xinglin/slides/ceph-bandiwdth-exp.pptx


I do not know why Ceph only gets about 60% of the disk bandwidth. To do a 
comparison, I ran tar to read every rbd object files to create a tarball and 
see how much bandwidth I can get from this workload. Interestingly, the tar 
workload actually gets a higher bandwidth (80% of block level bandwidth), even 
though it is accessing the disk more randomly (tar reads each object file in a 
dir sequentially while the object files were created in a different order.). 
For more detail, please goto my blog to have a read.
http://xinglin-system.blogspot.com/2014/04/ceph-lab-note-1-disk-read-bandwidth-in.html

Here are a few questions.
1. What are the maximum bandwidth people can get from each disk? I found 
Jiangang from Intel also reported 57% efficiency for disk bandwidth. He 
suggested one reason: interference among so many sequential read workloads. I 
agree but when I tried to run with one single workload, I still do not get a 
higher efficiency.
2. If the efficiency is about 60%, what are the reasons that cause this? Could 
it be because of the locks (futex as I mentioned in my previous email) or 
anything else?

Thanks very much for any feedback.

Thanks,
Xing




--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org

Re: [ceph-users] bandwidth with Ceph - v0.59 (Bobtail)

[Mark Nelson]

I don't have any recent results published, but you can see some of the 
older results from bobtail here:


http://ceph.com/performance-2/argonaut-vs-bobtail-performance-preview/

Specifically, look at the 256 concurrent 4MB rados bench tests.  In a 6 
disk, 2 SSD configuration we could push about 800MB/s for writes (no 
replication) and around 600-700MB/s for reads with BTRFS.  On this 
controller using a RAID0 configuration with WB cache helps quite a bit, 
but in other tests I've seen similar results with a 9207-8i that doesn't 
have WB cache when BTRFS filestores and SSD journals are used.


Regarding the drives, they can do somewhere around 140-150MB/s large 
block writes with fio.


Replication definitely adds additional latency so aggregate write 
throughput goes down, though it seems the penalty is worst after the 
first replica and doesn't hurt as much with subsequent ones.


Mark


On 04/25/2014 04:50 PM, Xing Lin wrote:

Hi Mark,

That seems pretty good. What is the block level sequential read
bandwidth of your disks? What configuration did you use? What was the
replica size, read_ahead for your rbds and what were the number of
workloads you used? I used btrfs in my experiments as well.

Thanks,
Xing

On 04/25/2014 03:36 PM, Mark Nelson wrote:

For what it's worth, I've been able to achieve up to around 120MB/s
with btrfs before things fragment.

Mark

On 04/25/2014 03:59 PM, Xing wrote:

Hi Gregory,

Thanks very much for your quick reply. When I started to look into
Ceph, Bobtail was the latest stable release and that was why I picked
that version and started to make a few modifications. I have not
ported my changes to 0.79 yet. The plan is if v-0.79 can provide a
higher disk bandwidth efficiency, I will switch to 0.79.
Unfortunately, that does not seem to be the case.

The futex trace was done with version 0.79, not 0.59. I did a profile
in 0.59 too. There are some improvements, such as the introduction of
fd cache. But lots of futex calls are still there in v-0.79. I also
measured the maximum bandwidth from each disk we can get in Version
0.79. It does not improve significantly: we can still only get 90~100
MB/s from each disk.

Thanks,
Xing


On Apr 25, 2014, at 2:42 PM, Gregory Farnum  wrote:


Bobtail is really too old to draw any meaningful conclusions from; why
did you choose it?

That's not to say that performance on current code will be better
(though it very much might be), but the internal architecture has
changed in some ways that will be particularly important for the futex
profiling you did, and are probably important for these throughput
results as well.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com


On Fri, Apr 25, 2014 at 1:38 PM, Xing  wrote:

Hi,

I also did a few other experiments, trying to get what the maximum
bandwidth we can get from each data disk. The output is not
encouraging: for disks that can provide 150 MB/s block-level
sequential read bandwidths, we can only get about 90MB/s from each
disk. Something that is particular interesting is that the replica
size also affects the bandwidth we could get from the cluster. It
seems that there is no such observation/conversations in the Ceph
community and I think it may be helpful to share my findings.

The experiment was run with two d820 machines in Emulab at
University of Utah. One is used as the data node and the other is
used as the client. They are connected by a 10 GB/s Ethernet. The
data node has 7 disks, one for OS and the rest 6 for OSDs. For the
rest 6 disks, we use one for journal and the other for data. Thus
in total we have 3 OSDs. The network bandwidth is sufficient to
support reading from 3 disks in full bandwidth.

I varied the read-ahead size for the rbd block device (exp1), osd
op threads for each osd (exp2), varied the replica size (exp3), and
object size (exp4). The most interesting is varying the replica
size. As I varied replica size from 1, to 2 and to 3, the
aggregated bandwidth dropped from 267 MB/s to 211 and 180. The
reason for the drop I believe is as we increase the number of
replicas, we store more data into each OSD. then when we need to
read it back, we have to read from a larger range (more seeks). The
fundamental problem is likely because we are doing replication
synchronously, and thus layout object files in a raid 10 - near
format, rather than the far format. For the difference between the
near format and far format for raid 10, you could have a look at
the link provided below.

http://lxr.free-electrons.com/source/Documentation/device-mapper/dm-raid.txt


For results about other experiments, you could download my slides
at the link provided below.
http://www.cs.utah.edu/~xinglin/slides/ceph-bandiwdth-exp.pptx


I do not know why Ceph only gets about 60% of the disk bandwidth.
To do a comparison, I ran tar to read every rbd object files to
create a tarball and see how much bandwidth I can get from this
workload. Interestingly, the tar workload actually gets a higher
b

Re: [ceph-users] Ceph mds laggy and failed assert in function replay mds/journal.cc

[Luke Jing Yuan]

HI Greg,

Actually the cluster that my colleague and I is working is rather new and still 
have plenty of space left (less than 7% used). What we noticed just before the 
MDS gave us this problem, was a temporary network issue in the data center so 
we are not sure that could have been the root cause.

Anyway, how may we create an empty sessionmap?

Regards,
Luke

-Original Message-
From: ceph-users-boun...@lists.ceph.com 
[mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Gregory Farnum
Sent: Saturday, 26 April, 2014 2:02 AM
To: Mohd Bazli Ab Karim
Cc: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Ceph mds laggy and failed assert in function replay 
mds/journal.cc

Hmm, it looks like your on-disk SessionMap is horrendously out of date. Did 
your cluster get full at some point?

In any case, we're working on tools to repair this now but they aren't ready 
for use yet. Probably the only thing you could do is create an empty sessionmap 
with a higher version than the ones the journal refers to, but that might have 
other fallout effects...
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com


On Fri, Apr 25, 2014 at 2:57 AM, Mohd Bazli Ab Karim  
wrote:
> More logs. I ran ceph-mds  with debug-mds=20.
>
> -2> 2014-04-25 17:47:54.839672 7f0d6f3f0700 10 mds.0.journal
> -2> EMetaBlob.replay inotable tablev 4316124 <= table 4317932
> -1> 2014-04-25 17:47:54.839674 7f0d6f3f0700 10 mds.0.journal
> -1> EMetaBlob.replay sessionmap v8632368 -(1|2) == table 7239603
> -1> prealloc [141df86~1] used 141db9e
>   0> 2014-04-25 17:47:54.840733 7f0d6f3f0700 -1 mds/journal.cc: In
> function 'void EMetaBlob::replay(MDS*, LogSegment*, MDSlaveUpdate*)'
> thread 7f0d6f3f0700 time 2014-04-25 17:47:54.839688 mds/journal.cc:
> 1303: FAILED assert(session)
>
> Please look at the attachment for more details.
>
> Regards,
> Bazli
>
> From: Mohd Bazli Ab Karim
> Sent: Friday, April 25, 2014 12:26 PM
> To: 'ceph-de...@vger.kernel.org'; ceph-users@lists.ceph.com
> Subject: Ceph mds laggy and failed assert in function replay
> mds/journal.cc
>
> Dear Ceph-devel, ceph-users,
>
> I am currently facing issue with my ceph mds server. Ceph-mds daemon does not 
> want to bring up back.
> Tried running that manually with ceph-mds -i mon01 -d but it shows that it 
> stucks at failed assert(session) line 1303 in mds/journal.cc and aborted.
>
> Can someone shed some light in this issue.
> ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60)
>
> Let me know if I need to send log with debug enabled.
>
> Regards,
> Bazli
>


DISCLAIMER:


This e-mail (including any attachments) is for the addressee(s) only and may be 
confidential, especially as regards personal data. If you are not the intended 
recipient, please note that any dealing, review, distribution, printing, 
copying or use of this e-mail is strictly prohibited. If you have received this 
email in error, please notify the sender immediately and delete the original 
message (including any attachments).


MIMOS Berhad is a research and development institution under the purview of the 
Malaysian Ministry of Science, Technology and Innovation. Opinions, conclusions 
and other information in this e-mail that do not relate to the official 
business of MIMOS Berhad and/or its subsidiaries shall be understood as neither 
given nor endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS 
Berhad nor its subsidiaries accepts responsibility for the same. All liability 
arising from or in connection with computer viruses and/or corrupted e-mails is 
excluded to the fullest extent permitted by law.

--
-
-
DISCLAIMER: 

This e-mail (including any attachments) is for the addressee(s) 
only and may contain confidential information. If you are not the 
intended recipient, please note that any dealing, review, 
distribution, printing, copying or use of this e-mail is strictly 
prohibited. If you have received this email in error, please notify 
the sender  immediately and delete the original message. 
MIMOS Berhad is a research and development institution under 
the purview of the Malaysian Ministry of Science, Technology and 
Innovation. Opinions, conclusions and other information in this e-
mail that do not relate to the official business of MIMOS Berhad 
and/or its subsidiaries shall be understood as neither given nor 
endorsed by MIMOS Berhad and/or its subsidiaries and neither 
MIMOS Berhad nor its subsidiaries accepts responsibility for the 
same. All liability arising from or in connection with computer 
viruses and/or corrupted e-mails is excluded to the fullest extent 
permitted by law.


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Copying RBD images between clusters?

[Vladislav Gorbunov]

rbd -m mon-cluster1 export rbd/one-1 - | rbd -m mon-cluster2 - rbd/one-1

пятница, 25 апреля 2014 г. пользователь Brian Rak написал:

> Is there a recommended way to copy an RBD image between two different
> clusters?
>
> My initial thought was 'rbd export - | ssh "rbd import -"', but I'm not
> sure if there's a more efficient way.
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Copying RBD images between clusters?

[Vladislav Gorbunov]

rbd -m mon-cluster1 export rbd/one-1 - | rbd -m mon-cluster2 import -
rbd/one-1

пятница, 25 апреля 2014 г. пользователь Brian Rak написал:

> Is there a recommended way to copy an RBD image between two different
> clusters?
>
> My initial thought was 'rbd export - | ssh "rbd import -"', but I'm not
> sure if there's a more efficient way.
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com