Re: [ceph-users] Cluster never reaching clean after osd out
- Mail original -
> I have a Cluster of 3 hosts, running Debian wheezy and Backports Kernel
> 3.16.0-0.bpo.4-amd64.
> For testing I did a
> ~# ceph osd out 20
> from a clean state.
> Ceph starts rebalancing, watching ceph -w one sees changing pgs stuck unclean
> to get up and then go down to about 11.
> Short after that the cluster keeps stuck forever in this state:
> health HEALTH_WARN 68 pgs stuck unclean; recovery 450/169647 objects degraded
> (0.265%); 3691/169647 objects misplaced (2.176%)
> According to the documentation at
> http://ceph.com/docs/master/rados/operations/add-or-rm-osds/ the Cluster
> should reach a clean state after an osd out.
> What am I doing wrong?
Hi Yves and Cephers,
I have a cluster with 6 nodes and 36 OSD. I have the same pb :
cluster 1d0503fb-36d0-4dbc-aabe-a2a0709163cd
health HEALTH_WARN 76 pgs stuck unclean; recovery 1/624 objects degraded
(0.160%); 7/624 objects misplaced (1.122%)
monmap e6: 6 mons
osdmap e616: 36 osds: 36 up, 35 in
pgmap v16344: 2048 pgs, 1 pools, 689 MB data, 208 objects
178 GB used, 127 TB / 127 TB avail
1/624 objects degraded (0.160%); 7/624 objects misplaced (1.122%)
76 active+remapped
1972 active+clean
After 'out' osd.15, ceph didn't return to health ok , and get misplaced object
... :-/
I noticed that this happen when i use a replicated 3 pool. When the pool use a
replicated 2, ceph returned to health ok... Have you try with a replicated 2
pool ?
In the same way, I wonder why he does not return to the status ok
CEPH OSD TREE
# id weight type name up/down reweight
-1000 144 root default
-200 48 datacenter mo
-133 48 rack mom02
-4 24 host mom02h01
12 4 osd.12 up 1
13 4 osd.13 up 1
14 4 osd.14 up 1
16 4 osd.16 up 1
17 4 osd.17 up 1
15 4 osd.15 up 0
-5 24 host mom02h02
18 4 osd.18 up 1
19 4 osd.19 up 1
20 4 osd.20 up 1
21 4 osd.21 up 1
22 4 osd.22 up 1
23 4 osd.23 up 1
-202 48 datacenter me
-135 48 rack mem04
-6 24 host mem04h01
24 4 osd.24 up 1
25 4 osd.25 up 1
26 4 osd.26 up 1
27 4 osd.27 up 1
28 4 osd.28 up 1
29 4 osd.29 up 1
-7 24 host mem04h02
30 4 osd.30 up 1
31 4 osd.31 up 1
32 4 osd.32 up 1
33 4 osd.33 up 1
34 4 osd.34 up 1
35 4 osd.35 up 1
-201 48 datacenter li
-134 48 rack lis04
-2 24 host lis04h01
0 4 osd.0 up 1
2 4 osd.2 up 1
3 4 osd.3 up 1
4 4 osd.4 up 1
5 4 osd.5 up 1
1 4 osd.1 up 1
-3 24 host lis04h02
6 4 osd.6 up 1
7 4 osd.7 up 1
8 4 osd.8 up 1
9 4 osd.9 up 1
10 4 osd.10 up 1
11 4 osd.11 up 1
Crushmap
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
# devices
device 0 osd.0
device 1 osd.1
device 2 osd.2
device 3 osd.3
device 4 osd.4
device 5 osd.5
device 6 osd.6
device 7 osd.7
device 8 osd.8
device 9 osd.9
device 10 osd.10
device 11 osd.11
device 12 osd.12
device 13 osd.13
device 14 osd.14
device 15 osd.15
device 16 osd.16
device 17 osd.17
device 18 osd.18
device 19 osd.19
device 20 osd.20
device 21 osd.21
device 22 osd.22
device 23 osd.23
device 24 osd.24
device 25 osd.25
device 26 osd.26
device 27 osd.27
device 28 osd.28
device 29 osd.29
device 30 osd.30
device 31 osd.31
device 32 osd.32
device 33 osd.33
device 34 osd.34
device 35 osd.35
# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root
# buckets
host lis04h01 {
id -2 # do not change unnecessarily
# weight 24.000
alg straw
hash 0 # rjenkins1
item osd.0 weight 4.000
item osd.2 weight 4.000
item osd.3 weight 4.000
item osd.4 weight 4.000
item osd.5 weight 4.000
item osd.1 weight 4.000
}
host lis04h02 {
id -3 # do not change unnecessarily
# weight 24.000
alg straw
hash 0 # rjenkins1
item osd.6 weight 4.000
item osd.7 weight 4.000
item osd.8 weight 4.000
item osd.9 weight 4.000
item osd.10 weight 4.000
item osd.11 weight 4.000
}
host mom02h01 {
id -4 # do not change unnecessarily
# weight 24.000
alg straw
hash 0 # rjenkins1
item osd.12 weight 4.000
item osd.13 weight 4.000
item osd.14 weight 4.000
item osd.16 weight 4.000
item osd.17 weight 4.000
item osd.15 weight 4.000
}
host mom02h02 {
id -5 # do not change unnecessarily
# weight 24.000
alg straw
hash 0 # rjenkins1
item osd.18 weight 4.000
item osd.19 weight 4.000
item osd.20 weight 4.000
item osd.21 weight 4.000
item osd.22 weight 4.000
item osd.23 weight 4.000
}
host mem04h01 {
id -6 # do not change unnecessarily
# weight 24.000
alg straw
hash 0 # rjenkins1
item osd.24 weight 4.000
item osd.25 weight 4.000
item osd.26 weight 4.000
item osd.27 weight 4.000
item osd.28 weight 4.000
item osd.29 weight 4.000
}
host mem04h02 {
id -7 # do not change unnecessarily
# weight 24.000
alg straw
hash 0 # rjenkins1
item osd.30 weight 4.000
item osd.31 weight 4.000
item osd.32 weight 4.000
item osd.33 weight 4.000
item osd.34 weight 4.000
it
Re: [ceph-users] client-ceph [can not connect from client][connect protocol feature mismatch]
Hi Sonal, You can refer to this doc to identify your problem. Your error code is 4204, so * 4000 upgrade to kernel 3.9 * 200 CEPH_FEATURE_CRUSH_TUNABLES2 * 4 CEPH_FEATURE_CRUSH_TUNABLES * http://ceph.com/planet/feature-set-mismatch-error-on-ceph-kernel-client/ Stephane. - Mail original - > Hi, > I am newbie for ceph, and ceph-user group. Recently I have been working on a > ceph client. It worked on all the environments while when i tested on the > production, it is not able to connect to ceph. > Following are the operating system details and error. If someone has seen > this problem before, any help is really appreciated. > OS - > lsb_release -a > No LSB modules are available. > Distributor ID: Ubuntu > Description: Ubuntu 12.04.2 LTS > Release: 12.04 > Codename: precise > 2015-03-05 13:37:16.816322 7f5191deb700 -- 10.8.25.112:0/2487 >> > 10.138.23.241:6789/0 pipe(0x12489f0 sd=3 pgs=0 cs=0 l=0).connect protocol > feature mismatch, my 1ffa < peer 42041ffa missing 4204 > 2015-03-05 13:37:17.635776 7f5191deb700 -- 10.8.25.112:0/2487 >> > 10.138.23.241:6789/0 pipe(0x12489f0 sd=3 pgs=0 cs=0 l=0).connect protocol > feature mismatch, my 1ffa < peer 42041ffa missing 4204 > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] OS file Cache, Ceph RBD cache and Network files systems
Hi Cephers, Our university could deploy ceph. The goal is to store datas for research laboratories (non-HPC) . To do this, we plan to use Ceph with RBD (mount block device) from a NFS ( or CIFS ) server (ceph client) to workstations in laboratories. According to our tests, the OS (ubuntu or centos...) that map the RBD block implements file system write cache (vm.dirty_ratio, etc ...). In that case, the NFS server will always perform writes to workstations whereas it has not finished writing datas to Ceph cluster - a nd regardless of whether the RBD cache is enabled or not in the config [client] section. My questions: 1. Does the activation of RBD cache is useful only when it combines Virtuals Machnies (where QEMU can access an image as a virtual block device directly via librbd) ? 2. Is it common to use Ceph, with RBD to share network file systems ? 3. And if so, what are the recommendations concerning the OS cache ? Thanks a lot. Stephane. -- Université de Lorraine Stéphane DUGRAVOT - Direction du numérique - Infrastructure Jabber : stephane.dugra...@univ-lorraine.fr Tél.: +33 3 83 68 20 98 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] OSD Forece Removal
- Mail original - > Hi all, can anybody tell me how can I force delete osds? the thing is that > one node got corrupted because of outage, so there is no way to get those > osd up and back, is there anyway to force the removal from ceph-deploy node? Hi, Try manual : * http://ceph.com/docs/master/rados/operations/add-or-rm-osds/#removing-osds-manual > Thanks > Jesus Chavez > SYSTEMS ENGINEER-C.SALES > jesch...@cisco.com > Phone: +52 55 5267 3146 > Mobile: +51 1 5538883255 > CCIE - 44433 > Cisco.com > Think before you print. > This email may contain confidential and privileged material for the sole use > of the intended recipient. Any review, use, distribution or disclosure by > others is strictly prohibited. If you are not the intended recipient (or > authorized to receive for the recipient), please contact the sender by reply > email and delete all copies of this message. > Please click here for Company Registration Information. > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] OSD Forece Removal
- Mail original - Thanks stephane the thing is that those steps needs to be run in the node where the osd lives, I dont have that node any more since the operating Systems got corrupted so I Couldnt make it work :( Jesus, On my test cluster, what i have done is (experimental) : * Edit crushmap and attach failed osd (osd.2) to a running host (previously attached to delta host) * reinject crushmap * then run the 2 commands : ceph osd down osd.2; ceph osd rm 2 * After that, ceph osd tree give me : * # id weight type name up/down reweight -100 2 root default -1 1 host bravo 0 1 osd.0 up 1 -2 1 host charlie 1 1 osd.1 up 1 2 1 osd.2 DNE * edit crushmap again, suppress the reference to osd.2 (DNE) * Reinject, and TADA ! : * # id weight type name up/down reweight -100 2 root default -1 1 host bravo 0 1 osd.0 up 1 -2 1 host charlie 1 1 osd.1 up 1 Stephane. Thanks Jesus Chavez SYSTEMS ENGINEER-C.SALES jesch...@cisco.com Phone: +52 55 5267 3146 Mobile: +51 1 5538883255 CCIE - 44433 On Mar 20, 2015, at 3:49 AM, Stéphane DUGRAVOT < stephane.dugra...@univ-lorraine.fr > wrote: - Mail original - Hi all, can anybody tell me how can I force delete osds? the thing is that one node got corrupted because of outage, so there is no way to get those osd up and back, is there anyway to force the removal from ceph-deploy node? Hi, Try manual : * http://ceph.com/docs/master/rados/operations/add-or-rm-osds/#removing-osds-manual Thanks Jesus Chavez SYSTEMS ENGINEER-C.SALES jesch...@cisco.com Phone: +52 55 5267 3146 Mobile: +51 1 5538883255 CCIE - 44433 Cisco.com Think before you print. This email may contain confidential and privileged material for the sole use of the intended recipient. Any review, use, distribution or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender by reply email and delete all copies of this message. Please click here for Company Registration Information. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] error creating image in rbd-erasure-pool
- Mail original - > Hi Markus, > On 24/03/2015 14:47, Markus Goldberg wrote: > > Hi, > > this is ceph version 0,93 > > I can't create an image in an rbd-erasure-pool: > > > > root@bd-0:~# > > root@bd-0:~# ceph osd pool create bs3.rep 4096 4096 replicated > > pool 'bs3.rep' created > > root@bd-0:~# rbd create --size 1000 --pool bs3.rep test > > root@bd-0:~# > > root@bd-0:~# ceph osd pool create bs3.era 4096 4096 erasure > > pool 'bs3.era' created > > root@bd-0:~# rbd create --size 1000 --pool bs3.era tapes2 > > rbd: create error: (95) Operation not supported2015-03-24 13:57:31.018411 > > 7fc186b77840 -1 > > librbd: error adding image to directory: (95) Operation not supported > RBD won't work with erasure coded pools. Instead you could try adding a > replicated cache pool and use it. > See http://docs.ceph.com/docs/master/rados/operations/cache-tiering/ for more > information. Hi Loic and Markus, By the way, Inktank do not support snapshot of a pool with cache tiering : * https://download.inktank.com/docs/ICE%201.2%20-%20Cache%20and%20Erasure%20Coding%20FAQ.pdf What's wrong exactly with that ? i suppose that some features is not possible ? Do you know what ? Thanks, Stephane. > Cheers > > > > Is this not possible at the moment or am i mistyping? > > > > BTW: Deleting or shrinking an empty image takes very, very lonng > > > > Thank you, > > Markus > > > > -- > > Markus Goldberg Universität Hildesheim > > Rechenzentrum > > Tel +49 5121 88392822 Universitätsplatz 1, D-31141 Hildesheim, Germany > > Fax +49 5121 88392823 email goldb...@uni-hildesheim.de > > -- > > > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- > Loïc Dachary, Artisan Logiciel Libre > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] when recovering start
- Le 7 Avr 15, à 14:57, lijian a écrit : > Haomai Wang, > the mon_osd_down_out_interval is 300, please refer to my settings, and I use > the > cli 'service ceph stop osd.X' to stop a osd > the pg status change to remap,backfill and recovering ... immediately > so other something wrong with my settings or operation? Hi, >From what daemon you get the mon_osd_down_out_interval value ? Is it from mon or osd ? Stephane. > Thanks, > Jian Ji > At 2015-04-07 20:38:29, "Haomai Wang" wrote: > >Whatever the version you tested, ceph won't recover data when you > >manually stop osd immediately. And it will trigger mark down osd out > >when it reach "mon_osd_down_out_interval" seconds. > >On Tue, Apr 7, 2015 at 8:33 PM, lijian wrote: > >> Hi, > >> The recovering start delay 300s after I stop a osd and the osd status > >> change > >> from in to out, the test ENV is Ceph 0.80.7 > >> But I test in ceph 0.87.1, the recovering start immediately after I stop a > >> OSD,all the settings is the default value,the following is mon_osd* > >> settings > >> in my test ENV: > >> "mon_osd_laggy_halflife": "3600", > >> "mon_osd_laggy_weight": "0.3", > >> "mon_osd_adjust_heartbeat_grace": "true", > >> "mon_osd_adjust_down_out_interval": "true", > >> "mon_osd_auto_mark_in": "false", > >> "mon_osd_auto_mark_auto_out_in": "true", > >> "mon_osd_auto_mark_new_in": "true", > >> "mon_osd_down_out_interval": "300", > >> "mon_osd_down_out_subtree_limit": "rack", > >> "mon_osd_min_up_ratio": "0.3", > >> "mon_osd_min_in_ratio": "0.3", > >> "mon_osd_max_op_age": "32", > >> "mon_osd_max_split_count": "32", > >> "mon_osd_allow_primary_temp": "false", > >> "mon_osd_allow_primary_affinity": "false", > >> "mon_osd_full_ratio": "0.95", > >> "mon_osd_nearfull_ratio": "0.85", > >> "mon_osd_report_timeout": "45000", > >> "mon_osd_min_down_reporters": "50", > >> "mon_osd_min_down_reports": "150", > >> "mon_osd_force_trim_to": "0", > >> so when the recovering start? why they are different with the two Ceph > >> version, or someting wrong with my settings > >> Thanks! > >> Jian Li > >> ___ > >> ceph-users mailing list > >> ceph-users@lists.ceph.com > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >-- > >Best Regards, > >Wheat > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] when recovering start
- Le 8 Avr 15, à 14:21, lijian a écrit : > Hi Stephane, > I dump from a osd deamon You have to apply the mon_osd_down_out_interval value to monitor and not osd. What is the value on the mon ? Stephane. > Thanks > Jian LI > At 2015-04-08 16:05:04, "Stéphane DUGRAVOT" > > wrote: >> - Le 7 Avr 15, à 14:57, lijian < blacker1...@163.com > a écrit : >>> Haomai Wang, >>> the mon_osd_down_out_interval is 300, please refer to my settings, and I >>> use the >>> cli 'service ceph stop osd.X' to stop a osd >>> the pg status change to remap,backfill and recovering ... immediately >>> so other something wrong with my settings or operation? >> Hi, >> From what daemon you get the mon_osd_down_out_interval value ? >> Is it from mon or osd ? >> Stephane. >>> Thanks, >>> Jian Ji >>> At 2015-04-07 20:38:29, "Haomai Wang" < haomaiw...@gmail.com > wrote: >>> >Whatever the version you tested, ceph won't recover data when you >>> >manually stop osd immediately. And it will trigger mark down osd out >>> >when it reach "mon_osd_down_out_interval" seconds. >>> >On Tue, Apr 7, 2015 at 8:33 PM, lijian < blacker1...@163.com > wrote: >>> >> Hi, >>> >> The recovering start delay 300s after I stop a osd and the osd status >>> >> change >>> >> from in to out, the test ENV is Ceph 0.80.7 >>> >> But I test in ceph 0.87.1, the recovering start immediately after I stop >>> >> a >>> >> OSD,all the settings is the default value,the following is mon_osd* >>> >> settings >>> >> in my test ENV: >>> >> "mon_osd_laggy_halflife": "3600", >>> >> "mon_osd_laggy_weight": "0.3", >>> >> "mon_osd_adjust_heartbeat_grace": "true", >>> >> "mon_osd_adjust_down_out_interval": "true", >>> >> "mon_osd_auto_mark_in": "false", >>> >> "mon_osd_auto_mark_auto_out_in": "true", >>> >> "mon_osd_auto_mark_new_in": "true", >>> >> "mon_osd_down_out_interval": "300", >>> >> "mon_osd_down_out_subtree_limit": "rack", >>> >> "mon_osd_min_up_ratio": "0.3", >>> >> "mon_osd_min_in_ratio": "0.3", >>> >> "mon_osd_max_op_age": "32", >>> >> "mon_osd_max_split_count": "32", >>> >> "mon_osd_allow_primary_temp": "false", >>> >> "mon_osd_allow_primary_affinity": "false", >>> >> "mon_osd_full_ratio": "0.95", >>> >> "mon_osd_nearfull_ratio": "0.85", >>> >> "mon_osd_report_timeout": "45000", >>> >> "mon_osd_min_down_reporters": "50", >>> >> "mon_osd_min_down_reports": "150", >>> >> "mon_osd_force_trim_to": "0", >>> >> so when the recovering start? why they are different with the two Ceph >>> >> version, or someting wrong with my settings >>> >> Thanks! >>> >> Jian Li >>> >> ___ >>> >> ceph-users mailing list >>>>> ceph-users@lists.ceph.com >> >>> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >-- >>> >Best Regards, >>> >Wheat >>> ___ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] cephfs ... show_layout deprecated ?
Hi all, When running command : cephfs /cephfs/ show_layout The result is : WARNING: This tool is deprecated. Use the layout.* xattrs to query and modify layouts. Error getting layout: (25) Inappropriate ioctl for device I didn't alternative for set and show layout nor another tool. is there a way to change the pool id ? How to use layout.* xattrs ? Thanks, Stephane. Ceph version : ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) The mount is a kernel mount : mount -t ceph 1.2.3.4:/data1 /cephfs -- Université de Lorraine Stéphane DUGRAVOT - Direction du numérique - Infrastructure Jabber : stephane.dugra...@univ-lorraine.fr Tél.: +33 3 83 68 20 98 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] cephfs ... show_layout deprecated ?
- Le 22 Avr 15, à 11:39, Wido den Hollander a écrit : > On 04/22/2015 11:22 AM, Stéphane DUGRAVOT wrote: > > Hi all, > > When running command : > > cephfs /cephfs/ show_layout > > The result is : > > WARNING: This tool is deprecated. Use the layout.* xattrs to query and > > modify > > layouts. > > Error getting layout: (25) Inappropriate ioctl for device > > I didn't alternative for set and show layout nor another tool. is there a > > way to > > change the pool id ? How to use layout.* xattrs ? > See the docs, they contain all you need: > http://ceph.com/docs/master/cephfs/file-layouts/ Thank you for pointing me the right url, it works as expected ! Stephane. > $ getfattr -n ceph.file.layout > Wido > > Thanks, > > Stephane. > > Ceph version : > > ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff) > > The mount is a kernel mount : > > mount -t ceph 1.2.3.4:/data1 /cephfs > > -- > > Université de Lorraine*/ > > /*Stéphane DUGRAVOT - Direction du numérique - Infrastructure > > Jabber : /stephane.dugra...@univ-lorraine.fr/ > > Tél.: /+33 3 83 68 20 98/ > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- > Wido den Hollander > 42on B.V. > Ceph trainer and consultant > Phone: +31 (0)20 700 9902 > Skype: contact42on > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] backup RGW in federated gateway
Hi all I am interested in RGW and I would make a backup of differents pools (objects from a s3 connector) in a second Ceph cluster. From this doc: * http://ceph.com/docs/master/radosgw/federated-config/ If I activate the synchronization agent, copying data will synchronously the main area to secondary area ? I wish the copy is realized for example once every night. Can i not use the agent in "automatic" mode, and start the sync on demand ? Stephane. -- Université de Lorraine Stéphane DUGRAVOT - Direction du numérique - Infrastructure Jabber : stephane.dugra...@univ-lorraine.fr Tél.: +33 3 83 68 20 98 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] ceph at "Universite de Lorraine"
Hi all, We (French University) plan to implement a storage platform (distributed of course ) for a volume of 750 TB. We are interesting in CEPH ... We wonder the availability of professional support in our project approach. Do you know a professional integrator that could assist us for : * early design , * deployment * and probably support ? I already know 2 of them. But ... Someone told me : " there is no need for professional support , just buy the equipment , install it and let it run ceph " -> This is probably true, but distributed storage technologies are so new to us that we should consider professional support in our project approach (especially at this point ) . Thanks. Stephane. -- Université de Lorraine Stéphane DUGRAVOT - Direction du numérique - Infrastructure Jabber : stephane.dugra...@univ-lorraine.fr Tél.: +33 3 83 68 20 98 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] erasure coded pool k=7,m=5
Hi all, Soon, we should have a 3 datacenters (dc) ceph cluster with 4 hosts in each dc. Each host will have 12 OSD. We can accept the loss of one datacenter and one host on the remaining 2 datacenters. In order to use erasure coded pool : 1. Is the solution for a strategy k = 7, m = 5 is acceptable ? 2. Is this is the only one that guarantees us our premise ? 3. And more generally, is there a formula (based on the number of dc, host and OSD) that allows us to calculate the profile ? Thanks. Stephane. -- Université de Lorraine Stéphane DUGRAVOT - Direction du numérique - Infrastructure Jabber : stephane.dugra...@univ-lorraine.fr Tél.: +33 3 83 68 20 98 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] backup of radosgw config
- Le 3 Nov 16, à 5:18, Thomas a écrit : > Hi guys, Hi Thomas, This is a question I also asked myself ... Maybe something like : radosgw-admin zonegroup get radosgw-admin zone get And for each user : radosgw-admin metadata get user:uid Anyone ? Stephane. > I'm not sure this was asked before as I wasn't able to find anything googling > (and the search function of the list is broken at > http://lists.ceph.com/pipermail/ceph-users-ceph.com/ ) - anyway: > - How would you backup the config of all users and bucket configurations for > the > radosgw so that in a disaster recovery we could import those again? > Cheers, > Thomas > -- > Thomas Gross > TGMEDIA Ltd. > p. +64 211 569080 | i...@tgmedia.co.nz > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Configuring Ceph RadosGW with SLA based rados pools
- Le 4 Nov 16, à 21:17, Andrey Ptashnik a écrit :
> Hello Ceph team!
> I’m trying to create different pools in Ceph in order to have different tiers
> (some are fast, small and expensive and others are plain big and cheap), so
> certain users will be tied to one pool or another.
> - I created two additional pools.
> .rgw.factor-2.buckets.data
> .rgw.factor-3.buckets.data
> .rgw.factor-2.buckets.index
> .rgw.factor-3.buckets.index
> - Edited region config file to include new placement target
> "placement_targets": [
> {
> "name": "default-placement",
> "tags": []
> },
> {
> "name": "factor-2",
> "tags": []
> },
> {
> "name": "factor-3",
> "tags": []
> }
> - Edited zone config file to tie placement targets with new pools:
> {
> "key": "factor-2",
> "val": {
> "index_pool": ".rgw.factor-2.buckets.index",
> "data_pool": ".rgw.factor-2.buckets.data",
> "data_extra_pool": ""
> }
> },
> {
> "key": "factor-3",
> "val": {
> "index_pool": ".rgw.factor-3.buckets.index",
> "data_pool": ".rgw.factor-3.buckets.data",
> "data_extra_pool": ""
> }
> }
> - Changed "default_placement": "factor-3” in one of the user configurations.
> Listing and viewing new pool/bucket contents using that modified user account
> is
> not problem at all. However when I’m trying to upload files to new pools or
> create a bucket using this modified user account I’m receiving an error " 400
> Bad Request”. Tools I’m using are S3cmd and rclone – both give me the same
> error.
> I’m trying to find root cause first square out any premision issues I may
> have.
> How can I make sure that users will have full access rights to new pools?
Hi,
Have you :
1. Update region map with command : radosgw-admin regionmap update
2. Restart rados Gateway
Stephane.
> === === ===
> P.S.: Procedure that I follow is described in document "Configuring Ceph
> RadosGW
> with SLA based rados pools” -> https://access.redhat.com/solutions/1532743 and
> his is exactly what I need.
> Regards,
> Andrey Ptashnik
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] maximum number of chunks/files with civetweb ? (status= -2010 http_status=400)
Hi all, In top of our ceph cluster, one application use the rados gateway/S3. This application did not use multipart s3 api, but it split files (for example 1 MB) into chunk of desire size (it have to work on top of several type of storage). For every tests, the application hang when uploading the 64th chunk ... And the upload failed. The chunk from 0 to 63 is well upload. For example, for a file of 100MB, the application would send 100 chunks of 1 MB. But it hang when uploading the chunk 64 ... And i don't know why ... I did not find setttings of civetweb, where can we specify parameters for civetweb ? For example a setting which limit for 64 upload at the same time for the same client ? s3cmd is able to send the same file with the multipart s3 api, and setting chunk (5MB). But in our case, we do not use this api ... Thank you. Stephane. Here is the log 2016-12-20 10:02:10.313809 7f00c52d9700 1 == starting new request req=0x7f00c52d3710 = 2016-12-20 10:02:26.702572 7f00c52d9700 1 == req done req=0x7f00c52d3710 op status=0 http_status=200 == 2016-12-20 10:02:26.702650 7f00c52d9700 1 civetweb: 0x7f01480743e0: 100.74.189.131 - - [20/Dec/2016:10:02:10 +0100] "PUT /test-bucket/Gros%20fichiers/10G.63_96 HTTP/1.1" 200 0 - - 2016-12-20 10:23:43.112493 7f00b1ab2700 1 == starting new request req=0x7f00b1aac710 = 2016-12-20 10:23:43.121837 7f00b1ab2700 1 == req done req=0x7f00b1aac710 op status=-2010 http_status=400 == 2016-12-20 10:23:43.121920 7f00b1ab2700 1 civetweb: 0x7f0138050490: 100.74.189.131 - - [20/Dec/2016:10:23:43 +0100] "PUT /test-bucket/Gros%20fichiers/10G.64_96 HTTP/1.1" 400 0 - - 2016-12-20 10:33:27.564398 7f00bc2c7700 1 == starting new request req=0x7f00bc2c1710 = 2016-12-20 10:33:27.570324 7f00bc2c7700 1 == req done req=0x7f00bc2c1710 op status=0 http_status=200 == 2016-12-20 10:33:27.570394 7f00bc2c7700 1 civetweb: 0x7f01752f8990: 100.74.189.131 - - [20/Dec/2016:10:33:27 +0100] "GET / HTTP/1.1" 200 0 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
