Re: [ceph-users] What is difference in storing data between rbd and rados ?

[debian Only]

Hope expert give me some light


2014-08-06 18:01 GMT+07:00 debian Only :

> I am confuse to understand how File store in Ceph.
>
> I do two test. where is the File or the object for the File
>
> ①rados put Python.msi Python.msi -p data
> ②rbd -p testpool create fio_test --size 2048
>
> rados command of ① means use Ceph as Object storage ?
> rbd command of ② means use Ceph as Block storage ?
>
> As i known, object in Ceph is 4M by default.  this Object will put in PG.
> so i try do test as blow.  the fio_test image store in Ceph by 512 object.
>  512（object） * 4 = 2048
> and i can get object in testpool.
>
> # rbd -p testpool info fio_test
> rbd image 'fio_test':
> size 2048 MB in 512 objects
> order 22 (4096 kB objects)
> block_name_prefix: rb.0.1b6f.2ae8944a
> format: 1
> # rados -p testpool ls |grep rb.0.1b6f.2ae8944a |wc -l
> 512
>
>
> but when i check the data pool, only one file :Python.msi (26M), why not
> split Python.msi to many object(4M)  ?
>
> t# rados ls -p pool-B
> python.msi
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Regarding cache tier understanding

[Somnath Roy]

Hi Sage,
I went through the tiering agent code base and here is my understanding on the 
agent behavior. Please let me know if that is correct.


1.   Agent will be always in idle state if target_max_bytes or 
target_max_objects not set on the pool irrespective of other tiering params set 
in the pool. dirty_micro and full_micro will not be calculated if those two 
params are zero which is by default I guess.

2.   Now, flush will be activated if dirty_micro is > flush_target. My 
understanding is, once it is activated it will iterate through all the dirty 
objects and flush all the dirty objects which is > cache_min_flush_age. Am I 
right ?

3.   For the eviction, if full_micro > 100 , the mode set as 
TierAgentState::EVICT_MODE_FULL and all the clean objects are flushed.

4.   if (full_micro > evict_target), the mode is set as 
TierAgentState::EVICT_MODE_SOME. In this scenario evict_effort is calculated 
and based on hit_set and temp calculation some clean objects are evicted. My 
question is , can we quantify this value ? For ex, if the target_full_ratio = 
50%, once the eviction is triggered, what %objects will be evicted ?

5.   Also, why we are multiplying with 100 always ? Is it because in 
the histogram it is positioned as 0..100 ?

6.   I saw the cache_min_evict_age is not been used anywhere, am I missing 
anything ?

7.   The cache_min_flush_age will only be applicable if the flush is 
triggered after crossing the dirty_threshold, right ? If dirty_threshold is not 
breached, the flush age param is never checked.

I need to understand the behavior so that we can test the Cache tiering 
properly.
Is there any behavior I am missing here which you want us to test out ?

Thanks & Regards
Somnath



PLEASE NOTE: The information contained in this electronic mail message is 
intended only for the use of the designated recipient(s) named above. If the 
reader of this message is not the intended recipient, you are hereby notified 
that you have received this message in error and that any review, 
dissemination, distribution, or copying of this message is strictly prohibited. 
If you have received this communication in error, please notify the sender by 
telephone or e-mail (as shown above) immediately and destroy any and all copies 
of this message in your possession (whether hard copies or electronically 
stored copies).

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] slow OSD brings down the cluster

[Luis Periquito]

Hi Mark,

I've been playing with the reweight on 3 of the OSDs (BTW each OSD is
backed by a HDD, with a SSD backing all the 4 journals on each host) and
these slower ones were given a reweight of 0.5, 0.66 and 0.66.

>From what I gathered the reweight would also reduce the number of I/O
directed at that OSD - as it did reduce the number of PGs mapped to that
OSD, and did all the rebalancing - however using iotstat I've seen the load
on these is usually on par or above the remaining of the cluster, thus
making them slower. I'm measuring the load as total IOPS. All these HDDs
are used exclusively for the OSD role.

Also I'm thinking of moving some smaller pools to be stored on SSDs and I'm
using "df detail" to try and figure out which should I move, but I'm unsure
what the READ and WRITE columns mean.

thanks,


On 6 August 2014 17:10, Mark Nelson  wrote:

> On 08/06/2014 03:43 AM, Luis Periquito wrote:
>
>> Hi,
>>
>> In the last few days I've had some issues with the radosgw in which all
>> requests would just stop being served.
>>
>> After some investigation I would go for a single slow OSD. I just
>> restarted that OSD and everything would just go back to work. Every
>> single time there was a deep scrub running on that OSD.
>>
>> This has happened in several different OSDs, running in different
>> machines. I currently have 32 OSDs on this cluster, with 4 OSD per host.
>>
>> First thing is should this happen? A single OSD with issues/slowness
>> shouldn't bring the whole cluster to a crawl...
>>
>
> When a client is writing data out to the cluster, it will issue some
> number of operations that it can have in flight at once.  This has to be
> bound at some level to avoid running out of memory.  Ceph will distribute
> those writes to some number of PGs in a psuedo-random like way, and those
> PGs map to specific OSDs where the data will be placed. One of the big
> advantages of crush is that it lets the client figure this mapping out
> itself based on the object name and the cluster topology, so you remove a
> centralized allocation table lookup from the data path which can be a huge
> win vs other large-scale distributed systems.
>
> The downside is that it means that in a setup where you have 1 disk behind
> each OSD (typically the best setup for Ceph right now), every disk will
> receive a relatively even (or potentially weighted) percentage of the
> writes regardless of how fast/slow/busy it is.  If a single OSD is slower
> than the others, over time it is likely to accumulate enough outstanding
> IOs that eventually nearly every client IO will be waiting on that OSD.
>  The rest of the OSDs in the cluster will only get new IOs once an IO
> completes on the slow one.
>
> Some day, maybe after the keyfilestore is implemented, I think it would be
> a very interesting experiment to try a hybrid approach where you use crush
> to distribute data to nodes, but behind the OSDs you use something like an
> allocation table and dynamically change the ratio of writes to different
> filesystems or key/value stores based on how slow/busy they are (especially
> during compaction, directory splitting, scrub, or if there's a really hot
> object on a specific disk).  You can still avoid the network allocation
> table lookup, but potentially within the node, if you can do it fast
> enough, you might be able to gain some level of adaptability and
> (hopefully) more consistent throughput.
>
> Mark
>
>
>> How can I make it stop happening? What kind of debug information can I
>> gather to stop this from happening?
>>
>> any further thoughts?
>>
>> I'm still running Emperor (0.72.2).
>>
>> --
>>
>> Luis Periquito
>>
>> Unix Engineer
>>
>>
>> Ocado.com 
>>
>>
>>
>> Head Office, Titan Court, 3 Bishop Square, Hatfield Business Park,
>> Hatfield, Herts AL10 9NE
>>
>>
>> Notice:  This email is confidential and may contain copyright material
>> of members of the Ocado Group. Opinions and views expressed in this
>> message may not necessarily reflect the opinions and views of the
>> members of the Ocado Group.
>>
>> If you are not the intended recipient, please notify us immediately and
>> delete all copies of this message. Please note that it is your
>> responsibility to scan this message for viruses.
>>
>> References to the “Ocado Group” are to Ocado Group plc (registered in
>>
>> England and Wales with number 7098618) and its subsidiary undertakings
>> (as that expression is defined in the Companies Act 2006) from time to
>> time.  The registered office of Ocado Group plc is Titan Court, 3
>> Bishops Square, Hatfield Business Park, Hatfield, Herts. AL10 9NE.
>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 

Lu

Re: [ceph-users] ceph rbd volume can't remove because image still has watchers

[Karan Singh]

Make sure the volume  "volume-17d9397b-d6e5-45e0-80fa-4bc7b7998842” does no 
have a snapshot or cloned linked to it  , it might sometimes cause problems 
while deletion.


- Karan Singh

On 07 Aug 2014, at 08:55, 杨万元  wrote:

> Hi all：
> we use ceph rbd with openstack ，recently there are some  dirty data in my 
> cinder-volume databases such as volumes status like error-deleting. So we 
> need manually delete this volumes。
> but when I delete the volume on ceph node，ceph tell me this error
> 
>   [root@ceph-node3 ~]# rbd -p glance rm 
> volume-17d9397b-d6e5-45e0-80fa-4bc7b7998842
> Removing image: 99% complete...failed.
> rbd: error: image still has watchers
> This means the image is still open or the client using it crashed. 
> Try again after   closing/unmapping it or waiting 30s for the crashed 
> client to timeout.
> 2014-08-07 11:25:42.793275 7faf8c58b760 -1 librbd: error removing 
> header: (16) Device or resource busy
> 
> 
>I google this problem and  find this  
> http://comments.gmane.org/gmane.comp.file-systems.ceph.user/9767
>I did it and got this:
> 
>  [root@ceph-node3 ~]# rbd info -p glance 
> volume-17d9397b-d6e5-45e0-80fa-4bc7b7998842
> rbd image 'volume-17d9397b-d6e5-45e0-80fa-4bc7b7998842':
> size 51200 MB in 12800 objects
> order 22 (4096 kB objects)
> block_name_prefix: rbd_data.3b1464f8e96d5
> format: 2
> features: layering
>  [root@ceph-node3 ~]# rados -p glance listwatchers 
> rbd_header.3b1464f8e96d5 
> watcher=192.168.39.116:0/1032797 client.252302 cookie=1
> 
>   192.168.39.116 is my nova compute node ,so i can't reboot this server,
>   what can i do to delete this volume without reboot my  compute-node?
> 
>   my ceph version is 0.72.1.
> 
>  thanks very much!
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph writes stall for long perioids with no disk/network activity

[Mariusz Gronczewski]

> > 
> > I've often wished for some sort of bottleneck finder for ceph. An easy
> > way for the system to say where it is experiencing critical latencies
> > e.g. network, journals, osd data disks, etc. This would assist
> > troubleshooting and initial deployments immensely.
> 
> As mentioned above, it's tricky. 
> Most certainly desirable, but the ole Mark I eyeball and wetware is quite
> good at spotting these when presented with appropriate input like atop.
> 

Is there any stats from OSD perf dump that could help with that ?
I've wrote simple wrapper to collectd to get op_ and subop_
rw/w/r_latency but I'm not certain if it will show problems with
underlying storage, so far everytime I evicted "slow" (3-5x times
bigger latency than other ones) osd another took its place.

I'm guessing probably because that OSD got "short end of the CRUSH" and
got loaded with a bit more requests so other OSDs were waiting for that
one.

Also, is there any way to correlate results of
dump_historic_ops between OSDs ? I've noticed that in my case longest one are 
usually "waiting for subops from X, Y" and except for time there is no other 
information to correlate that for example op on osd.1 waited for subop on osd.5 
and that subop on osd.5 was slow because of y
-- 
Mariusz Gronczewski, Administrator

Efigence S. A.
ul. Wołoska 9a, 02-583 Warszawa
T: [+48] 22 380 13 13
F: [+48] 22 380 13 14
E: mariusz.gronczew...@efigence.com



signature.asc
Description: PGP signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph writes stall for long perioids with no disk/network activity

[Christian Balzer]

On Thu, 7 Aug 2014 14:21:30 +0200 Mariusz Gronczewski wrote:

> 
> > > 
> > > I've often wished for some sort of bottleneck finder for ceph. An
> > > easy way for the system to say where it is experiencing critical
> > > latencies e.g. network, journals, osd data disks, etc. This would
> > > assist troubleshooting and initial deployments immensely.
> > 
> > As mentioned above, it's tricky. 
> > Most certainly desirable, but the ole Mark I eyeball and wetware is
> > quite good at spotting these when presented with appropriate input
> > like atop.
> > 
> 
> Is there any stats from OSD perf dump that could help with that ?

Help, yes. By itself, no.
Firstly those values are transient, so need to be sampled frequently and
put into the right correlation.
I suppose if the OP had used "ceph osd perf" during the tests, spotting
the suspect OSD and confirming it with atop or iostat might have been
quicker.

> I've wrote simple wrapper to collectd to get op_ and subop_
> rw/w/r_latency but I'm not certain if it will show problems with
> underlying storage, so far everytime I evicted "slow" (3-5x times
> bigger latency than other ones) osd another took its place.
> 
> I'm guessing probably because that OSD got "short end of the CRUSH" and
> got loaded with a bit more requests so other OSDs were waiting for that
> one.
> 

If the problem, hotspot just migrates to another OSD that is pretty
probable and a reasonable assumption. 
How many OSDs are we talking about in your case?

I got a 2 node cluster, each with 2 OSDs, SSD journals backed by 11 disk
RAID6 behind a 4GB HW cache RAID controller. 
So things normally look very impressive like this (average of 1MBs/200IOPS
at the time):
---
# ceph osd perf
osdid fs_commit_latency(ms) fs_apply_latency(ms) 
0222 
1234 
2243 
3225 
---

In this particular setup, knowing very well the capabilities of the
hardware involved, if OSDs would vary slightly (10% or so) it is probably
PG imbalance (bad luck of the CRUSH draw). 
Not surprising with 4 OSDs and depending on the test or use case
something I've seen and could reproduce. 
A much larger imbalance would suggest a wonky HDD in one of the OSD RAID
sets or a RAID rebuild (something I would of course know already about).
^o^

In the case of the OP the problem stuck to a specific OSD (and probably
would have been verifiable with speed tests of that disk) and went away
when it got removed.

Ceph could (using probably not insignificant computational resources) take
into account all the ops issued to an OSD in  and then put the
performance numbers in a relation to it. 
So if with 10 OSDs 9 got 5% of all ops and one got 55% in the sample
period, crappy latency is to be expected and should be corrected for.
If however the distribution was equal AND the hardware is equal (something
Ceph is likely never to know) then sanitized performance counters would
pick up a slow OSD easily.

> Also, is there any way to correlate results of
> dump_historic_ops between OSDs ? I've noticed that in my case longest
> one are usually "waiting for subops from X, Y" and except for time there
> is no other information to correlate that for example op on osd.1 waited
> for subop on osd.5 and that subop on osd.5 was slow because of y

No idea, this calls for the Ceph engineers. ^o^

Christian
-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Fusion Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] What is difference in storing data between rbd and rados ?

[Sage Weil]

On Thu, 7 Aug 2014, debian Only wrote:
> Hope expert give me some light
> 
> 
> 2014-08-06 18:01 GMT+07:00 debian Only :
>   I am confuse to understand how File store in Ceph.
> 
> I do two test. where is the File or the object for the File
> 
> ?rados put Python.msi Python.msi -p data
> ?rbd -p testpool create fio_test --size 2048
>  
> rados command of ? means use Ceph as Object storage ?
> rbd command of ? means use Ceph as Block storage ?
> 
> As i known, object in Ceph is 4M by default.  this Object will put in
> PG. 
> so i try do test as blow.  the fio_test image store in Ceph by 512
> object.  512?object? * 4 = 2048
> and i can get object in testpool.
> 
> # rbd -p testpool info fio_test
> rbd image 'fio_test':
>         size 2048 MB in 512 objects
>         order 22 (4096 kB objects)
>         block_name_prefix: rb.0.1b6f.2ae8944a
>         format: 1
> # rados -p testpool ls |grep rb.0.1b6f.2ae8944a |wc -l
> 512   
> 
> 
> but when i check the data pool, only one file :Python.msi (26M), why
> not split Python.msi to many object(4M)  ?
> 
> t# rados ls -p pool-B
> python.msi

The striping happens *above* the rados layer.  In this case, it is librbd 
that is striping a large image across lots of smaller objects (that it is 
naming).  CephFS clients do the same thing.  The rados CLI is talking 
directly to that lower layer and storing a single (large) object.

There is a new librados_striper library that provides lightweight striping 
functionality and handles things like delete in a reasonable way for 
applications that need that functionality...

sage___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Regarding cache tier understanding

[Sage Weil]

On Thu, 7 Aug 2014, Somnath Roy wrote:
> Hi Sage,
> 
> I went through the tiering agent code base and here is my understanding on
> the agent behavior. Please let me know if that is correct.
> 
>  
> 
> 1.   Agent will be always in idle state if target_max_bytes or
> target_max_objects not set on the pool irrespective of other tiering params
> set in the pool. dirty_micro and full_micro will not be calculated if those
> two params are zero which is by default I guess.

Yeah

> 2.   Now, flush will be activated if dirty_micro is > flush_target. My
> understanding is, once it is activated it will iterate through all the dirty
> objects and flush all the dirty objects which is > cache_min_flush_age. Am I
> right ?

Yeah

> 3.   For the eviction, if full_micro > 100 , the mode set as
> TierAgentState::EVICT_MODE_FULL and all the clean objects are flushed.

Yeah

> 4.   if (full_micro > evict_target), the mode is set as
> TierAgentState::EVICT_MODE_SOME. In this scenario evict_effort is calculated
> and based on hit_set and temp calculation some clean objects are evicted. My
> question is , can we quantify this value ? For ex, if the target_full_ratio
> = 50%, once the eviction is triggered, what %objects will be evicted ?

The effort is calculated based on how far between target_full and 100% we 
are.  This is mapped onto the estimation of atime.  We generate a 
histogram of age for the objects we have examined, so after the agent has 
run a bit we'll have a reasonable idea how the current object's age 
compares to others and can decide whether this is older or newer than the 
others; based on that we decide what to do.

> 5.   Also, why we are multiplying with 100 always ? Is it because in
> the histogram it is positioned as 0..100 ?

Yeah, and we use "micro" units throughout to avoid doing any floating 
point.

> 6.   I saw the cache_min_evict_age is not been used anywhere, am I
> missing anything ?

It's possible.  The _min_ params are a bit dangerous because they can 
potentially confuse the cache agent (e.g., what if *all* objects are under 
the min?  How/when do we decide to ignore the min, or, how/when do we 
give up trying to find an older object?).

> 7.   The cache_min_flush_age will only be applicable if the flush is
> triggered after crossing the dirty_threshold, right ? If dirty_threshold is
> not breached, the flush age param is never checked.

Right.

> I need to understand the behavior so that we can test the Cache tiering
> properly.
> 
> Is there any behavior I am missing here which you want us to test out ?

There is a new behavior just merged into master that controls our decision 
to promote data into the cache on read; see maybe_handle_cache().  That's 
not strictly agent behavior per se.  Also, there is now a readforward mode 
that doesn't promote on read ever, based on our discussion about the 
performance of flash on read.

Hope this helps!
sage___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph can't seem to forget

[Sean Sullivan]

I think I have a split issue or I can't seem to get rid of these objects.
How can I tell ceph to forget the objects and revert?

How this happened is that due to the python 2.7.8/ceph bug ( a whole rack
of ceph went town (it had ubuntu 14.10 and that seemed to have 2.7.8 before
14.04). I didn't know what was going on and tried re-installing which
killed the vast majority of the data. 2/3. The drives are gone and the data
on them is lost now.

I tried deleting them via rados but that didn't seem to work either and
just froze there.  Any help would be much appreciated.


Pastebin data below
http://pastebin.com/HU8yZ1ae


cephuser@host:~/CephPDC$ ceph --version
ceph version 0.82-524-gbf04897 (bf048976f50bd0142f291414ea893ef0f205b51a)

cephuser@host:~/CephPDC$ ceph -s
cluster 9e0a4a8e-91fa-4643-887a-c7464aa3fd14
 health HEALTH_WARN 2 pgs recovering; 2 pgs stuck unclean; 5 requests
are blocked > 32 sec; recovery 478/15386946 objects degraded (0.003%);
23/5128982 unfound (0.000%)
 monmap e9: 5 mons at {kg37-12=
10.16.0.124:6789/0,kg37-17=10.16.0.129:6789/0,kg37-23=10.16.0.135:6789/0,kg37-28=10.16.0.140:6789/0,kg37-5=10.16.0.117:6789/0},
election epoch 1450, quorum 0,1,2,3,4 kg37-5,kg37-12,kg37-17,kg37-23,kg37-28
 mdsmap e100: 1/1/1 up {0=kg37-5=up:active}
 osdmap e46061: 245 osds: 245 up, 245 in
  pgmap v3268915: 22560 pgs, 19 pools, 20020 GB data, 5008 kobjects
61956 GB used, 830 TB / 890 TB avail
478/15386946 objects degraded (0.003%); 23/5128982 unfound
(0.000%)
   22558 active+clean
   2 active+recovering
  client io 95939 kB/s rd, 80854 B/s wr, 795 op/s


cephuser@host:~/CephPDC$ ceph health detail
HEALTH_WARN 2 pgs recovering; 2 pgs stuck unclean; 5 requests are blocked >
32 sec; 1 osds have slow requests; recovery 478/15386946 objects degraded
(0.003%); 23/5128982 unfound (0.000%)
pg 5.f4f is stuck unclean since forever, current state active+recovering,
last acting [279,115,78]
pg 5.27f is stuck unclean since forever, current state active+recovering,
last acting [213,0,258]
pg 5.f4f is active+recovering, acting [279,115,78], 10 unfound
pg 5.27f is active+recovering, acting [213,0,258], 13 unfound
5 ops are blocked > 67108.9 sec
5 ops are blocked > 67108.9 sec on osd.279
1 osds have slow requests
recovery 478/15386946 objects degraded (0.003%); 23/5128982 unfound (0.000%)

cephuser@host:~/CephPDC$ ceph pg 5.f4f mark_unfound_lost revert
2014-08-06 12:59:42.282672 7f7d4a6fb700  0 -- 10.16.0.117:0/1005129 >>
10.16.64.29:6844/718 pipe(0x7f7d4005c120 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7f7d4005c3b0).fault
2014-08-06 12:59:51.890574 7f7d4a4f9700  0 -- 10.16.0.117:0/1005129 >>
10.16.64.29:6806/7875 pipe(0x7f7d4005f180 sd=4 :0 s=1 pgs=0 cs=0 l=1
c=0x7f7d4005fae0).fault
pg has no unfound objects
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Openstack Havana root fs resize don't work

[Hauke Bruno Wollentin]

Yes, I use cloud-init with cloud-initramfs-growroot without other scripts.

I tested your advices within a fresh image, but poorly I do not see any 
result. The only thing the log said about my mounts oder FS' is

[CLOUDINIT] helpers.py[DEBUG]: Running config-mounts using lock ()
[CLOUDINIT] DataSourceEc2.py[DEBUG]: Unable to convert ephemeral0 to a device
[CLOUDINIT] cc_mounts.py[DEBUG]: Ignoring nonexistant named default mount 
ephemeral0
[CLOUDINIT] DataSourceEc2.py[DEBUG]: Unable to convert swap to a device
[CLOUDINIT] cc_mounts.py[DEBUG]: Ignoring nonexistant named default mount swap
[CLOUDINIT] cc_mounts.py[DEBUG]: No modifications to fstab needed.

There is also no output in the log relating to "growroot", "growpart" oder 
"resizefs" module.

Can I run those modules manually just to check wether they'll work or not?

Kind regards,
Hauke

---
original message
timestamp: Wednesday, August 06, 2014 10:46:27 AM
from: Jeremy Hanmer 
to: Hauke Bruno Wollentin 
cc: ceph-users@lists.ceph.com 
subject: Re: [ceph-users] Openstack Havana root fs resize don't work
message id: 

> And you're using cloud-init in these cases, or are you executing
> growrootfs via some other means?
> 
> If you're using cloud-init, you should see some useful messages in
> /var/log/cloud-init.log (particularly on debian/ubuntu; I've found
> centos' logs to not be as helpful).
> 
> Also, if you're using cloud-init, you want to make sure that you've
> got it configured to do the right thing.  That changes a bit depending
> on the version, but with debian wheezy, I make sure to have "growpart"
> and "resizefs" in cloud_init_modules.  Also, I don't believe it's
> strictly necessary, but I add this too because I'm not a big fan of
> implicit defaults:
> 
> growpart:
>   mode: auto
>   devices: ['/']
> 
> resize_rootfs: True
> 
> On Wed, Aug 6, 2014 at 12:45 AM, Hauke Bruno Wollentin
> 
>  wrote:
> > Hi,
> > 
> > 1) I have flavors like 1 vCPU, 2GB memory, 20GB root disk. No swap + no
> > ephemeral disk. Then I just create an instance via horizon choosing an
> > image + a flavor.
> > 
> > 2) OpenStack itselfs runs on Ubuntu 12.04.4 LTS, for the instances I have
> > some Ubuntu 12.04/14.04s, Debians and CentOS'.
> > 
> > 3) In the spawned instances I see that the partition wasn't resized.
> > /proc/partions + fdisk -l show the size of the image partition, not the
> > instance partition specified by the flavor.
> > 
> > 
> > 
> > ---
> > original message
> > timestamp: Tuesday, August 05, 2014 03:50:55 PM
> > from: Jeremy Hanmer 
> > to: Dinu Vlad 
> > cc: ceph-users@lists.ceph.com 
> > subject: Re: [ceph-users] Openstack Havana root fs resize don't work
> > message id:  > ch8qy...@mail.gmail.com>
> > 
> >> This is *not* a case of that bug.  That LP bug is referring to an
> >> issue with the 'nova resize' command and *not* with an instance
> >> resizing its own root filesystem.  I can confirm that the latter case
> >> works perfectly fine in Havana if you have things configured properly.
> >> 
> >> A few questions:
> >> 
> >> 1) What workflow are you using?  (Create a volume from an image ->
> >> boot from that volume, ceps-backed ephemeral, or some other patch?)
> >> 2) What OS/release are you running?  I've gotten it to work with
> >> recent versions Centos, Debian, Fedora, and Ubuntu.
> >> 3) What are you actually seeing on the image?  Is the *partition* not
> >> being resized at all (as referenced by /proc/partions), or is it just
> >> the filesystem that isn't being resized (as referenced by df)?
> >> 
> >> On Tue, Aug 5, 2014 at 3:41 PM, Dinu Vlad  wrote:
> >> > There’s a known issue with Havana’s rbd driver in nova and it has
> >> > nothing
> >> > to do with ceph. Unfortunately, it is only fixed in icehouse. See
> >> > https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1219658 for more
> >> > details.
> >> > 
> >> > I can confirm that applying the patch manually works.
> >> > 
> >> > On 05 Aug 2014, at 11:00, Hauke Bruno Wollentin  > 
> > bruno.wollen...@innovo-cloud.de> wrote:
> >> >> Hi folks,
> >> >> 
> >> >> we use Ceph Dumpling as storage backend for Openstack Havana. However
> >> >> our
> >> >> instances are not able to resize its root filesystem.
> >> >> 
> >> >> This issue just occurs for the virtual root disk. If we start
> >> >> instances
> >> >> with an attached volume, the virtual volume disks size is correct.
> >> >> 
> >> >> Our infrastructure:
> >> >> - 1 OpenStack Controller
> >> >> - 1 OpenStack Neutron Node
> >> >> - 1 OpenStack Cinder Node
> >> >> - 4 KVM Hypervisors
> >> >> - 4 Ceph-Storage Nodes including mons
> >> >> - 1 dedicated mon
> >> >> 
> >> >> As OS we use Ubuntu 12.04.
> >> >> 
> >> >> Our cinder.conf on Cinder Node:
> >> >> 
> >> >> volume_driver = cinder.volume.driver.RBDDriver
> >> >> rbd_pool = volumes
> >> >> rbd_secret = SECRET
> >> >> rbd_user = cinder
> >> >> rbd_ceph_conf = /etc/ceph/ceph.conf
> >> >> rbd_max_clone_depth = 5
> >> >> glance_api_version = 2
> >> >> 
> >> >> Our nova.conf on hyperv

Re: [ceph-users] Regarding cache tier understanding

[Somnath Roy]

Thanks Sage, this helps !

Regards
Somnath

-Original Message-
From: Sage Weil [mailto:sw...@redhat.com]
Sent: Thursday, August 07, 2014 7:12 AM
To: Somnath Roy
Cc: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com; Anirban Ray; Allen 
Samuels
Subject: Re: Regarding cache tier understanding

On Thu, 7 Aug 2014, Somnath Roy wrote:
> Hi Sage,
>
> I went through the tiering agent code base and here is my
> understanding on the agent behavior. Please let me know if that is correct.
>
>
>
> 1.   Agent will be always in idle state if target_max_bytes or
> target_max_objects not set on the pool irrespective of other tiering
> params set in the pool. dirty_micro and full_micro will not be
> calculated if those two params are zero which is by default I guess.

Yeah

> 2.   Now, flush will be activated if dirty_micro is >
> flush_target. My understanding is, once it is activated it will
> iterate through all the dirty objects and flush all the dirty objects
> which is > cache_min_flush_age. Am I right ?

Yeah

> 3.   For the eviction, if full_micro > 100 , the mode set as
> TierAgentState::EVICT_MODE_FULL and all the clean objects are flushed.

Yeah

> 4.   if (full_micro > evict_target), the mode is set as
> TierAgentState::EVICT_MODE_SOME. In this scenario evict_effort is
> calculated and based on hit_set and temp calculation some clean
> objects are evicted. My question is , can we quantify this value ? For
> ex, if the target_full_ratio = 50%, once the eviction is triggered, what 
> %objects will be evicted ?

The effort is calculated based on how far between target_full and 100% we are.  
This is mapped onto the estimation of atime.  We generate a histogram of age 
for the objects we have examined, so after the agent has run a bit we'll have a 
reasonable idea how the current object's age compares to others and can decide 
whether this is older or newer than the others; based on that we decide what to 
do.

> 5.   Also, why we are multiplying with 100 always ? Is it
> because in the histogram it is positioned as 0..100 ?

Yeah, and we use "micro" units throughout to avoid doing any floating point.

> 6.   I saw the cache_min_evict_age is not been used anywhere, am I
> missing anything ?

It's possible.  The _min_ params are a bit dangerous because they can 
potentially confuse the cache agent (e.g., what if *all* objects are under the 
min?  How/when do we decide to ignore the min, or, how/when do we give up 
trying to find an older object?).

> 7.   The cache_min_flush_age will only be applicable if the flush
> is triggered after crossing the dirty_threshold, right ? If
> dirty_threshold is not breached, the flush age param is never checked.

Right.

> I need to understand the behavior so that we can test the Cache
> tiering properly.
>
> Is there any behavior I am missing here which you want us to test out ?

There is a new behavior just merged into master that controls our decision to 
promote data into the cache on read; see maybe_handle_cache().  That's not 
strictly agent behavior per se.  Also, there is now a readforward mode that 
doesn't promote on read ever, based on our discussion about the performance of 
flash on read.

Hope this helps!
sage



PLEASE NOTE: The information contained in this electronic mail message is 
intended only for the use of the designated recipient(s) named above. If the 
reader of this message is not the intended recipient, you are hereby notified 
that you have received this message in error and that any review, 
dissemination, distribution, or copying of this message is strictly prohibited. 
If you have received this communication in error, please notify the sender by 
telephone or e-mail (as shown above) immediately and destroy any and all copies 
of this message in your possession (whether hard copies or electronically 
stored copies).

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Is there a Ceph quick reference card?

[O'Reilly, Dan]

See the above subject.

AKA a "cheat sheet"?

Dan O'Reilly
UNIX Systems Administration
[cid:image001.jpg@01CFB232.9F0DE2B0]
9601 S. Meridian Blvd.
Englewood, CO 80112
720-514-6293


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph-deploy activate actually didn't activate the OSD

[German Anders]

Hi to all,

 I'm having some issues while trying to deploy a OSD:

ceph@cephdeploy01:~/ceph-deploy$ ceph-deploy osd prepare --fs-type 
btrfs cephosd01:sdd:sde
[ceph_deploy.cli][INFO  ] Invoked (1.4.0): /usr/bin/ceph-deploy osd 
prepare --fs-type btrfs cephosd01:sdd:sde
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks 
cephosd01:/dev/sdd:/dev/sde

[cephosd01][DEBUG ] connected to host: cephosd01
[cephosd01][DEBUG ] detect platform information from remote host
[cephosd01][DEBUG ] detect machine type
[ceph_deploy.osd][INFO  ] Distro info: Ubuntu 14.04 trusty
[ceph_deploy.osd][DEBUG ] Deploying osd to cephosd01
[cephosd01][DEBUG ] write cluster configuration to 
/etc/ceph/{cluster}.conf
[cephosd01][INFO  ] Running command: sudo udevadm trigger 
--subsystem-match=block --action=add
[ceph_deploy.osd][DEBUG ] Preparing host cephosd01 disk /dev/sdd 
journal /dev/sde activate False
[cephosd01][INFO  ] Running command: sudo ceph-disk-prepare --fs-type 
btrfs --cluster ceph -- /dev/sdd /dev/sde
[cephosd01][WARNIN] WARNING:ceph-disk:OSD will not be hot-swappable if 
journal is not the same device as the osd data
[cephosd01][WARNIN] Turning ON incompat feature 'extref': increased 
hardlink limit per file to 65536

[cephosd01][DEBUG ] The operation has completed successfully.
[cephosd01][DEBUG ] Creating new GPT entries.
[cephosd01][DEBUG ] The operation has completed successfully.
[cephosd01][DEBUG ]
[cephosd01][DEBUG ] WARNING! - Btrfs v3.12 IS EXPERIMENTAL
[cephosd01][DEBUG ] WARNING! - see http://btrfs.wiki.kernel.org before 
using

[cephosd01][DEBUG ]
[cephosd01][DEBUG ] fs created label (null) on /dev/sdd1
[cephosd01][DEBUG ] nodesize 32768 leafsize 32768 sectorsize 4096 
size 2.73TiB

[cephosd01][DEBUG ] Btrfs v3.12
[cephosd01][DEBUG ] The operation has completed successfully.
[ceph_deploy.osd][DEBUG ] Host cephosd01 is now ready for osd use.
ceph@cephdeploy01:~/ceph-deploy$

ceph@cephdeploy01:~/ceph-deploy$ ceph-deploy osd activate --fs-type 
btrfs cephosd01:sdd1:sde2
[ceph_deploy.cli][INFO  ] Invoked (1.4.0): /usr/bin/ceph-deploy osd 
activate --fs-type btrfs cephosd01:sdd1:sde2
[ceph_deploy.osd][DEBUG ] Activating cluster ceph disks 
cephosd01:/dev/sdd1:/dev/sde2

[cephosd01][DEBUG ] connected to host: cephosd01
[cephosd01][DEBUG ] detect platform information from remote host
[cephosd01][DEBUG ] detect machine type
[ceph_deploy.osd][INFO  ] Distro info: Ubuntu 14.04 trusty
[ceph_deploy.osd][DEBUG ] activating host cephosd01 disk /dev/sdd1
[ceph_deploy.osd][DEBUG ] will use init type: upstart
[cephosd01][INFO  ] Running command: sudo ceph-disk-activate 
--mark-init upstart --mount /dev/sdd1
[cephosd01][WARNIN] INFO:ceph-disk:ceph osd.0 already mounted in 
position; unmounting ours.

ceph@cephdeploy01:~/ceph-deploy$


Ok, it appear to have no errors..but then when i run a ceph osd tree I 
found that the osd.X is down and out:


ceph@cephmon01:~$ sudo ceph osd tree
# idweighttype nameup/downreweight
-12.73root default
-22.73host cephosd01
02.73osd.0down0


From the ceph -w output


...
2014-08-07 15:01:26.844127 mon.0 [INF] pgmap v15: 192 pgs: 192 
creating; 0 bytes data, 0 kB used, 0 kB / 0 kB avail

2014-08-07 15:03:04.762202 mon.0 [INF] osdmap e15: 1 osds: 0 up, 0 in
...

ceph@cephosd01:/var/lib/ceph/bootstrap-osd$ ls -ltr
total 4
-rw--- 1 root root 71 Aug  7 14:58 ceph.keyring

ceph@cephosd01:/etc/ceph$ ls -l
total 12
-rw-r--r-- 1 root root  63 Aug  7 14:58 ceph.client.admin.keyring
-rw-r--r-- 1 root root 302 Aug  7 15:02 ceph.conf
-rw-r--r-- 1 root root  92 May 12 11:14 rbdmap

Any ideas? I'm stuck here and can't go any further.

Thanks in advance,

Best regards,


German Anders
















___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph can't seem to forget

[Craig Lewis]

Have you re-formatted and re-added all of the lost OSDs?  I've found that
if you lose an OSD, you can tell Ceph the data is gone (ceph osd lost
), but it won't believe you until it can talk to that OSDID again.

If you have OSDs that are offline, you can verify that Ceph is waiting on
them with ceph pg  query, looking at the recovery_state section.
You're looking for probing_osds and down_osds_we_would_probe.  If
down_osds_we_would_probe isn't empty, then Ceph won't do anything until
it's can talk to those down OSDs again.

Once Ceph is able to probe all of the OSDs, you'll need to tell it to
create PGs that have lost all copies.  ceph pg  mark_unfound_lost
revert is used when the latest version of an object was lost, but previous
versions are still available.  ceph pg force_create_pg  is used when
all copies of a PG have been deleted. You may need to scrub the PGs too.
 I've been through this once, and I ran these commands many times before
Ceph finally agreed to re-create the missing PGs.


What your data looks like once you get the cluster healthy again, I can't
say.  I don't know how RDB or RadosGW will handle it.  Based on the number
of pools you have, I'm guessing you're using RadosGW.  If so, once this is
healthy, I would try to retrieve a copy of every object that RadosGW claims
to have, and verify it's MD5 hash.  In my case, after finally getting the
PGs created, I ended up deleting the whole pool, and re-running replication.






On Wed, Aug 6, 2014 at 11:33 AM, Sean Sullivan  wrote:

> I think I have a split issue or I can't seem to get rid of these objects.
> How can I tell ceph to forget the objects and revert?
>
> How this happened is that due to the python 2.7.8/ceph bug ( a whole rack
> of ceph went town (it had ubuntu 14.10 and that seemed to have 2.7.8 before
> 14.04). I didn't know what was going on and tried re-installing which
> killed the vast majority of the data. 2/3. The drives are gone and the data
> on them is lost now.
>
> I tried deleting them via rados but that didn't seem to work either and
> just froze there.  Any help would be much appreciated.
>
>
> Pastebin data below
> http://pastebin.com/HU8yZ1ae
>
>
> cephuser@host:~/CephPDC$ ceph --version
> ceph version 0.82-524-gbf04897 (bf048976f50bd0142f291414ea893ef0f205b51a)
>
> cephuser@host:~/CephPDC$ ceph -s
> cluster 9e0a4a8e-91fa-4643-887a-c7464aa3fd14
>  health HEALTH_WARN 2 pgs recovering; 2 pgs stuck unclean; 5 requests
> are blocked > 32 sec; recovery 478/15386946 objects degraded (0.003%);
> 23/5128982 unfound (0.000%)
>  monmap e9: 5 mons at {kg37-12=
> 10.16.0.124:6789/0,kg37-17=10.16.0.129:6789/0,kg37-23=10.16.0.135:6789/0,kg37-28=10.16.0.140:6789/0,kg37-5=10.16.0.117:6789/0},
> election epoch 1450, quorum 0,1,2,3,4 kg37-5,kg37-12,kg37-17,kg37-23,kg37-28
>  mdsmap e100: 1/1/1 up {0=kg37-5=up:active}
>  osdmap e46061: 245 osds: 245 up, 245 in
>   pgmap v3268915: 22560 pgs, 19 pools, 20020 GB data, 5008 kobjects
> 61956 GB used, 830 TB / 890 TB avail
> 478/15386946 objects degraded (0.003%); 23/5128982 unfound
> (0.000%)
>22558 active+clean
>2 active+recovering
>   client io 95939 kB/s rd, 80854 B/s wr, 795 op/s
>
>
> cephuser@host:~/CephPDC$ ceph health detail
> HEALTH_WARN 2 pgs recovering; 2 pgs stuck unclean; 5 requests are blocked
> > 32 sec; 1 osds have slow requests; recovery 478/15386946 objects degraded
> (0.003%); 23/5128982 unfound (0.000%)
> pg 5.f4f is stuck unclean since forever, current state active+recovering,
> last acting [279,115,78]
> pg 5.27f is stuck unclean since forever, current state active+recovering,
> last acting [213,0,258]
> pg 5.f4f is active+recovering, acting [279,115,78], 10 unfound
> pg 5.27f is active+recovering, acting [213,0,258], 13 unfound
> 5 ops are blocked > 67108.9 sec
> 5 ops are blocked > 67108.9 sec on osd.279
> 1 osds have slow requests
> recovery 478/15386946 objects degraded (0.003%); 23/5128982 unfound
> (0.000%)
>
> cephuser@host:~/CephPDC$ ceph pg 5.f4f mark_unfound_lost revert
> 2014-08-06 12:59:42.282672 7f7d4a6fb700  0 -- 10.16.0.117:0/1005129 >>
> 10.16.64.29:6844/718 pipe(0x7f7d4005c120 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7f7d4005c3b0).fault
> 2014-08-06 12:59:51.890574 7f7d4a4f9700  0 -- 10.16.0.117:0/1005129 >>
> 10.16.64.29:6806/7875 pipe(0x7f7d4005f180 sd=4 :0 s=1 pgs=0 cs=0 l=1
> c=0x7f7d4005fae0).fault
> pg has no unfound objects
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Can't start OSD

[O'Reilly, Dan]

# idweight  type name   up/down reweight
-1  7.2 root default
-2  1.8 host tm1cldosdl01
0   0.45osd.0   up  1
1   0.45osd.1   up  1
2   0.45osd.2   up  1
3   0.45osd.3   up  1
-3  1.8 host tm1cldosdl02
4   0.45osd.4   up  1
5   0.45osd.5   up  1
6   0.45osd.6   up  1
7   0.45osd.7   up  1
-4  1.8 host tm1cldosdl03
8   0.45osd.8   up  1
9   0.45osd.9   up  1
10  0.45osd.10  up  1
11  0.45osd.11  up  1
-5  1.8 host tm1cldosdl04
12  0.45osd.12  down0
13  0.45osd.13  down0
14  0.45osd.14  down0
15  0.45osd.15  down0
[ceph@tm1cldosdl04 ~]$ sudo /etc/init.d/ceph start osd.12
/etc/init.d/ceph: osd.12 not found (/etc/ceph/ceph.conf defines , /var/lib/ceph 
defines )

What am I missing?  Specifically, what would need to be in ceph.conf or 
/var/lib/ceph?

Dan O'Reilly
UNIX Systems Administration
[cid:image001.jpg@01CFB257.7F6F8E70]
9601 S. Meridian Blvd.
Englewood, CO 80112
720-514-6293


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Start clients during boot

[O'Reilly, Dan]

How do I get ceph clients to start after a system boot?

Dan O'Reilly
UNIX Systems Administration
[cid:image001.jpg@01CFB258.1469ADD0]
9601 S. Meridian Blvd.
Englewood, CO 80112
720-514-6293


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-deploy activate actually didn't activate the OSD

[Mark Kirkwood]

On 08/08/14 07:07, German Anders wrote:

Hi to all,

   I'm having some issues while trying to deploy a OSD:



> ceph@cephmon01:~$ *sudo ceph osd tree*
> # idweighttype nameup/downreweight
> -12.73root default
> -22.73host cephosd01
> 02.73osd.0
>

Any ideas? I'm stuck here and can't go any further.



It appears your osd is failing to start for some reason - is there 
anything interesting in /var/log/ceph/ceph-osd.0.log ?


Regards

Mark

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph can't seem to forget

[Craig Lewis]

For your RDB volumes, you've lost random 4MiB chunks from your virtual
disks.  Think of it as unrecoverable bad sectors on the HDD.  It was only a
few unfound objects though (ceph status said 23 out of 5128982).  You can
probably recovery from that.

I'd fsck all of the volumes, and perform any application level checks for
anything high level (database tests for MySQL, stuff like that).

If you still have the list of unfound objects, you might be able to trace
it back to the specific RDB volume.  That would give you a short list of
volumes to check, instead of doing them all.




On Thu, Aug 7, 2014 at 3:54 PM, Sean Sullivan  wrote:

> Thanks craig! I think I got it back up. The odd thing is that only 2 of
> the pgs using the osds on the downed nodes were corrupted.
>
> I ended up forcing all of the osds in the pool groups down, rebooting the
> hosts. Then restarting the osds and bringing them back up to get it
> working.
>
> I had previously rebooted the osds in the pgs but something must have been
> stuck.
>
> Now I am seeing corrupt data like you mentioned and am beginning to
> question the integrity of the pool.
>
> So far the cinder volume for our main login node had some corruption but
> no complaints so far. Repaired without issue.
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] rados bench no clean cleanup

[zhu qiang]

Not "cleanup", it is "--no-cleanup"

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
Kenneth Waegeman
Sent: Wednesday, August 06, 2014 4:49 PM
To: ceph-users
Subject: [ceph-users] rados bench no clean cleanup

Hi,

I did a test with 'rados -p ecdata bench 10 write' on an ECpool with a 
cache replicated pool over it (ceph 0.83).
The benchmark wrote about 12TB of data. After the 10 seconds run, rados 
started to delete his benchmark files.
But only about 2,5TB got deleted, then rados returned. I tried to do it with 
the cleanup function 'rados -p ecdata cleanup --prefix bench'
and after a lot of time, it returns:

  Warning: using slow linear search
  Removed 2322000 objects

But rados df showed the same statistics as before.
I ran it again, and it again showed 'Removed 2322000 objects', without any 
change in the rados df statistics.
It is probably the 'lazy deletion', because if I try to do a 'rados get' on it, 
there is 'No such file or directory'. But I still see the objects when I do 
'rados -p ecdata ls'.

Is this indeed because of the lazy deletion?  Is there a way to see how much 
not-deleted objects are in the pool? And is there then a reason why rados did 
remove the first 2,5TB? Or is this just a rados bench issue?:)

Thanks again!

Kenneth

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com