Re: [ceph-users] Potential OSD deadlock?

2015-09-25 Thread Robert LeBlanc
TIME_WAIT at this point. There are 130 OSDs, 8 KVM hosts hosting about 150 VMs. Open files is set at 32K for the OSD processes and 16K system wide. Does this seem like the right spot to be looking? What are some configuration items we should be looking at? Thanks, Robert LeBlanc PGP

[ceph-users] Ceph Consulting

2015-09-28 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Ceph consulting was provided by Inktank[1], but the Inktank website is down. How do we go about getting consulting services now? [1] http://ceph.com/help/professional/ Thanks, - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD

Re: [ceph-users] Potential OSD deadlock?

2015-10-03 Thread Robert LeBlanc
kHI1H5fZ5b2SIDz32/d5ywG4Gay9Rk6hp8VanvIrBbev+JYEoYT 8/WX+fhueHt4dqUYWIl3HZ0CEzbXbug0xmFvhrbmL2f3t9XOkDZRbAjlYrGm lswiJMDueY8JkxSnPvCQrHXqjbCcy9rMG7nTnLFz98rTcHNCwtpv0qVYhheg 4YRNRVMbfNP/6xsJvG1wVOSQPwxZSPqJh42pDqMRePJl3Zn66MTx5wvdNDpk l7OF =OI++ -----END PGP SIGNATURE- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6

Re: [ceph-users] Potential OSD deadlock?

2015-10-04 Thread Robert LeBlanc
0.00 5.507.50 298.0068.00 5275.25 34.9875.31 174.85 26.73 178.58 2.67 81.60 sdh 0.00 0.002.502.0028.0024.00 23.11 0.012.785.000.00 2.78 1.25 - -------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD

Re: [ceph-users] Potential OSD deadlock?

2015-10-04 Thread Robert LeBlanc
LVv9B+YI8i8mUxxv1j5 PHFi7KTgRUPm1FPMJDSyzvOgqyMj9AzaESl1Na6k529ILFIcyfko0niTT1oZ 3EPx =UDIV -END PGP SIGNATURE- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Sun, Oct 4, 2015 at 7:48 AM, Sage Weil wrote: > On Sat, 3 Oct 2015, Robert

Re: [ceph-users] Potential OSD deadlock?

2015-10-05 Thread Robert LeBlanc
Hfh/qZmJbEB+zHaqJNyg CN6w6xu1FHLeVobKAWe5ZzKY5lxw6b8YG+ce/E2dvW73gSASPTvtv68gaT04 2SPF9Ql0fERL5EDY9Pc4MHpQVcS0XxxJA69CgnWgaG6fzq2eY7fALeMBVWlB fRj3zQwqJls/X8JZ3c4P4G0R6DP9bmMwGr++oYc3gWGrvgzxw3N7+ornd0jd GdXC =Aigq -END PGP SIGNATURE----- -------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654

Re: [ceph-users] Potential OSD deadlock?

2015-10-06 Thread Robert LeBlanc
available. I'll also review the document to make sure I didn't miss anything else. Robert LeBlanc Sent from a mobile device please excuse any typos. On Oct 6, 2015 6:37 AM, "Sage Weil" wrote: > On Mon, 5 Oct 2015, Robert LeBlanc wrote: > > -BEGIN PGP SIGNED MES

Re: [ceph-users] Potential OSD deadlock?

2015-10-06 Thread Robert LeBlanc
c0 -1 error checking features: (1) Operation not permitted Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Tue, Oct 6, 2015 at 8:38 AM, Sage Weil wrote: > On Tue, 6 Oct 2015, Robert LeBlanc wrote: >> Thanks for your time Sage. It sou

Re: [ceph-users] Placement rule not resolved

2015-10-06 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 I've only done a 'step take ' where is a root entry. I haven't tried with it being under the root. I would suspect it would work, but you can try to put your tiers in a root section and test it there. - -------

Re: [ceph-users] Potential OSD deadlock?

2015-10-06 Thread Robert LeBlanc
U4lWH/ynVLLY8fOKb2/x1GEKYNz3cGKx3M0u S5wRtYvOOZEwb/B2bhvCSNtb2FgqpS4INfgKn+334ibd2X1o42oj52SA1lz/ baNh =LmYc -END PGP SIGNATURE- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Tue, Oct 6, 2015 at 10:19 AM, Sage Weil wrote: > On Tue, 6

Re: [ceph-users] Potential OSD deadlock?

2015-10-06 Thread Robert LeBlanc
NuKii4xfjI32M/ luUMXHZ8lTHk9C8MfZcpyPGvwp2FliD9LqaWOVPWtWZJcerEWcZVlEApg4qb fo5a =ahEi -END PGP SIGNATURE- ---- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Tue, Oct 6, 2015 at 6:37 AM, Sage Weil wrote: > On Mon, 5 Oct 2015, Robert LeBlanc wrote: >> -BE

Re: [ceph-users] Potential OSD deadlock?

2015-10-06 Thread Robert LeBlanc
prevents a clean build as that is my most limiting factor? Thanks, - -------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Tue, Oct 6, 2015 at 12:32 PM, Sage Weil wrote: > On Tue, 6 Oct 2015, Robert LeBlanc wrote: >> -BEGIN PGP SIGNED MES

Re: [ceph-users] Potential OSD deadlock?

2015-10-06 Thread Robert LeBlanc
ondary OSD logs. - -------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Tue, Oct 6, 2015 at 1:34 PM, Sage Weil wrote: > On Tue, 6 Oct 2015, Robert LeBlanc wrote: >> -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA256 >> >> I

Re: [ceph-users] Potential OSD deadlock?

2015-10-06 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On my second test (a much longer one), it took nearly an hour, but a few messages have popped up over a 20 window. Still far less than I have been seeing. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2

Re: [ceph-users] Potential OSD deadlock?

2015-10-06 Thread Robert LeBlanc
D6hcAZGtdvpiBRXBC5gORfiFWFxwu5kQ+daUhgUIe/O/EWyeD0rirZoqlLnZ EDrG =BZVw -END PGP SIGNATURE- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Tue, Oct 6, 2015 at 2:36 PM, Robert LeBlanc wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256

Re: [ceph-users] Q on the hererogenity

2015-10-07 Thread Robert LeBlanc
ll be transitioning this cluster back today. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Wed, Oct 7, 2015 at 3:43 AM, Andrey Shevel wrote: > Hello everybody, > > we are discussing experimental installation ceph cluster under > S

Re: [ceph-users] Potential OSD deadlock?

2015-10-07 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 We forgot to upload the ceph.log yesterday. It is there now. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Tue, Oct 6, 2015 at 5:40 PM, Robert LeBlanc wrote: > -BEGIN PGP SIG

Re: [ceph-users] Potential OSD deadlock?

2015-10-08 Thread Robert LeBlanc
e Jgs+aT2m1erdSyZn0ZC9a6CjWmwJXY6FCSGhE53BbefBxmCFxn+8tVav+Q8W 7s14TntP6ex4ca7eTwGuSXC9FU5fAVa+3+3aXDAC1QPAkeVkXyB716W1XG6b BCFo =GJL4 -END PGP SIGNATURE- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Wed, Oct 7, 2015 at 1:25 PM, Robert Le

Re: [ceph-users] Potential OSD deadlock?

2015-10-14 Thread Robert LeBlanc
i+pMy XU8gYbG1um5YY8jrtvvkhnbHdeO/k24/cH7MGslxeezBPnMNzmqj3qVdiX1H EBbyBBtp8OF+pKExrmZc2w01W/Nxl6GbVoG+IKJ61FgwKOXEiMwb0wv5mu30 eP3D =R0O9 -END PGP SIGNATURE- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Wed, Oct 14, 2015 at 12:00 AM, Haomai

Re: [ceph-users] Potential OSD deadlock?

2015-10-14 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 I'm sure I have a log of a 1,000 second block somewhere, I'll have to look around for it. I'll try turning that knob and see what happens. I'll come back with the results. Thanks, - -------- Robert LeBlanc PGP Finger

[ceph-users] Cache Tiering Question

2015-10-15 Thread Robert LeBlanc
%. Will this cause problems with the relative cache tier settings? Do I need to set the percentage based on what Ceph is reporting here? Thanks, - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 -BEGIN PGP SIGNATURE- Version: Mailvelope

Re: [ceph-users] Cache Tiering Question

2015-10-15 Thread Robert LeBlanc
that cache_target_{dirty,dirty_high,full}_ratio works as a ratio of target_max_bytes set? - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Thu, Oct 15, 2015 at 3:32 PM, Nick Fisk wrote: > > > > >> -Original Message- >>

Re: [ceph-users] Cache Tiering Question

2015-10-15 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 One more question. Is max_{bytes,objects} before or after replication factor? - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Thu, Oct 15, 2015 at 4:42 PM, LOPEZ Jean-Charles wrote: >

Re: [ceph-users] Cache Tiering Question

2015-10-15 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Thanks - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Thu, Oct 15, 2015 at 5:38 PM, JC Lopez wrote: > Hi Robert > > usable bytes so before replication. The size of the actual

Re: [ceph-users] Error after upgrading to Infernalis

2015-10-16 Thread Robert LeBlanc
. {} +) - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Fri, Oct 16, 2015 at 7:41 AM, German Anders wrote: > Hi all, > > I'm trying to upgrade a ceph cluster (prev hammer release 0.94.3) to the > last release of infernalis (9.1.0-61-gf2b9f89

Re: [ceph-users] Cache Tiering Question

2015-10-16 Thread Robert LeBlanc
jeSfDf0MmK3S5bXtLfKKZTKYv0m/+B8PY KzZcuVTavBhFSLWiT3L2U1OOyPz5AEu2ezE2Y6ElFePc+g38eO/I7kuTSixV +0yZL1tO6vEYZLnwWTWgYFmmrOA5yTBvssGpjpZVPe7swkJG97kvqe/bh2/W OqQ5PEnhn5Gx3vIDHJwvI/PT4MXZk2VU9cpPMPs7PeIQBPZYPi0/WcfT8m+g oclkznsM+BSLMiTT8yBc7/T1kLFQXS42jVXEFAKYnJj8LIk0aMc54Gu25g0w PM6+IFROsMQlGdybbWCPXIXsZ

Re: [ceph-users] Cache Tiering Question

2015-10-16 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 I started another fio test to one of the same RBDs (leaving the hung ones still hung) and it is working OK, but the hungs ones are still just hung. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

Re: [ceph-users] Cache Tiering Question

2015-10-16 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Is the only option to restart the librbd client in this case? Anything I can do to help resolve it? Thanks, - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Fri, Oct 16, 2015 at 10:17 AM

Re: [ceph-users] Ceph journal - isn't it a bit redundant sometimes?

2015-10-19 Thread Robert LeBlanc
d. I still might try... - -------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Mon, Oct 19, 2015 at 12:18 PM, Jan Schermer wrote: > I'm sorry for appearing a bit dull (on purpose), I was hoping I'd hear what > other people using Ceph think. > >

Re: [ceph-users] Ceph journal - isn't it a bit redundant sometimes?

2015-10-20 Thread Robert LeBlanc
can do. You are still ultimately bound by the back end disk. Robert LeBlanc Sent from a mobile device please excuse any typos. On Oct 20, 2015 2:34 AM, "Luis Periquito" wrote: > On Tue, Oct 20, 2015 at 3:26 AM, Haomai Wang wrote: > > The fact is that journal could help a lo

Re: [ceph-users] upgrading to major releases

2015-10-23 Thread Robert LeBlanc
safe. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Fri, Oct 23, 2015 at 10:27 AM, John-Paul Robinson wrote: > Hi, > > When upgrading to the next release, is it necessary to first upgrade to > the most recent point release of the prior release or can o

[ceph-users] PG won't stay clean

2015-10-25 Thread Robert LeBlanc
clean. I shut down each OSD in turn and removed any PG_TEMP directories. I'm just not sure why the cluster is so confused as to the status of this PG. Any ideas? - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 -BEGIN PGP SIGNATURE- Ve

Re: [ceph-users] PG won't stay clean

2015-10-25 Thread Robert LeBlanc
. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Sun, Oct 25, 2015 at 5:51 PM, Robert LeBlanc wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > I have a 0.94.4 cluster that when I repair/deep-scrub a PG, it comes > back clean, b

Re: [ceph-users] PG won't stay clean

2015-10-26 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 That seems to have resolved the problem. Thanks! I guess the repair does not increase the epoc of the PG or something so it loses it's brain about it being clean. I'm glad it was a simple resolution. - -------- Robert LeBlanc PGP F

[ceph-users] Not possible to remove cache tier with RBDs open?

2015-10-26 Thread Robert LeBlanc
o not have to shut down VMs to remove a cache tier. Thanks, - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 -BEGIN PGP SIGNATURE- Version: Mailvelope v1.2.2 Comment: https://www.mailvelope.com wsFcBAEBCAAQBQJWLlMrCRDmVDuy+mK58QA

Re: [ceph-users] creating+incomplete issues

2015-10-28 Thread Robert LeBlanc
Please paste 'ceph osd tree'. Robert LeBlanc Sent from a mobile device please excuse any typos. On Oct 28, 2015 6:54 PM, "Wah Peng" wrote: > Hello, > > Just did it, but still no good health. can you help? thanks. > > ceph@ceph:~/my-cluster$ ceph osd stat &g

Re: [ceph-users] creating+incomplete issues

2015-10-28 Thread Robert LeBlanc
You need to change the CRUSH map to select osd instead of host. Robert LeBlanc Sent from a mobile device please excuse any typos. On Oct 28, 2015 7:00 PM, "Wah Peng" wrote: > $ ceph osd tree > # idweight type name up/down reweight > -1 0.24root def

Re: [ceph-users] creating+incomplete issues

2015-10-28 Thread Robert LeBlanc
ing-a-crush-map Robert LeBlanc Sent from a mobile device please excuse any typos. On Oct 28, 2015 7:46 PM, "Wah Peng" wrote: > Is there a ceph sub-command existing instead of changing the config file? > :) > > > On 2015/10/29 星期四 9:24, Li, Chengyuan wrote: > >>

Re: [ceph-users] rados bench leaves objects in tiered pool

2015-11-03 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Try: rados -p {cachepool} cache-flush-evict-all and see if the objects clean up. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Tue, Nov 3, 2015 at 8:02 AM, Gregory Farnum wrote: > W

Re: [ceph-users] ceph-deploy on lxc container - 'initctl: Event failed'

2015-11-06 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 I've put monitors in LXC but I haven't done it with ceph-deploy. I've had no problems with it. - -------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Fri, Nov 6, 2015 at 12:55 P

Re: [ceph-users] Ceph performances

2015-11-07 Thread Robert LeBlanc
er so there was no spindle activity. Short answer is that your will need a lot more SSDS to hit your target with hammer. Or if you can wait for Jewel you may be able to get by with only needing a little bit more. Robert LeBlanc Sent from a mobile device please excuse any typos. On Nov 7, 2015

Re: [ceph-users] Issue activating OSDs

2015-11-07 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 It can't talk to the monitor at 192.168.107.67. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Sat, Nov 7, 2015 at 1:23 PM, James Gallagher wrote: > Hi, > > I've rec

Re: [ceph-users] (no subject)

2015-11-12 Thread Robert LeBlanc
firewall. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Thu, Nov 12, 2015 at 7:21 AM, James Gallagher wrote: > Hi, I'm having issues activating my OSDs. I have provided the output of the > fault. I can see that the error messa

Re: [ceph-users] about PG_Number

2015-11-13 Thread Robert LeBlanc
similar reasons. Since increasing the PG number things have gotten a lot better, there is still a pretty big descrepency between the active and idle disks and after we get our SSD cache tier in we will change to straw2 and do some balancing. Robert LeBlanc Sent from a mobile device please excuse any

[ceph-users] SSD Caching Mode Question

2015-11-17 Thread Robert LeBlanc
[3] http://robert.leblancnet.us/files/promote_evict.png Thanks, - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 -BEGIN PGP SIGNATURE- Version: Mailvelope v1.2.3 Comment: https://www.mailvelope.com wsFcB

[ceph-users] OSD Recovery Delay Start

2015-11-17 Thread Robert LeBlanc
get back to normal performance. [1] http://docs.ceph.com/docs/v0.94/rados/configuration/osd-config-ref/#recovery Thanks, - -------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 -BEGIN PGP SIGNATURE- Version: Mailvelope v1.2.3 Comment: https://

Re: [ceph-users] High load during recovery (after disk placement)

2015-11-20 Thread Robert LeBlanc
NQRBFfjdpYzoLv MreC+bHCpA/gN3KMFe4quNfUTAUrVDQ1rMDX3hNTqrIJjXg77ZFL0UMGiGwG 8B6j =dG2z -END PGP SIGNATURE- ---- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Fri, Nov 20, 2015 at 10:33 AM, Simon Engelsman wrote: > Hi, > > We've experienced a very weird pr

Re: [ceph-users] SSD journals killed by VMs generating 500 IOPs (4kB) non-stop for a month, seemingly because of a syslog-ng bug

2015-11-22 Thread Robert LeBlanc
sd63XGfau3Oku59+vvb9MxmrFwEMtCqGlS/sKYP0 NhLG5wmV6vEfsgyzpUaTio+Fws0juj8GeUL6a9Yp1JaNGxN2HIyCFwD8FzTp 5oBU =scws -END PGP SIGNATURE- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Sun, Nov 22, 2015 at 10:40 AM, Alex Moore wrote: > I just had 2 of th

Re: [ceph-users] SSD Caching Mode Question

2015-11-23 Thread Robert LeBlanc
SIGNATURE- -------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Mon, Nov 23, 2015 at 9:49 AM, Samuel Just wrote: > My read of that doc is that you still need to either set the configs > to force all objects to be flushed or use the rados command to > flus

Re: [ceph-users] CEPH over SW-RAID

2015-11-23 Thread Robert LeBlanc
ave a specific need, it is best to forgo the Linux SW RAID or even HW RAIDs too with Ceph. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Mon, Nov 23, 2015 at 10:09 AM, Jose Tavares wrote: > Hi guys ... > > Is there any advantage in run

Re: [ceph-users] Ceph 0.94.5 with accelio

2015-11-23 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Are you using unconnected mode or connected mode? With connected mode you can up your MTU to 64K which may help on the network side. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Mon, Nov

Re: [ceph-users] Ceph 0.94.5 with accelio

2015-11-24 Thread Robert LeBlanc
ZGmPVDhmVt7fRYz/+rARkN/jwXH4qG x/Dk =/88f -END PGP SIGNATURE----- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Tue, Nov 24, 2015 at 8:24 AM, German Anders wrote: > Another test make between two HP blades with QDR (

Re: [ceph-users] Ceph 0.94.5 with accelio

2015-11-24 Thread Robert LeBlanc
ck is my disks and CPU. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Tue, Nov 24, 2015 at 10:26 AM, German Anders wrote: > Thanks a lot Robert for the explanation. I understand what you are saying > and I'm also excited to s

Re: [ceph-users] Scrubbing question

2015-11-25 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Since the one that is different is not your primary for the pg, then pg repair is safe. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Wed, Nov 25, 2015 at 2:42 AM, Major Csaba wrote: >

Re: [ceph-users] Ceph performances

2015-11-25 Thread Robert LeBlanc
---END PGP SIGNATURE----- -------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Wed, Nov 25, 2015 at 5:40 AM, Rémi BUISSON wrote: > > > > > Hello Robert, > > Sorry for late answer. > > Thanks for your reply. I updated to infernalis and I ap

Re: [ceph-users] Cache Tiering Investigation and Potential Patch

2015-11-25 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 I think if it is not too much, we would like N out of M. I don't know specifically about only building the one package, but I build locally with make to shake out any syntax bugs, then I run make-debs.sh which takes about 10-15 minutes to build to i

Re: [ceph-users] Upgrade to hammer, crush tuneables issue

2015-11-25 Thread Robert LeBlanc
or queued. - -------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Wed, Nov 25, 2015 at 2:42 PM, Tomasz Kuzemko wrote: > To ease on clients you can change osd_backfill_scan_min and > osd_backfill_scan_max to 1. It's possible to change thi

Re: [ceph-users] Upgrade to hammer, crush tuneables issue

2015-11-26 Thread Robert LeBlanc
DbNaK QjWQGbAbz15QrcCmeAY+79fq3Ndl5PhUfIt5tlW2+jKJgK27YaAhoTyVx7Rm IXC8 =JwlH -END PGP SIGNATURE- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Thu, Nov 26, 2015 at 1:52 AM, Tomasz Kuzemko wrote: > This has nothing to do with the number of

Re: [ceph-users] OSD on a partition

2015-12-01 Thread Robert LeBlanc
j7DoaBMxTT4igWfHwWb5/I 0R5bYkheBkYxLlVaf7faUWcjySwunW1SY/rc2FkUFe52VlZ5cbFfJ+ym0an5 i5SdfLd0gk4zR5l35j7svdJZU9+QIZLcz/S12Nx5mwUxhnhEeqYMBS/ENSca tKq4nlqyIGaCyDaLlcaECRLBjskrNRMeV7vnNUQ59BzJuMWOHhq571zHeXYO tezS =mxz9 -END PGP SIGNATURE----- -------- Robert LeBlanc PGP Fingerprint 79A2

Re: [ceph-users] OSD on a partition

2015-12-01 Thread Robert LeBlanc
Hammer as recently as two weeks ago. - -------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Tue, Dec 1, 2015 at 3:06 PM, Marek Dohojda wrote: > > Didn’t mean to send previous email, I appologize for the spam. > > Anyway Thank you, I

Re: [ceph-users] Remap PGs with size=1 on specific OSD

2015-12-03 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Reweighting the OSD to 0.0 or setting the osd out (but not terminating the process) should allow it to backfill the PGs to a new OSD. I would try the reweight first (and in a test environment). - Robert LeBlanc PGP Fingerprint 79A2

Re: [ceph-users] mark out vs crush weight 0

2016-05-23 Thread Robert LeBlanc
+3MejQPzr9UDay/k4 7gxrNrB7YJ7YIX5i2yGYfE+tNVNUD4nGBgPCcBY7yDAzvbBKM5HzZSxWfYv6 JULq+EVc592gGjUx8BI+vJnckV3yGABCrVdUda2xxYwjMkIHbnoQtL7yi3DL W7Y5Z5iIDGSSpDcMOIEzSCiABzuKJHQC+EPf1NHGbEtK7ZGFPqmVx98eREgO oyjl =U0LK -END PGP SIGNATURE- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4

Re: [ceph-users] Blocked ops, OSD consuming memory, hammer

2016-05-26 Thread Robert LeBlanc
zMaAkja95aC7mxzfXTsxQj68iIAKUp 47Nwaz4ln2A5f20SQe3W4jxp33MKsAJYej2/xn/B0roxH7ZTAhXlcpYhU8Ni s5aw =X+rk -END PGP SIGNATURE- ---- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Tue, May 24, 2016 at 3:16 PM, Heath Albritton wrote: > Hav

Re: [ceph-users] deep-scrubbing has large impact on performance

2016-11-22 Thread Robert LeBlanc
If you use wpq, I recommend also setting "osd_op_queue_cut_off = high" as well, otherwise replication OPs are not weighted and really reduces the benefit of wpq. -------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Tue, Nov 22, 2016

Re: [ceph-users] problem after reinstalling system

2015-12-09 Thread Robert LeBlanc
idn't require it, but going back required the backfill. - -------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Wed, Dec 9, 2015 at 5:25 AM, Jacek Jarosiewicz wrote: > Hi, > > I have a working ceph cluster with storage nodes running Ubun

Re: [ceph-users] Blocked requests after "osd in"

2015-12-09 Thread Robert LeBlanc
uSnL4eFGIe9yg2jHSGVy fmDFEirT3DlxFEDWja8uNFGdJ8rMYjTqOMdyOCS3SLtizTmC/+SF00kk0m9A sB8x =i9F/ -END PGP SIGNATURE- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Wed, Dec 9, 2015 at 7:33 AM, Christian Kauhaus wrote: > Am 09.12.201

Re: [ceph-users] building ceph rpms, "ceph --version" returns no version

2015-12-09 Thread Robert LeBlanc
/4RGxfr9O4T1LYR3FvBCd8VWHZn+kbvrx9L2Gydz+hjlcuS yztGzzjuXUiCpnmRi7I6z1Myf2+D2ZbGkx4q+8s3c8zg/7TwrNdEXG+l9Qqw +q1CbKWyX8u3MJfHWB5S6UMc3tPkMgWml7sV4OUEbEfeMU/znbKTniqa9C9M Po6n =nE/8 -END PGP SIGNATURE- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

Re: [ceph-users] All pgs stuck peering

2015-12-13 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 I've had something similar to this when there was an MTU mismatch, the smaller I/O would get through, but the larger I/O would be blocked and prevent peering. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654

Re: [ceph-users] How to configure if there are tow network cards in Client

2015-12-25 Thread Robert LeBlanc
network through NIC1. You may need a corresponding route on the router to send traffic to that NIC, but I assume that is already working. There is no Ceph configuration that you need to do. I'm moving this to the ceph-users list where it is more appropriate. - Robert LeBlan

Re: [ceph-users] cephfs, low performances

2015-12-31 Thread Robert LeBlanc
tter drive utilization across the cluster as well as 3-17% performance increase and a substantial reduction in client performance deviation (all clients are getting the same amount of performance). Hopefully we will be able to get that into Jewel. Robert LeBlanc Sent from a mobile device please excuse any

Re: [ceph-users] Need suggestions for using ceph as reliable block storage

2016-01-03 Thread Robert LeBlanc
t bad. Robert LeBlanc Sent from a mobile device please excuse any typos. On Jan 3, 2016 9:04 PM, "Kalyana sundaram" wrote: > Hi > We are building our private cloud. We have decided to use ceph to provide > features like ebs. How can we make it transparent for vms when one ceph &

Re: [ceph-users] Need suggestions for using ceph as reliable block storage

2016-01-03 Thread Robert LeBlanc
ith the VM or RBD, or is less likely to lock up the whole box. Robert LeBlanc Sent from a mobile device please excuse any typos. On Jan 3, 2016 10:39 PM, "Kalyana sundaram" wrote: > As far as I read as soon as a osd is marked down writes won't recover > because pgs have to

Re: [ceph-users] KVM problems when rebalance occurs

2016-01-06 Thread Robert LeBlanc
GXlbWCxoI7+StJDHPnY2k7NHLvbN2yG3f5PZvZaBgqqyAP8Fnq6CDtTIE vZ/p+ZcuRw8lqoDgjjdiFyMmhQnFcCtDo3vtIy/UXDw23AVsI5edUyyP/sHt ruPt =X7SH -END PGP SIGNATURE- -------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Wed, Jan 6, 2016 at 7:13 AM, nick wrote: > Heya, > we are using a ceph

Re: [ceph-users] rbd partition table

2016-01-06 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 It is just a block device, you can use it with or without a partition. I should be careful with that statement as bcache looks like a block device, but you can not partition it directly. - Robert LeBlanc PGP Fingerprint 79A2 9CA4

Re: [ceph-users] Any suggestion to deal with slow request?

2016-01-07 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 What is the file system on the OSDs? Anything interesting in iostat/atop? What are the drives backing the OSDs? A few more details would be helpful. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

Re: [ceph-users] using cache-tier with writeback mode, raods bench result degrade

2016-01-08 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Are you backporting that to hammer? We'd love it. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Fri, Jan 8, 2016 at 9:28 AM, Nick Fisk wrote: > There was/is a bug in Infernalis a

Re: [ceph-users] using cache-tier with writeback mode, raods bench result degrade

2016-01-11 Thread Robert LeBlanc
nt to take credit for Nick's work, so I've been waiting. Thanks - -------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Mon, Jan 11, 2016 at 4:29 AM, Nick Fisk wrote: > Looks like it has been done > > https://github.

Re: [ceph-users] using cache-tier with writeback mode, raods bench result degrade

2016-01-11 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 https://github.com/ceph/ceph/pull/7024 - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Mon, Jan 11, 2016 at 1:47 PM, Robert LeBlanc wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash:

Re: [ceph-users] Ceph cache tier and rbd volumes/SSD primary, HDD replica crush rule!

2016-01-12 Thread Robert LeBlanc
7grO68DTR1E26GEoINv7q3UhpQGsXzrnvYJwZoVv cabSHDm9TIF0hlorQRTdvzElALuoxrB/rfpxsGhC3FFlrkfgsNA4QottF+dv AbcxnIfMD+HoOxLadL4xKiRUVHOtgUtKuRCFGqEL7FaagyE165PiiTmR0tJk H+Cz6wz/fW7CnoaxBE+M733hkdTb4QYnf0wqdkJIQ7Flec988/Ds+fxPcE8o Hx7S =3WAd -END PGP SIGNATURE- Robert LeBlanc PGP Fi

[ceph-users] Odd single VM ceph error

2016-01-14 Thread Robert LeBlanc
em is EXT4 and this is 0.94.5. Thanks, Robert LeBlanc Sent from a mobile device please excuse any typos. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Odd single VM ceph error

2016-01-15 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Exactly the problem. Thanks for the point in the right direction. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Fri, Jan 15, 2016 at 10:28 AM, Sage Weil wrote: > On Thu, 14 Jan 2

Re: [ceph-users] Infernalis upgrade breaks when journal on separate partition

2016-01-15 Thread Robert LeBlanc
converted from MBR to GPT, but it didn't require any resizing that I remember. I'd test it in a VM or similar to make sure you understand the process. You will also have to manually add the Ceph Journal UUID to the partition after the conversion for it to all work automatically. - ---

Re: [ceph-users] Ceph Cache pool redundancy requirements.

2016-01-18 Thread Robert LeBlanc
YDmmtouMvN0KuEqR+HHErVzifjOX7D5QXNdjlAtMhzmPB45D/zcrrEXk JYpiDTWCCoADtIai+uyqZaXoE311nne4lv89gzaqZQTjXep5bLvkPHQwkUPn GF6aOgGIZjZf199EkImqsQYmoIyTCXKlwwxcDyFqGcqk5/m43IapcpZ/SnN/ lXPM =iUPg -END PGP SIGNATURE- -------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On S

Re: [ceph-users] CRUSH Rule Review - Not replicating correctly

2016-01-18 Thread Robert LeBlanc
VKKd lQJdxcwXOW7/v5oayY4wiEE6NF4rCupcqt0nPxxmbehmeRPxgkWCKJJs3FNr VmUdnrdpfxzR5c8dmOELJnpNS6MTT56B8A4kKmqbbHCEKpZ83piG7uwqc+6f RKkQ =gp/0 -END PGP SIGNATURE- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Sun, Jan 17, 2016 at 6:31 PM, deeepdish wrote: > Hi Everyone, >

Re: [ceph-users] CRUSH Rule Review - Not replicating correctly

2016-01-18 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Not that I know of. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Mon, Jan 18, 2016 at 10:33 AM, deeepdish wrote: > Thanks Robert. Will definitely try this. Is there a way to implem

Re: [ceph-users] How to set a new Crushmap in production

2016-01-20 Thread Robert LeBlanc
pool, but it is in the man page. You learn something new everyday. [1] https://www.mail-archive.com/ceph-users@lists.ceph.com/msg26017.html - -------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Wed, Jan 20, 2016 at 7:11 AM, Vincent Godin wrote: &

Re: [ceph-users] SSD OSDs - more Cores or more GHz

2016-01-20 Thread Robert LeBlanc
Ku+h8GiyGrY+CctNRB9rxeHJ9uMyhh+DcRUFXLAQf0YO gvUX =E4LJ -END PGP SIGNATURE- ---- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Wed, Jan 20, 2016 at 2:01 AM, Götz Reinicke - IT Koordinator wrote: > Hi folks, > > we plan to use more ssd O

Re: [ceph-users] Ceph OSD network configuration

2016-01-24 Thread Robert LeBlanc
If you want redundancy and link aggregation then you will use LACP (or one of the Linux pseudo modes) and bonding in linux. This is common and not Ceph specific. Robert LeBlanc Sent from a mobile device please excuse any typos. On Jan 23, 2016 6:39 PM, "名花" wrote: > > Hi, I hav

Re: [ceph-users] Ceph Cache Tiering Error error listing images

2016-01-27 Thread Robert LeBlanc
time (luckily we tested on a dev cluster first). Robert LeBlanc Sent from a mobile device please excuse any typos. On Jan 27, 2016 3:26 AM, "Ferhat Ozkasgarli" wrote: > Hello, > > I have installed OpenStack cluster with Mirantis Fuel 7.0. Back end > storage is ceph and work

Re: [ceph-users] Lost access when removing cache pool overlay

2016-01-29 Thread Robert LeBlanc
cache tier, it could no longer access the pool. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Fri, Jan 29, 2016 at 8:47 AM, Gerd Jakobovitsch wrote: > Dear all, > > I had to move .rgw.buckets.index pool to another structure; the

Re: [ceph-users] SSD Journal

2016-01-29 Thread Robert LeBlanc
from, only when the OSD process crashes. I'm happy to be corrected if I've misstated anything. Robert LeBlanc -BEGIN PGP SIGNATURE- Version: Mailvelope v1.3.4 Comment: https://www.mailvelope.com wsFcBAEBCAAQBQJWq5u8CRDmVDuy+mK58QAALaoP/3r6oB7cZepby/KRbEGO pXfZ0X9bDW5S55Kg

Re: [ceph-users] Install ceph with infiniband

2016-02-01 Thread Robert LeBlanc
a lot about the progress lately, but that doesn't mean nothing is going on. - -------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Mon, Feb 1, 2016 at 6:37 AM, 1 <10...@candesoft.com> wrote: > Hello, > I am going to instal

Re: [ceph-users] Optimal OSD count for SSDs / NVMe disks

2016-02-03 Thread Robert LeBlanc
n the cache, there is still a performance penalty for having the caching tier vs. a native SSD pool. So if you are not using the tiering, move to a straight SSD pool. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Wed, Feb 3, 2016 at 5:

Re: [ceph-users] Set cache tier pool forward state automatically!

2016-02-03 Thread Robert LeBlanc
gTJLHH+dbrW32MZEl1Gd yTKyzFarbae6tbAmaMPC8l9vaj15t7bAB0KOokMqZied7EcM1ZoFVqKRahrm 73mIeHiDUwZ8gi+BHKX7OwqKt3VZJYf/+rNJx+g4kp5WN0FEkUMoqF75qO4p 62+PuQIwh6jUpB4cDsbEJd78UGbCptJBojmsNVogU+xiSXTKQmEduP0HqQfG JhTLg3Un2C4/MSGbhRI26csFCzEi66iRXQWdfCITP4Um70KO6dE2C1MAveYg hJ7b =CaRF -END PGP SIGNATURE----- -------- Robert LeBlanc

Re: [ceph-users] Set cache tier pool forward state automatically!

2016-02-03 Thread Robert LeBlanc
+/D8dt4lb6OSvxYk983RuY4SSWjwIc sJMzBXVsQfuzE0iF0BuuqXdgRoPmr8FofDN/Vm6eF6wz6nGPGlUjT6t13b0y 1RLhEPtq2YcYNkiE6dT97g16PRCW7R7h1bnQ5EVUIzFb+Dzh7rtjTjyYmm2z geuA =gMKn -END PGP SIGNATURE- ---- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Wed, Feb 3, 2016 at 5:07 PM, Mih

Re: [ceph-users] Set cache tier pool forward state automatically!

2016-02-03 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On Wed, Feb 3, 2016 at 9:00 PM, Christian Balzer wrote: > On Wed, 3 Feb 2016 16:57:09 -0700 Robert LeBlanc wrote: > That's an interesting strategy, I suppose you haven't run into the issue I > wrote about 2 days ago when s

Re: [ceph-users] Upgrading with mon & osd on same host

2016-02-04 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Just make sure that your monitors and OSDs are on the very latest of Hammer or else your Infernalis OSDs won't activate. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Thu, Feb 4, 2016

Re: [ceph-users] Set cache tier pool forward state automatically!

2016-02-04 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On Thu, Feb 4, 2016 at 8:32 PM, Christian Balzer wrote: > On Wed, 3 Feb 2016 22:42:32 -0700 Robert LeBlanc wrote: > I just finished downgrading my test cluster from testing to Jessie and > then upgrading Ceph from Firefly to Hammer (tha

Re: [ceph-users] Unified queue in Infernalis

2016-02-05 Thread Robert LeBlanc
I believe this is referring to combining the previously separate queues into a single queue (PrioritizedQueue and soon to be WeightedPriorityQueue) in ceph. That way client IO and recovery IO can be better prioritized in the Ceph code. This is all before the disk queue. Robert LeBlanc Sent from

Re: [ceph-users] K is for Kraken

2016-02-08 Thread Robert LeBlanc
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Too bad K isn't an LTS. It was be fun to release the Kraken many times. I like liliput - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Mon, Feb 8, 2016 at 11:36 AM, Sage Weil wrote

<    1   2   3   4   5   >