[ceph-users] Cache Tier 1 vs. Journal

Hello! If I using cache tier 1 pool in writeback mode, it is a good idea turn off journal on OSDs? I think in this sutuation journal can help if you are hit a rebalance procedure in a "cold" storage. In outer situation the journal is useless, I think. Any comments? __

Re: [ceph-users] Ceph Supermicro hardware recommendation

17.02.2015 04:11, Christian Balzer пишет: > > Hello, > > re-adding the mailing list. > > On Mon, 16 Feb 2015 17:54:01 +0300 Mike wrote: > >> Hello >> >> 05.02.2015 08:35, Christian Balzer пишет: >>> >>> Hello, >>> >>&

[ceph-users] What a maximum theoretical and practical capacity in ceph cluster?

Hello, My company is plaining to build a big Ceph cluster for achieving and storing data. By requirements from customer - 70% of capacity is SATA, 30% SSD. First day data is storing in SSD storage, on next day moving SATA storage. By now we decide use a SuperMicro's SKU with 72 bays for HDD = 22 S

[ceph-users] Again: full ssd ceph cluster

Hello all! Some our customer asked for only ssd storage. By now we looking to 2027R-AR24NV w/ 3 x HBA controllers (LSI3008 chip, 8 internal 12Gb ports on each), 24 x Intel DC S3700 800Gb SSD drives, 2 x mellanox 40Gbit ConnectX-3 (maybe newer ConnectX-4 100Gbit) and Xeon e5-2660V2 with 64Gb RAM. Re

Re: [ceph-users] Again: full ssd ceph cluster

Hello, On 12/11/2014 11:35 AM, Christian Balzer wrote: > > Hello, > > On Wed, 10 Dec 2014 18:08:23 +0300 Mike wrote: > >> Hello all! >> Some our customer asked for only ssd storage. >> By now we looking to 2027R-AR24NV w/ 3 x HBA controllers (LSI3008 chip, >

Re: [ceph-users] Number of SSD for OSD journal

15.12.2014 23:45, Sebastien Han пишет: > Salut, > > The general recommended ratio (for me at least) is 3 journals per SSD. Using > 200GB Intel DC S3700 is great. > If you’re going with a low perf scenario I don’t think you should bother > buying SSD, just remove them from the picture and do 12 S

Re: [ceph-users] Number of SSD for OSD journal

16.12.2014 10:53, Daniel Schwager пишет: > Hallo Mike, > >> This is also have another way. >> * for CONF 2,3 replace 200Gb SSD to 800Gb and add another 1-2 SSD to >> each node. >> * make tier1 read-write cache on SSDs >> * also you can add journal partition on

[ceph-users] Change size journal's blocks from 4k to another.

sdr2410.00 301.25 0.00301 0 I think my cluster have a bottle neck in journal block size. How I can increase the size of block for journal? -- Best regards, Mike. ___ ceph-users mailing list ceph-users@lists.ceph.com h

[ceph-users] 5Tb useful space based on Erasure Coded Pool

t yet learned much about the difference erasure plugins, performance, low level configuration. Have you some advice about it? It's can work at all or not? Erasure and this implementation it Ceph can solve the task? For any advice thanks. --

Re: [ceph-users] Building a Pb EC cluster for a cheaper cold storage

10.11.2015 19:40, Paul Evans пишет: > Mike - unless things have changed in the latest versions(s) of Ceph, I *not* > believe CRUSH will be successful in creating a valid PG map if the ’n' value > is 10 (k+m), your host count is 6, and your failure domain is set to host. &g

Re: [ceph-users] Building a Pb EC cluster for a cheaper cold storage

10.11.2015 19:40, Paul Evans пишет: > Mike - unless things have changed in the latest versions(s) of Ceph, I *not* > believe CRUSH will be successful in creating a valid PG map if the ’n' value > is 10 (k+m), your host count is 6, and your failure domain is set to host. &g

Re: [ceph-users] Building a Pb EC cluster for a cheaper cold storage

11.11.2015 06:14, Christian Balzer пишет: > > Hello, > > On Tue, 10 Nov 2015 13:29:31 +0300 Mike Almateia wrote: > >> Hello. >> >> For our CCTV storing streams project we decided to use Ceph cluster with >> EC pool. >> Input requirements is not

[ceph-users] OSD/BTRFS: OSD didn't start after change btrfs mount options

d to interpret this. *** I'm stuck. I can't understand what I did wrong and how recover the OSDs? Googling didn't help me. -- Mike, runs! ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] PG calculate for cluster with a huge small objects

pool is just 4.2k. If we just will be increase PG/PGP num for this pool we stuck to limit a PGP number per OSD after some time. What we will do? Count of objects are increasing, space - a few. What we living witch cluster of huge small size objects? Any suggestion? -- Mike, run

Re: [ceph-users] Multiple L2 LAN segments with Ceph

- node1: 10.2.1.1/24 - node2: 10.2.1.2/24 - public-leaf2: 10.2.2.0/24 ceph.conf would be: cluster_network: 10.1.0.0/255.255.0.0 public_network: 10.2.0.0/255.255.0.0 - Mike Dawson On 5/28/2014 1:01 PM, Travis Rhoden wrote: Hi folks, Does anybody know if there are any issues ru

Re: [ceph-users] Calamari Goes Open Source

Great work Inktank / Red Hat! An open source Calamari will be a great benefit to the community! Cheers, Mike Dawson On 5/30/2014 6:04 PM, Patrick McGarry wrote: Hey cephers, Sorry to push this announcement so late on a Friday but... Calamari has arrived! The source code bits have been

Re: [ceph-users] How to avoid deep-scrubbing performance hit?

is each individual object inside the PG locked as it is processed? Some of my PGs will be in deep-scrub for minutes at a time. 0: http://ceph.com/docs/master/dev/osd_internals/scrub/ Thanks, Mike Dawson On 6/9/2014 6:22 PM, Craig Lewis wrote: I've correlated a large deep scrubbing

Re: [ceph-users] Best practice K/M-parameters EC pool

7; metric from something like 'iostat -xt 2' during a single disk backfill to get a very clear view that spindle contention is the true limiting factor. It'll be pegged at or near 100% if spindle contention is the issue. - Mike Another example if you please: My shitty test cl

Re: [ceph-users] Best practice K/M-parameters EC pool

On 8/28/2014 11:17 AM, Loic Dachary wrote: On 28/08/2014 16:29, Mike Dawson wrote: On 8/28/2014 12:23 AM, Christian Balzer wrote: On Wed, 27 Aug 2014 13:04:48 +0200 Loic Dachary wrote: On 27/08/2014 04:34, Christian Balzer wrote: Hello, On Tue, 26 Aug 2014 20:21:39 +0200 Loic Dachary

Re: [ceph-users] Best practice K/M-parameters EC pool

very (i.e. the "wrongly marked me down" log messages), I've found far superior results by tuning the recovery knobs than by permanently setting the nodown flag. - Mike Recovery of a single OSD might cause other OSDs to crash. In the primary cluster, I was always able to get

Re: [ceph-users] ISCSI LIO hang after 2-3 days of working

Not sure if there are multiple problems. On 02/05/2015 04:46 AM, reistlin87 wrote: > Feb 3 13:17:01 is-k13bi32e2s6vdi-002 CRON[10237]: (root) CMD ( cd / && > run-parts --report /etc/cron.hourly) > Feb 3 13:25:01 is-k13bi32e2s6vdi-002 CRON[10242]: (root) CMD (command -v > debian-sa1 > /dev/nu

Re: [ceph-users] rbd: I/O Errors in low memory situations

On 02/18/2015 06:05 PM, "Sebastian Köhler [Alfahosting GmbH]" wrote: > Hi, > > yesterday we had had the problem that one of our cluster clients > remounted a rbd device in read-only mode. We found this[1] stack trace > in the logs. We investigated further and found similar traces on all > other ma

Re: [ceph-users] tgt and krbd

On 03/06/2015 06:51 AM, Jake Young wrote: > > > On Thursday, March 5, 2015, Nick Fisk > wrote: > > Hi All, > > __ __ > > Just a heads up after a day’s experimentation. > > __ __ > > I believe tgt with its default settings has a small write

Re: [ceph-users] tgt and krbd

On 03/09/2015 11:15 AM, Nick Fisk wrote: > Hi Mike, > > I was using bs_aio with the krbd and still saw a small caching effect. I'm > not sure if it was on the ESXi or tgt/krbd page cache side, but I was > definitely seeing the IO's being coalesced into larger ones on the

Re: [ceph-users] tgt and krbd

On 03/15/2015 07:54 PM, Mike Christie wrote: > On 03/09/2015 11:15 AM, Nick Fisk wrote: >> Hi Mike, >> >> I was using bs_aio with the krbd and still saw a small caching effect. I'm >> not sure if it was on the ESXi or tgt/krbd page cache side, but I was >

Re: [ceph-users] tgt and krbd

On 03/15/2015 08:42 PM, Mike Christie wrote: > On 03/15/2015 07:54 PM, Mike Christie wrote: >> On 03/09/2015 11:15 AM, Nick Fisk wrote: >>> Hi Mike, >>> >>> I was using bs_aio with the krbd and still saw a small caching effect. I'm >>> not sure

[ceph-users] Ceph migration to AWS

To those interested in a tricky problem, We have a Ceph cluster running at one of our data centers. One of our client's requirements is to have them hosted at AWS. My question is: How do we effectively migrate our data on our internal Ceph cluster to an AWS Ceph cluster? Ideas currently on the ta

Re: [ceph-users] Discuss: New default recovery config settings

lls 3 --osd-recovery-max-active 3' If I see slow requests, I drop them down. The biggest downside to setting either to 1 seems to be the long tail issue detailed in: http://tracker.ceph.com/issues/9566 Thanks, Mike Dawson On 6/3/2015 6:44 PM, Sage Weil wrote: On Mon, 1 Jun 2015, Greg

[ceph-users] .New Ceph cluster - cannot add additional monitor

atus args=[]: finished 2015-06-09 11:33:27.950014 7fef2eb83700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd='mon_status' args=[]: dispatch 2015-06-09 11:33:27.950113 7fef2eb83700 0 log_channel(audit) log [DBG] : from='admin socket' entity='admin socket' cmd=mon_status args=[]: finished All of our google searching seems to indicate that there may be a clock skew, but, the clocks are matched within .001 seconds Any assistance is much appreciated, thanks, Mike C ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] .New Ceph cluster - cannot add additional monitor

Thank you for the reply Alex, I'm going to check into that and see if it helps resolve the issue. Mike C On Fri, Jun 12, 2015 at 11:57 PM, Alex Muntada wrote: > We've recently found similar problems creating a new cluster over an older > one, even after using "ceph-deploy

Re: [ceph-users] .New Ceph cluster - cannot add additional monitor

Just to follow up, I started from scratch, and I think the key was to run ceph-deploy purge (nodes) , ceph-deploy purgdata (nodes) and finally ceph-deploy forgetkeys Thanks for the replies Alex and Alex! Mike C ___ ceph-users mailing list ceph-users

Re: [ceph-users] What is this HEALTH_WARN indicating?

Run "ceph health detail" and it should give you more information. (I'd guess an osd or mon has a full hard disk) Cheers Mike On 8 July 2013 21:16, Jordi Llonch wrote: > Hello, > > I am testing ceph using ubuntu raring with ceph version 0.61.4 > (1669132fcfc27d0c0b5

Re: [ceph-users] ceph & hbase:

once a day) to mitigate the effects of bug #5039, being that your data pool is growing much faster than you might expect, and it being much larger than the visible filesize in cephfs. With those workarounds in place we're running a stable install of openTSDB on top of hbase. Mike On 17 July 2013

Re: [ceph-users] Unclean PGs in active+degrared or active+remapped

elov wrote: > On Fri, Jul 19, 2013 at 3:54 PM, Mike Lowe wrote: > I'm not sure how to get you out of the situation you are in but what you have > in your crush map is osd 2 and osd 3 but ceph starts counting from 0 so I'm > guessing it's probably gotten confused. Some h

Re: [ceph-users] Cinder volume creation issues

OpenStack Dashboard lists the machine running the Cinder APIs, not specifically the server hosting the storage. Like Greg stated, Ceph stripes the storage across your cluster. Fix your uuids and cinder.conf any you'll be moving in the right direction. Cheers, Mike On 7/26/2013 1:32 PM,

Re: [ceph-users] Defective ceph startup script

ot;} # ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok version {"version":"0.61.7"} Also, I use 'service ceph restart' on Ubuntu 13.04 running a mkcephfs deployment. It may be different when using ceph-deploy. Thanks, Mike Dawson Co-Founder & Director of

Re: [ceph-users] Production/Non-production segmentation

production services. A separate non-production cluster will allow you to test and validate new versions (including point releases within a stable series) before you attempt to upgrade your production cluster. Cheers, Mike Dawson Co-Founder & Director of Cloud Architecture Cloudapt LLC 6330

Re: [ceph-users] Production/Non-production segmentation

On 7/31/2013 3:34 PM, Greg Poirier wrote: On Wed, Jul 31, 2013 at 12:19 PM, Mike Dawson mailto:mike.daw...@cloudapt.com>> wrote: Due to the speed of releases in the Ceph project, I feel having separate physical hardware is the safer way to go, especially in light of your ment

Re: [ceph-users] Why is my mon store.db is 220GB?

are required anymore though. See some history here: http://tracker.ceph.com/issues/4895 Thanks, Mike Dawson Co-Founder & Director of Cloud Architecture Cloudapt LLC 6330 East 75th Street, Suite 170 Indianapolis, IN 46250 On 8/1/2013 6:52 PM, Jeppesen, Nelson wrote: My Mon store.db has been a

Re: [ceph-users] qemu-1.4.0 and onwards, linux kernel 3.2.x, ceph-RBD, heavy I/O leads to kernel_hung_tasks_timout_secs message and unresponsive qemu-process

debug rbd = 20, debug ms = 1, and debug objectcacher = 30 that would be great" We'll do that over the weekend. If you could as well, we'd love the help! [1] http://www.gammacode.com/kvm/wedged-with-timestamps.txt [2] http://www.gammacode.com/kvm/not-wedged.txt Thanks, Mike Daws

Re: [ceph-users] Large storage nodes - best practices

/Erasure_encoding_as_a_storage_backend [4]: http://wiki.ceph.com/01Planning/02Blueprints/Dumpling/Erasure_encoding_as_a_storage_backend [5]: http://www.inktank.com/about-inktank/roadmap/ Cheers, Mike Dawson On 8/5/2013 9:50 AM, Brian Candler wrote: I am looking at evaluating ceph for use with

Re: [ceph-users] Large storage nodes - best practices

On 8/5/2013 12:51 PM, Brian Candler wrote: On 05/08/2013 17:15, Mike Dawson wrote: Short answer: Ceph generally is used with multiple OSDs per node. One OSD per storage drive with no RAID is the most common setup. At 24- or 36-drives per chassis, there are several potential bottlenecks to

Re: [ceph-users] qemu-1.4.0 and onwards, linux kernel 3.2.x, ceph-RBD, heavy I/O leads to kernel_hung_tasks_timout_secs message and unresponsive qemu-process, [Qemu-devel] [Bug 1207686]

SDs) - rbd cache = true and cache=writeback - qemu 1.4.0 1.4.0+dfsg-1expubuntu4 - Ubuntu Raring with 3.8.0-25-generic This issue is reproducible in my environment, and I'm willing to run any wip branch you need. What else can I provide to help? Thanks, Mike Dawson On 8/5/2013 3:48 AM, Stefan

Re: [ceph-users] Openstack glance ceph rbd_store_user authentification problem

Steffan, It works for me. I have: user@node:/etc/ceph# cat /etc/glance/glance-api.conf | grep rbd default_store = rbd # glance.store.rbd.Store, rbd_store_ceph_conf = /etc/ceph/ceph.conf rbd_store_user = images rbd_store_pool = images rbd_store_chunk_size = 4 Thanks, Mike Dawson

Re: [ceph-users] how to recover the osd.

Looks like you didn't get osd.0 deployed properly. Can you show: - ls /var/lib/ceph/osd/ceph-0 - cat /etc/ceph/ceph.conf Thanks, Mike Dawson Co-Founder & Director of Cloud Architecture Cloudapt LLC 6330 East 75th Street, Suite 170 Indianapolis, IN 46250 On 8/8/2013 9:13 AM, Sur

Re: [ceph-users] how to recover the osd.

On 8/8/2013 12:30 PM, Suresh Sadhu wrote: Thanks Mike,Please find the output of two commands sadhu@ubuntu3:~$ ls /var/lib/ceph/osd/ceph-0 ^^^ that is a problem. It appears that osd.0 didn't get deployed properly. To see an example of what structure should be there, do: ls /var/lib

Re: [ceph-users] Storage, File Systems and Data Scrubbing

I think you are missing the distinction between metadata journaling and data journaling. In most cases a journaling filesystem is one that journal's it's own metadata but your data is on its own. Consider the case where you have a replication level of two, the osd filesystems have journaling d

Re: [ceph-users] Storage, File Systems and Data Scrubbing

lose data with ceph. > Journaling your metadata is the absolute minimum level of assurance you need > to make a transactional system like ceph work. > > Hey Mike J > > I get your point. However, isn’t it then possible to authoritatively say > which one is the correct one i

Re: [ceph-users] RBD hole punching

There is TRIM/discard support and I use it with some success. There are some details here http://ceph.com/docs/master/rbd/qemu-rbd/ The one caveat I have is that I've sometimes been able to crash an osd by doing fstrim inside a guest. On Aug 22, 2013, at 10:24 AM, Guido Winkelmann wrote: > H

Re: [ceph-users] Significant slowdown of osds since v0.67 Dumpling

Jumping in pretty late on this thread, but I can confirm much higher CPU load on ceph-osd using 0.67.1 compared to 0.61.7 under a write-heavy RBD workload. Under my workload, it seems like it might be 2x-5x higher CPU load per process. Thanks, Mike Dawson On 8/22/2013 4:41 AM, Oliver Daudey

Re: [ceph-users] Significant slowdown of osds since v0.67 Dumpling

perf2 appear very promising. Thanks for your work! I'll report back tomorrow if I have any new results. Thanks, Mike Dawson Co-Founder & Director of Cloud Architecture Cloudapt LLC 6330 East 75th Street, Suite 170 Indianapolis, IN 46250 On 8/29/2013 2:52 PM, Oliver Daudey wrote: Hey Mark

Re: [ceph-users] status of glance/cinder/nova integration in openstack grizzly

Darren, I can confirm Copy on Write (show_image_direct_url = True) does work in Grizzly. It sounds like you are close. To check permissions, run 'ceph auth list', and reply with "client.images" and "client.volumes" (or whatever keys you use in Glance and Cinde

Re: [ceph-users] status of glance/cinder/nova integration in openstack grizzly

On 9/10/2013 4:50 PM, Darren Birkett wrote: Hi Mike, That led me to realise what the issue was. My cinder (volumes) client did not have the correct perms on the images pool. I ran the following to update the perms for that client: ceph auth caps client.volumes mon 'allow r' osd &#

Re: [ceph-users] Pause i/o from time to time

cause. To re-enable scrub and deep-scrub: # ceph osd unset noscrub # ceph osd unset nodeep-scrub Because you seem to only have two OSDs, you may also be saturating your disks even without scrub or deep-scrub. http://tracker.ceph.com/issues/6278 Cheers, Mike Dawson On 9/16/2013 12:30 PM

Re: [ceph-users] OSD and Journal Files

partition for each journal and leave the rest of the SSD unallocated (it will be used for wear-leveling). If you use high-endurance SSDs, you could certainly consider smaller drives as long as they maintain sufficient performance characteristics. Thanks, Mike Dawson Co-Founder & Director

Re: [ceph-users] OSD and Journal Files

returns. Thanks, Mike Dawson Co-Founder & Director of Cloud Architecture Cloudapt LLC On 9/18/2013 1:27 PM, Gruher, Joseph R wrote: -Original Message- From: ceph-users-boun...@lists.ceph.com [mailto:ceph-users- boun...@lists.ceph.com] On Behalf Of Mike Dawson you need to unders

Re: [ceph-users] Ceph performance with 8K blocks.

Well, in a word, yes. You really expect a network replicated storage system in user space to be comparable to direct attached ssd storage? For what it's worth, I've got a pile of regular spinning rust, this is what my cluster will do inside a vm with rbd writeback caching on. As you can see, l

[ceph-users] radosgw-admin unable to list or store user info after upgrade

er, unable to store user info Has anyone else experienced this problem? Thanks, Mike ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RBD Snap removal priority

acker.ceph.com/issues/6278 [2] http://tracker.ceph.com/issues/6333 I think this family of issues speak to the need for Ceph to have more visibility into the underlying storage's limitations (especially spindle contention) when performing known expensive maintainance operations. Thanks, Mike Dawson On

Re: [ceph-users] issues with 'https://ceph.com/git/?p=ceph.git; a=blob_plain; f=keys/release.asc'

I have had the same issues. From: qgra...@onq.com.au To: ceph-users@lists.ceph.com Date: Mon, 30 Sep 2013 00:01:11 + Subject: [ceph-users] issues with 'https://ceph.com/git/?p=ceph.git; a=blob_plain; f=keys/release.asc' Hey Guys, Looks like 'https://ceph.com/git/?p=ceph.git;a=blo

Re: [ceph-users] Weird behavior of PG distribution

tep chooseleaf" lines inside the rule for each pool. Under certain configurations, I believe the placement that you describe is in fact the expected behavior. Thanks, Mike Dawson Co-Founder, Cloudapt LLC On 10/1/2013 10:46 AM, Chen, Ching-Cheng (KFRM 1) wrote: Found a weird behavior (or l

Re: [ceph-users] Ceph and RAID

/01Planning/02Blueprints/Emperor/Erasure_coded_storage_backend_%28step_2%29 Initial release is scheduled for Ceph's Firefly release in February 2014. Thanks, Mike Dawson Co-Founder & Director of Cloud Architecture Cloudapt LLC On 10/3/2013 2:44 PM, Aronesty, Erik wrote: Does Ceph really h

Re: [ceph-users] About Ceph SSD and HDD strategy

Based on my experience I think you are grossly underestimating the expense and frequency of flushes issued from your vm's. This will be especially bad if you aren't using the async flush from qemu >= 1.4.2 as the vm is suspended while qemu waits for the flush to finish. I think your best cours

Re: [ceph-users] Expanding ceph cluster by adding more OSDs

You can add PGs, the process is called splitting. I don't think PG merging, the reduction in the number of PGs, is ready yet. On Oct 8, 2013, at 11:58 PM, Guang wrote: > Hi ceph-users, > Ceph recommends the PGs number of a pool is (100 * OSDs) / Replicas, per my > understanding, the number o

[ceph-users] indexing object store with SOLR

All, I have been prototyping an object store and am looking at a way to index content and metadata. Has anyone looked at doing anything similar? I would be interested in kicking around some ideas. I'd really like to implement something with Apache Solr or something similar. Thanks,

Re: [ceph-users] retrieving usage information via admin API

Yes, you need to enabling usage logging in your ceph.conf ... something like: rgw enable usage log = truergw usage log tick interval = 30rgw usage log flush threshold = 1024rgw usage max shards = 32rgw usage max user shards = 1 You can find more info here: http://ceph.com/docs/master/radosgw/conf

[ceph-users] osds and gateway not coming up on restart

; 1 osds have slow requests ceph osd state134: 18 osds: 15 up, 15 in Thanks, Mike ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osds and gateway not coming up on restart

c0 1 -- 10.10.2.202:6809/15169 shutdown complete. > Date: Thu, 10 Oct 2013 23:04:05 +0200 > From: w...@42on.com > To: ceph-users@lists.ceph.com > CC: mike.oto...@outlook.com > Subject: Re: [ceph-users] osds and gateway not coming up on restart > > On 10/10/2013 11:01 PM, Mike O'

Re: [ceph-users] osds and gateway not coming up on restart

3:04:05 +0200 > From: w...@42on.com > To: ceph-users@lists.ceph.com > CC: mike.oto...@outlook.com > Subject: Re: [ceph-users] osds and gateway not coming up on restart > > On 10/10/2013 11:01 PM, Mike O'Toole wrote: > > > > I created them with ceph-deploy and there

Re: [ceph-users] kvm live migrate wil ceph

I wouldn't go so far as to say putting a vm in a file on a networked filesystem is wrong. It is just not the best choice if you have a ceph cluster at hand, in my opinion. Networked filesystems have a bunch of extra stuff to implement posix semantics and live in kernel space. You just need si

Re: [ceph-users] Multiply OSDs per host strategy ?

Andrija, You can use a single pool and the proper CRUSH rule step chooseleaf firstn 0 type host to accomplish your goal. http://ceph.com/docs/master/rados/operations/crush-map/ Cheers, Mike Dawson On 10/16/2013 5:16 PM, Andrija Panic wrote: Hi, I have 2 x 2TB disks, in 3 servers, so

Re: [ceph-users] poor read performance on rbd+LVM, LVM overload

On Wed, Oct 16 2013 at 12:16pm -0400, Sage Weil wrote: > Hi, > > On Wed, 16 Oct 2013, Ugis wrote: > > > > What could make so great difference when LVM is used and what/how to > > tune? As write performance does not differ, DM extent lookup should > > not be lagging, where is the trick? > > My

Re: [ceph-users] poor read performance on rbd+LVM, LVM overload

On Mon, Oct 21 2013 at 11:01am -0400, Mike Snitzer wrote: > On Mon, Oct 21 2013 at 10:11am -0400, > Christoph Hellwig wrote: > > > On Sun, Oct 20, 2013 at 08:58:58PM -0700, Sage Weil wrote: > > > It looks like without LVM we're getting 128KB requests (which IIRC

Re: [ceph-users] poor read performance on rbd+LVM, LVM overload

On Mon, Oct 21 2013 at 10:11am -0400, Christoph Hellwig wrote: > On Sun, Oct 20, 2013 at 08:58:58PM -0700, Sage Weil wrote: > > It looks like without LVM we're getting 128KB requests (which IIRC is > > typical), but with LVM it's only 4KB. Unfortunately my memory is a bit > > fuzzy here, but I

Re: [ceph-users] poor read performance on rbd+LVM, LVM overload

On Mon, Oct 21 2013 at 12:02pm -0400, Sage Weil wrote: > On Mon, 21 Oct 2013, Mike Snitzer wrote: > > On Mon, Oct 21 2013 at 10:11am -0400, > > Christoph Hellwig wrote: > > > > > On Sun, Oct 20, 2013 at 08:58:58PM -0700, Sage Weil wrote: > > > > It

Re: [ceph-users] poor read performance on rbd+LVM, LVM overload

On Mon, Oct 21 2013 at 2:06pm -0400, Christoph Hellwig wrote: > On Mon, Oct 21, 2013 at 11:01:29AM -0400, Mike Snitzer wrote: > > It isn't DM that splits the IO into 4K chunks; it is the VM subsystem > > no? > > Well, it's the block layer based on wh

Re: [ceph-users] saucy salamander support?

For the time being, you can install the Raring debs on Saucy without issue. echo deb http://ceph.com/debian-dumpling/ raring main | sudo tee /etc/apt/sources.list.d/ceph.list I'd also like to register a +1 request for official builds targeted at Saucy. Cheers, Mike On 10/22/2013 11:

Re: [ceph-users] saucy salamander support?

And a +1 from me as well. It would appear that ubuntu has picked up the 0.67.4 source and included a build of it in their official repo, so you may be able to get by until the next point release with those. http://packages.ubuntu.com/search?keywords=ceph On Oct 22, 2013, at 11:46 AM, Mike

Re: [ceph-users] About use same SSD for OS and Journal

were you seeing on the cluster during the periods where things got laggy due to backfills, etc? Last, did you attempt to throttle using ceph config setting in the old setup? Do you need to throttle in your current setup? Thanks, Mike Dawson On 10/24/2013 10:40 AM, Kurt Bauer wrote: Hi, we

Re: [ceph-users] How can I check the image's IO ?

, --no-rbd-cache, and --io-pattern {seq|rand} Cheers, Mike On 10/30/2013 3:23 AM, vernon1987 wrote: Hi cephers, I use "qemu-img create -f rbd rbd:test-pool/test-image" to create a image. I want to know how can I check this image's IO. Or how to check the IO for each block? Tha

Re: [ceph-users] OSD numbering

You really should, I believe the osd number is used in computing crush. Bad things will happen if you don't use sequential numbers. On Oct 30, 2013, at 11:37 AM, Glen Aidukas wrote: > I wanted to know, does the OSD numbering half to be sequential and what is > the highest usable number (2^16

Re: [ceph-users] Red Hat clients

If you were to run your Red Hat based client in a vm you could run run unmodified an unmodified kernel. if you are using rhel 6.4 then you get the extra goodies in the virtio-scsi qemu driver. On Oct 30, 2013, at 2:47 PM, wrote: > Now that my ceph cluster seems to be happy and stable, I hav

Re: [ceph-users] Ceph monitor problems

total monitors. -Mike On 10/30/2013 4:46 PM, Aaron Ten Clay wrote: On Wed, Oct 30, 2013 at 1:43 PM, Joao Eduardo Luis mailto:joao.l...@inktank.com>> wrote: A quorum of 2 monitors is completely fine as long as both monitors are up. A quorum is always possible regardless of how ma

Re: [ceph-users] Ceph health checkup

Narendra, This is an issue. You really want your cluster to he HEALTH_OK with all PGs active+clean. Some exceptions apply (like scrub / deep-scrub). What do 'ceph health detail' and 'ceph osd tree' show? Thanks, Mike Dawson Co-Founder & Director of Cloud Architectur

Re: [ceph-users] Ceph User Committee

I also have time I could spend. Thanks for getting this started Loic! Thanks, Mike Dawson On 11/6/2013 12:35 PM, Loic Dachary wrote: Hi Ceph, I would like to open a discussion about organizing a Ceph User Committee. We briefly discussed the idea with Ross Turk, Patrick McGarry and Sage Weil

Re: [ceph-users] ceph cluster performance

We just fixed a performance issue on our cluster related to spikes of high latency on some of our SSDs used for osd journals. In our case, the slow SSDs showed spikes of 100x higher latency than expected. What SSDs were you using that were so slow? Cheers, Mike On 11/6/2013 12:39 PM, Dinu

Re: [ceph-users] ceph cluster performance

https://github.com/gregsfortytwo/fsync-tester Thanks, Mike Dawson On 11/6/2013 4:18 PM, Dinu Vlad wrote: ST240FN0021 connected via a SAS2x36 to a LSI 9207-8i. By "fixed" - you mean replaced the SSDs? Thanks, Dinu On Nov 6, 2013, at 10:25 PM, Mike Dawson wrote: We just fixed a pe

Re: [ceph-users] Running on disks that lose their head

Thanks, Mike Dawson Co-Founder & Director of Cloud Architecture Cloudapt LLC 6330 East 75th Street, Suite 170 Indianapolis, IN 46250 On 11/7/2013 2:12 PM, Kyle Bader wrote: Once I know a drive has had a head failure, do I trust that the rest of the drive isn't going to go at an inc

Re: [ceph-users] how to enable rbd cache

/ceph-devel@vger.kernel.org/msg16168.html 4) Once you get an RBD admin socket, query it like: ceph --admin-daemon /var/run/ceph/rbd-29050.asok config show | grep rbd Cheers, Mike Dawson On 11/25/2013 11:12 AM, Gregory Farnum wrote: On Mon, Nov 25, 2013 at 5:58 AM, Mark Nelson wrote: On

Re: [ceph-users] CEPH HA with Ubuntu OpenStack and Highly Available Controller Nodes

ss racks rather than hosts if your cluster will be large enough. - Don't set the "ceph osd set nodown" flag on your cluster, as it will prevent osds from being marked as down automatically if unavailable, substantially diminishing the HA capabilities. Cheers, Mike Dawson On 1

Re: [ceph-users] Adding new OSDs, need to increase PGs?

have? Any RAID involved under your OSDs? Thanks, Mike Dawson On 12/3/2013 1:31 AM, Robert van Leeuwen wrote: On 2 dec. 2013, at 18:26, "Brian Andrus" wrote: Setting your pg_num and pgp_num to say... 1024 would A) increase data granularity, B) likely lend no noticeable i

Re: [ceph-users] Adding new OSDs, need to increase PGs?

Robert, Do you have rbd writeback cache enabled on these volumes? That could certainly explain the higher than expected write performance. Any chance you could re-test with rbd writeback on vs. off? Thanks, Mike Dawson On 12/3/2013 10:37 AM, Robert van Leeuwen wrote: Hi Mike, I am using

Re: [ceph-users] v0.67.11 dumpling released

C, 0.67.11 does not include the proposed changes to address #9487 or #9503, right? Thanks, Mike Dawson * osd: fix mount/remount sync race (#9144 Sage Weil) Getting Ceph * Git at git://github.com/ceph/ceph.git * Tarball at http://ceph.com/download/ceph-0.67.11.tar.gz * For pac

Re: [ceph-users] v0.67.11 dumpling released

://ceph.com/debian-dumpling/pool/main/c/ceph/libcephfs1_0.67.11-1precise_amd64.deb 404 Not Found Based on the timestamps of the files that made it, it looks like the process to publish the packages isn't still in process, but rather failed yesterday. Thanks, Mike Dawson On 9/25/2014 11:

Re: [ceph-users] converting legacy puppet-ceph configured OSDs to look like ceph-deployed OSDs

this very issue earlier this year, but got pulled in another direction before completing the work. I'd like to bring a production cluster deployed with mkcephfs out of the stone ages, so your work will be very useful to me. Thanks again, Mike Dawson ___

Re: [ceph-users] Poor RBD performance as LIO iSCSI target

On 10/27/2014 04:24 PM, Christopher Spearman wrote: > > - What tested with bad performance (Reads ~25-50MB/s - Writes ~25-50MB/s) >* RBD setup as target using LIO >* RBD -> LVM -> LIO target >* RBD -> RAID0/1 -> LIO target > - What tested with good performance (Reads ~700-800MB/s - W

Re: [ceph-users] Negative amount of objects degraded

on as your. Your results may vary. - Mike Dawson On 10/30/2014 4:50 PM, Erik Logtenberg wrote: Thanks for pointing that out. Unfortunately, those tickets contain only a description of the problem, but no solution or workaround. One was opened 8 months ago and the other more than a year ago. No

Re: [ceph-users] Poor RBD performance as LIO iSCSI target

On 11/13/2014 10:17 AM, David Moreau Simard wrote: > Running into weird issues here as well in a test environment. I don't have a > solution either but perhaps we can find some things in common.. > > Setup in a nutshell: > - Ceph cluster: Ubuntu 14.04, Kernel 3.16.7, Ceph 0.87-1 (OSDs with separa

Re: [ceph-users] Poor RBD performance as LIO iSCSI target

lems. I want to run a few >>>>>> more tests with different settings to see if I can reproduce your >>>>>> problem. I will let you know if I find anything. >>>>>> >>>>>> If there is anything you would like me to try, pleas

Re: [ceph-users] Poor RBD performance as LIO iSCSI target

Oh yeah, for the iscsi fio full write test, did you experiment with bs and numjobs? For just 10 GB iscsi, I think numjibs > 1 (around 4 is when I stop seeing benefits) and bs < 1MB (around 64K to 256K) works better. On 12/08/2014 05:22 PM, Mike Christie wrote: > Some distros have LIO

  1   2   3   4   5   >