Thanks for the tips.
Could anyone share their experience building a SSD pool or a SSD cache
tier with HP SL4540 server?
rgds,
Sreenath
On 4/2/15, Christian Balzer wrote:
>
> Hello,
>
> On Wed, 1 Apr 2015 18:40:10 +0530 Sreenath BH wrote:
>
>> Hi all,
>>
>> we are considering building all SSD OSD
Fortunately Ceph Giant + OpenStack Juno works flawlessly for me.
If you have configured cinder / glance correctly , then after restarting
cinder and glance services , you should see something like this in cinder and
glance logs.
Cinder logs :
volume.log:2015-04-02 13:20:43.943 2085 INFO cin
On Wed, Apr 1, 2015 at 12:31 AM, Barclay Jameson
wrote:
> Here is the mds output from the command you requested. I did this
> during the small data run . ( time cp small1/* small2/ )
> It is 20MB in size so I couldn't find a place online that would accept
> that much data.
>
> Please find attached
Hello everyone,
Is there anyone having experience to try to tune Kernel RBD device by changing
scheduler and other settings?
Currently we are trying it on RHEL 7.1 bundled rbd module, to change the
following setting under /sys/block/rbdX/queue:
1) scheduler: noop vs deadline, deadline seems bett
Thanks Karan/Quentin/Erik,
I admit up front that this is all new to me as my background is optical
transport rather than server/storage admin!
I'm reassured to know that it should work and this is why I'm completely
willing to believe that it's something that I'm doing wrong ... but
unfortunately
Hi,
>>with HP SL4540 server?
this model http://www8.hp.com/h20195/v2/GetDocument.aspx?docname=c04128155 ?
(60 drives ?)
I think for a full ssd node, it'll be impossible to reach max performance,
you'll be cpu bound.
I think a small node with 6-8 ssd osd for 20cores should be ok.
- Ma
Oh, apologies, I missed the versions ...
# glance --version : 0.14.2
# cinder --version : 1.1.1
# ceph -v: ceph version 0.87.1
(283c2e7cfa2457799f534744d7d549f83ea1335e)
>From rpm I can confirm that Cinder and Glance are both of the February 2014
vintage:
# rpm -qa |grep -e ceph -e
We have the model with 25 disks per node.
We have just two 10G network interfaces per node. Does that not limit
the thgouthput and hence the load on the CPUs?
-Sreenath
On 4/2/15, Alexandre DERUMIER wrote:
> Hi,
>
>>>with HP SL4540 server?
>
> this model http://www8.hp.com/h20195/v2/GetDocument
It's probably more a question of IOPS unless you have really slow SSDs. :)
Mark
On 04/02/2015 07:22 AM, Sreenath BH wrote:
We have the model with 25 disks per node.
We have just two 10G network interfaces per node. Does that not limit
the thgouthput and hence the load on the CPUs?
-Sreenath
I'm probably going to get shot down for saying this...but here goes.
As a very rough guide, think of it more as you need around 10Mhz for every IO,
whether that IO is 4k or 4MB it uses roughly the same amount of CPU, as most of
the CPU usage is around ceph data placement rather than the actual r
Hi,
I just bought the book "Learning Ceph". I'm trying to go through the pages
and set up a virtual lab. I managed to set up the servers etc but when it
comes to using ceph-deploy mon create to other nodes - it fails badly. I
don't see anything that I missed on the books.
Here's the output:
[roo
i first had some major disaster i had 12 incomplete pgs that couldn't be fixed.
(due to several harddisk failures at once)
alls these incomplete pgs where all in the ".rgw" and ".rgw.buckets" pools
so the only option i could think of is to take my loses and delete and recreate
those pools.
the
Yes. I have setup ceph and hadoop in each node. ceph health is OK and the
hadoop works fine when I use HDFS (I have ran the same command with HDFS
and it works). One node is the admin(job tracker running), other 4 are
slaves(tasktracker running). The problem occurs when I change the
hadoop/conf/co
Hi Greg,
Thank you for your response. I have understood that I should be starting
only the mapred daemons when using cephFS instead of HDFS. I have fixed
that and trying to run hadoop wordcount job using this instruction:
bin/hadoop jar hadoop*examples*.jar wordcount /tmp/wc-input /tmp/wc-output
Hello Ceph Community,
I have an issue whenever I am creating an object (media file) on ceph using
the s3 api. I get the error message :
Tue Mar 24 14:41:00.079071 2015] [fastcgi:error] [pid 1606:tid
139809104328448] [client 192.168.88.1:18644] FastCGI: comm with server
"/var/www/s3gw.fcgi" aborte
Hi,
Currently I am trying to figure out how to work with RADOS Gateway (ceph
0.87) limits and I've managed to produce such strange behavior:
{ "bucket": "test1-8",
"pool": ".rgw.buckets",
"index_pool": ".rgw.buckets.index",
"id": "default.17497.14",
"marker": "default.17497.14",
"owner"
Hi, can't pretend that I have all the answers (or any of them!) but I've
also been unable to deploy a mon node that doesn't appear in the 'mon
initial members' list. However, the "No such file or directory" is
something that I don't remember. Did you run "ceph-deploy install" against
the node first
I am using the Giant release. The OSDs and MON/MDS are using default
RHEL 7 kernel. Client is using elrepo 3.19 kernel. I am also using
cephaux.
I may have found something.
I did the build manually as such I did _NOT_ set up these config settings:
filestore xattr use omap = false
filestore max inl
Yes, I did.
Furthermore, looking at node2, the ceph service is not starting, giving me
this error.
# service ceph start
=== mon.ceph-node2 ===
Starting Ceph mon.ceph-node2 on ceph-node2...
2015-04-02 18:08:50.387800 7f819f2667a0 -1 no public_addr or public_network
specified, and mon.ceph-node2 no
- Original Message -
> From: "Sergey Arkhipov"
> To: ceph-users@lists.ceph.com
> Sent: Monday, March 30, 2015 2:55:33 AM
> Subject: [ceph-users] RADOS Gateway quota management
> Hi,
> Currently I am trying to figure out how to work with RADOS Gateway (ceph
> 0.87) limits and I've manage
As expected I had a typo in my config for cinder. Correcting that got
cinder working. Everything in glance looks correct according to the above
referenced page, but I'm not seeing any mention of rbd in the logs, and I
notice that the cephx authentication pieces that are present for cinder and
consp
The RDO glance-store package had a bug in it that miscalculated the chunk
size. I should hope that it's been patched by Redhat now since the fix was
committed upstream before the first Juno rleease, but perhaps not. The
symptom of the bug was horribly slow uploads to glance.
Run this and send back
Glance should just require something like the following under [default]
rbd_store_user=glance
rbd_store_pool=images
rbd_store_ceph_conf=/etc/ceph/ceph.conf
rbd_store_chunk_size=8
default_store=rbd
Also make sure the keyring is in /etc/ceph and you may want to explicitly
define the user and keyrin
also from my tests, writes use a lot more cpus than read.
I think they are some optimisations in hammer for this.
(and also the tcmalloc bug which seem to impact cpu)
I will redo bench soon when hammer will be released.
- Mail original -
De: "Nick Fisk"
À: "ceph-users"
Envoyé: Jeudi
Hm, even lacking the mentions of rbd in the glance docs, and the lack of
cephx auth information in the config, glance seems to be working after all.
S, hooray! It was probably working all along, I just hadn't gotten to
really testing it since I was getting blocked by my typo on the cinder
confi
On Thu, Apr 2, 2015 at 12:18 PM, Quentin Hartman <
qhart...@direwolfdigital.com> wrote:
> Hm, even lacking the mentions of rbd in the glance docs, and the lack of
> cephx auth information in the config, glance seems to be working after all.
> S, hooray! It was probably working all along, I jus
Thanks Erik,
Maybe this is related as I have:
[DEFAULT]
verbose = True
notification_driver = noop
default_store = rbd
show_image_direct_url = true
debug=True
[database]
connection = mysql://glance:glancepw@ps-sw-ctrl1/glance
[keystone_authtoken]
auth_uri = http://ps-sw-ctrl
Well, 100% may be overstating things. When I try to create a volume from an
image it fails. I'm digging through the logs right now. glance alone works
(I can upload and delete images) and cinder alone works (I can create and
delete volumes) but when cinder tries to get the glance service it fails,
Thought folks might like to see this:
http://hothardware.com/reviews/intel-ssd-750-series-nvme-pci-express-solid-state-drive-review
Quick summary:
- PCIe SSD based on the P3700
- 400GB for $389!
- 1.2GB/s writes and 2.4GB/s reads
- power loss protection
- 219TB write endurance
So basically loo
On Thu, Apr 2, 2015 at 10:03 AM, Mark Nelson wrote:
> Thought folks might like to see this:
>
> http://hothardware.com/reviews/intel-ssd-750-series-nvme-pci-express-solid-state-drive-review
>
> Quick summary:
>
> - PCIe SSD based on the P3700
> - 400GB for $389!
> - 1.2GB/s writes and 2.4GB/s read
All,
Whenever we're doing some kind of recovery operation on our ceph
clusters (cluster expansion or dealing with a drive failure), there
seems to be a fairly noticable performance drop while it does the
backfills (last time I measured it the performance during recovery was
something like 20% of a
Hi,
On 04/02/15 19:31, Stillwell, Bryan wrote:
> All,
>
> Whenever we're doing some kind of recovery operation on our ceph
> clusters (cluster expansion or dealing with a drive failure), there
> seems to be a fairly noticable performance drop while it does the
> backfills (last time I measured it
On Thursday, April 2, 2015, Nick Fisk wrote:
> I'm probably going to get shot down for saying this...but here goes.
>
> As a very rough guide, think of it more as you need around 10Mhz for every
> IO, whether that IO is 4k or 4MB it uses roughly the same amount of CPU, as
> most of the CPU usage
>Recovery creates I/O performance drops in our VM too but it's manageable.
>What really hurts us are deep scrubs.
>Our current situation is Firefly 0.80.9 with a total of 24 identical OSDs
>evenly distributed on 4 servers with the following relevant configuration:
>
>osd recovery max active
>
>
> On Thursday, April 2, 2015, Nick Fisk wrote:
> I'm probably going to get shot down for saying this...but here goes.
>
> As a very rough guide, think of it more as you need around 10Mhz for every
> IO, whether that IO is 4k or 4MB it uses roughly the same amount of CPU, as
> most of the CP
On 04/02/15 21:02, Stillwell, Bryan wrote:
>> With these settings and no deep-scrubs the load increased a bit in the
>> VMs doing non negligible I/Os but this was manageable. Even disk thread
>> ioprio settings (which is what you want to get the ionice behaviour for
>> deep scrubs) didn't seem to m
On 04/02/15 21:02, Stillwell, Bryan wrote:
>
> I'm pretty sure setting 'nodeep-scrub' doesn't cancel any current
> deep-scrubs that are happening,
Indeed it doesn't.
> but something like this would help prevent
> the problem from getting worse.
If the cause of the recoveries/backfills are an OS
Geddes
If you are still struggling with this , ping me in IRC #CEPH ( ksingh )
Karan Singh
Systems Specialist , Storage Platforms
CSC - IT Center for Science,
Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland
mobile: +358 50
Hi,
I am building Ceph Debian Packages off of the 0.80.9 (latest firefly) and on
top of that I am applying an optimization patch.
I am following the standard instructions from the README file and effectively
running commands in this order:
$ ./autogen.sh
$ ./confi
Nope,
I redid the cluster with the above config options and it did not fix it.
It must have cached the files from the first copy.
Any thoughts on this?
On Thu, Apr 2, 2015 at 10:18 AM, Barclay Jameson
wrote:
> I am using the Giant release. The OSDs and MON/MDS are using default
> RHEL 7 kernel.
Try changing src/ceph_ver.h and repeat what you did earlier.-Krishna
On Thursday, April 2, 2015 2:17 PM, "Garg, Pankaj"
wrote:
Hi, I am
building Ceph Debian Packages off of the 0.80.9 (latest firefly) and on top of
that I am applying an optimization patch. I am following the stan
No, but I've seen it in RadosGW too. I've been meaning to post about it.
I get about ten a day, out of about 50k objects/day.
clewis@clewis-mac ~ (-) $ s3cmd ls s3://live32/ | grep '1970-01' | head -1
1970-01-01 00:00 0
s3://live-32/39020f17716a18b39efd8daa96e8245eb2901f353ba1004e724cb56
You need to create both system users in both zones, with the same access
and secret keys. The replication process needs these users to do the
replication.
Location support isn't currently supported... I think that's targeted for
Hammer maybe? http://ceph.com/docs/master/release-notes/ indicates
On 04/01/2015 02:42 AM, Jimmy Goffaux wrote:
English Version :
Hello,
I found a strange behavior in Ceph. This behavior is visible on Buckets
(RGW) and pools (RDB).
pools:
``
root@:~# qemu-img info rbd:pool/kibana2
image: rbd:pool/kibana2
file format: raw
virtual size: 30G (32212254720 bytes)
On Thu, Apr 2, 2015 at 11:18 PM, Barclay Jameson
wrote:
> I am using the Giant release. The OSDs and MON/MDS are using default
> RHEL 7 kernel. Client is using elrepo 3.19 kernel. I am also using
> cephaux.
I reproduced this issue by using giant release. It's a bug in the MDS
code. Could you try
I have a cluster running 0.80.9 on Ubuntu 14.04. A couple nights ago I lost two
disks from a pool with size=2. :(
I replaced the two failed OSDs and I now have two PGs which are marked as
incomplete in an otherwise healthy cluster. Following this page (
https://ceph.com/community/incomplete-pgs
Hi Chris,
according to your ceph osd tree capture, although the OSD reweight is set to 1,
the OSD CRUSH weight is set to 0 (2nd column). You need to assign the OSD a
CRUSH weight so that it can be selected by CRUSH: ceph osd crush reweight
osd.30 x.y (where 1.0=1TB)
Only when this is done will
I think you want to do:
$ dch
$ dpkg-buildpackage
You can muck about with what the package is gonna be called (versions,
revisions etc) from dch, without changing the src.
Cheers
Mark
On 03/04/15 10:17, Garg, Pankaj wrote:
Hi,
I am building Ceph Debian Packages off of the 0.80.9 (latest f
performance seem quite impressive (40iops 4k) :)
for endurance, It's looking like intel s3500
I still preferer s3610 for now,
400GB for $419 , with 3.0PB endurance.
- Mail original -
De: "Mark Nelson"
À: "ceph-users"
Envoyé: Jeudi 2 Avril 2015 19:03:14
Objet: [ceph-users] New
49 matches
Mail list logo
