Hi,
I have a crushmap structure likes root->rack->host->osds. I designed the rule
below, since I used "chooseleaf...rack" in rule definition, if there is only
one rack in the cluster, the ceph gps will always stay at stuck unclean state
(that is because the default metadata/data/rbd pool set 2
perhaps group sets of hosts into racks in crushmap. The crushmap doesn't
have to strictly map the real world.
On 05/13/2014 08:52 AM, Cao, Buddy wrote:
Hi,
I have a crushmap structure likes root->rack->host->osds. I designed
the rule below, since I used "chooseleaf...rack" in rule definition
I'm clearly talking to myself, but whatever.
For Greg, I've played with all the pertinent journal and filestore options
and TCP nodelay, no changes at all.
Is there anybody on this ML who's running a Ceph cluster with a fast
network and FAST filestore, so like me with a big HW cache in front of
Hello,
No actual question, just some food for thought and something that later
generations can scour from the ML archive.
I'm planning another Ceph storage cluster, this time a "classic" Ceph
design, 3 storage nodes with 8 HDDs for OSDs and 4 SSDs for OS and journal.
When juggling the budget fo
Hi Christian,
I'm going to test a full ssd cluster in coming months,
I'll send result on the mailing.
Do you have tried to use 1 osd by physical disk ? (without raid6)
Maybe they are bottleneck in osd daemon,
and using osd daemon by disk could help.
- Mail original -
De: "Christi
Hello,
On Tue, 13 May 2014 11:33:27 +0200 (CEST) Alexandre DERUMIER wrote:
> Hi Christian,
>
> I'm going to test a full ssd cluster in coming months,
> I'll send result on the mailing.
>
Looking forward to that.
>
> Do you have tried to use 1 osd by physical disk ? (without raid6)
>
No, if y
On thing that would put me off the 530 is lack on power off safety
(capacitor or similar). Given the job of the journal, I think an SSD
that has some guarantee of write integrity is crucial - so yeah the
DC3500 or DC3700 seem like the best choices.
Regards
Mark
On 13/05/14 21:31, Christian B
El 13/05/14 11:31, Christian Balzer escribió:
> Hello,
>
> No actual question, just some food for thought and something that later
> generations can scour from the ML archive.
>
> I'm planning another Ceph storage cluster, this time a "classic" Ceph
> design, 3 storage nodes with 8 HDDs for OSDs an
Begin forwarded message:
> From: Wenjun Huang
> Subject: What is link and unlink options used for in radosgw-admin
> Date: May 13, 2014 at 2:55:18 PM GMT+8
> To: ceph-us...@ceph.com
>
> Hello, everyone
>
> I am now confused with the options of link & unlink in radosgw-admin utility.
>
> In m
Thanks for your answers Craig, it seems this is a niche use case for Ceph, not
a lot of replies from the ML.
Cheers
--
Cédric Lemarchand
> Le 11 mai 2014 à 00:35, Craig Lewis a écrit :
>
>> On 5/10/14 12:43 , Cédric Lemarchand wrote:
>> Hi Craig,
>>
>> Thanks, I really appreciate the well d
>>It might, but at the IOPS I'm seeing anybody using SSD for file storage
>>should have screamed out already.
>>Also given the CPU usage I'm seeing during that test run such a setup
>>would probably require 32+ cores.
Just found this:
https://objects.dreamhost.com/inktankweb/Inktank_Hardware_
Hi,
I think you're not getting many replies simply because those are rather
large servers and not many have such hardware in prod.
We run with 24x3TB drives, 64GB ram, one 10Gbit NIC. Memory-wise there
are no problems. Throughput-wise, the bottleneck is somewhere between
the NIC (~1GB/s) and
On Tue, 13 May 2014 22:03:11 +1200 Mark Kirkwood wrote:
> On thing that would put me off the 530 is lack on power off safety
> (capacitor or similar). Given the job of the journal, I think an SSD
> that has some guarantee of write integrity is crucial - so yeah the
> DC3500 or DC3700 seem like
On Tue, 13 May 2014 12:07:12 +0200 Xabier Elkano wrote:
> El 13/05/14 11:31, Christian Balzer escribió:
> > Hello,
> >
> > No actual question, just some food for thought and something that later
> > generations can scour from the ML archive.
> >
> > I'm planning another Ceph storage cluster, this
Hello,
System Ubuntu 14.04
Ceph 0.80
I'm getting either a 405 Method Not Allowed or a 403 Permission Denied
from Radosgw.
Here is what I get from radosgw:
HTTP/1.1 405 Method Not Allowed
Date: Tue, 13 May 2014 12:21:43 GMT
Server: Apache
Accept-Ranges: bytes
Content-Length: 82
Content-Type:
Hello,
On Tue, 13 May 2014 13:36:49 +0200 (CEST) Alexandre DERUMIER wrote:
> >>It might, but at the IOPS I'm seeing anybody using SSD for file
> >>storage should have screamed out already.
> >>Also given the CPU usage I'm seeing during that test run such a setup
> >>would probably require 32+
El 13/05/14 14:23, Christian Balzer escribió:
> On Tue, 13 May 2014 12:07:12 +0200 Xabier Elkano wrote:
>
>> El 13/05/14 11:31, Christian Balzer escribió:
>>> Hello,
>>>
>>> No actual question, just some food for thought and something that later
>>> generations can scour from the ML archive.
>>>
>>
On Tue, 13 May 2014 14:46:23 +0200 Xabier Elkano wrote:
> El 13/05/14 14:23, Christian Balzer escribió:
> > On Tue, 13 May 2014 12:07:12 +0200 Xabier Elkano wrote:
> >
> >> El 13/05/14 11:31, Christian Balzer escribió:
> >>> Hello,
> >>>
> >>> No actual question, just some food for thought and som
Hi All,
Is there a way by which we can measure the performance of Ceph block devices ?
(Example : I/O stats, data to identify bottlenecks etc).
Also what are the available ways in which we can compare Ceph storage
performance with other storage solutions ?
Thanks
Kumar
_
Le 12/05/2014 15:45, Uwe Grohnwaldt a écrit :
Hi,
yes, we use it in production. I can stop/kill the tgt on one server and
XenServer goes to the second one. We enabled multipathing in xenserver. In our
setup we haven't multiple ip-ranges so we scan/login the second target on
xenserverstartup w
Hi, i am working with test Ceph cluster and now I want to implement Zabbix
monitoring with items such as:
- whoe cluster IO (for example ceph -s -> recovery io 143 MB/s, 35
objects/s)
- pg statistics
I would like to create single script in python to retrive values using
rados python module, but t
>>For what it's worth, my cluster gives me 4100 IOPS with the sequential fio
>>run below and 7200 when doing random reads (go figure). Of course I made
>>sure these came come the pagecache of the storage nodes, no disk I/O
>>reported at all and the CPUs used just 1 core per OSD.
>>---
>>fio --
> Anyway replacing set of monitors means downtime for every client, so
> I`m in doubt if 'no outage' word is still applicable there.
Taking the entire quorum down for migration would be bad. It's better
to add one in the new location, remove one at the old, ad infinitum.
--
Kyle
___
On Tue, 13 May 2014 16:09:28 +0200 (CEST) Alexandre DERUMIER wrote:
> >>For what it's worth, my cluster gives me 4100 IOPS with the sequential
> >>fio run below and 7200 when doing random reads (go figure). Of course
> >>I made sure these came come the pagecache of the storage nodes, no
> >>disk I
Dima, do you have any examples / howtos for this? I would love to give it a go.
Cheers
- Original Message -
From: "Dimitri Maziuk"
To: ceph-users@lists.ceph.com
Sent: Monday, 12 May, 2014 3:38:11 PM
Subject: Re: [ceph-users] NFS over CEPH - best practice
On 5/12/2014 4:52 AM, Andr
Uwe, do you mind sharing your storage and xenserver iscsi config files?
Also, what is your performance like?
Thanks
- Original Message -
From: "Uwe Grohnwaldt"
To: ceph-users@lists.ceph.com
Sent: Monday, 12 May, 2014 2:45:43 PM
Subject: Re: [ceph-users] Ceph with VMWare / XenServ
Hi, i am working with test Ceph cluster and now I want to implement Zabbix
monitoring with items such as:
- whoe cluster IO (for example ceph -s -> recovery io 143 MB/s, 35
objects/s)
- pg statistics
I would like to create single script in python to retrive values using
rados python module, but t
On 5/13/2014 9:43 AM, Andrei Mikhailovsky wrote:
Dima, do you have any examples / howtos for this? I would love to give
it a go.
Not really: I haven't done this myself. Google for "tgtd failover with
heartbeat", you should find something useful.
The setups I have are heartbeat (3.0.x) managi
>>Actually check your random read output again, you gave it the wrong
>>parameter, it needs to be randread, not rand-read.
oops, sorry. I got around 7500iops with randread.
>>Your cluster isn't that old (the CPUs are in the same ballpark)
Yes, this is 6-7 year old server. (this xeons were release
Not sure your demand.
I use "ceph --admin-daemon /var/run/ceph/ceph-osd.x.asok perf dump" to
get the monitor infos. And the result can be parsed by simplejson
easily via python.
On Tue, May 13, 2014 at 10:56 PM, Adrian Banasiak wrote:
> Hi, i am working with test Ceph cluster and now I want to i
I hit a "bug" where radosgw crashed with
-101> 2014-05-13 15:26:07.188494 7fde82886820 0 ERROR: FCGX_Accept_r
returned -24
0> 2014-05-13 15:26:07.193772 7fde82886820 -1 rgw/rgw_main.cc: In
function 'virtual void RGWProcess::RGWWQ::_clear()' thread 7fde82886820
time 2014-05-13 15:26:07.1
I've upgraded to 0.80.1 on a testing instance: the cluster gets
cyclically active+clean+deep scrubbing for a little while and then
reaches active+clean status. I'm not worried about this, I think it's
normal, but I didn't have this behaviour on emperor 0.72.2.
Cheers,
Fabrizio
On 13 May 2014 06:0
All,
I have a recurring issue where the admin sockets
(/var/run/ceph/ceph-*.*.asok) may vanish on a running cluster while the
daemons keep running (or restart without my knowledge). I see this issue
on a dev cluster running Ubuntu and Ceph Emperor/Firefly, deployed with
ceph-deploy using Upst
I have just done some test,
with fio-rbd,
(http://telekomcloud.github.io/ceph/2014/02/26/ceph-performance-analysis_fio_rbd.html)
directly from the kvm host,(not from the vm).
1 fio job: around 8000iops
2 differents parralel fio job (on different rbd volume) : around 8000iops by
fio job !
cpu
On Tue, May 13, 2014 at 8:52 AM, Brian Rak wrote:
> I hit a "bug" where radosgw crashed with
>
> -101> 2014-05-13 15:26:07.188494 7fde82886820 0 ERROR: FCGX_Accept_r
> returned -24
too many files opened. You probably need to adjust your limits.
>
> 0> 2014-05-13 15:26:07.193772 7fde828868
I upgraded to 0.80.1 to see if that helped. It didn't change anything,
but I'm now seeing more useful errors:
2014-05-13 16:27:32.954007 7f5183cfc700 0 RGWGC::process() failed to
acquire lock on gc.10
2014-05-13 16:27:48.098428 7f5183cfc700 0 RGWGC::process() failed to
acquire lock on gc.14
On Tue, 13 May 2014 18:10:25 +0200 (CEST) Alexandre DERUMIER wrote:
> I have just done some test,
>
> with fio-rbd,
> (http://telekomcloud.github.io/ceph/2014/02/26/ceph-performance-analysis_fio_rbd.html)
>
> directly from the kvm host,(not from the vm).
>
>
> 1 fio job: around 8000iops
> 2 d
Thanks for sugestion with admin daemon but it looks like single osd
oriented. I have used perf dump on mon socket and it output some
interesting data in case of monitoring whole cluster:
{ "cluster": { "num_mon": 4,
"num_mon_quorum": 4,
"num_osd": 29,
"num_osd_up": 29,
"num_
On 5/13/2014 12:29 PM, Yehuda Sadeh wrote:
On Tue, May 13, 2014 at 8:52 AM, Brian Rak wrote:
I hit a "bug" where radosgw crashed with
-101> 2014-05-13 15:26:07.188494 7fde82886820 0 ERROR: FCGX_Accept_r
returned -24
too many files opened. You probably need to adjust your limits.
0>
>>You didn't specify what you did, but i assume you did read test.
yes, indeed
>>Those scale, as in running fio in multiple VMs in parallel gives me about
>>6200 IOPS each, so much better than the 7200 for a single one.
>>And yes, the client CPU is quite busy.
oh ok !
>>However my real, orig
This turns out to have been a configuration change to nginx that I
forgot I had made. It wasn't passing all the http options through any
more, so authentication was failing.
On 5/13/2014 12:43 PM, Brian Rak wrote:
On 5/13/2014 12:29 PM, Yehuda Sadeh wrote:
On Tue, May 13, 2014 at 8:52 AM, B
Adrian,
Yes, it is single OSD oriented.
Like Haomai, we monitor perf dumps from individual OSD admin sockets. On
new enough versions of ceph, you can do 'ceph daemon osd.x perf dump',
which is a shorter way to ask for the same output as 'ceph
--admin-daemon /var/run/ceph/ceph-osd.x.asok perf
python-cephclient may be of some use to you
https://github.com/dmsimard/python-cephclient
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Mike Dawson
> Sent: Tuesday, May 13, 2014 10:04 AM
> To: Adrian Banasiak; Haomai Wang
> Cc: ceph-us
By way of follow-up, I have done quite a bit more additional testing here and
my problem seems to be exclusive to rbd-fuse on Ubuntu 14.04. So probably not
related to this thread I chimed in on last week.
I am able to get read and write speeds in the hundreds of megabytes per second
with this s
On Mon, May 12, 2014 at 11:55 PM, wsnote wrote:
> Hi, everyone!
> Where can I find the SDK of ceph object storage?
> Python: boto
> C++: libs3 which I found in the src of ceph and github.com/ceph/libs3.
> where are that of other language? Does ceph supply them?
> Otherwise I use the SDK of Amazon
You just use a type other than "rack" in your chooseleaf rule. In your
case, "host". When using chooseleaf, the bucket type you specify is
the failure domain which it must segregate across.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
On Tue, May 13, 2014 at 12:52 AM, Cao, B
On Tue, May 13, 2014 at 9:06 AM, Mike Dawson wrote:
> All,
>
> I have a recurring issue where the admin sockets
> (/var/run/ceph/ceph-*.*.asok) may vanish on a running cluster while the
> daemons keep running
Hmm.
>(or restart without my knowledge).
I'm guessing this might be involved:
> I see
I have to say I'm shocked to see the suggestion is rbd import/export if
'you care the data'. These kind of operation is common use case and should
be an essential part of any distributed storage. What if I have a hundred
node cluster running for years and need to do hardware refresh? There are
no c
Greg/Loic,
I can confirm that "logrotate --force /etc/logrotate.d/ceph" removes the
monitor admin socket on my boxes running 0.80.1 just like the
description in Issue 7188 [0].
0: http://tracker.ceph.com/issues/7188
Should that bug be reopened?
Thanks,
Mike Dawson
On 5/13/2014 2:10 PM, Gr
On Tue, May 13, 2014 at 11:36 AM, Fred Yang wrote:
> I have to say I'm shocked to see the suggestion is rbd import/export if 'you
> care the data'. These kind of operation is common use case and should be an
> essential part of any distributed storage. What if I have a hundred node
> cluster runni
Yeah, I just did so. :(
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com
On Tue, May 13, 2014 at 11:41 AM, Mike Dawson wrote:
> Greg/Loic,
>
> I can confirm that "logrotate --force /etc/logrotate.d/ceph" removes the
> monitor admin socket on my boxes running 0.80.1 just like th
Anyone still seeing this issue on 0.80.1 you'll probable need to dump
out your scrub list "ceph pg dump | grep scrub" then find the OSD listed
as the acting primary for the PG stuck scrubbing and stop it a bit more
aggressively. I found that the acting primary for a PG stuck in scrub
status was
Hi All,
Seems commit 2adc534a72cc199c8b11dbdf436258cbe147101b has removed the
ability to delete and recreate the data and metadata pools using osd
pool delete (Returns Error EBUSY - Is in use by CephFS). Currently have
no mds running as I'm no longer using CephFS and so it's not mounted
anywh
Answered my own question. Created two new pools, used mds newfs on them
and then deleted the original pools and renamed the new ones.
-Michael
On 13/05/2014 22:20, Michael wrote:
Hi All,
Seems commit 2adc534a72cc199c8b11dbdf436258cbe147101b has removed the
ability to delete and recreate the
2014-05-13 21:21 GMT+02:00 Gregory Farnum :
> You misunderstand. Migrating between machines for incrementally
> upgrading your hardware is normal behavior and well-tested (likewise
> for swapping in all-new hardware, as long as you understand the IO
> requirements involved). So is decommissioning o
Assuming you have the spare throughput-/IOPS for Ceph to do its thing
without disturbing your clients, this will work fine.
-Greg
On Tuesday, May 13, 2014, Gandalf Corvotempesta <
gandalf.corvotempe...@gmail.com> wrote:
> 2014-05-13 21:21 GMT+02:00 Gregory Farnum
> >:
> > You misunderstand. Migr
On 5/13/14 05:15 , Christian Balzer wrote:
On Tue, 13 May 2014 22:03:11 +1200 Mark Kirkwood wrote:
On thing that would put me off the 530 is lack on power off safety
(capacitor or similar). Given the job of the journal, I think an SSD
that has some guarantee of write integrity is crucial - so y
Georg,
I've had similar issues when I had a "+" character in my secret key. Not all
clients support it. You might need to escape this with \ and see if it works.
Andrei
- Original Message -
From: "Georg Höllrigl"
To: ceph-users@lists.ceph.com
Sent: Tuesday, 13 May, 2014 1:30:14
> TL;DR: Power outages are more common than your colo facility will admit.
Seconded. I've seen power failures in at least 4 different facilities
and all of them had the usual gamut of batteries/generators/etc. Some
of those facilities I've seen problems multiple times in a single
year. Even a data
On 05/13/2014 05:49 PM, Kyle Bader wrote:
>> TL;DR: Power outages are more common than your colo facility will admit.
>
> Seconded. I've seen power failures in at least 4 different facilities
> and all of them had the usual gamut of batteries/generators/etc. Some
> of those facilities I've seen pr
On 5/13/14 09:33 , Adrian Banasiak wrote:
Thanks for sugestion with admin daemon but it looks like single osd
oriented. I have used perf dump on mon socket and it output some
interesting data in case of monitoring whole cluster:
{ "cluster": { "num_mon": 4,
"num_mon_quorum": 4,
"num
Hi Adrian,
You may be interested in "rados -p poo_name df --format json", although it's
pool oriented, you could probably add the values together :)
Regards,
Kai
在 2014-05-13 08:33:11，"Adrian Banasiak" 写道：
Thanks for sugestion with admin daemon but it looks like single osd oriented. I
have u
Hi All,
I am following manual steps to create osd node.
while executing below command, i am facing error like below
#ceph-osd -i 1 --mkfs --mkkey
2014-05-14 05:04:12.097585 7f91c99007c0 -1 ** ERROR: error creating empty
object store in /var/lib/ceph/osd/-: (2) No such file or directory
But dir
Thanks Gregory so much，it solved the problem!
Wei Cao (Buddy)
-Original Message-
From: Gregory Farnum [mailto:g...@inktank.com]
Sent: Wednesday, May 14, 2014 2:00 AM
To: Cao, Buddy
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] crushmap question
You just use a type other than
BTW, I'd like to know, after I change the "from rack" to "from host", if I add
more racks with host/osds in the cluster, will ceph choose the osds for pg only
from one zone? or ceph will randomly choose from several different zones?
Wei Cao (Buddy)
-Original Message-
From: Cao, Buddy
65 matches
Mail list logo
