Not much suggestion from my side (I never ran Ceph on complete HDDs), other
than probably running more OSDs/HDDs. More OSDs should help specially if you
can spread these on many nodes.
I would say try with fio-rbd (librbd) first (rbd_cache = false) as it may give
you some boost over kernel rbd s
Hi ,
I 'm in the process of upgrading my ceph cluster from Firefly to Hammer.
The ceph cluster has 12 OSD spread across 4 nodes.
Mons have been upgraded to hammer, since I have created pools with value
512 and 256 , so am bit confused with the warning message.
--snip--
ceph -s
cluster a71
Hi!
>Hi, I can reach 60 iops 4k read with 3 nodes (6ssd each).
It is very interesting! Can you give any details about your config?
We can't get more than ~40kiops 4k random reads from 2node x 2ssd pool. :(
Under load our SSDs give ~8kiops each, and that is far too low for Intel DC
S3700 400
On 08/31/2015 08:06 AM, Alexandre DERUMIER wrote:
>>> True, true. But I personally think that Ceph doesn't perform well on
>>> small <10 node clusters.
>
> Hi, I can reach 60 iops 4k read with 3 nodes (6ssd each).
>
True, but your performance is greatly impacted during recovery. So a
three n
Can you share your SSD products and ceph.conf?
In a test 3node cluster, 2ssds each intel s3500 i see very disappointing
numbers.
I maintain a 6 node cluster mixed ssd and sata pools. IOPS are not enough for a
kvm hosting company unless you have really low values at disk io throttling. A
big clu
On Mon, Aug 31, 2015 at 5:07 AM, Christian Balzer wrote:
>
> Hello,
>
> I'm about to add another storage node to small firefly cluster here and
> refurbish 2 existing nodes (more RAM, different OSD disks).
>
> Insert rant about not going to start using ceph-deploy as I would have to
> set the clus
On Mon, Aug 31, 2015 at 8:30 AM, 10 minus wrote:
> Hi ,
>
> I 'm in the process of upgrading my ceph cluster from Firefly to Hammer.
>
> The ceph cluster has 12 OSD spread across 4 nodes.
>
> Mons have been upgraded to hammer, since I have created pools with value
> 512 and 256 , so am bit confus
Hello,
I'm getting a segmentation fault error from the monitor of our test cluster.
The cluster was in a bad state because I have recently removed three hosts from
it. Now I started cleaning it up and first marked the removed osd's as lost
(ceph osd lost), and then I tried to remove the osd's f
On Mon, Aug 31, 2015 at 9:33 AM, Eino Tuominen wrote:
> Hello,
>
> I'm getting a segmentation fault error from the monitor of our test cluster.
> The cluster was in a bad state because I have recently removed three hosts
> from it. Now I started cleaning it up and first marked the removed osd's
This generally shouldn't be a problem at your bucket sizes. Have you
checked that the cluster is actually in a healthy state? The sleeping
locks are normal but should be getting woken up; if they aren't it
means the object access isn't working for some reason. A down PG or
something would be the si
On Sat, Aug 29, 2015 at 11:50 AM, Gerd Jakobovitsch wrote:
> Dear all,
>
> During a cluster reconfiguration (change of crush tunables from legacy to
> TUNABLES2) with large data replacement, several OSDs get overloaded and had
> to be restarted; when OSDs stabilize, I got a number of PGs marked st
On Sat, Aug 29, 2015 at 3:32 PM, Евгений Д. wrote:
> I'm running 3-node cluster with Ceph (it's Deis cluster, so Ceph daemons are
> containerized). There are 3 OSDs and 3 mons. After rebooting all nodes one
> by one all monitors are up, but only two OSDs of three are up. 'Down' OSD is
> really run
Maybe it's just a precision problem?
I calculate the durability from PL(*) columns with the formula:
1-PL(site)-PL(copy)-PL(NRE).
Result:
2-cp is 0.99896562
3-cp is 0.99900049
Both of them are approximates to 99.9%
Actually the model result is 99.900%. Maybe the author wants us to ignore
the la
Hi Greg,
(gdb) bt
#0 0x7f071a05020b in raise () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x009a996d in ?? ()
#2
#3 0x0085ada2 in crush_do_rule ()
#4 0x007a85cb in OSDMap::_pg_to_osds(pg_pool_t const&, pg_t,
std::vector >*, int*, unsigned int*) const ()
#5 0
Oh whoops, can you install the ceph-debug packages as well? That will
provide line numbers on the call sites. :)
-Greg
On Mon, Aug 31, 2015 at 10:25 AM, Eino Tuominen wrote:
> Hi Greg,
>
> (gdb) bt
> #0 0x7f071a05020b in raise () from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x009
Hi Greg,
Sure, should have gathered that myself...
(gdb) bt
#0 0x7f071a05020b in raise () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x009a996d in reraise_fatal (signum=11) at
global/signal_handler.cc:59
#2 handle_fatal_signal (signum=11) at global/signal_handler.cc:109
#3
#
On Mon, Aug 24, 2015 at 6:38 PM, Gregory Farnum wrote:
> On Mon, Aug 24, 2015 at 11:35 AM, Simon Hallam wrote:
>> Hi Greg,
>>
>> The MDS' detect that the other one went down and started the replay.
>>
>> I did some further testing with 20 client machines. Of the 20 client
>> machines, 5 hung wi
Hi there,
I have a ceph-cluster (0.94-2) with >100 rbd kvm images.
Most vms are running rock-solid but 7 vms are hanging about once a week.
I found out the hanging machines have
features: layering, exclusive, object map while all other vms do not have
exclusive and object map set.
Now I want
On Mon, Aug 31, 2015 at 12:16 PM, Yan, Zheng wrote:
> On Mon, Aug 24, 2015 at 6:38 PM, Gregory Farnum wrote:
>> On Mon, Aug 24, 2015 at 11:35 AM, Simon Hallam wrote:
>>> Hi Greg,
>>>
>>> The MDS' detect that the other one went down and started the replay.
>>>
>>> I did some further testing with
Hi,
I have been trying to understand the default pools used by the default zone
in a radosgw only configuration and have got a confusion regarding the same.
The documentation here (
http://ceph.com/docs/master/radosgw/config-ref/#pools) says that the set of
default pools include .rgw, while the d
Unfortunately, the tool the dynamically enable/disable image features (rbd
feature disable ) was added during the Infernalis
development cycle. Therefore, in the short-term you would need to recreate the
images via export/import or clone/flatten.
There are several object map / exclusive loc
On 08/31/2015 10:37 AM, Eino Tuominen wrote:
> Hi Greg,
>
> Sure, should have gathered that myself...
>
> (gdb) bt
> #0 0x7f071a05020b in raise () from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x009a996d in reraise_fatal (signum=11) at
> global/signal_handler.cc:59
> #2 handle_
Sorry about the repost from the cbt list, but it was suggested I post here as
well:
I am attempting to track down some performance issues in a Ceph cluster
recently deployed. Our configuration is as follows:
3 storage nodes, each with:
- 8 Cores
- 64GB of
No, it really was in the cluster. Before reboot cluster had HEALTH_OK.
Though now I've checked `current` directory and it doesn't contain any data:
root@staging-coreos-1:/var/lib/ceph/osd/ceph-0# ls current
commit_op_seq meta nosnap omap
while other OSDs do. It really looks lik
Is it possible that something else was mounted there?
Or is it possible nothing was mounted there?
That would explain such behaviour...
Jan
> On 31 Aug 2015, at 17:07, Евгений Д. wrote:
>
> No, it really was in the cluster. Before reboot cluster had HEALTH_OK.
> Though now I've checked `current
On 08/31/2015 10:37 AM, Eino Tuominen wrote:
> Hi Greg,
>
> Sure, should have gathered that myself...
>
> (gdb) bt
> #0 0x7f071a05020b in raise () from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x009a996d in reraise_fatal (signum=11) at
> global/signal_handler.cc:59
> #2 handle_
Hi cephers,
What's the recommended version for new productive clusters?
Thanks in advanced,
Best regards,
*German*
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Hello,
On Mon, 31 Aug 2015 08:31:57 -0500 Kenneth Van Alstyne wrote:
> Sorry about the repost from the cbt list, but it was suggested I post
> here as well:
>
I wasn't even aware a CBT (what the heck does that acronym stand for?)
existed...
> I am attempting to track down some performance issu
I see that the objects that were deleted last Friday are indeed gone now (via
gc I guess).
gc list does not show anything even after right after objects are deleted.
I couldn't get temp remove to do anything.
-- Tom
> -Original Message-
> From: Ben Hines [mailto:bhi...@gmail.com]
> Sent
Hammer should be very stable at this point.
*Kobi Laredo*
*Cloud Systems Engineer* | (*408) 409-KOBI*
On Mon, Aug 31, 2015 at 8:51 AM, German Anders wrote:
> Hi cephers,
>
>What's the recommended version for new productive clusters?
>
> Thanks in advanced,
>
> Best regards,
>
> *German*
>
>
Thanks a lot Kobi
*German*
2015-08-31 14:20 GMT-03:00 Kobi Laredo :
> Hammer should be very stable at this point.
>
> *Kobi Laredo*
> *Cloud Systems Engineer* | (*408) 409-KOBI*
>
> On Mon, Aug 31, 2015 at 8:51 AM, German Anders
> wrote:
>
>> Hi cephers,
>>
>>What's the recommended version
Christian, et al:
Sorry for the lack of information. I wasn’t sure what of our hardware
specifications or Ceph configuration was useful information at this point.
Thanks for the feedback — any feedback, is appreciated at this point, as I’ve
been beating my head against a wall trying to figure
Hello Brad,
Sorry for the delay in replying. As you mentioned earlier,
$ `which rados` -v
Returns a command not found error
$ strings `which rados`|grep "^ceph version" -A5
Returns no results.
Thanks a lot!
Aakanksha
-Original Message-
From: Brad Hubbard [mailto:bhubb...@redhat.com]
On one of our staging ceph clusters (firefly 0.80.10) I've noticed that
some
of the statistics in the 'ceph df' output don't seem to match up. For
example
in the output below the amount of raw used is 8,402G, which with triple
replication would be 2,800.7G used (all the pools are triple replicatio
- Original Message -
> From: "Aakanksha Pudipeddi-SSI"
> To: "Brad Hubbard"
> Cc: ceph-us...@ceph.com
> Sent: Tuesday, 1 September, 2015 3:33:38 AM
> Subject: RE: [ceph-users] Rados: Undefined symbol error
>
> Hello Brad,
>
> Sorry for the delay in replying. As you mentioned earlier,
>
Hello Brad,
When I type "which rados" it returns /usr/bin/rados. I am using Ubuntu 14.04. I
follow these steps in installing ceph from source:
1. ./autogen.sh
2. Going to be using rocksdb, so: ./configure --with-librocksdb-static
3. make
4. sudo dpkg-buildpackage
Then I use ceph-deploy to compl
Hey Kenneth, it looks like you¹re just down the tollroad from me. I¹m in
Reston Town Center.
Just as a really rough estimate, I¹d say this is your max IOPS:
80 IOPS/spinner * 6 drives / 3 replicas = 160ish max sustained IOPS
It¹s more complicated than that, since you have a reasonable solid state
I tried pg query, but it doesn't return, it hungs forever. As I
understand it, when the PG is stale, there is no OSD to get the query.
Am I right?
I did the tunables in 2 steps, but didn't wait for all the data being
moved before doing the second step.
I rolled back to intermediate tunables -
No input, eh? (or maybe TL,DR for everyone)
Short version: Presuming the bucket index shows blank/empty, which it
does and is fine, would me manually deleting the rados objects with
the prefix matching the former bucket's ID cause any problems?
thanks,
-Ben
On Fri, Aug 28, 2015 at 4:22 PM, Ben
- Original Message -
> From: "Aakanksha Pudipeddi-SSI"
> To: "Brad Hubbard"
> Cc: ceph-us...@ceph.com
> Sent: Tuesday, 1 September, 2015 7:27:04 AM
> Subject: RE: [ceph-users] Rados: Undefined symbol error
>
> Hello Brad,
>
> When I type "which rados" it returns /usr/bin/rados.
Ah, I t
As long as you're 100% sure that the prefix is only being used for the
specific bucket that was previously removed, then it is safe to remove
these objects. But please do double check and make sure that there's
no other bucket that matches this prefix somehow.
Yehuda
On Mon, Aug 31, 2015 at 2:42
Brad,
Yes, you are right. Sorry about that! This is what I get when I try with the
back ticks:
$ `which rados` -v
/usr/bin/rados: symbol lookup error: /usr/bin/rados: undefined symbol:
_ZN5MutexC1ERKSsbbbP11CephContext
$ strings `which rados`|grep "^ceph version"
$
$ strings $(which rados)|grep
When we know we need to off a node, we weight it down over time. Depending
on your cluster, you may need to do this over days or hours.
In theory, you could do the same when putting OSDs in, by setting noin,
and then setting weight to something very low, and going up over time. I
haven¹t tried thi
Ok. I'm not too familiar with the inner workings of RGW, but i would
assume that for a bucket with these parameters:
"id": "default.8873277.32",
"marker": "default.8873277.32",
Tha it would be the only bucket using the files that start with
"default.8873277.32"
default.8873277.32__shadow_.
Make sure you use the underscore also, e.g., "default.8873277.32_".
Otherwise you could potentially erase objects you did't intend to,
like ones who start with "default.8873277.320" and such.
On Mon, Aug 31, 2015 at 3:20 PM, Ben Hines wrote:
> Ok. I'm not too familiar with the inner workings of R
I would say you are probably simply IO starved because you're running too
many VMs.
To follow on from Warren's response, if you spread those 160 available iops
across 15 VMs, you are talking about roughly 10 iops per vm, assuming they
have similar workloads. That's almost certainly too little. I w
Good call, thanks!
Is there any risk of also deleting parts of the bucket index? I'm not
sure what the objects for the index itself look like, or if they are
in the .rgw.buckets pool.
On Mon, Aug 31, 2015 at 3:23 PM, Yehuda Sadeh-Weinraub
wrote:
> Make sure you use the underscore also, e.g., "d
- Original Message -
> From: "Aakanksha Pudipeddi-SSI"
> To: "Brad Hubbard"
> Cc: "ceph-users"
> Sent: Tuesday, 1 September, 2015 7:58:33 AM
> Subject: RE: [ceph-users] Rados: Undefined symbol error
>
> Brad,
>
> Yes, you are right. Sorry about that! This is what I get when I try with
We have the following in our ceph.conf to bring in new OSDs with a weight
of 0:
[osd]
osd_crush_initial_weight = 0
We then set 'nobackfill' and bring in each OSD at full weight one at a
time (letting things settle down before bring in the next OSD). Once all
the OSDs are brought in we unset 'no
- Original Message -
> From: "Brad Hubbard"
> To: "Aakanksha Pudipeddi-SSI"
> Cc: "ceph-users"
> Sent: Tuesday, 1 September, 2015 8:36:33 AM
> Subject: Re: [ceph-users] Rados: Undefined symbol error
>
> - Original Message -
> > From: "Aakanksha Pudipeddi-SSI"
> > To: "Brad Hu
The bucket index objects are most likely in the .rgw.buckets.index pool.
Yehuda
On Mon, Aug 31, 2015 at 3:27 PM, Ben Hines wrote:
> Good call, thanks!
>
> Is there any risk of also deleting parts of the bucket index? I'm not
> sure what the objects for the index itself look like, or if they are
On Mon, 31 Aug 2015 08:57:23 +0200 Udo Lembke wrote:
> Hi Christian,
> for my setup "b" takes too long - too much data movement and stress to
> all nodes. I have simply (with replica 3) "set noout", reinstall one
> node (with new filesystem on the OSDs, but leave them in the crushmap)
> and start
Hello,
On Mon, 31 Aug 2015 22:44:05 + Stillwell, Bryan wrote:
> We have the following in our ceph.conf to bring in new OSDs with a weight
> of 0:
>
> [osd]
> osd_crush_initial_weight = 0
>
>
> We then set 'nobackfill' and bring in each OSD at full weight one at a
> time (letting things se
Hi,
Lets assume, I have an 40MB object stripped using librados stripper to ten
4MB sub objects. If I add data to the middle of the object, then how does
RADOS handle it ?
Concretely, say object A is stripped to A0 - A9. I add 2MB of data starting
at 8MB. This means, I now have A2 increased from 4M
Hello,
On Mon, 31 Aug 2015 12:28:15 -0500 Kenneth Van Alstyne wrote:
In addition to the spot on comments by Warren and Quentin, verify this by
watching your nodes with atop, iostat, etc.
The culprit (HDDs) should be plainly visible.
More inline:
> Christian, et al:
>
> Sorry for the lack of
I had a question regarding how OSD locations are determined by CRUSH.
>From the CRUSH paper I gather that the replica locations of an object (A)
is a vector (v) that is got by the function *c(r,x) = (hash (x) + rp) mod
m)*.
Now when new OSDs are added, objects are shuffled to maintain uniform dat
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Nick Fisk
> Sent: Wednesday, August 19, 2015 5:25 AM
> To: 'Samuel Just'
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] any recommendation of using EnhanceIO?
>
> Hi Sam,
>
> > -
Hello,
I use librados C API rados_append() to append object data in erasure coded
pool, it always return -95 (Operation not supported).
Buf if i use the same code to append object data in replicated pool, it
works fine.
Does erasure coded pool not support append write?
Below is my erasure coded p
I think this could be misaligned writes.
Is it multiple of 4k ?? Its just a wild guess.
thanks,
Shylesh
On Tue, Sep 1, 2015 at 9:17 AM, Hercules wrote:
> Hello,
>
> I use librados C API rados_append() to append object data in erasure coded
> pool, it always return -95 (Operation not supported).
59 matches
Mail list logo
