Hi Alexandre,
Yeah we are using filestore for the moment with luminous. With regards to
client, I tried both jewel and luminous librbd versions against the
luminous cluster - similar results.
I am running fio on a physical machine with fio rbd engine. This is a
snippet of the fio config for the r
On 17-09-20 08:06, nokia ceph wrote:
Hello,
Env:- RHEL 7.2 , 3.10.0-327.el7.x86_64 , EC 4+1 , bluestore
We are writing to ceph via librados C API . Testing with rados no
issues.
The same we tested with Jewel/kraken without any issue. Need your view
how to debug this issue?
maybe similar to
Hi
so, you use also filestore on luminous ?
do you have also upgraded librbd on client ? (are you benching inside a qemu
machine ? or directly with fio-rbd ?)
(I'm going to do a lot of benchmarks in coming week, I'll post results on
mailing soon.)
- Mail original -
De: "Rafael Lop
hey guys.
wondering if anyone else has done some solid benchmarking of jewel vs
luminous, in particular on the same cluster that has been upgraded (same
cluster, client and config).
we have recently upgraded a cluster from 10.2.9 to 12.2.0, and
unfortunately i only captured results from a single
Hello,
Env:- RHEL 7.2 , 3.10.0-327.el7.x86_64 , EC 4+1 , bluestore
We are writing to ceph via librados C API . Testing with rados no issues.
The same we tested with Jewel/kraken without any issue. Need your view how
to debug this issue?
>>
OSD.log
==
~~~
2017-09-18 14:51:59.895746 7f1e7
We had suicide timeouts, but unfortunately I can't remember the specific
root cause at this point.
It was definitely one of two things:
* too low of net.netfilter.nf_conntrack_max (preventing the osds from
opening new connections to each other)
* too low of kernel.pid_max or kernel.threads-m
We don't use EC pools, but my experience with similar slow requests on
RGW+replicated_pools is that in the logs you need to find out the first
slow request and identify where it's from, for example, is it
deep-scrub, or some client accessing corrupted objects, disk errors etc.
On 20/09/17 8:1
I like this, there is some similar ideas we probably can borrow from
Cassandra on disk failure
# policy for data disk failures:
# die: shut down gossip and Thrift and kill the JVM for any fs errors or
# single-sstable errors, so the node can be replaced.
# stop_paranoid: shut down gossip a
Can you please provide the output of `ceph status`, `ceph osd tree`, and
`ceph health detail`? Thank you.
On Tue, Sep 19, 2017 at 2:59 PM Jonas Jaszkowic <
jonasjaszkowic.w...@gmail.com> wrote:
> Hi all,
>
> I have setup a Ceph cluster consisting of one monitor, 32 OSD hosts (1 OSD
> of size 320
Just starting 3 nights ago we started seeing OSDs randomly going down in
our cluster (Jewel 10.2.7). At first I saw that each OSD that was recently
marked down in the cluster (`ceph osd dump | grep -E '^osd\.[0-9]+\s' |
sort -nrk11` sorted list of OSDs by which OSDs have been marked down in the
mo
Hi all,
I have setup a Ceph cluster consisting of one monitor, 32 OSD hosts (1 OSD of
size 320GB per host) and 16 clients which are reading
and writing to the cluster. I have one erasure coded pool (shec plugin) with
k=8, m=4, c=3 and pg_num=256. Failure domain is host.
I am able to reach a HEA
You've probably run in to http://tracker.ceph.com/issues/16010 — do you
have very large directories? (Or perhaps just a whole bunch of unlinked
files which the MDS hasn't managed to trim yet?)
On Tue, Sep 19, 2017 at 11:51 AM Christian Salzmann-Jäckel <
christian.salzm...@fu-berlin.de> wrote:
> H
Hi,
we run cephfs (10.2.9 on Debian jessie; 108 OSDs on 9 nodes) as scratch
filesystem for a HPC cluster using IPoIB interconnect with kernel client
(Debian backports kernel version 4.9.30).
Our clients started blocking on file system access.
Logs show 'mds0: Behind on trimming' and slow reque
On Tue, Sep 19, 2017 at 9:38 AM Kees Meijs wrote:
> Hi Jake,
>
> On 19-09-17 15:14, Jake Young wrote:
> > Ideally you actually want fewer disks per server and more servers.
> > This has been covered extensively in this mailing list. Rule of thumb
> > is that each server should have 10% or less of
Adding the old OSD back in with its data shouldn't help you at all. Your
cluster has finished backfilling and has the proper amount of copies of all
of its data. The time you would want to add a removed OSD back to a
cluster is when you have unfound objects.
The scrub errors and inconsistent PGs
Hi David,
What I want is to add the OSD back with its data yes. But avoiding any
troubles that can happen from the time it was out.
Is it possible? I suppose that some pg has been updated after. Will ceph
manage it gracefully?
Ceph status is getting worse every day.
ceph status
cluster
On Mon, Sep 18, 2017 at 2:02 PM, Christian Theune wrote:
> Hi,
>
> and here’s another update which others might find quite interesting.
>
> Florian and I spend some time discussing the issue further, face to face. I
> had one switch that I brought up again (—osd-recovery-start-delay) which I
> l
Hi Jake,
On 19-09-17 15:14, Jake Young wrote:
> Ideally you actually want fewer disks per server and more servers.
> This has been covered extensively in this mailing list. Rule of thumb
> is that each server should have 10% or less of the capacity of your
> cluster.
That's very true, but let's f
On Tue, Sep 19, 2017 at 7:34 AM Kees Meijs wrote:
> Hi list,
>
> It's probably something to discuss over coffee in Ede tomorrow but I'll
> ask anyway: what HBA is best suitable for Ceph nowadays?
>
> In an earlier thread I read some comments about some "dumb" HBAs running
> in IT mode but still b
> Op 19 september 2017 om 13:34 schreef Kees Meijs :
>
>
> Hi list,
>
> It's probably something to discuss over coffee in Ede tomorrow but I'll
> ask anyway: what HBA is best suitable for Ceph nowadays?
>
I still prefer LSI (Avago) in most systems. A 8-port or 16-port controller and
a bunch
Are you asking to add the osd back with its data or add it back in as a
fresh osd. What is your `ceph status`?
On Tue, Sep 19, 2017, 5:23 AM Gonzalo Aguilar Delgado <
gagui...@aguilardelgado.com> wrote:
> Hi David,
>
> Thank you for the great explanation of the weights, I thought that ceph
> was
Hi list,
It's probably something to discuss over coffee in Ede tomorrow but I'll
ask anyway: what HBA is best suitable for Ceph nowadays?
In an earlier thread I read some comments about some "dumb" HBAs running
in IT mode but still being able to use cache on the HBA. Does it make
sense? Or, is th
On Tue, 19 Sep 2017, Yoann Moulin said:
> Hello,
>
> Does anyone have tested s3cmd or other tools to manage ACL on luminous
> radosGW ?
Don't know about ACL, but s3cmd for other things works for me. Version 1.6.1
My config file includes (but is not limited to):
access_key = yourkey
secret_ke
> Op 19 september 2017 om 10:24 schreef Adrian Saul
> :
>
>
> > I understand what you mean and it's indeed dangerous, but see:
> > https://github.com/ceph/ceph/blob/master/systemd/ceph-osd%40.service
> >
> > Looking at the systemd docs it's difficult though:
> > https://www.freedesktop.org/soft
Op 19-9-2017 om 11:24 schreef Yoann Moulin:
Hello,
Does anyone have tested s3cmd or other tools to manage ACL on luminous radosGW ?
I have opened an issue on s3cmd too
https://github.com/s3tools/s3cmd/issues/919
Just an extra option. Have you tried --signature-v2 on s3cmd?
Thanks for your h
Hello,
Does anyone have tested s3cmd or other tools to manage ACL on luminous radosGW ?
I have opened an issue on s3cmd too
https://github.com/s3tools/s3cmd/issues/919
Thanks for your help
Yoann
> I have a fresh luminous cluster in test and I made a copy of a bucket (4TB
> 1.5M files) with r
Hi David,
Thank you for the great explanation of the weights, I thought that ceph
was adjusting them based on disk. But it seems it's not.
But the problem was not that I think the node was failing because a
software bug because the disk was not full anymeans.
/dev/sdb1 9
Am Tue, 19 Sep 2017 08:24:48 +
schrieb Adrian Saul :
> > I understand what you mean and it's indeed dangerous, but see:
> > https://github.com/ceph/ceph/blob/master/systemd/ceph-osd%40.service
> >
> > Looking at the systemd docs it's difficult though:
> > https://www.freedesktop.org/software/s
> I understand what you mean and it's indeed dangerous, but see:
> https://github.com/ceph/ceph/blob/master/systemd/ceph-osd%40.service
>
> Looking at the systemd docs it's difficult though:
> https://www.freedesktop.org/software/systemd/man/systemd.service.ht
> ml
>
> If the OSD crashes due to ano
> Op 19 september 2017 om 10:02 schreef Manuel Lausch :
>
>
> Hi,
>
> I see a issue with systemd's restart behaviour and disk IO-errors
> If a disk fails with IO-errors ceph-osd stops running. Systemd detects
> this and starts the daemon again. In our cluster I did see some loops
> with osd cr
Hi,
I see a issue with systemd's restart behaviour and disk IO-errors
If a disk fails with IO-errors ceph-osd stops running. Systemd detects
this and starts the daemon again. In our cluster I did see some loops
with osd crashes caused by disk failure and restarts triggerd by
systemd. Every time w
31 matches
Mail list logo
