Hi everyone,
I read an earlier thread [1] that made a good explanation on the 'step
choose|chooseleaf' option. Could someone further help me to understand
the 'firstn|indep' part? Also, what is the relationship between 'step
take' and 'step choose|chooseleaf' when it comes to define a failure
dom
On Wed, Aug 22, 2018 at 7:57 AM mj wrote:
>
> Hi,
>
> This morning I woke up, seeing my ceph jewel 10.2.10 cluster in
> HEALTH_ERR state. That helps you getting out of bed. :-)
>
> Anyway, much to my surprise, all VMs running on the cluster were still
> working like nothing was going on. :-)
>
>
[sent both to ceph-devel and ceph-users lists, as it might be of
interest for both audiences]
Hi all,
This e-mail is just to announce the WIP on backporting dashboard_v2
(http://docs.ceph.com/docs/master/mgr/dashboard/) from master to
Luminous release.
The ultimate goal for this backport is to r
On 22/08/2018 12:16, Ernesto Puerta wrote:
[sent both to ceph-devel and ceph-users lists, as it might be of
interest for both audiences]
Hi all,
This e-mail is just to announce the WIP on backporting dashboard_v2
(http://docs.ceph.com/docs/master/mgr/dashboard/) from master to
Luminous release.
Hi!
I am in the progress of moving a local ("large", 24x1TB) ZFS RAIDZ2 to
CephFS.
This storage is used for backup images (large sequential reads and writes).
To save space and have a RAIDZ2 (RAID6) like setup, I am planning the
following profile:
ceph osd erasure-code-profile set myprofile \
Not 3+2, but we run 4+2, 6+2, 6+3, 5+3, and 8+3 with cephfs in
production. Most of them are HDDs without separate DB devices.
Paul
2018-08-22 14:27 GMT+02:00 Kevin Olbrich :
> Hi!
>
> I am in the progress of moving a local ("large", 24x1TB) ZFS RAIDZ2 to
> CephFS.
> This storage is used for bac
I would suggest having some flash media for their own OSDs to put the
cephfs metadata pool onto. That was a pretty significant boost for me when
I moved the metadata pool onto flash media. My home setup is only 3 nodes
and is running EC 2+1 on pure HDD OSDs with metadata on SSDs. It's been
runni
Hi everyone,
We have a hard time figuring out a behaviour encountered after upgrading
the monitors of one of our cluster from Jewel to Luminous yesterday.
The cluster is composed of 14 OSDs hosts (2xE5-2640 v3 and 64 GB of RAM),
each containing 12x4TB OSD with journals on DC grade SSDs). The clus
I also have 2+1 (still only 3 nodes), and 3 replicated. I also moved the
meta datapool to ssds.
What is nice with the cephfs, you can have folders in your filesystem on
the ec21 pool for not so important data and the rest will be 3x
replicated.
I think the single session performance is not
Hello *,
we have an issue with a Luminous cluster (all 12.2.5, except one on
12.2.7) for RBD (OpenStack) and CephFS. This is the osd tree:
host1:~ # ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 22.57602 root default
-41.81998 host host5
14
On Wed, Aug 22, 2018 at 1:28 PM Kevin Olbrich wrote:
>
> Hi!
>
> I am in the progress of moving a local ("large", 24x1TB) ZFS RAIDZ2 to CephFS.
> This storage is used for backup images (large sequential reads and writes).
>
> To save space and have a RAIDZ2 (RAID6) like setup, I am planning the
>
Hi all,
For those still using filestore and running clusters with a large number of
objects, I am seeking some thoughts on increasing the filestore split
settings. Currently we have:
filestore merge threshold = 70
filestore split multiple = 20
Has anyone gone higher than this?
We are hitting th
Adjusting CRUSH weight shouldn't have caused this. Unfortunately the logs
don't have a lot of hints — the thread that crashed doesn't have any output
except for the Crashed state. If you can reproduce this with more debugging
on we ought to be able to track it down; if not it seems we missed a
stra
Hi,
I was testing versioning and autosharding in luminous 12.2.5 upgrading to
12.2.7 I wanted to know if the upgraded autosharded bucket is still usable.
Looks like it is, but a bucket limit check seems to show too many objects.
On my test servers, I created a bucket using 12.2.5, turned on
Hi,
I am trying to enable prometheus on Mimic so I can use it with cephmetrics
But it fails with the following error
Any help troubleshooting this will be very appreciated
"..
Module 'prometheus' has failed: error('No socket could be created',)
.."
here is some info ( all commons ran on MON whe
Hi ceph-users,
A few weeks ago, I had an OSD node -- ceph02 -- lock up hard with no
indication why. I reset the system and everything came back OK, except
that I now get intermittent warnings about slow/blocked requests from
OSDs on the other nodes, waiting for a "subop" to complete on one of
ceph
I have created new OSDs for Ceph Luminous. In my Ceph.conf I have
specified that the db size be 10GB, and the wal size be 1GB. However when
I type ceph daemon osd.0 perf dump I get: bluestore_allocated": 5963776
I think this means that the bluestore db is using the default, and not the
value o
Thanks for your feedback, Willem!
> The old dashboard does not need any package fetch while
> building/installing. Something that is not very handy when building
> FreeBSD packages. And I haven't gotten around to determining how to > get
> around that.
I thought that https://github.com/ceph/ceph
Hi Robert,
bluestore_allocated just tracks how many space was allocated from
slow(aka block) device to keep actual user data. It has nothing about DB
and/or WAL.
There are counters in bluefs section which track corresponding DB/WAL usage.
Thanks,
Igor
On 8/22/2018 8:34 PM, Robert Stanford
Some follow-up.
After doubling the RAM (so 128GB for 12x4TB OSD), all the RAM is still
consumed a bit after the OSD restart.
We are considering going through with the update of the OSDs to luminous
(or maybe go back to jewel on the mons...) but the
cluster is in bad shape...
health: HEALTH_
The config settings for DB and WAL size don't do anything. For journal
sizes they would be used for creating your journal partition with
ceph-disk, but ceph-volume does not use them for creating bluestore OSDs.
You need to create the partitions for the DB and WAL yourself and supply
those partitio
My initial reaction to this PR/backport was questioning why such a major
update would happen on a dot release of Luminous. Your reaction to keeping
both dashboards viable goes to support that. Should we really be
backporting features into a dot release that force people to change how
they use the
On Wed, Aug 22, 2018 at 2:48 PM, David Turner wrote:
> The config settings for DB and WAL size don't do anything. For journal
> sizes they would be used for creating your journal partition with ceph-disk,
> but ceph-volume does not use them for creating bluestore OSDs. You need to
> create the p
In my case I am using the same values for lvcreate and in the ceph.conf
(bluestore* settings). Since my lvs are the size I want the db to be, and
since I'm told that the wal will live in the same place automatically, it
sounds like setting my lv to be xGB ensures Ceph will use all of this for
db/
Yes, whatever you set your DB LV to at the time you create the Bluestore
OSD, it will use all of that space for the db/wal. If you increase the
size after the initial creation, the space will not be used for the the
DB. You cannot resize it.
On Wed, Aug 22, 2018 at 3:39 PM Robert Stanford
wrote
David - thanks again. Your input the last week has been invaluable.
On Wed, Aug 22, 2018 at 2:41 PM David Turner wrote:
> Yes, whatever you set your DB LV to at the time you create the Bluestore
> OSD, it will use all of that space for the db/wal. If you increase the
> size after the initial
Hi Adrien,
I don't expect I can fully explain what happened to your cluster, but
since you got no other feedback so far I'll try my best.
So you have 517 million RADOS objects. Assuming at least 3 copies each
for normal replication or 5 "shards" for EC pools, there are somewhere
between 1.5 to 2.
We had DevConf in Boston over last weekend and they've posted videos of
each talk on Youtube. Quality isn't perfect but is reasonable.
I spoke on the Teuthology testing framework (
https://devconfus2018.sched.com/event/FNMj/pains-pleasures-of-automatic-software-testing)
and the talk is at https://
Thank you so much for your feedback. It always helps with this kind of
situation.
We fixed the network issue, added as much RAM and as much swap as possible
but
were still far from anything with OOM killer decimating the OSDs which at
times
used more than 35GB of memory.
We decided to shut half o
I swear I remember a thread on the ML that talked about someone having
increased memory usage on their OSDs after upgrading their MONs to Luminous
as well, but I can't seem to find it. Iirc the problem for them was
resolved when they finished the upgrade to Luminous. It might not be a bad
idea fo
On Wed, Aug 22, 2018 at 6:35 AM Adrien Gillard
wrote:
> Hi everyone,
>
> We have a hard time figuring out a behaviour encountered after upgrading
> the monitors of one of our cluster from Jewel to Luminous yesterday.
>
> The cluster is composed of 14 OSDs hosts (2xE5-2640 v3 and 64 GB of RAM),
>
On 22.08.2018 20:57, David Turner wrote:
> does it remove any functionality of the previous dashboard?
No it doesn't. All dashboard_v1 features are integrate and part of the
dashboard_v2 as well.
--
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284
(AG Nürnberg)
On Wed, Aug 22, 2018 at 6:46 AM Eugen Block wrote:
> Hello *,
>
> we have an issue with a Luminous cluster (all 12.2.5, except one on
> 12.2.7) for RBD (OpenStack) and CephFS. This is the osd tree:
>
> host1:~ # ceph osd tree
> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
> -1
On Wed, Aug 22, 2018 at 2:46 AM John Spray wrote:
> On Wed, Aug 22, 2018 at 7:57 AM mj wrote:
> >
> > Hi,
> >
> > This morning I woke up, seeing my ceph jewel 10.2.10 cluster in
> > HEALTH_ERR state. That helps you getting out of bed. :-)
> >
> > Anyway, much to my surprise, all VMs running on
On Wed, Aug 22, 2018 at 12:56 AM Konstantin Shalygin wrote:
> > Hi everyone,
> >
> > I read an earlier thread [1] that made a good explanation on the 'step
> > choose|chooseleaf' option. Could someone further help me to understand
> > the 'firstn|indep' part? Also, what is the relationship betwee
This is quite strange. Given that you have a log, I think what you want to
do is find one request in the log, trace it through its lifetime, and see
where the time is elapsed. You may find a bifurcation, where some
categories of requests happen instantly but other categories take a second
or more;
So, I've finally journeyed deeper into the depths of ceph and discovered a
grand mistake that is likely the root cause of many woeful nights of blocked
requests. To start off, I'm running jewel, and I know that is dated and I need
to upgrade (if anyone knows if this is a seamless upgrade even th
We 'paused' the cluster early in our investigation to avoid unnecessary IO.
We also set the nodown flag but the OOM rate was really sustained and we
got servers that stopped responding from time to time, so we decided to
lower the number of OSDs up and let them peer.
I don't know if it is the best
Did I say Jewel? I was too hopeful. I meant hammer. This particular cluster is
hammer :(
-Russ
From: ceph-users on behalf of Russell
Holloway
Sent: Wednesday, August 22, 2018 8:49:19 PM
To: ceph-users@lists.ceph.com
Subject: [ceph-users] Ceph RGW Index Shardi
On Wed, Aug 22, 2018 at 6:02 PM Adrien Gillard
wrote:
> We 'paused' the cluster early in our investigation to avoid unnecessary
> IO.
> We also set the nodown flag but the OOM rate was really sustained and we
> got servers that stopped responding from time to time, so we decided to
> lower the nu
Hi, I've been fighting to get good stability on my cluster for about
3 weeks now. I am running into intermittent issues with OSD flapping
marking other OSD down then going back to a stable state for hours and
days.
The cluster is 4x Cisco UCS S3260 with dual E5-2660, 256GB ram, 40G
Network to 4
Hello,
On Wed, 22 Aug 2018 23:00:24 -0400 Tyler Bishop wrote:
> Hi, I've been fighting to get good stability on my cluster for about
> 3 weeks now. I am running into intermittent issues with OSD flapping
> marking other OSD down then going back to a stable state for hours and
> days.
>
> The
During high load testing I'm only seeing user and sys cpu load around
60%... my load doesn't seem to be anything crazy on the host and iowait
stays between 6 and 10%. I have very good `ceph osd perf` numbers too.
I am using 10.2.11 Jewel.
On Wed, Aug 22, 2018 at 11:30 PM Christian Balzer wrote
The release notes for 0.94.10 mention the introduction of the `radosgw-admin
bucket reshard` command. Redhat [1] documentation for their Enterprise
version of Jewel goes into detail for the procedure. You can also search
the ML archives for the command to find several conversations about the
proces
Greg, thanks for your reply.
So, this is actually just noisy logging from the client processing an
OSDMap. That should probably be turned down, as it's not really an
indicator of...anything...as far as I can tell.
I usually stick with the defaults:
host4:~ # ceph daemon osd.21 config show | g
45 matches
Mail list logo
