Hello,
until Bluestore gets caching that is
a) self-tuning (within definable limits) so that a busy OSD can consume
more cache than ones that are idle AND
b) the cache will be as readily evicted as pagecache in low memory
situations
you're essential SoL, having the bad choices of increasing pe
Your claim that all cache is used for K/V cache is false (with default
settings).
K/V cache is maximized to 500Meg. :
bluestore_cache_kv_max
Description:The maximum amount of cache devoted to key/value data
(rocksdb).
Type: Unsigned Integer
Required: Yes
Default:512 *
Hello all!
I had a electric power problem. After this I have 2 incomplete pg. But
all RBD volumes are work.
But not work my CephFS. MDS load stop at "replay" state and MDS related
commands hangs:
cephfs-journal-tool journal export backup.bin - freeze;
cephfs-journal-tool event recover_dent
Also,
I tried to add a client admin socket, but I am getting this message:
[client]
admin socket = /var/run/ceph/$name.$pid.asok
2018-08-30 20:04:03.337 7f4641ffb700 -1 set_mon_vals failed to set
admin_socket = $run_dir/$cluster-$name.asok: Configuration option
'admin_socket' may not be modif
I keep getting the following error message:
018-08-30 18:52:37.882 7fca9df7c700 -1 asok(0x7fca98000fe0)
AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed
to bind the UNIX domain socket to
'/var/run/ceph/ceph-client.admin.asok': (17) File exists
Otherwise things seem to
Be very careful trying to utilize more RAM while your cluster is healthy.
When you're going to need the extra RAM is when you're closer is unhealthy
and your osds are peering, recovering, backfilling, etc. That's when the
osd daemons start needing the RAM that is recommended in the docs.
On Thu, A
Hi,
My OSD host has 256GB of ram and I have 52 OSD. Currently I have the cache
set to 1GB and the system only consumes around 44GB of ram and the other
ram sits as unallocated because I am using bluestore vs filestore.
The documentation:
http://docs.ceph.com/docs/master/rados/configuration/blues
This moved to the PG map in luminous. I think it might have been there in
Jewel as well.
http://docs.ceph.com/docs/luminous/man/8/ceph/#pg
ceph pg set_full_ratio
ceph pg set_backfillfull_ratio
ceph pg set_nearfull_ratio
On Thu, Aug 30, 2018, 1:57 PM David C wrote:
> Hi All
>
> I feel like t
Hi All
I feel like this is going to be a silly query with a hopefully simple
answer. I don't seem to have the osd_backfill_full_ratio config option on
my OSDs and can't inject it. This a Lumimous 12.2.1 cluster that was
upgraded from Jewel.
I added an OSD to the cluster and woke up the next day t
LOn Thu, Aug 30, 2018 at 12:46 PM William Lawton
wrote:
Oh i see. We’d taken steps to reduce the risk of losing the active mds and
> mon leader instances at the same time in the hope that it would prevent
> this issue. Do you know if the mds always connects to a specific mon
> instance i.e. the m
Oh i see. We’d taken steps to reduce the risk of losing the active mds and mon
leader instances at the same time in the hope that it would prevent this issue.
Do you know if the mds always connects to a specific mon instance i.e. the mon
provider and can it be determined which mon instance that
Hi Eugen.
Ok, edited the file /etc/salt/minion, uncommented the "log_level_logfile"
line and set it to "debug" level.
Turned off the computer, waited a few minutes so that the time frame would
stand out in the /var/log/messages file, and restarted the computer.
Using vi I "greped out" (awful wor
Okay, well that will be the same reason then. If the active MDS is
connected to a monitor and they fail at the same time, the monitors can’t
replace the mds until they’ve been through their own election and a full
mds timeout window.
On Thu, Aug 30, 2018 at 11:46 AM William Lawton
wrote:
> Thanks
Hi!
During our move from filestore to bluestore, we removed several Intel P3700
NVMe from the nodes.
Is someone running a SPDK/DPDK NVMe-only EC pool? Is it working well?
The docs are very short about the setup:
http://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#spdk-usage
The Hammer ticket was https://tracker.ceph.com/issues/13990. The problem
here was when OSDs asked each other for which map they needed to keep and a
leak would set it to NULL then that OSD would never delete an OSD map again
until it was restarted.
On Thu, Aug 30, 2018 at 3:09 AM Joao Eduardo Lui
Thanks for the response Greg. We did originally have co-located mds and mon but
realised this wasn't a good idea early on and separated them out onto different
hosts. So our mds hosts are on ceph-01 and ceph-02, and our mon hosts are on
ceph-03, 04 and 05. Unfortunately we see this issue occurri
I have 2 OSDs failing to start due to this [1] segfault. What is happening
matches what Sage said about this [2] bug. The OSDs are NVMe disks and
rocksdb is compacting omaps. I attempted setting `bluestore_bluefs_min_free
= 10737418240` and then start the OSDs, but they both segfaulted with the
Thanks for the thought. It’s mounted with this entry in fstab (one line, if
email wraps it):
cephmon-s01,cephmon-s02,cephmon-s03:/ /loam ceph
noauto,name=clientname,secretfile=/etc/ceph/secret,noatime,_netdev 0 2
Pretty plain, but I'm open to tweaking!
peter
Peter Eisch
v
I'm glad you asked this, because it was on my to-do list. I know that based
on our not existing in the bucket marker does not mean it's safe to
delete. I have an index pool with 22k objects in it. 70 objects match
existing bucket markers. I was having a problem on the cluster and started
deleting
On Thu, Aug 30, 2018 at 5:24 AM, Wolfgang Lendl
wrote:
> Hi Alfredo,
>
>
> caught some logs:
> https://pastebin.com/b3URiA7p
That looks like there is an issue with bluestore. Maybe Radoslaw or
Adam might know a bit more.
>
> br
> wolfgang
>
> On 2018-08-29 15:51, Alfredo Deza wrote:
>> On Wed,
How are you mounting CephFS? It may be that the cache settings are just set
very badly for a 10G pipe. Plus rados bench is a very parallel large-IO
benchmark and many benchmarks you might dump into a filesystem are
definitely not.
-Greg
On Thu, Aug 30, 2018 at 7:54 AM Peter Eisch
wrote:
> Hi,
>
Yes, this is a consequence of co-locating the MDS and monitors — if the MDS
reports to its co-located monitor and both fail, the monitor cluster has to
go through its own failure detection and then wait for a full MDS timeout
period after that before it marks the MDS down. :(
We might conceivably
Hi.
We have a 5 node Ceph cluster (refer to ceph -s output at bottom of email).
During resiliency tests we have an occasional problem when we reboot the active
MDS instance and a MON instance together i.e. dub-sitv-ceph-02 and
dub-sitv-ceph-04. We expect the MDS to failover to the standby inst
Hi,
I have a cluster serving cephfs and it works. It’s just slow. Client is using
the kernel driver. I can ‘rados bench’ writes to the cephfs_data pool at wire
speeds (9580Mb/s on a 10G link) but when I copy data into cephfs it is rare to
get above 100Mb/s. Large file writes may start fast
Correct, except it doesn't have to be a specific host or a specific
OSD. What matters here is whether the client is idle. As soon as the
client is woken up and sends a request to _any_ OSD, it receives a new
osdmap and applies it, possibly emitting those dmesg entries.
Thanks for the clarifica
On Thu, Aug 30, 2018 at 1:04 PM Eugen Block wrote:
>
> Hi again,
>
> we still didn't figure out the reason for the flapping, but I wanted
> to get back on the dmesg entries.
> They just reflect what happened in the past, they're no indicator to
> predict anything.
The kernel client is just that,
Replying to self...
On Wed, Aug 1, 2018 at 11:56 AM Dan van der Ster wrote:
>
> Dear rgw friends,
>
> Somehow we have more than 20 million objects in our
> default.rgw.buckets.index pool.
> They are probably leftover from this issue we had last year:
> http://lists.ceph.com/pipermail/ceph-users-c
How is it going with this? Are we getting close to a state where we can
store a mailbox on ceph with this librmb?
-Original Message-
From: Wido den Hollander [mailto:w...@42on.com]
Sent: maandag 25 september 2017 9:20
To: Gregory Farnum; Danny Al-Gaaf
Cc: ceph-users
Subject: Re: [ce
Hi Alfredo,
caught some logs:
https://pastebin.com/b3URiA7p
br
wolfgang
On 2018-08-29 15:51, Alfredo Deza wrote:
> On Wed, Aug 29, 2018 at 2:06 AM, Wolfgang Lendl
> wrote:
>> Hi,
>>
>> after upgrading my ceph clusters from 12.2.5 to 12.2.7 I'm experiencing
>> random crashes from SSD OSDs (bl
On 08/30/2018 09:28 AM, Dan van der Ster wrote:
> Hi,
>
> Is anyone else seeing rocksdb mon stores slowly growing to >15GB,
> eventually triggering the 'mon is using a lot of disk space' warning?
>
> Since upgrading to luminous, we've seen this happen at least twice.
> Each time, we restart all t
Hi all,
I just finished setting up a new Ceph cluster (Luminous 12.2.7, 3xMON
nodes and 6xOSDs nodes, BlueStore OSD on sata hdd with WAL/DB on
separated NVMe devices, 2x10 Gbs network per node, 3 replicas by pool)
I created a CephFS pool : data pool uses hdd OSDs and metadata pool uses
dedic
Hi again,
we still didn't figure out the reason for the flapping, but I wanted
to get back on the dmesg entries.
They just reflect what happened in the past, they're no indicator to
predict anything.
For example, when I changed the primary-affinity of OSD.24 last week,
one of the clients
Hi,
Is anyone else seeing rocksdb mon stores slowly growing to >15GB,
eventually triggering the 'mon is using a lot of disk space' warning?
Since upgrading to luminous, we've seen this happen at least twice.
Each time, we restart all the mons and then stores slowly trim down to
<500MB. We have 'm
Welcome Mike! You're the perfect person for this role!
--
Tomasz Kuzemko
tomasz.kuze...@corp.ovh.com
Od: ceph-users w imieniu użytkownika Sage
Weil
Wysłane: środa, 29 sierpnia 2018 03:13
Do: ceph-de...@vger.kernel.org; ceph-users@lists.ceph.com;
ceph-c
34 matches
Mail list logo
