Hi,
I am currently evaluating ceph and stumbled across an odd issue when an osd
comes back online.
The osd was taken offline, but is still "in" and is brought back online before
it is marked "out".
As a test I run a fio job with 4k rand I/O on a 10G rbd volume during the OSD
down and up proced
Hi all,
the rados_stat() function has a TODO in the comments:
* TODO: when are these set, and by whom? can they be out of date?
Can anyone help with this? How reliably is the pmtime updated? Is there a
minimum update interval?
Thank you,
Peter
__
Am 02.12.20 um 15:04 schrieb Seena Fallah:
> I don't think so! I want to slow down the recovery not speed up and it says
> I should reduce these values.
I read the documentation the same. Low value = low weight, High value = high
weight. [1]
Operations with higher weight get easier dispatched.
Am 01.12.20 um 17:32 schrieb Peter Lieven:
> Hi all,
>
>
> the rados_stat() function has a TODO in the comments:
>
>
> * TODO: when are these set, and by whom? can they be out of date?
>
> Can anyone help with this? How reliably is the pmtime updated? Is there a
Hi,
we currently run into an issue where a rbd ls for a namespace returns ENOENT
for some of the images in that namespace.
/usr/bin/rbd --conf=XXX --id XXX ls
'mypool/28ef9470-76eb-4f77-bc1b-99077764ff7c' -l --format=json
2021-06-09 11:03:34.916 7f2225ffb700 -1 librbd::io::AioCompletion:
0x5
Am 09.06.21 um 13:28 schrieb Ilya Dryomov:
> On Wed, Jun 9, 2021 at 11:24 AM Peter Lieven wrote:
>> Hi,
>>
>>
>> we currently run into an issue where a rbd ls for a namespace returns ENOENT
>> for some of the images in that namespace.
>>
>>
>&
Am 10.06.21 um 11:08 schrieb Manuel Lausch:
> Hi,
>
> has no one a idea what could cause this issue. Or how I could debug it?
>
> In some days I have to go live with this cluster. If I don't have a
> solution I have to go live with nautilus.
Hi Manuel,
I had similar issues with Octopus and i a
Am 09.06.21 um 13:52 schrieb Ilya Dryomov:
> On Wed, Jun 9, 2021 at 1:36 PM Peter Lieven wrote:
>> Am 09.06.21 um 13:28 schrieb Ilya Dryomov:
>>> On Wed, Jun 9, 2021 at 11:24 AM Peter Lieven wrote:
>>>> Hi,
>>>>
>>>>
>>>> we
Am 10.06.21 um 17:45 schrieb Manuel Lausch:
> Hi Peter,
>
> your suggestion pointed me to the right spot.
> I didn't know about the feature, that ceph will read from replica
> PGs.
>
> So on. I found two functions in the osd/PrimaryLogPG.cc:
> "check_laggy" and "check_laggy_requeue". On both is fi
Am 11.06.21 um 11:48 schrieb Dan van der Ster:
> On Fri, Jun 11, 2021 at 11:08 AM Peter Lieven wrote:
>> Am 10.06.21 um 17:45 schrieb Manuel Lausch:
>>> Hi Peter,
>>>
>>> your suggestion pointed me to the right spot.
>>> I didn't know about the
Have you tried setting
osd op queue cut off to high?
Peter
> Am 11.08.2021 um 15:24 schrieb Frank Schilder :
>
> The recovery_sleep options are the next choice to look at. Increase it and
> clients will get more I/O time slots. However, with your settings, I'm
> surprised clients are impac
Am 12.08.21 um 17:25 schrieb Frank Schilder:
Wow, that is impressive and sounds opposite of what we see around
here. Often rebalances directly and strongly impact client I/O.
It might be the missing settings:
osd_op_queue = wpq
osd_op_queue_cut_off = high
Afaik, the default for osd_op_queue_
Hi Ceph users,
one of our backup jobs ran into the following assertion. Is this something
known?
/build/ceph-14.2.21/src/msg/async/Event.cc: In function 'int
EventCenter::create_file_event(int, int, EventCallbackRef)' thread 7f3c5d893700
time 2021-08-19 01:29:23.981240
/build/ceph-14.2.21/s
Am 23.08.21 um 00:53 schrieb Kai Börnert:
As far as i understand, more important factor (for the ssds) is if they have
power loss protections (so they can use their ondevice write cache) and how
many iops they have when using direct writes with queue depth 1
I just did a test for a hdd with bl
Am 26.09.21 um 19:08 schrieb Alexandre Marangone:
> Thanks for the feedback Alex! If you have any issue or ideas for
> improvements please do submit them on the GH repo:
> https://github.com/digitalocean/pgremapper/
>
> Last Thursday I did a Ceph at DO tech talk, I talked about how we use
> pgremap
Am 27.09.21 um 22:38 schrieb Josh Baergen:
>> I have a question regarding the last step. It seems to me that the ceph
>> balancer is not able to remove the upmaps
>> created by pgremapper, but instead creates new upmaps to balance the pgs
>> among osds.
> The balancer will prefer to remove existi
Am 01.10.21 um 16:52 schrieb Josh Baergen:
Hi Peter,
When I check for circles I found that running the upmap balancer alone never
seems to create
any kind of circle in the graph
By a circle, do you mean something like this?
pg 1.a: 1->2 (upmap to put a chunk on 2 instead of 1)
pg 1.b: 2->3
pg
Am 24.06.22 um 16:13 schrieb Peter Lieven:
Am 23.06.22 um 12:59 schrieb Ilya Dryomov:
On Thu, Jun 23, 2022 at 11:32 AM Peter Lieven wrote:
Am 22.06.22 um 15:46 schrieb Josh Baergen:
Hey Peter,
I found relatively large allocations in the qemu smaps and checked the
contents. It contained
Am 19.07.22 um 17:57 schrieb Ilya Dryomov:
On Tue, Jul 19, 2022 at 5:10 PM Peter Lieven wrote:
Am 24.06.22 um 16:13 schrieb Peter Lieven:
Am 23.06.22 um 12:59 schrieb Ilya Dryomov:
On Thu, Jun 23, 2022 at 11:32 AM Peter Lieven wrote:
Am 22.06.22 um 15:46 schrieb Josh Baergen:
Hey Peter
Am 21.07.22 um 17:50 schrieb Ilya Dryomov:
> On Thu, Jul 21, 2022 at 11:42 AM Peter Lieven wrote:
>> Am 19.07.22 um 17:57 schrieb Ilya Dryomov:
>>> On Tue, Jul 19, 2022 at 5:10 PM Peter Lieven wrote:
>>>> Am 24.06.22 um 16:13 schrieb Peter Lieven:
>>>>&
Am 21.07.22 um 17:50 schrieb Ilya Dryomov:
On Thu, Jul 21, 2022 at 11:42 AM Peter Lieven wrote:
Am 19.07.22 um 17:57 schrieb Ilya Dryomov:
On Tue, Jul 19, 2022 at 5:10 PM Peter Lieven wrote:
Am 24.06.22 um 16:13 schrieb Peter Lieven:
Am 23.06.22 um 12:59 schrieb Ilya Dryomov:
On Thu, Jun
> Am 22.10.2021 um 11:12 schrieb Tommy Sway :
>
> Even hypervisor support is useless if ceph itself does not support it.
Thick provisioning is supported from Octopus onwards. If you are using Qemu I
can look into adding support for preprovisioning in the Qemu driver.
Peter
>
>
>
> -
> Am 22.10.2021 um 13:48 schrieb Tommy Sway :
>
> Yes, I am using qemu. And what I need to config to enable it ?
The current QEMU red driver currently does not support preprovisioning modes
for qemu-img create or expand.
>From octopus onwards there is an option for thick-provisioning. To get
Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---
-Original Message-
From: Peter Lieven
Sent: Thursday, November 26, 2020 11:37 PM
To: ceph-users@ceph.io
Subject: [ceph
Am 02.11.21 um 15:02 schrieb Sage Weil:
On Tue, Nov 2, 2021 at 8:29 AM Manuel Lausch wrote:
Hi Sage,
The "osd_fast_shutdown" is set to "false"
As we upgraded to luminous I also had blocked IO issuses with this
enabled.
Some weeks ago I tried out the options "osd_fast_shutdown" and
"osd_fast_
Am 04.11.21 um 23:51 schrieb Sage Weil:
> Can you try setting paxos_propose_interval to a smaller number, like .3 (by
> default it is 2 seconds) and see if that has any effect.
>
> It sounds like the problem is not related to getting the OSD marked down (or
> at least that is not the only thing g
Am 08.11.21 um 23:59 schrieb Igor Fedotov:
> Hi folks,
>
> having a LTS release cycle could be a great topic for upcoming "Ceph User +
> Dev Monthly meeting".
>
> The first one is scheduled on November 18, 2021, 14:00-15:00 UTC
>
> https://pad.ceph.com/p/ceph-user-dev-monthly-minutes
>
> Any volu
Am 10.11.21 um 09:57 schrieb Manuel Lausch:
> Hi Sage,
>
>
> thank you for your help.
>
>
> My origin issue with slow ops on osd restarts are gone too. Even with default
> values for paxos_proposal_interval.
>
>
> Its a bit annoying, that I spent many hours to debug this and finally I
> missed on
Am 10.11.21 um 11:35 schrieb Manuel Lausch:
> oh shit,
>
> I patched in a switch to deactivate the read_lease feature. This is only a
> hack to test a bit around. But accidentally I had this switch enabled for my
> last tests done here in this mail-thread.
>
> The bad news. The require_osd_releas
> Am 09.11.2021 um 00:01 schrieb Igor Fedotov :
>
> Hi folks,
>
> having a LTS release cycle could be a great topic for upcoming "Ceph User +
> Dev Monthly meeting".
>
> The first one is scheduled on November 18, 2021, 14:00-15:00 UTC
>
> https://pad.ceph.com/p/ceph-user-dev-monthly-minute
Am 17.11.21 um 12:20 schrieb Igor Fedotov:
> Hi Peter,
>
> sure, why not...
See [1]. I read it that it is not wanted by upstream developers. If we want it
the community has to do it.
Nevertheless, I have put it on the list.
Peter
[1]
https://lists.ceph.io/hyperkitty/list/d...@ceph.io/thr
Am 17.11.21 um 18:09 schrieb Janne Johansson:
>> * I personally wouldn't want to run an LTS release based on ... what would
>> that be now.. Luminous + security patches??. IMO, the new releases really
>> are much more performant, much more scalable. N, O, and P are really much
>> much *much* better
Am 17.11.21 um 20:14 schrieb Marc:
a good choice. It lacks RBD encryption and read leases. But for us
upgrading from N to O or P is currently not
what about just using osd encryption with N?
That would be Data at Rest encryption only. The keys for the OSDs are stored on
the mons. Data is tr
Am 22.11.21 um 16:25 schrieb Luke Hall:
> On 22/11/2021 15:18, mj wrote:
>> Hi,
>>
>> We were in the same position as you, and replaced our 24 4TB harddisks with
>> Samsung PM883 .
>>
>> They seem to work quite nicely, and their wearout (after one year) is still
>> at 1% for our use.
>
> Thanks,
Am 22.12.21 um 16:39 schrieb J-P Methot:
> So, from what I understand from this, neither the latest nautilus client nor
> the latest octopus client has the fix? Only the latest pacific?
Are you sure that this issue is reproducible in Nautilus?
I tried it with a Nautilus 14.2.22 client and it wo
Am 13.01.22 um 08:37 schrieb Szabo, Istvan (Agoda):
> Hi,
>
> I can see a lot of message regarding the rotating key, but not sure this is
> the root cause.
>
> 2022-01-13 03:21:57.156 7fe7e085e700 -1 monclient: _check_auth_rotating
> possible clock skew, rotating keys expired way too early (befor
a00700 0 client.0 ms_handle_reset on
> v2:10.121.58.222:6800/1141825
> 2022-01-13 14:16:08.119 7f28b8a00700 0 client.0 ms_handle_reset on
> v2:10.121.58.222:6800/1141825
> 2022-01-13 14:31:08.125 7f28b8a00700 0 client.0 ms_handle_reset on
> v2:10.121.58.222:6800/1141825
> 2022
Am 13.01.22 um 09:19 schrieb Szabo, Istvan (Agoda):
> But in your case the election is successful to the other mgr, am I correct?
> So the dash always up for you? Not sure for me why not, maybe I need to
> disable it really :/
I can't tell you why its not failing over. But if you don't track th
Am 13.01.22 um 09:19 schrieb Szabo, Istvan (Agoda):
But in your case the election is successful to the other mgr, am I correct? So
the dash always up for you? Not sure for me why not, maybe I need to disable it
really :/
Has disabling the prometheus module prevented further crashes?
Best,
Hi,
we noticed that some of our long running VMs (1 year without migration) seem to
have a very slow memory leak. Taking a dump of the leaked memory revealed that
it seemed to contain osd and pool information so we
concluded that it must have something to do with crush map updates. We then
wr
Von meinem iPhone gesendet
> Am 22.06.2022 um 10:35 schrieb Ilya Dryomov :
>
> On Tue, Jun 21, 2022 at 8:52 PM Peter Lieven wrote:
>>
>> Hi,
>>
>>
>> we noticed that some of our long running VMs (1 year without migration) seem
>> to have
> Am 22.06.2022 um 12:52 schrieb Janne Johansson :
>
>
>>
>> I found relatively large allocations in the qemu smaps and checked the
>> contents. It contained several hundred repetitions of osd and pool names. We
>> use the default builds on Ubuntu 20.04. Is there a special memory allocator
> Am 22.06.2022 um 14:28 schrieb Ilya Dryomov :
>
> On Wed, Jun 22, 2022 at 11:14 AM Peter Lieven wrote:
>>
>>
>>
>> Von meinem iPhone gesendet
>>
>>>> Am 22.06.2022 um 10:35 schrieb Ilya Dryomov :
>>>
>>&g
Am 22.06.22 um 15:46 schrieb Josh Baergen:
Hey Peter,
I found relatively large allocations in the qemu smaps and checked the
contents. It contained several hundred repetitions of osd and pool names. We
use the default builds on Ubuntu 20.04. Is there a special memory allocator in
place that
Am 23.06.22 um 12:59 schrieb Ilya Dryomov:
On Thu, Jun 23, 2022 at 11:32 AM Peter Lieven wrote:
Am 22.06.22 um 15:46 schrieb Josh Baergen:
Hey Peter,
I found relatively large allocations in the qemu smaps and checked the
contents. It contained several hundred repetitions of osd and pool
Am 23.06.22 um 12:59 schrieb Ilya Dryomov:
> On Thu, Jun 23, 2022 at 11:32 AM Peter Lieven wrote:
>> Am 22.06.22 um 15:46 schrieb Josh Baergen:
>>> Hey Peter,
>>>
>>>> I found relatively large allocations in the qemu smaps and checked the
>>>>
46 matches
Mail list logo
