Hello,
I have encountered a strange I/O freeze while rebooting one OSD node
for maintenance purpose. It was one of the 3 Nodes in the entire
cluster. Before this rebooting or shutting down and entire node just
slowed down the ceph, but not completely froze it.
Can you please share the ceph -s
> o/p?
>
> Thanks
> Swami
>
> On Tue, Sep 13, 2016 at 3:54 PM, Daznis wrote:
>>
>> Hello,
>>
>>
>> I have encountered a strange I/O freeze while rebooting one OSD node
>> for maintenance purpose. It was one of
it?
>
> Have a look at the down PG's '18 pgs down' - I suspect this will be what is
> causing the I/O freeze.
>
> Is your cursh map setup correctly to split data over different hosts?
>
> Thanks
>
> On Tue, Sep 13, 2016 at 11:45 AM, Daznis wrote:
>>
&g
Hello,
The story goes like this.
I have added another 3 drives to the caching layer. OSDs were added to
crush map one by one after each successful rebalance. When I added the
last OSD and went away for about an hour I noticed that it's still not
finished rebalancing. Further investigation showed
ng or fix the issue.
On Wed, Nov 23, 2016 at 12:04 PM, Nick Fisk wrote:
> Hi Daznis,
>
> I'm not sure how much help I can be, but I will try my best.
>
> I think the post-split stats error is probably benign, although I think this
> suggests you also increased the number
OSD::init()+0x181a) [0x6c0e8a]
5: (main()+0x29dd) [0x6484bd]
6: (__libc_start_main()+0xf5) [0x7f36b916bb15]
7: /usr/bin/ceph-osd() [0x661ea9]
On Wed, Nov 23, 2016 at 12:31 PM, Nick Fisk wrote:
>> -Original Message-
>> From: Daznis [mailto:daz...@gmail.com]
>> S
e cluster have all PG's recovered now? Unless anyone else can comment,
> you might be best removing/wiping and then re-adding the OSD.
>
>> -Original Message-
>> From: Daznis [mailto:daz...@gmail.com]
>> Sent: 23 November 2016 12:55
>> To: Nick Fisk
>
involved in recovering objects with the object
> store tool.
>
>> -Original Message-
>> From: Daznis [mailto:daz...@gmail.com]
>> Sent: 23 November 2016 13:56
>> To: Nick Fisk
>> Cc: ceph-users
>> Subject: Re: [ceph-users] Ceph strange issu
gt; -Original Message-
>> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
>> Daznis
>> Sent: 24 November 2016 15:43
>> To: Nick Fisk
>> Cc: ceph-users
>> Subject: Re: [ceph-users] Ceph strange issue after adding a cache OSD.
>
he cause, but whether this on its own would cause the problem
> or also needs the introduction of new OSD's at the same time, might make
> tracing the cause hard.
>
>> -Original Message-
>> From: Daznis [mailto:daz...@gmail.com]
>> Sent: 24 November 2016
Hello,
I'm running preliminary test on cache tier removal on a live cluster,
before I try to do that on a production one. I'm trying to avoid
downtime, but from what I noticed it's either impossible or I'm doing
something wrong. My cluster is running Centos 7.2 and 0.94.9 ceph.
Example 1:
I'm s
ts you originally posted.
>
>> -----Original Message-
>> From: Daznis [mailto:daz...@gmail.com]
>> Sent: 25 November 2016 13:59
>> To: Nick Fisk
>> Cc: ceph-users
>> Subject: Re: [ceph-users] Ceph strange issue after adding a cache OSD.
>>
>
Hi,
Recently about ~2 weeks ago something strange started happening with
one of the ceph cluster I'm managing. It's running ceph jewel 10.2.10
with cache layer. Some OSD's started crashing with "too many open
files error". From looking at the issue I have found that it keeps a
lot of links in /pro
Hello,
Yesterday I have encountered a strange osd crash which led to cluster
flapping. I had to force nodown flag on the cluster to finish the
flapping. The first osd that crashed with:
2018-08-02 17:23:23.275417 7f87ec8d7700 1 heartbeat_map is_healthy
'OSD::osd_op_tp thread 0x7f8803dfb700' had
Hello,
I remember watching one of the ceph monthly videos on youtube and
there was a talk that pg_num would be available in mimic, but I can't
find any info about it? Was this feature delayed?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http:/
Hi,
We used a PXE boot with NFS server, but had some issues if NFS server
crapped out and dropped connections or needed a reboot for
maintenance. If I remember it correctly it sometimes took out some of
the rebooted servers. So we switched to PXE with livecd based images.
You basically create a li
Hello,
I have replace one of our failed OSD drives and recreated a new osd
with ceph-deploy and it failes to start.
Command: ceph-deploy --overwrite-conf osd create --filestore
--zap-disk --data /dev/bcache0 --journal /dev/nvme0n1p13
Output off ceph-deploy:
[ceph_deploy.conf][DEBUG ] found con
Hello,
It appears that something is horribly wrong with the cluster itself. I
can't create or add any new osds to it at all.
On Mon, Aug 20, 2018 at 11:04 AM Daznis wrote:
>
> Hello,
>
>
> Zapping the journal didn't help. I tried to create the journal after
> zapp
=
-- managers --
mon03
mon02
mon01
osds
node03
node02
node01
node05
node04
mons
mon01
mon03
mon02
107 passed, on 11 hosts
On Mon, Aug 20, 2018 at 6
Thanks for the all the help. For some bizarre reason I had an empty
host inside default root. Once I dumped a "fake" osd into it
everything started working.
On Mon, Aug 20, 2018 at 7:36 PM Daznis wrote:
>
> Hello,
>
>
> Medic shows everything fine. Whole cluster is on t
Hello,
On Tue, Jan 10, 2017 at 11:11 PM, Nick Fisk wrote:
>> -Original Message-
>> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
>> Daznis
>> Sent: 09 January 2017 12:54
>> To: ceph-users
>> Subject: [ceph-users]
21 matches
Mail list logo
