Re: [ceph-users] Mimic offline problem

2018-10-04 Thread Goktug Yildirim
This is ceph-object-store tool logs for OSD.0. https://paste.ubuntu.com/p/jNwf4DC46H/ There is something wrong. But we are not sure if we cant use the tool or there is something wrong with OSD. > On 4 Oct 2018, at 06:17, Sage Weil wrote: > > On Thu, 4 Oct 2018, Goktug Yildi

Re: [ceph-users] Mimic offline problem

2018-10-03 Thread Goktug Yildirim
then inject it on the 3 mons, that should get you past this > hump (and hopefully back up!). > > sage > > >> >> Sage Weil şunları yazdı (3 Eki 2018 21:13): >> >>> I bet the kvstore output it in a hexdump format? There is another option >>> to ge

Re: [ceph-users] Mimic offline problem

2018-10-03 Thread Goktug YILDIRIM
an find the "dump" link below. > > > > dump: > https://drive.google.com/file/d/1ZLUiQyotQ4-778wM9UNWK_TLDAROg0yN/view?usp=sharing > > > > > > Sage Weil şunları yazdı (3 Eki 2018 18:45): > > > > >> On Wed, 3 Oct 2018, Goktug Yildirim wrote:

Re: [ceph-users] Mimic offline problem

2018-10-03 Thread Goktug Yildirim
We are starting to work on it. First step is getting the structure out and dumping the current value as you say. And you were correct we did not run force_create_pg. > On 3 Oct 2018, at 17:52, Sage Weil wrote: > > On Wed, 3 Oct 2018, Goktug Yildirim wrote: >> Sage, >> &

Re: [ceph-users] Mimic offline problem

2018-10-03 Thread Goktug Yildirim
lete > it? If yes, I'm ~70% sure that will then get cleaned out at the mon end > and restarting the OSDs will make these pg_creates go away. > > s > > On Wed, 3 Oct 2018, Goktug Yildirim wrote: > >> Hello, >> >> It seems nothing has changed. >>

Re: [ceph-users] Mimic offline problem

2018-10-03 Thread Goktug Yildirim
: > from='osd.47 10.10.112.17:6803/64652' entity='osd.47' cmd=[{"prefix": > "osd crush create-or-move", "id": 47, "weight":3.6396, "args": > ["host=SRV-SEKUARK8", "root=default"]}]: dispatch

Re: [ceph-users] Mimic offline problem

2018-10-03 Thread Goktug Yildirim
ha > t the mons have. Were they really caught up when you unset noup? > > I'm just catching up and haven't read the whole thread but I haven't seen > anything that explains why teh OSDs are dong lots of disk IO. Catching up > on maps could explain it but not why they

Re: [ceph-users] Help! OSDs across the cluster just crashed

2018-10-02 Thread Goktug Yildirim
Hi, Sorry to hear that. I’ve been battling with mine for 2 weeks :/ I’ve corrected mine OSDs with the following commands. My OSD logs (/var/log/ceph/ceph-OSDx.log) has a line including log(EER) with the PG number besides and before crash dump. ceph-objectstore-tool --data-path /var/lib/ceph/os

Re: [ceph-users] Mimic offline problem

2018-10-02 Thread Goktug Yildirim
> have lots of osd map loading and decoding in the log. Are you sure all > monitors/managers/osds are up to date? Plus make sure you aren't > forcing jemalloc loading. I had a funny interaction after upgrading to > mimic. > On Tue, Oct 2, 2018 at 9:02 PM Goktug Yildirim >

Re: [ceph-users] Mimic offline problem

2018-10-02 Thread Goktug Yildirim
lly I can run daemon and see status. I must have checked it but will do again. > > > Paul > > Am Di., 2. Okt. 2018 um 20:02 Uhr schrieb Goktug Yildirim > : >> >> Hello Darius, >> >> Thanks for reply! >> >> The main problem is we can not query PGs

Re: [ceph-users] Mimic offline problem

2018-10-02 Thread Goktug Yildirim
epair. But there are limited >> documentation. >> Anyone who has experince with it? >> Anyone lead/help to a proper check would be great. >> Goktug Yildirim , 1 Eki 2018 Pzt, 22:55 >> tarihinde şunu yazdı: >>> >>> Hi all, >>> >>>