Re: [ceph-users] Mimic offline problem

Quick update here: The problem with the OSDs that are throwing rocksdb errors (missing SST files) is that ceph-kvstore-tool bluestore-kv ... repair was run on OSDs, and it looks like the rocksdb repair function actually broke the (non-broken) rocksdb instance. I'm not quite sure why that is th

Re: [ceph-users] Mimic offline problem

This is ceph-object-store tool logs for OSD.0. https://paste.ubuntu.com/p/jNwf4DC46H/ There is something wrong. But we are not sure if we cant use the tool or there is something wrong with OSD. > On 4 Oct 2018, at 06:17, Sage Weil wrote: > > On Thu, 4 Oct 2018, Goktug Yildirim wrote: >> This

Re: [ceph-users] Mimic offline problem

On Thu, 4 Oct 2018, Goktug Yildirim wrote: > This is our cluster state right now. I can reach rbd list and thats good! > Thanks a lot Sage!!! > ceph -s: https://paste.ubuntu.com/p/xBNPr6rJg2/ Progress! Not out of the woods yet, though... > As you can see we have 2 unfound pg since some of our O

Re: [ceph-users] Mimic offline problem

This is our cluster state right now. I can reach rbd list and thats good! Thanks a lot Sage!!! ceph -s: https://paste.ubuntu.com/p/xBNPr6rJg2/ As you can see we have 2 unfound pg since some of our OSDs can not start. 58 OSD gives different errors. How can I fix these OSD's? If I remember correct

Re: [ceph-users] Mimic offline problem

Also you was asking the RAW output. I've been trying to fix it for days and I didn't sleep. Forgive the dumb mistakes. RAW dump output: https://drive.google.com/file/d/1SzFNNjSK9Q_j4iyYJTRqOYuLWJcsFX9C/view?usp=sharing Göktuğ Yıldırım şunları yazdı (3 Eki 2018 21:34): > I'm so sorry about tha

Re: [ceph-users] Mimic offline problem

On Wed, 3 Oct 2018, Göktuğ Yıldırım wrote: > I'm so sorry about that I missed "out" parameter. My bad.. > This is the output: https://paste.ubuntu.com/p/KwT9c8F6TF/ Excellent, thanks. That looks like it confirms the problem is that teh recovery tool didn't repopulate the creating pgs properly.

Re: [ceph-users] Mimic offline problem

I'm so sorry about that I missed "out" parameter. My bad.. This is the output: https://paste.ubuntu.com/p/KwT9c8F6TF/ Sage Weil şunları yazdı (3 Eki 2018 21:13): > I bet the kvstore output it in a hexdump format? There is another option to > get the raw data iirc > > > >> On October 3, 201

Re: [ceph-users] Mimic offline problem

I bet the kvstore output it in a hexdump format? There is another option to get the raw data iirc On October 3, 2018 3:01:41 PM EDT, Goktug YILDIRIM wrote: >I changed the file name to make it clear. >When I use your command with "+decode" I'm getting an error like this: > >ceph-dencoder type

Re: [ceph-users] Mimic offline problem

I changed the file name to make it clear. When I use your command with "+decode" I'm getting an error like this: ceph-dencoder type creating_pgs_t import DUMPFILE decode dump_json error: buffer::malformed_input: void creating_pgs_t::decode(ceph::buffer::list::iterator&) no longer understand old e

Re: [ceph-users] Mimic offline problem

On Wed, 3 Oct 2018, Göktuğ Yıldırım wrote: > If I didn't do it wrong, I got the output as below. > > ceph-kvstore-tool rocksdb /var/lib/ceph/mon/ceph-SRV-SBKUARK14/store.db/ get > osd_pg_creating creating > dump > 2018-10-03 20:08:52.070 7f07f5659b80 1 rocksdb: do_open column families: > [defau

Re: [ceph-users] Mimic offline problem

On Wed, 3 Oct 2018, Goktug Yildirim wrote: > We are starting to work on it. First step is getting the structure out and > dumping the current value as you say. > > And you were correct we did not run force_create_pg. Great. So, eager to see what the current structure is... please attach once yo

Re: [ceph-users] Mimic offline problem

We are starting to work on it. First step is getting the structure out and dumping the current value as you say. And you were correct we did not run force_create_pg. > On 3 Oct 2018, at 17:52, Sage Weil wrote: > > On Wed, 3 Oct 2018, Goktug Yildirim wrote: >> Sage, >> >> Pool 66 is the only p

Re: [ceph-users] Mimic offline problem

On Wed, 3 Oct 2018, Goktug Yildirim wrote: > Sage, > > Pool 66 is the only pool it shows right now. This a pool created months ago. > ceph osd lspools > 66 mypool > > As we recreated mon db from OSDs, the pools for MDS was unusable. So we > deleted them. > After we create another cephfs fs and p

Re: [ceph-users] Mimic offline problem

Sage, Pool 66 is the only pool it shows right now. This a pool created months ago. ceph osd lspools 66 mypool As we recreated mon db from OSDs, the pools for MDS was unusable. So we deleted them. After we create another cephfs fs and pools we started MDS and it stucked on creation. So we stoppe

Re: [ceph-users] Mimic offline problem

Oh... I think this is the problem: 2018-10-03 16:37:04.284 7efef2ae0700 20 slow op osd_pg_create(e72883 66.af:60196 66.ba:60196 66.be:60196 66.d8:60196 66.f8:60196 66.124:60196 66.14c:60196 66.1ac:60196 66.223:60196 66.248:60196 66.271:60196 66.2d1:60196 66.47a:68641) initiated 2018-10-03 16:20

Re: [ceph-users] Mimic offline problem

Hello, It seems nothing has changed. OSD config: https://paste.ubuntu.com/p/MtvTr5HYW4/ OSD debug log: https://paste.ubuntu.com/p/7Sx64xGzkR/ > On 3 Oct 2018, at 14:27, Darius Kasparavičius wrote: > > Hello, >

Re: [ceph-users] Mimic offline problem

Hello, You can also reduce the osd map updates by adding this to your ceph config file. "osd crush update on start = false". This should remove and update that is generated when osd starts. 2018-10-03 14:03:21.534 7fe15eddb700 0 mon.SRV-SBKUARK14@0(leader) e14 handle_command mon_command({"prefi

Re: [ceph-users] Mimic offline problem

Hi Sage, Thank you for your response. Now I am sure this incident is going to be resolved. The problem started when 7 server crashed same time and they came back after ~5 minutes. Two of our 3 mon services were restarted in this crash. Since mon services are enabled they should be started ne

Re: [ceph-users] Mimic offline problem

osd_find_best_info_ignore_history_les is a dangerous option and you should only use it in very specific circumstances when directed by a developer. In such cases it will allow a stuck PG to peer. But you're not getting to that point...you're seeing some sort of resource exhaustion. The noup t

Re: [ceph-users] Mimic offline problem

Anyone heart about osd_find_best_info_ignore_history_les = true ? Is that be usefull here? There is such a less information about it. Goktug Yildirim şunları yazdı (2 Eki 2018 22:11): > Hi, > > Indeed I left ceph-disk to decide the wal and db partitions when I read > somewhere that that will d

Re: [ceph-users] Mimic offline problem

Hi, Indeed I left ceph-disk to decide the wal and db partitions when I read somewhere that that will do the proper sizing. For the blustore cache size I have plenty of RAM. I will increase 8GB for each and decide a more calculated numberafter cluster settles. For the osd map loading I’ve a

Re: [ceph-users] Mimic offline problem

Thanks for the reply! My answers are inline. > On 2 Oct 2018, at 21:51, Paul Emmerich wrote: > > (Didn't follow the whole story, so you might have already answered that) > Did you check what the OSDs are doing during the period of high disk > utilization? > As in: > > * running perf top Did not

Re: [ceph-users] Mimic offline problem

Hi, I can see some issues from the osd log file. You have an extremely low size db and wal partitions. Only 1GB for DB and 576MB for wal. I would recommend cranking up rocksdb cache size as much as possible. If you have RAM you can also increase bluestores cache size for hdd. Default is 1GB be as

Re: [ceph-users] Mimic offline problem

(Didn't follow the whole story, so you might have already answered that) Did you check what the OSDs are doing during the period of high disk utilization? As in: * running perf top * sampling a few stack traces from procfs or gdb * or just high log settings * running "status" on the admin socket l

Re: [ceph-users] Mimic offline problem

Hello Darius, Thanks for reply! The main problem is we can not query PGs. “ceph pg 67.54f query” does stucks and wait forever since OSD is unresponsive. We are certain that OSD gets unresponsive as soon as it UP. And we are certain that OSD responds again after its disk utilization stops. So

Re: [ceph-users] Mimic offline problem

Hello, Currently you have 15 objects missing. I would recommend finding them and making backups of them. Ditch all other osds that are failing to start and concentrate on bringing online those that have missing objects. Then slowly turn off nodown and noout on the cluster and see if it stabilises

Re: [ceph-users] Mimic offline problem

One of ceph experts indicated that bluestore is somewhat preview tech (as for Redhat). So it could be best to checkout bluestore and rocksdb. There are some tools to check health and also repair. But there are limited documentation. Anyone who has experince with it? Anyone lead/help to a proper che

[ceph-users] Mimic offline problem

I mistyped the user list mail address. I am correcting and sending again. Apologies for the noise. My mail is below. İleti başlangıcı: > Kimden: Goktug Yildirim > Tarih: 1 Ekim 2018 21:54:31 GMT+2 > Kime: ceph-users-j...@lists.ceph.com > Bilgi: ceph-de...@vger.kernel.org > Konu: Mimic offline