Hello, We have two ceph luminous clusters (12.2.5).
recently one of our big buckets stopped syncing properly. We have a one specific bucket which is around 30TB in size consisting of alot of directories with each one having files of 10-20MB. The secondary zone is often completly missing multiple days of data in this bucket, while all other smaller buckets sync just fine. Even with the complete data missing radosgw-admin sync status always says everything is fine. the sync error log doesnt show anything for those days. Running radosgw-admin metadata sync and data sync also doesnt solve the issue. The only way of making it sync again is to disable and re-eanble the sync. That needs to be done as often as like 10 times in an hour to make it sync properly. radosgw-admin bucket sync disable radosgw-admin bucket sync enable when i run data init i sometimes get this: radosgw-admin data sync init --source-zone berlin 2018-06-24 07:55:46.337858 7fe7557fa700 0 ERROR: failed to distribute cache for amsterdam.rgw.log:datalog.sync-status.6a9448d2-bdba-4bec-aad6-aba72cd8eac6 Sometimes when really alot of data is missing (yesterday it was more then 1 month) this helps making them get in sync again when run on the secondary zone: radosgw-admin bucket check --fix --check-objects how can i debug that problem further? We have so many requests on the cluster that is is hard to dig something out of the log files.. Given all the smaller buckets are perfectly in sync i suspect some problem because of the size of the bucket. Any points to the right direction are greatly appreciated. Regards, Enrico -- *Enrico Kern* VP IT Operations enrico.k...@glispa.com +49 (0) 30 555713017 / +49 (0) 152 26814501 skype: flyersa LinkedIn Profile <https://www.linkedin.com/in/enricokern> <http://goog_59398030/> <https://www.glispa.com/> *Glispa GmbH* | Berlin Office Sonnenburger Straße 73 <https://maps.google.com/?q=Sonnenburger+Stra%C3%9Fe+73+10437+Berlin%C2%A0%7C+%3Chttps://maps.google.com/?q%3DSonnenburgerstra%25C3%259Fe%2B73%2B10437%2BBerlin%25C2%25A0%257C%25C2%25A0Germany%26entry%3Dgmail%26source%3Dg%3E%C2%A0Germany&entry=gmail&source=g> 10437 Berlin <https://maps.google.com/?q=Sonnenburger+Stra%C3%9Fe+73+10437+Berlin%C2%A0%7C+%3Chttps://maps.google.com/?q%3DSonnenburgerstra%25C3%259Fe%2B73%2B10437%2BBerlin%25C2%25A0%257C%25C2%25A0Germany%26entry%3Dgmail%26source%3Dg%3E%C2%A0Germany&entry=gmail&source=g> | <https://maps.google.com/?q=Sonnenburgerstra%C3%9Fe+73+10437+Berlin%C2%A0%7C%C2%A0Germany&entry=gmail&source=g> Germany <https://maps.google.com/?q=Sonnenburger+Stra%C3%9Fe+73+10437+Berlin%C2%A0%7C+%3Chttps://maps.google.com/?q%3DSonnenburgerstra%25C3%259Fe%2B73%2B10437%2BBerlin%25C2%25A0%257C%25C2%25A0Germany%26entry%3Dgmail%26source%3Dg%3E%C2%A0Germany&entry=gmail&source=g> Managing Director Din Karol-Gavish Registered in Berlin AG Charlottenburg | <https://maps.google.com/?q=Sonnenburger+Stra%C3%9Fe+73+10437+Berlin%C2%A0%7C+%3Chttps://maps.google.com/?q%3DSonnenburgerstra%25C3%259Fe%2B73%2B10437%2BBerlin%25C2%25A0%257C%25C2%25A0Germany%26entry%3Dgmail%26source%3Dg%3E%C2%A0Germany&entry=gmail&source=g> HRB 114678B –––––––––––––––––––––––––––––
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
