[ceph-users] Re: Question about per MDS journals

osdc/Journaler is for RDB, client/Journaler is for CephFS. On Thu, Feb 25, 2021 at 8:26 AM 조규진 wrote: > > Hi, John. > > Thanks for your kind reply! > > While i'm checking the code that you recommend to check and other .cc files > about journal, I find that there is two Journaler class. > One is

[ceph-users] Re: ceph slow at 80% full, mds nodes lots of unused memory

On 24/02/2021 22:28, Patrick Donnelly wrote: > Hello Simon, > > On Wed, Feb 24, 2021 at 7:43 AM Simon Oosthoek > wrote: >> >> On 24/02/2021 12:40, Simon Oosthoek wrote: >>> Hi >>> >>> we've been running our Ceph cluster for nearly 2 years now (Nautilus) >>> and recently, due to a temporary situa

[ceph-users] Re: ceph slow at 80% full, mds nodes lots of unused memory

Simon Oosthoek wrote: > On 24/02/2021 22:28, Patrick Donnelly wrote: > > Hello Simon, > > > > On Wed, Feb 24, 2021 at 7:43 AM Simon Oosthoek > > wrote: > > > > On 24/02/2021 12:40, Simon Oosthoek wrote: > > Hi > > > > we've been running our Ceph cluster for near

[ceph-users] Re: [Suspicious newsletter] Re: List number of buckets owned per user

Maybe this one? radosgw-admin bucket list --uid= And after linux commands, grep, wc -l. Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com --- -Ori

[ceph-users] Re: ceph slow at 80% full, mds nodes lots of unused memory

On 25/02/2021 11:19, Dylan McCulloch wrote: > Simon Oosthoek wrote: >> On 24/02/2021 22:28, Patrick Donnelly wrote: >> >   Hello Simon, >> >   >> >  On Wed, Feb 24, 2021 at 7:43 AM Simon Oosthoek > wrote: >> >   >> >  On 24/02/2021 12:40, Simon Oosthoek wrote: >> >   Hi

[ceph-users] Erasure coded calculation

Hello everyone! I'm trying to calculate the theoretical usable storage of a ceph cluster with erasure coded pools. I have 8 nodes and the profile for all data pools will be k=6 m=2. If every node has 6 x 1TB wouldn't the calculation be like this: RAW capacity: 8Nodes x 6Disks x 1TB = 48TB Loss t

[ceph-users] Re: Newbie Requesting Help - Please, This Is Driving Me Mad/Crazy!

> The line in the steps I did is: OK, missed that one. Should be the right location. The logs I'm referring to are the ordinary stdout/stderr of the osd process. Just start the daemon in foreground by hand if this output is not available otherwise. Simplest form is: /usr/bin/ceph-osd -f -i

[ceph-users] Re: Newbie Requesting Help - Please, This Is Driving Me Mad/Crazy!

I think it is this here: https://docs.ceph.com/en/latest/install/manual-deployment/#long-form . As far as I can tell, this is only for educational purposes and not intended for real deployment. It is not a manual deployment for lvm OSDs though, so yeah, this should be updated. I think the inte

[ceph-users] Re: Erasure coded calculation

109 is 81.75 yes the rest of them some bluestore stuffs I guess. Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com --- -Original Message- From:

[ceph-users] Re: February 2021 Tech Talk and Code Walk-through

Just a reminder, Sage is live now, giving an update on the Pacific Release. This will be recorded and posted later to the Ceph Youtube channel. https://ceph.io/ceph-tech-talks/ On Tue, Feb 16, 2021 at 11:14 PM Mike Perez wrote: > > Hi everyone! > > I'm excited to announce two talks we have on th

[ceph-users] RGW: Multiple Site does not sync olds data

Hi all: ceph version: 15.2.7 (88e41c6c49beb18add4fdb6b4326ca466d931db8) I have a strange question, I just create a multiple site for Ceph cluster. But I notice the old data of source cluster is not synced. Only new data will be synced into second zone cluster. Is there anything I need to do to e

[ceph-users] Re: Question about per MDS journals

On Thu, Feb 25, 2021 at 09:59:41AM +, John Spray wrote: > osdc/Journaler is for RDB, client/Journaler is for CephFS. Actually, src/journal/(Journaler.h) is for RBD (it is more generic, but currently is used by RBD only). And src/osdc/Journaler.h is for cephfs. > > > On Thu, Feb 25, 2021 at

[ceph-users] Re: Erasure coded calculation

On Thu, Feb 25, 2021 at 10:55:05AM +, Simon Sutter wrote: > The output of ceph df detail is: > --- RAW STORAGE --- > CLASS SIZE AVAILUSED RAW USED %RAW USED > hdd109 TiB 103 TiB 5.8 TiB 5.9 TiB 5.41 > TOTAL 109 TiB 103 TiB 5.8 TiB 5.9 TiB 5.41 > > --- PO

[ceph-users] MON slow ops and growing MON store

Hi, All of a sudden, we are experiencing very concerning MON behaviour. We have five MONs and all of them have thousands up to tens of thousands of slow ops, the oldest one blocking basically indefinitely (at least the timer keeps creeping up). Additionally, the MON stores keep inflating heavil

[ceph-users] Re: MON slow ops and growing MON store

ceph daemon mon.`hostname -s` ops That should show you the accumulating ops. .. dan On Thu, Feb 25, 2021, 8:23 PM Janek Bevendorff < janek.bevendo...@uni-weimar.de> wrote: > Hi, > > All of a sudden, we are experiencing very concerning MON behaviour. We > have five MONs and all of them have tho

[ceph-users] MDSs report damaged metadata

Hi all, My cephfs MDS is reporting damaged metadata following the addition (and remapping) of 12 new OSDs. `ceph tell mds.database-0 damage ls` reports ~85 files damaged. All of type "backtrace" which is very concerning. ` ceph tell mds.database-0 scrub start / recursive repair` seems to have

[ceph-users] Re: MON slow ops and growing MON store

Thanks, Dan. On the first MON, the command doesn’t even return, but I was able to get a dump from the one I restarted most recently. The oldest ops look like this: { "description": "log(1000 entries from seq 17876238 at 2021-02-25T15:13:20.306487+0100)", "initiat

[ceph-users] Re: MDSs report damaged metadata

Hello Ricardo, On Thu, Feb 25, 2021 at 11:51 AM wrote: > > Hi all, > > > > My cephfs MDS is reporting damaged metadata following the addition (and > remapping) of 12 new OSDs. > `ceph tell mds.database-0 damage ls` reports ~85 files damaged. All of type > "backtrace" which is very concerning. It

[ceph-users] Re: MON slow ops and growing MON store

> "source": "osd.104... What's happening on that osd? Is it something new which corresponds to when your mon started growing? Are other OSDs also flooding the mons with logs? I'm mobile so can't check... Are those logging configs the defaults? If not revert to default... BTW do your mons ha

[ceph-users] Re: MON slow ops and growing MON store

Nothing special is going on that OSD as far as I can tell and the OSD number of each op is different. The config isn’t entirely default, but we have been using it successfully for quite a bit. It basically just redirects everything to journald so that we don’t have log creep. I reverted it nonet

[ceph-users] Re: MON slow ops and growing MON store

I had the same problem in my cluster and it was because of insights mgr module that was storing lots of data to the RocksDB because mu cluster was degraded. If you have degraded pgs try to disable insights module. On Thu, Feb 25, 2021 at 11:40 PM Dan van der Ster wrote: > > "source": "osd.104...

[ceph-users] Re: MON slow ops and growing MON store

Thanks for the tip, but I do not have degraded PGs and the module is already disabled. > On 25. Feb 2021, at 21:17, Seena Fallah wrote: > > I had the same problem in my cluster and it was because of insights mgr > module that was storing lots of data to the RocksDB because mu cluster was > d

[ceph-users] Re: MON slow ops and growing MON store

Maybe the debugging steps in that insights tracker can be helpful anyway: https://tracker.ceph.com/issues/39955 -- dan On Thu, Feb 25, 2021 at 9:27 PM Janek Bevendorff wrote: > > Thanks for the tip, but I do not have degraded PGs and the module is already > disabled. > > > On 25. Feb 2021, at 2

[ceph-users] Re: MON slow ops and growing MON store

Thanks, I’ll try that tomorrow. > On 25. Feb 2021, at 21:59, Dan van der Ster wrote: > > Maybe the debugging steps in that insights tracker can be helpful > anyway: https://tracker.ceph.com/issues/39955 > > -- dan > > On Thu, Feb 25, 2021 at 9:27 PM Janek Bevendorff > wrote: >> >> Thanks fo

[ceph-users] Re: MON slow ops and growing MON store

Also did you solve your log spam issue here? https://tracker.ceph.com/issues/49161 Surely these things are related? You might need to share more full logs from cluster, mon, osd, mds, mgr so that we can help get to the bottom of this. -- dan On Thu, Feb 25, 2021 at 10:04 PM Janek Bevendorff wro

[ceph-users] Re: MON slow ops and growing MON store

> On 25. Feb 2021, at 22:17, Dan van der Ster wrote: > > Also did you solve your log spam issue here? > https://tracker.ceph.com/issues/49161 > Surely these things are related? No. But I noticed that DBG log spam only happens when log_to_syslog is enabled. systemd is smart enough to avoid fi

[ceph-users] Re: Question about per MDS journals

Quite right, I flipped them while writing the mail - oops. John On Thu, Feb 25, 2021 at 6:04 PM Mykola Golub wrote: > > On Thu, Feb 25, 2021 at 09:59:41AM +, John Spray wrote: > > osdc/Journaler is for RDB, client/Journaler is for CephFS. > > Actually, src/journal/(Journaler.h) is for RBD (i

[ceph-users] Slow cluster / misplaced objects - Ceph 15.2.9

Hi, We've got an interesting issue we're running into on Ceph 15.2.9. We're experiencing VERY slow performance from the cluster, and extremely slow misplaced object correction, with very little cpu/disk/network utilization (almost idle) across all nodes in the cluster. We have 7 servers in this c

[ceph-users] Re: [Suspicious newsletter] RGW: Multiple Site does not sync olds data

Same for me, 15.2.8 also. I’m trying directional sync now, looks like symmetrical has issue. Istvan Szabo Senior Infrastructure Engineer --- Agoda Services Co., Ltd. e: istvan.sz...@agoda.com ---

[ceph-users] Newbie Requesting Help - Please, This Is Driving Me Mad/Crazy! - A Follow Up

Hi Everyone, Thanks to all for both the online and PM help - once it was pointed out that the existing (Octopus) Documentation was... less than current I ended up using the ceph-volume command. A couple of follow-up questions: When using ceph-volume lvm create: 1. Can you specify an osd num

[ceph-users] Newbie Help With ceph-mgr

Hi All, My ceph-mgr keeps stopping (for some unknown reason) after about an hour or so (but has run for up to 2-3 hours before stopping). Up till now I've simple restarted it with 'ceph-mgr -i ceph01'. Is this normal behaviour, or if it isn't, what should I be looking for in the logs? I wa