[ceph-users] Re: 16.2.11 branch

[Christian Rohmann]

Hey Laura, Greg, all,

On 31/10/2022 17:15, Gregory Farnum wrote:

If you don't mind me asking Laura, have those issues regarding the

testing lab been resolved yet?


There are currently a lot of folks working to fix the testing lab issues.
Essentially, disk corruption affected our ability to reach quay.ceph.io.
We've made progress this morning, but we are still working to understand
the root cause of the corruption. We expect to re-deploy affected services
soon so we can resume testing for v16.2.11.

We got a note about this today, so I wanted to clarify:

For Reasons, the sepia lab we run teuthology in currently uses a Red
Hat Enterprise Virtualization stack — meaning, mostly KVM with a lot
of fancy orchestration all packaged up, backed by Gluster. (Yes,
really — a full Ceph integration was never built and at one point this
was deemed the most straightforward solution compared to running
all-up OpenStack backed by Ceph, which would have been the available
alternative.) The disk images stored in Gluster started reporting
corruption last week (though Gluster was claiming to be healthy), and
with David's departure and his backup on vacation it took a while for
the remaining team members to figure out what was going on and
identify strategies to resolve or work around it.

The relevant people have figured out a lot more of what was going on,
and Adam (David's backup) is back now so we're expecting things to
resolve more quickly at this point. And indeed the team's looking at
other options for providing this infrastructure going forward. 😄
-Greg



May I kindly ask for an update on how things are progressing? Mostly I 
am interested on the (persisting) implications for testing new point 
releases (e.g. 16.2.11) with more and more bugfixes in them.



Thanks a bunch!


Christian
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph-volume inventory reports available devices as unavailable

[Stefan Kooman]

On 12/14/22 15:18, Eugen Block wrote:

Hi,

I haven't been dealing with ceph-volume too much lately, but I remember 
seeing that when I have multiple DB devices on SSD and wanted to replace 
only one failed drive. Although ceph-volume inventory reported the disk 
as unavailable the actual create command was successful. But I don't 
remember which versions were okay and which weren't, there were multiple 
regressions in ceph-volume IIRC, it seems to be a very complex 
structure. But apparently '... batch --report' is more reliable than 
'... inventory'.


I can confirm this behaviour for 16.2.10. The disk is not seen as 
available, but cephadm will recreate the OSD on the LVM according to the 
spec (and that in turn invokes ceph-volume with the batch lvm command).


Gr. Stefan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 16.2.11 branch

[Christian Rohmann]

On 15/12/2022 10:31, Christian Rohmann wrote:


May I kindly ask for an update on how things are progressing? Mostly I 
am interested on the (persisting) implications for testing new point 
releases (e.g. 16.2.11) with more and more bugfixes in them.


I guess I just have not looked on the right ML, it's being worke on 
already ... 
https://lists.ceph.io/hyperkitty/list/d...@ceph.io/thread/CQPQJXD6OVTZUH43I4U3GGOP2PKYOREJ/




Sorry for the nagging,


Christian

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Cephadm recreating osd with multiple block devices

[Ali Akil]

Hallo folks,

i am encountering a weird behavior from Ceph when i try to remove an OSD 
to replace it with an encrypted one. Where the OSD is being directly 
recreated with an additional block device after removal.
So, the idea is to remove an OSD which was created without encryption 
enabled and re-create another encrapted one.


My idea was to process this way

1- I am setting noout first on the osd :
`ceph osd add-noout osd.9`

2- removing the osd from crush with replace flag in-order to retain the 
id 22 :

`ceph orch osd rm 9 --replace --force`

3- checking if it's safe to destroy:
`ceph osd safe-to-destroy osd.9`

4- zap the lvm partitions
`cephadm shell ceph-volume lvm zap --destroy --osd-id 9`

5- re-apply the osd service spec with encryption enabled

6- unset noout

But after removing the osd in step 2 the OSD is being direktly recreated 
and when i list logical volumes and devices i can see that the osd now 
has two block devices

```
== osd.9 ===

  [block] 
/dev/ceph-0074651e-25fa-462d-af82-44ea7e0c7866/osd-block-fe12826f-f3e4-4106-b323-1e880471b243


  block device 
/dev/ceph-0074651e-25fa-462d-af82-44ea7e0c7866/osd-block-fe12826f-f3e4-4106-b323-1e880471b243

  block uuid 9UBLAv-giHf-pGUy-Ir5f-XQSe-nIjE-X7vkTP
  cephx lockbox secret AQDie5hj/QS/IBAA/Fpb9TfqZJSiRUG7haxRUw==
  cluster fsid 42c6ceac-d549-11ec-9db5-b549f63e669c
  cluster name  ceph
  crush device class
  encrypted 1
  osd fsid fe12826f-f3e4-4106-b323-1e880471b243
  osd id    9
  osdspec affinity  osd_spec_nvme
  type  block
  vdo   0
  devices   /dev/sde

  [block] 
/dev/ceph-38f39e5e-3cb2-4a38-8cea-84a91bb5b755/osd-block-adf2c0a6-3912-4eea-b094-7a39f010b25d


  block device 
/dev/ceph-38f39e5e-3cb2-4a38-8cea-84a91bb5b755/osd-block-adf2c0a6-3912-4eea-b094-7a39f010b25d

  block uuid A4tFzY-jxbM-gHvd-eTfI-14Lh-YTK8-Pm1YEA
  cephx lockbox secret
  cluster fsid 42c6ceac-d549-11ec-9db5-b549f63e669c
  cluster name  ceph
  crush device class
  db device 
/dev/ceph-ee8df5e7-ee7a-4ee7-acd9-a8fef79893b5/osd-db-39ddd23c-94bb-4c50-a326-0798265fb696

  db uuid DlWk5x-EQQD-EZlP-GH56-T7Ea-3Y1x-Lqkr2x
  encrypted 0
  osd fsid adf2c0a6-3912-4eea-b094-7a39f010b25d
  osd id    9
  osdspec affinity  osd_spec_nvme
  type  block
  vdo   0
  devices   /dev/sdi


  [db] 
/dev/ceph-ee8df5e7-ee7a-4ee7-acd9-a8fef79893b5/osd-db-39ddd23c-94bb-4c50-a326-0798265fb696


  block device 
/dev/ceph-38f39e5e-3cb2-4a38-8cea-84a91bb5b755/osd-block-adf2c0a6-3912-4eea-b094-7a39f010b25d

  block uuid A4tFzY-jxbM-gHvd-eTfI-14Lh-YTK8-Pm1YEA
  cephx lockbox secret
  cluster fsid 42c6ceac-d549-11ec-9db5-b549f63e669c
  cluster name  ceph
  crush device class
  db device 
/dev/ceph-ee8df5e7-ee7a-4ee7-acd9-a8fef79893b5/osd-db-39ddd23c-94bb-4c50-a326-0798265fb696

  db uuid DlWk5x-EQQD-EZlP-GH56-T7Ea-3Y1x-Lqkr2x
  encrypted 0
  osd fsid adf2c0a6-3912-4eea-b094-7a39f010b25d
  osd id    9
  osdspec affinity  osd_spec_nvme
  type  db
  vdo   0
  devices   /dev/nvme0n1
```

I am unable to explain this behavior! How can i stop cephadm from 
recreating the osd. I thought that setting the noout would be sufficient.


I am running Ceph quincy version 17.2.0

Best regards,
Ali Akil

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: SLOW_OPS

[Murilo Morais]

Eugen, thanks for answering.

I understand that there is not enough memory, but I managed to recover a
lot of the memory that was in use.
Right now I can't upgrade to 48, but it's already planned.

After yesterday's episode I managed to recover a lot of the memory that was
in use.

Until then everything was normal, but at this very moment it happened
again, but without high disk traffic and not much RAM in use. This leads me
to believe that the problem would not be due to lack of memory, as there is
a lot of free memory at the moment.

Em qua., 14 de dez. de 2022 às 13:34, Eugen Block  escreveu:

> With 12 OSDs and a default of 4 GB RAM per OSD you would at least
> require 48 GB, usually a little more. Even if you reduced the memory
> target per OSD it doesn’t mean they can deal with the workload. There
> was a thread explaining that a couple of weeks ago.
>
> Zitat von Murilo Morais :
>
> > Good morning everyone.
> >
> > Guys, today my cluster had a "problem", it was showing SLOW_OPS, when
> > restarting the OSDs that were showing this problem everything was solved
> > (there were VMs stuck because of this), what I'm breaking my head is to
> > know the reason for having SLOW_OPS.
> >
> > In the logs I saw that the problem started at 04:00 AM and continued
> until
> > 07:50 AM (when I restarted the OSDs).
> >
> > I'm suspicious of some exaggerated settings that I applied and forgot
> there
> > in the initial setup while performing a test, which may have caused a
> high
> > use of RAM leaving a maximum of 400 MB of 32 GB free memory, which in
> this
> > case was to put 512 PGs in two pools, one of which was affected.
> >
> > In the logs I saw that the problem started when some VMs started to
> perform
> > backup actions, increasing the writing a little (to a maximum of 300
> MBps),
> > after a few seconds a disk started to show this WARN and also this line:
> > Dec 14 04:01:01 dcs1.evocorp ceph-mon[639148]: 69 slow requests (by type
> [
> > 'delayed' : 65 'waiting for sub ops' : 4 ] most affected pool [
> > 'cephfs.ds_disk.data' : 69])
> >
> > Then he presented these:
> > Dec 14 04:01:02 dcs1.evocorp ceph-mon[639148]: log_channel(cluster) log
> > [WRN] : Health check update: 0 slow ops, oldest one blocked for 36 sec,
> > daemons [osd.20,osd.5 ] have slow ops. (SLOW_OPS)
> > [...]
> > Dec 14 05:52:01 dcs1.evocorp ceph-mon[639148]: log_channel(cluster) log
> > [WRN] : Health check update: 149 slow ops, oldest one blocked for 6696
> sec,
> > daemons [osd.20,osd.5 ,osd.50] have slow ops. (SLOW_OPS)
> >
> > I've already checked the SMART, they're all OK, I've checked the graphs
> > generated in Grafana and none of the disks saturate, there haven't been
> any
> > incidents related to the network, that is, I haven't identified any other
> > problem that could cause this.
> >
> > What could have caused this event? What can I do to prevent it from
> > happening again?
> >
> > Below is some information about the cluster:
> > 5 machines with 32GB RAM, 2 processors and 12 3TB SAS disks and connected
> > through 40Gb interfaces.
> >
> > # ceph osd tree
> > ID   CLASS  WEIGHT TYPE NAME   STATUS  REWEIGHT  PRI-AFF
> >  -1 163.73932  root default
> >  -3  32.74786  host dcs1
> >   0hdd2.72899  osd.0   up   1.0  1.0
> >   1hdd2.72899  osd.1   up   1.0  1.0
> >   2hdd2.72899  osd.2   up   1.0  1.0
> >   3hdd2.72899  osd.3   up   1.0  1.0
> >   4hdd2.72899  osd.4   up   1.0  1.0
> >   5hdd2.72899  osd.5   up   1.0  1.0
> >   6hdd2.72899  osd.6   up   1.0  1.0
> >   7hdd2.72899  osd.7   up   1.0  1.0
> >   8hdd2.72899  osd.8   up   1.0  1.0
> >   9hdd2.72899  osd.9   up   1.0  1.0
> >  10hdd2.72899  osd.10  up   1.0  1.0
> >  11hdd2.72899  osd.11  up   1.0  1.0
> >  -5  32.74786  host dcs2
> >  12hdd2.72899  osd.12  up   1.0  1.0
> >  13hdd2.72899  osd.13  up   1.0  1.0
> >  14hdd2.72899  osd.14  up   1.0  1.0
> >  15hdd2.72899  osd.15  up   1.0  1.0
> >  16hdd2.72899  osd.16  up   1.0  1.0
> >  17hdd2.72899  osd.17  up   1.0  1.0
> >  18hdd2.72899  osd.18  up   1.0  1.0
> >  19hdd2.72899  osd.19  up   1.0  1.0
> >  20hdd2.72899  osd.20  up   1.0  1.0
> >  21hdd2.72899  osd.21  up   1.0  1.0
> >  22hdd2.72899  osd.22  up   1.0  1.0
> >  23hdd2.72899  osd.23  up   1.0  1.0
> >  -7  32.74786  host dcs3
> >  24hdd  

[ceph-users] Re: mds stuck in standby, not one active

[Mevludin Blazevic]

Hi,

while upgrading to ceph pacific 6.2.7, the upgrade process stuck exactly 
at the mds daemons. Before, I have tried to increase/shrink the 
placement size of them, but nothing happens. Currently I have 4/3 
running daemons. One daemon should be stopped and removed.


Do you suggest to force remove these daemons or what could be the 
preferred workaround?


Regards,

Mevludin

Am 13.12.2022 um 20:32 schrieb Patrick Donnelly:

On Tue, Dec 13, 2022 at 2:21 PM Mevludin Blazevic
 wrote:

Hi,

thanks for the quick response!

CEPH STATUS:

cluster:
  id: 8c774934-1535-11ec-973e-525400130e4f
  health: HEALTH_ERR
  7 failed cephadm daemon(s)
  There are daemons running an older version of ceph
  1 filesystem is degraded
  1 filesystem has a failed mds daemon
  1 filesystem is offline
  1 filesystem is online with fewer MDS than max_mds
  23 daemons have recently crashed

services:
  mon: 2 daemons, quorum cephadm-vm,store2 (age 12d)
  mgr: store1.uevcpd(active, since 34m), standbys: cephadm-vm.zwagng
  mds: 0/1 daemons up (1 failed), 4 standby
  osd: 324 osds: 318 up (since 3h), 318 in (since 2h)

data:
  volumes: 0/1 healthy, 1 failed
  pools:   6 pools, 257 pgs
  objects: 2.61M objects, 9.8 TiB
  usage:   29 TiB used, 2.0 PiB / 2.0 PiB avail
  pgs: 257 active+clean

io:
  client:   0 B/s rd, 2.8 MiB/s wr, 435 op/s rd, 496 op/s wr

FS DUMP:

e60
enable_multiple, ever_enabled_multiple: 1,1
default compat: compat={},rocompat={},incompat={1=base v0.20,2=client
writeable ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
legacy client fscid: 1

Filesystem 'ceph_fs' (1)
fs_name ceph_fs
epoch   58
flags   32
created 2022-11-28T12:05:17.203346+
modified2022-12-13T19:03:46.707236+
tableserver 0
root0
session_timeout 60
session_autoclose   300
max_file_size   1099511627776
required_client_features{}
last_failure0
last_failure_osd_epoch  196035
compat  compat={},rocompat={},incompat={1=base v0.20,2=client writeable
ranges,3=default file layouts on dirs,4=dir inode in separate
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
anchor table,9=file layout v2,10=snaprealm v2}
max_mds 2
in  0
up  {}
failed  0
damaged
stopped
data_pools  [4]
metadata_pool   5
inline_data disabled
balancer
standby_count_wanted1


Standby daemons:

[mds.ceph_fs.store5.gnlqqm{-1:152180029} state up:standby seq 1
join_fscid=1 addr
[v2:192.168.50.135:6800/3548272808,v1:192.168.50.135:6801/3548272808]
compat {c=[1],r=[1],i=[1]}]
[mds.ceph_fs.store6.fxgvoj{:915af89} state up:standby seq 1
join_fscid=1 addr
[v2:192.168.50.136:1b70/4fde2aa0,v1:192.168.50.136:1b71/4fde2aa0] compat
{c=[1],r=[1],i=[1]}]
[mds.ceph_fs.store4.mhvpot{:916a09d} state up:standby seq 1
join_fscid=1 addr
[v2:192.168.50.134:1a90/b8b1f33c,v1:192.168.50.134:1a91/b8b1f33c] compat
{c=[1],r=[1],i=[1]}]
[mds.ceph_fs.store3.vcnwzh{:916aff7} state up:standby seq 1
join_fscid=1 addr
[v2:192.168.50.133:1a90/49cb4e4,v1:192.168.50.133:1a91/49cb4e4] compat
{c=[1],r=[1],i=[1]}]
dumped fsmap epoch 60

You're encountering a bug fixed in v16.2.7. Please upgrade to the
latest version.


--
Mevludin Blazevic, M.Sc.

University of Koblenz-Landau
Computing Centre (GHRKO)
Universitaetsstrasse 1
D-56070 Koblenz, Germany
Room A023
Tel: +49 261/287-1326

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephfs snap-mirror stalled

[Venky Shankar]

Hi Holger,

(sorry for the late reply)

On Fri, Dec 9, 2022 at 6:22 PM Holger Naundorf  wrote:
>
> As an update:
> After the third restart now the mirror-daemon is running normal again -
> only change to the restarts before was that during the restart
> dbug_client was set to 20. (First restart was after ~48h of no data
> movemnent to the receiver side and no changes, 2nd was with debug_mirror
> set to 20 and the thisrd - maybe to quickly - after only ~12h of no data
> movement with debug_mirror + _client set to 20). Currently I reset the
> debug levels back down to reduce the load on the system disks.

Do you see ESTALE related messages in mirror daemon logs (with
debug_client: 20)? We have run into a couple of instances where the
mirror daemon would be stuck on a directory entry. The workaround for
that is to find the directory path (where the daemon is stuck) and
from another client (mount), list the entries in that directory (or at
times restarting the daemon works, as in your case).

This will be fixed in the next pacific release (tracker:
https://tracker.ceph.com/issues/55935).

>
> Regards,
> Holger Naundorf
>
>
> On 07.12.22 15:53, Holger Naundorf wrote:
> > On 06.12.22 14:17, Venky Shankar wrote:
> >> On Tue, Dec 6, 2022 at 6:34 PM Holger Naundorf
> >>  wrote:
> >>>
> >>>
> >>>
> >>> On 06.12.22 09:54, Venky Shankar wrote:
>  Hi Holger,
> 
>  On Tue, Dec 6, 2022 at 1:42 PM Holger Naundorf
>   wrote:
> >
> > Hello,
> > we have set up a snap-mirror for a directory on one of our clusters -
> > running ceph version
> >
> > ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific
> > (stable)
> >
> > to get mirrorred our other cluster - running ceph version
> >
> > ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific
> > (stable)
> >
> > The initial setup went ok, when the first snapshot was created data
> > started to flow at a decent (for our HW) rate of 100-200MB/s. As the
> > directory contains  ~200TB this was expected to take some time -
> > but now
> > the process has stalled completely after ~100TB were mirrored and ~7d
> > running.
> >
> > Up to now I do not have any hints why it has stopped - I do not see
> > any
> > error messages from the cephfs-mirror daemon. Can the small version
> > mismatch be a problem?
> >
> > Any hints where to look to find out what has got stuck are welcome.
> 
>  I'd look at the mirror daemon logs for any errors to start with. You
>  might want to crank up the log level for debugging (debug
>  cephfs_mirror=20).
> 
> >>>
> >>> Even on max debug I do not see anything which looks like an error - but
> >>> as this is the first time I try to dig into any cephfs-mirror logs I
> >>> might not notice (as long as it is not red and flashing).
> >>>
> >>> The Log basically this type of sequence, repeating forever:
> >>>
> >>> (...)
> >>> cephfs::mirror::MirrorWatcher handle_notify
> >>> cephfs::mirror::Mirror update_fs_mirrors
> >>> cephfs::mirror::Mirror schedule_mirror_update_task: scheduling fs mirror
> >>> update (0x556fe3a7f130) after 2 seconds
> >>> cephfs::mirror::Watcher handle_notify: notify_id=751516198184655,
> >>> handle=93939050205568, notifier_id=25504530
> >>> cephfs::mirror::MirrorWatcher handle_notify
> >>> cephfs::mirror::PeerReplayer(19361031-928d-4366-99bd-50df70d3adf1) run:
> >>> trying to pick from 1 directories
> >>> cephfs::mirror::PeerReplayer(19361031-928d-4366-99bd-50df70d3adf1)
> >>> pick_directory
> >>> cephfs::mirror::Watcher handle_notify: notify_id=751516198184656,
> >>> handle=93939050205568, notifier_id=25504530
> >>> cephfs::mirror::MirrorWatcher handle_notify
> >>> cephfs::mirror::Mirror update_fs_mirrors
> >>> cephfs::mirror::Mirror schedule_mirror_update_task: scheduling fs mirror
> >>> update (0x556fe3a7fc70) after 2 seconds
> >>> cephfs::mirror::Watcher handle_notify: notify_id=751516198184657,
> >>> handle=93939050205568, notifier_id=25504530
> >>> cephfs::mirror::MirrorWatcher handle_notify
> >>> (...)
> >>
> >> Basically, the interesting bit is not captured since it probably
> >> happened sometime back. Could you please set the following:
> >>
> >> debug cephfs_mirror = 20
> >> debug client = 20
> >>
> >> and restart the mirror daemon? The daemon would start synchronizing
> >> again. When synchronizing stalls, please share the daemon logs. If the
> >> log is huge, you could upload them via ceph-post-file.
> >>
> > If I set debug_client to 20 'huge' is an understatement.
> >
> > I now have three huge logfiles - one pair with debug_mirror set to 20
> > capturing the restart and the point where the sync stalls again and one
> > with both mirror and client debug at 20 capturing the  restart - but as
> > this setting created ~10GB logs within 20min I reset the client logging
> > again to spare our small system disks - if these logs are needed I think
> > I will have to

[ceph-users] not all pgs not evicted after reweight

[Ali Akil]

Hallo folks,

i want to replace an OSD, so i reweight it to 0 `ceph osd reweight 
osd. 0`.

The OSD hat 24 PGs and the number went down to 7, but stuck there.

`ceph osd tree` shows:

22    hdd  0 0  0 B  0 B  0 B  0 B  
0 B  0 B 0 0    7  up  osd.22


Any idea to debug that?

Regards,
Ali

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Cephadm recreating osd with multiple block devices

[Ali Akil]

I think the issue has been described in this note 
 in 
the documentation.


On 15.12.22 11:47, Ali Akil wrote:

Hallo folks,

i am encountering a weird behavior from Ceph when i try to remove an 
OSD to replace it with an encrypted one. Where the OSD is being 
directly recreated with an additional block device after removal.
So, the idea is to remove an OSD which was created without encryption 
enabled and re-create another encrapted one.


My idea was to process this way

1- I am setting noout first on the osd :
`ceph osd add-noout osd.9`

2- removing the osd from crush with replace flag in-order to retain 
the id 22 :

`ceph orch osd rm 9 --replace --force`

3- checking if it's safe to destroy:
`ceph osd safe-to-destroy osd.9`

4- zap the lvm partitions
`cephadm shell ceph-volume lvm zap --destroy --osd-id 9`

5- re-apply the osd service spec with encryption enabled

6- unset noout

But after removing the osd in step 2 the OSD is being direktly 
recreated and when i list logical volumes and devices i can see that 
the osd now has two block devices

```
== osd.9 ===

  [block] 
/dev/ceph-0074651e-25fa-462d-af82-44ea7e0c7866/osd-block-fe12826f-f3e4-4106-b323-1e880471b243


  block device 
/dev/ceph-0074651e-25fa-462d-af82-44ea7e0c7866/osd-block-fe12826f-f3e4-4106-b323-1e880471b243

  block uuid 9UBLAv-giHf-pGUy-Ir5f-XQSe-nIjE-X7vkTP
  cephx lockbox secret AQDie5hj/QS/IBAA/Fpb9TfqZJSiRUG7haxRUw==
  cluster fsid 42c6ceac-d549-11ec-9db5-b549f63e669c
  cluster name  ceph
  crush device class
  encrypted 1
  osd fsid fe12826f-f3e4-4106-b323-1e880471b243
  osd id    9
  osdspec affinity  osd_spec_nvme
  type  block
  vdo   0
  devices   /dev/sde

  [block] 
/dev/ceph-38f39e5e-3cb2-4a38-8cea-84a91bb5b755/osd-block-adf2c0a6-3912-4eea-b094-7a39f010b25d


  block device 
/dev/ceph-38f39e5e-3cb2-4a38-8cea-84a91bb5b755/osd-block-adf2c0a6-3912-4eea-b094-7a39f010b25d

  block uuid A4tFzY-jxbM-gHvd-eTfI-14Lh-YTK8-Pm1YEA
  cephx lockbox secret
  cluster fsid 42c6ceac-d549-11ec-9db5-b549f63e669c
  cluster name  ceph
  crush device class
  db device 
/dev/ceph-ee8df5e7-ee7a-4ee7-acd9-a8fef79893b5/osd-db-39ddd23c-94bb-4c50-a326-0798265fb696

  db uuid DlWk5x-EQQD-EZlP-GH56-T7Ea-3Y1x-Lqkr2x
  encrypted 0
  osd fsid adf2c0a6-3912-4eea-b094-7a39f010b25d
  osd id    9
  osdspec affinity  osd_spec_nvme
  type  block
  vdo   0
  devices   /dev/sdi


  [db] 
/dev/ceph-ee8df5e7-ee7a-4ee7-acd9-a8fef79893b5/osd-db-39ddd23c-94bb-4c50-a326-0798265fb696


  block device 
/dev/ceph-38f39e5e-3cb2-4a38-8cea-84a91bb5b755/osd-block-adf2c0a6-3912-4eea-b094-7a39f010b25d

  block uuid A4tFzY-jxbM-gHvd-eTfI-14Lh-YTK8-Pm1YEA
  cephx lockbox secret
  cluster fsid 42c6ceac-d549-11ec-9db5-b549f63e669c
  cluster name  ceph
  crush device class
  db device 
/dev/ceph-ee8df5e7-ee7a-4ee7-acd9-a8fef79893b5/osd-db-39ddd23c-94bb-4c50-a326-0798265fb696

  db uuid DlWk5x-EQQD-EZlP-GH56-T7Ea-3Y1x-Lqkr2x
  encrypted 0
  osd fsid adf2c0a6-3912-4eea-b094-7a39f010b25d
  osd id    9
  osdspec affinity  osd_spec_nvme
  type  db
  vdo   0
  devices   /dev/nvme0n1
```

I am unable to explain this behavior! How can i stop cephadm from 
recreating the osd. I thought that setting the noout would be sufficient.


I am running Ceph quincy version 17.2.0

Best regards,
Ali Akil

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: User + Dev Monthly Meeting happening tomorrow, December 15th!

[Laura Flores]

Hi everyone,

Since there are no topics on the agenda for today, the User + Dev Monthly
Meeting is cancelled. We will reconvene next year!

- Laura Flores

On Wed, Dec 14, 2022 at 11:52 AM Laura Flores  wrote:

> Hi Ceph Users,
>
> The User + Dev Monthly Meeting is coming up tomorrow, *Thursday, December
> 15th* *@* *3:00pm UTC* (time conversions below). See meeting details at
> the bottom of this email.
>
> Please add any topics you'd like to discuss to the agenda:
> https://pad.ceph.com/p/ceph-user-dev-monthly-minutes
> 
>
> See you there,
> Laura Flores
>
> Meeting link: https://meet.jit.si/ceph-user-dev-monthly
>
> Time conversions:
> UTC:   Thursday, December 15, 15:00 UTC
> Mountain View, CA, US: Thursday, December 15,  7:00 PST
> Phoenix, AZ, US:   Thursday, December 15,  8:00 MST
> Denver, CO, US:Thursday, December 15,  8:00 MST
> Huntsville, AL, US:Thursday, December 15,  9:00 CST
> Raleigh, NC, US:   Thursday, December 15, 10:00 EST
> London, England:   Thursday, December 15, 15:00 GMT
> Paris, France: Thursday, December 15, 16:00 CET
> Helsinki, Finland: Thursday, December 15, 17:00 EET
> Tel Aviv, Israel:  Thursday, December 15, 17:00 IST
> Pune, India:   Thursday, December 15, 20:30 IST
> Brisbane, Australia:   Friday, December 16,  1:00 AEST
> Singapore, Asia:   Thursday, December 15, 23:00 +08
> Auckland, New Zealand: Friday, December 16,  4:00 NZDT
>
>
> --
>
> Laura Flores
>
> She/Her/Hers
>
> Software Engineer, Ceph Storage
>
> Red Hat Inc. 
>
> Chicago, IL
>
> lflo...@redhat.com
> M: +17087388804
> @RedHat    Red Hat
>   Red Hat
> 
> 
>
> --

Laura Flores

She/Her/Hers

Software Engineer, Ceph Storage

Red Hat Inc. 

Chicago, IL

lflo...@redhat.com
M: +17087388804
@RedHat    Red Hat
  Red Hat


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MDS crashes to damaged metadata

[Stolte, Felix]

Hi Patrick,

we used your script to repair the damaged objects on the weekend and it went 
smoothly. Thanks for your support.

We adjusted your script to scan for damaged files on a daily basis, runtime is 
about 6h. Until thursday last week, we had exactly the same 17 Files. On 
thursday at 13:05 a snapshot was created and our active mds crashed once at 
this time (snapshot was created):

2022-12-08T13:05:48.919+0100 7f440afec700 -1 
/build/ceph-16.2.10/src/mds/ScatterLock.h: In function 'void 
ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f440afec700 time 
2022-12-08T13:05:48.921223+0100
/build/ceph-16.2.10/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state 
LOCK_XLOCK || state LOCK_XLOCKDONE)

12 Minutes lates the unlink_local error crashes appeared again. This time with 
a new file. During debugging we noticed a MTU mismatch between MDS (1500) and 
client (9000) with cephfs kernel mount. The client is also creating the 
snapshots via mkdir in the .snap directory.

We disabled snapshot creation for now, but really need this feature. I uploaded 
the mds logs of the first crash along with the information above to 
https://tracker.ceph.com/issues/38452

I would greatly appreciate it, if you could answer me the following question:

Is the Bug related to our MTU Mismatch? We fixed the MTU Issue going back to 
1500 on all nodes in the ceph public network on the weekend also.

If you need a debug level 20 log of the ScatterLock for further analysis, i 
could schedule snapshots at the end of our workdays and increase the debug 
level 5 Minutes arround snap shot creation.

Regards
Felix
-
-
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior
-
-

Am 02.12.2022 um 20:08 schrieb Patrick Donnelly :

On Thu, Dec 1, 2022 at 5:08 PM Stolte, Felix  wrote:

Script is running for ~2 hours and according to the line count in the memo file 
we are at 40% (cephfs is still online).

We had to modify the script putting a try/catch arround the for loop in line 78 
to 87. For some reasons there are some objects (186 at this moment) which throw 
an UnicodeDecodeError exception during the iteration:

 Traceback (most recent call 
last): File "first-damage.py", line 138, in  traverse(f, ioctx) File 
"first-damage.py", line 79, in traverse for (dnk, val) in it: File "rados.pyx", 
line 1382, in rados.OmapIterator.__next__ File "rados.pyx", line 311, in 
rados.decode_cstr UnicodeDecodeError: 'utf-8' codec can't decode bytes in 
position 10-11: invalid continuation byte

Don’t know if this is because of the filesystem still running. We saved the 
object names in a separate file and i will investigate further tomorrow. We 
should be able to modify the script to only check for the objects which threw 
the exception instead of searching through the whole pool again.

That shouldn't be caused by teh fs running. It may be you have some
file names which have invalid unicode characters?

Regarding the mds logfiles with debug 20:
We cannot run this debug level for longer than one hour since the logfile size 
increase is to high for the local storage on the mds servers where logs are 
stored (don’t have a central logging yet).

Okay.

But if you are just interested in the time frame arround the crash, i could set 
the debug level to 20, trigger the crash on the weekend and sent you the logs.

The crash is unlikely to point to what causes the corruption. I was
hoping we could locate an instance of damage while the MDS is running.

Regards Felix


-
-
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior
-
-

Am 01.12.2022 um 20:51 schrieb Patrick Donnelly :

On Thu, De

[ceph-users] Removing OSD very slow (objects misplaced)

[E Taka]

Hi,

when removing some OSD with the command `ceph orch osd rm X`, the
rebalancing starts very fast, but after a while it almost stalls with a
very low recovering rate:

Dec 15 18:47:17 … : cluster [DBG] pgmap v125312: 3361 pgs: 13
active+clean+scrubbing+deep, 4 active+remapped+backfilling, 3344
active+clean; 95 TiB data, 298 TiB used, 320 TiB / 618 TiB avail; 13 MiB/s
rd, 3.9 MiB/s wr, 610 op/s; 403603/330817302 objects misplaced (0.122%);
1.1 MiB/s, 2 objects/s recovering

As you can see, the rate is 2 Objects/s for over 40 objects. `ceph orch
osd rm status` shows long running draining processes (now over 4 days):

OSD  HOSTSTATE PGS  REPLACE  FORCE  ZAPDRAIN STARTED AT
64   ceph05  draining1  FalseFalse  False  2022-12-11
16:18:14.692636+00:00
…

Is there y way to increase the speed of the draining/rebalancing?

Thanks!
Erich
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: mds stuck in standby, not one active

[Patrick Donnelly]

On Thu, Dec 15, 2022 at 7:24 AM Mevludin Blazevic
 wrote:
>
> Hi,
>
> while upgrading to ceph pacific 6.2.7, the upgrade process stuck exactly
> at the mds daemons. Before, I have tried to increase/shrink the
> placement size of them, but nothing happens. Currently I have 4/3
> running daemons. One daemon should be stopped and removed.
>
> Do you suggest to force remove these daemons or what could be the
> preferred workaround?

Hard to say without more information. Please share:

ceph fs dump
ceph status
ceph health detail

-- 
Patrick Donnelly, Ph.D.
He / Him / His
Principal Software Engineer
Red Hat, Inc.
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: rgw: "failed to read header: bad method" after PutObject failed with 404 (NoSuchBucket)

[Stefan Reuter]

Hi,

I noticed that the error behavior is actually depending on the length of 
the PutObject request. The subsequent request
* results in a 400 Bad Request error (with log message "failed to read 
header: bad method") for small files or
* a read timeout on the connection is encountered by the client for 
larger files


In both cases the cause seems to be an issue with parsing the HTTP request.

I have created an issue at https://tracker.ceph.com/issues/58286

Best regards,

Stefan

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: mds stuck in standby, not one active

[Mevludin Blazevic]

Ceph fs dump:

e62
enable_multiple, ever_enabled_multiple: 1,1
default compat: compat={},rocompat={},incompat={1=base v0.20,2=client 
writeable ranges,3=default file layouts on dirs,4=dir inode in separate 
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no 
anchor table,9=file layout v2,10=snaprealm v2}legacy client fscid: 1


Filesystem 'ceph_fs' (1)
fs_name ceph_fs
epoch   62
flags   12
created 2022-11-28T12:05:17.203346+
modified    2022-12-15T12:09:14.091724+
tableserver 0
root    0
session_timeout 60
session_autoclose   300
max_file_size   1099511627776
required_client_features    {}
last_failure    0
last_failure_osd_epoch  196035
compat  compat={},rocompat={},incompat={1=base v0.20,2=client writeable 
ranges,3=default file layouts on dirs,4=dir inode in separate 
object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no 
anchor table,9=file layout v2,10=snaprealm v2}

max_mds 1
in  0
up  {}
failed  0
damaged
stopped
data_pools  [4]
metadata_pool   5
inline_data disabled
balancer
standby_count_wanted    1

f
Standby daemons:

[mds.ceph_fs.store5.gnlqqm{-1:152180029} state up:standby seq 1 
join_fscid=1 addr 
[v2:192.168.50.135:6800/3548272808,v1:192.168.50.135:6801/3548272808] 
compat {c=[1],r=[1],i=[1]}]
[mds.ceph_fs.store6.fxgvoj{-1:152416137} state up:standby seq 1 
join_fscid=1 addr 
[v2:192.168.50.136:7024/1339959968,v1:192.168.50.136:7025/1339959968] 
compat {c=[1],r=[1],i=[1]}]
[mds.ceph_fs.store4.mhvpot{-1:152477853} state up:standby seq 1 
join_fscid=1 addr 
[v2:192.168.50.134:6800/3098669884,v1:192.168.50.134:6801/3098669884] 
compat {c=[1],r=[1],i=[1]}]
[mds.ceph_fs.store3.vcnwzh{-1:152481783} state up:standby seq 1 
join_fscid=1 addr 
[v2:192.168.50.133:6800/77378788,v1:192.168.50.133:6801/77378788] compat 
{c=[1],r=[1],i=[1]}]

dumped fsmap epoch 62

Ceph Status:

  cluster:
    id: 8c774934-1535-11ec-973e-525400130e4f
    health: HEALTH_ERR
    1 filesystem is degraded
    1 filesystem has a failed mds daemon
    1 filesystem is offline
    26 daemons have recently crashed

  services:
    mon: 2 daemons, quorum cephadm-vm,store2 (age 2d)
    mgr: store1.uevcpd(active, since 2d), standbys: cephadm-vm.zwagng
    mds: 0/1 daemons up (1 failed), 4 standby
    osd: 312 osds: 312 up (since 8h), 312 in (since 17h)

  data:
    volumes: 0/1 healthy, 1 failed
    pools:   7 pools, 289 pgs
    objects: 2.62M objects, 9.8 TiB
    usage:   29 TiB used, 1.9 PiB / 1.9 PiB avail
    pgs: 286 active+clean
 3   active+clean+scrubbing+deep

  io:
    client:   945 KiB/s rd, 3.3 MiB/s wr, 516 op/s rd, 562 op/s wr

Ceph Health detail:

HEALTH_ERR 1 filesystem is degraded; 1 filesystem has a failed mds 
daemon; 1 filesystem is offline; 26 daemons have recently crashed

[WRN] FS_DEGRADED: 1 filesystem is degraded
    fs ceph_fs is degraded
[WRN] FS_WITH_FAILED_MDS: 1 filesystem has a failed mds daemon
    fs ceph_fs has 1 failed mds
[ERR] MDS_ALL_DOWN: 1 filesystem is offline
    fs ceph_fs is offline because no MDS is active for it.
[WRN] RECENT_CRASH: 26 daemons have recently crashed
    osd.323 crashed on host store7 at 2022-12-12T14:03:23.857874Z
    osd.323 crashed on host store7 at 2022-12-12T14:03:43.945625Z
    osd.323 crashed on host store7 at 2022-12-12T14:04:03.282797Z
    osd.323 crashed on host store7 at 2022-12-12T14:04:22.612037Z
    osd.323 crashed on host store7 at 2022-12-12T14:04:41.630473Z
    osd.323 crashed on host store7 at 2022-12-12T14:34:49.237008Z
    osd.323 crashed on host store7 at 2022-12-12T14:35:09.903922Z
    osd.323 crashed on host store7 at 2022-12-12T14:35:28.621955Z
    osd.323 crashed on host store7 at 2022-12-12T14:35:46.985517Z
    osd.323 crashed on host store7 at 2022-12-12T14:36:05.375758Z
    osd.323 crashed on host store7 at 2022-12-12T15:01:57.235785Z
    osd.323 crashed on host store7 at 2022-12-12T15:02:16.581335Z
    osd.323 crashed on host store7 at 2022-12-12T15:02:33.212653Z
    osd.323 crashed on host store7 at 2022-12-12T15:02:49.775560Z
    osd.323 crashed on host store7 at 2022-12-12T15:03:06.303861Z
    mgr.cephadm-vm.zwagng crashed on host cephadm-vm at 
2022-12-13T13:21:41.149773Z
    mgr.cephadm-vm.zwagng crashed on host cephadm-vm at 
2022-12-13T13:22:15.413105Z
    mgr.cephadm-vm.zwagng crashed on host cephadm-vm at 
2022-12-13T13:23:39.888401Z
    mgr.cephadm-vm.zwagng crashed on host cephadm-vm at 
2022-12-13T13:27:56.458529Z
    mgr.cephadm-vm.zwagng crashed on host cephadm-vm at 
2022-12-13T13:31:03.791532Z
    mgr.cephadm-vm.zwagng crashed on host cephadm-vm at 
2022-12-13T13:34:24.023106Z

    osd.98 crashed on host store3 at 2022-12-13T16:11:38.064735Z
    mgr.store1.uevcpd crashed on host store1 at 2022-12-13T18:39:33.091261Z
    osd.322 crashed on host store6 at 2022-12-14T06:06:14.193437Z
    osd.234 crashed on host store8 at 2022-12-15T02:32:13.009795Z
    osd.311 crashed on host store8 at 2022-12-15T02:32:18.40

[ceph-users] Re: mds stuck in standby, not one active

[Patrick Donnelly]

On Thu, Dec 15, 2022 at 3:17 PM Mevludin Blazevic
 wrote:
>
> Ceph fs dump:
>
> e62
> enable_multiple, ever_enabled_multiple: 1,1
> default compat: compat={},rocompat={},incompat={1=base v0.20,2=client
> writeable ranges,3=default file layouts on dirs,4=dir inode in separate
> object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
> anchor table,9=file layout v2,10=snaprealm v2}legacy client fscid: 1
>
> Filesystem 'ceph_fs' (1)
> fs_name ceph_fs
> epoch   62
> flags   12
> created 2022-11-28T12:05:17.203346+
> modified2022-12-15T12:09:14.091724+
> tableserver 0
> root0
> session_timeout 60
> session_autoclose   300
> max_file_size   1099511627776
> required_client_features{}
> last_failure0
> last_failure_osd_epoch  196035
> compat  compat={},rocompat={},incompat={1=base v0.20,2=client writeable
> ranges,3=default file layouts on dirs,4=dir inode in separate
> object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no
> anchor table,9=file layout v2,10=snaprealm v2}
> max_mds 1
> in  0
> up  {}
> failed  0
> damaged
> stopped
> data_pools  [4]
> metadata_pool   5
> inline_data disabled
> balancer
> standby_count_wanted1
>
> f
> Standby daemons:
>
> [mds.ceph_fs.store5.gnlqqm{-1:152180029} state up:standby seq 1
> join_fscid=1 addr
> [v2:192.168.50.135:6800/3548272808,v1:192.168.50.135:6801/3548272808]
> compat {c=[1],r=[1],i=[1]}]
> [mds.ceph_fs.store6.fxgvoj{-1:152416137} state up:standby seq 1
> join_fscid=1 addr
> [v2:192.168.50.136:7024/1339959968,v1:192.168.50.136:7025/1339959968]
> compat {c=[1],r=[1],i=[1]}]
> [mds.ceph_fs.store4.mhvpot{-1:152477853} state up:standby seq 1
> join_fscid=1 addr
> [v2:192.168.50.134:6800/3098669884,v1:192.168.50.134:6801/3098669884]
> compat {c=[1],r=[1],i=[1]}]
> [mds.ceph_fs.store3.vcnwzh{-1:152481783} state up:standby seq 1
> join_fscid=1 addr
> [v2:192.168.50.133:6800/77378788,v1:192.168.50.133:6801/77378788] compat
> {c=[1],r=[1],i=[1]}]
> dumped fsmap epoch 62
>
> Ceph Status:
>
>cluster:
>  id: 8c774934-1535-11ec-973e-525400130e4f
>  health: HEALTH_ERR
>  1 filesystem is degraded
>  1 filesystem has a failed mds daemon
>  1 filesystem is offline
>  26 daemons have recently crashed
>
>services:
>  mon: 2 daemons, quorum cephadm-vm,store2 (age 2d)
>  mgr: store1.uevcpd(active, since 2d), standbys: cephadm-vm.zwagng
>  mds: 0/1 daemons up (1 failed), 4 standby
>  osd: 312 osds: 312 up (since 8h), 312 in (since 17h)
>
>data:
>  volumes: 0/1 healthy, 1 failed
>  pools:   7 pools, 289 pgs
>  objects: 2.62M objects, 9.8 TiB
>  usage:   29 TiB used, 1.9 PiB / 1.9 PiB avail
>  pgs: 286 active+clean
>   3   active+clean+scrubbing+deep
>
>io:
>  client:   945 KiB/s rd, 3.3 MiB/s wr, 516 op/s rd, 562 op/s wr
>
> Ceph Health detail:
>
> HEALTH_ERR 1 filesystem is degraded; 1 filesystem has a failed mds
> daemon; 1 filesystem is offline; 26 daemons have recently crashed
> [WRN] FS_DEGRADED: 1 filesystem is degraded
>  fs ceph_fs is degraded
> [WRN] FS_WITH_FAILED_MDS: 1 filesystem has a failed mds daemon
>  fs ceph_fs has 1 failed mds
> [ERR] MDS_ALL_DOWN: 1 filesystem is offline
>  fs ceph_fs is offline because no MDS is active for it.
> [WRN] RECENT_CRASH: 26 daemons have recently crashed
>  osd.323 crashed on host store7 at 2022-12-12T14:03:23.857874Z
>  osd.323 crashed on host store7 at 2022-12-12T14:03:43.945625Z
>  osd.323 crashed on host store7 at 2022-12-12T14:04:03.282797Z
>  osd.323 crashed on host store7 at 2022-12-12T14:04:22.612037Z
>  osd.323 crashed on host store7 at 2022-12-12T14:04:41.630473Z
>  osd.323 crashed on host store7 at 2022-12-12T14:34:49.237008Z
>  osd.323 crashed on host store7 at 2022-12-12T14:35:09.903922Z
>  osd.323 crashed on host store7 at 2022-12-12T14:35:28.621955Z
>  osd.323 crashed on host store7 at 2022-12-12T14:35:46.985517Z
>  osd.323 crashed on host store7 at 2022-12-12T14:36:05.375758Z
>  osd.323 crashed on host store7 at 2022-12-12T15:01:57.235785Z
>  osd.323 crashed on host store7 at 2022-12-12T15:02:16.581335Z
>  osd.323 crashed on host store7 at 2022-12-12T15:02:33.212653Z
>  osd.323 crashed on host store7 at 2022-12-12T15:02:49.775560Z
>  osd.323 crashed on host store7 at 2022-12-12T15:03:06.303861Z
>  mgr.cephadm-vm.zwagng crashed on host cephadm-vm at
> 2022-12-13T13:21:41.149773Z
>  mgr.cephadm-vm.zwagng crashed on host cephadm-vm at
> 2022-12-13T13:22:15.413105Z
>  mgr.cephadm-vm.zwagng crashed on host cephadm-vm at
> 2022-12-13T13:23:39.888401Z
>  mgr.cephadm-vm.zwagng crashed on host cephadm-vm at
> 2022-12-13T13:27:56.458529Z
>  mgr.cephadm-vm.zwagng crashed on host cephadm-vm at
> 2022-12-13T13:31:03.791532Z
>  mgr.cephadm-vm.zwagng crashed on host cephadm-vm at
> 2022-12-13T13:34:24.023106Z
>  osd.98 crashe

[ceph-users] Re: 16.2.11 pacific QE validation status

[Brad Hubbard]

On Fri, Dec 16, 2022 at 3:15 AM Yuri Weinstein  wrote:
>
> Details of this release are summarized here:
>
> https://tracker.ceph.com/issues/58257#note-1
> Release Notes - TBD
>
> Seeking approvals for:
>
> rados - Neha (https://github.com/ceph/ceph/pull/49431 is still being
> tested and will be merged soon)
> rook - Sébastien Han
> cephadm - Adam
> dashboard - Ernesto
> rgw - Casey (rwg will be rerun on the latest SHA1)
> rbd - Ilya, Deepika
> krbd - Ilya, Deepika
> fs - Venky, Patrick
> upgrade/nautilus-x (pacific) - Neha, Laura
> upgrade/octopus-x (pacific) - Neha, Laura
> upgrade/pacific-p2p - Neha - Neha, Laura
> powercycle - Brad

The failure here is due to fallout from the recent lab issues and was
fixed in main by https://github.com/ceph/ceph/pull/49021 I'm waiting
to see if there are plans to backport this to pacific and quincy since
that will be needed.

> ceph-volume - Guillaume, Adam K
>
> Thx
> YuriW
>
> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email to dev-le...@ceph.io



-- 
Cheers,
Brad

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 16.2.11 pacific QE validation status

[Laura Flores]

I reviewed the upgrade runs:

https://pulpito.ceph.com/yuriw-2022-12-13_15:57:57-upgrade:nautilus-x-pacific_16.2.11_RC-distro-default-smithi/
https://pulpito.ceph.com/yuriw-2022-12-13_21:47:46-upgrade:nautilus-x-pacific_16.2.11_RC-distro-default-smithi/
https://pulpito.ceph.com/yuriw-2022-12-13_15:58:18-upgrade:octopus-x-pacific_16.2.11_RC-distro-default-smithi/
https://pulpito.ceph.com/yuriw-2022-12-14_15:41:10-upgrade:octopus-x-pacific_16.2.11_RC-distro-default-smithi/

Failures:
  1. https://tracker.ceph.com/issues/50618 -- known bug assigned to Ilya;
assuming it's not a big deal since it's been around for over a year

Details:
  1. qemu_xfstests_luks1 failed on xfstest 168 - Ceph - RBD


https://pulpito.ceph.com/yuriw-2022-12-13_15:58:24-upgrade:pacific-p2p-pacific_16.2.11_RC-distro-default-smithi/
https://pulpito.ceph.com/yuriw-2022-12-14_15:40:37-upgrade:pacific-p2p-pacific_16.2.11_RC-distro-default-smithi/

Failures, unrelated:
  1. https://tracker.ceph.com/issues/58223 -- new failure reported by me 7
days ago; seems infrastructure related and not regression-related
  2. https://tracker.ceph.com/issues/52590 -- closed by Casey; must not be
of importance
  3. https://tracker.ceph.com/issues/58289 -- new failure raised by me
today; seems related to other "wait_for_recovery" failures, which are
generally not cause for concern since they're so infrequent.
  4. https://tracker.ceph.com/issues/51652 -- known bug from over a year ago

Details;
  1. failure on `sudo fuser -v /var/lib/dpkg/lock-frontend` - Infrastructure
  2. "[ FAILED ] CmpOmap.cmp_vals_u64_invalid_default" in
upgrade:pacific-p2p-pacific - Ceph - RGW
  3. "AssertionError: wait_for_recovery: failed before timeout expired"
from down pg in pacific-p2p-pacific - Ceph - RADOS
  4. heartbeat timeouts on filestore OSDs while deleting objects in
upgrade:pacific-p2p-pacific - Ceph - RADOS

On Thu, Dec 15, 2022 at 4:34 PM Brad Hubbard  wrote:

> On Fri, Dec 16, 2022 at 3:15 AM Yuri Weinstein 
> wrote:
> >
> > Details of this release are summarized here:
> >
> > https://tracker.ceph.com/issues/58257#note-1
> > Release Notes - TBD
> >
> > Seeking approvals for:
> >
> > rados - Neha (https://github.com/ceph/ceph/pull/49431 is still being
> > tested and will be merged soon)
> > rook - Sébastien Han
> > cephadm - Adam
> > dashboard - Ernesto
> > rgw - Casey (rwg will be rerun on the latest SHA1)
> > rbd - Ilya, Deepika
> > krbd - Ilya, Deepika
> > fs - Venky, Patrick
> > upgrade/nautilus-x (pacific) - Neha, Laura
> > upgrade/octopus-x (pacific) - Neha, Laura
> > upgrade/pacific-p2p - Neha - Neha, Laura
> > powercycle - Brad
>
> The failure here is due to fallout from the recent lab issues and was
> fixed in main by https://github.com/ceph/ceph/pull/49021 I'm waiting
> to see if there are plans to backport this to pacific and quincy since
> that will be needed.
>
> > ceph-volume - Guillaume, Adam K
> >
> > Thx
> > YuriW
> >
> > ___
> > Dev mailing list -- d...@ceph.io
> > To unsubscribe send an email to dev-le...@ceph.io
>
>
>
> --
> Cheers,
> Brad
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>


-- 

Laura Flores

She/Her/Hers

Software Engineer, Ceph Storage

Red Hat Inc. 

Chicago, IL

lflo...@redhat.com
M: +17087388804
@RedHat    Red Hat
  Red Hat


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph-iscsi lock ping pong

[Xiubo Li]

On 15/12/2022 02:46, Joe Comeau wrote:

That's correct - we use the kernel target not tcmu-runner


Okay.

There are some difference for the configurations between kernel target 
and the ceph-iscsi target.


Thanks,

- Xiubo



>>> Xiubo Li  12/13/2022 6:02 PM >>>

On 14/12/2022 06:54, Joe Comeau wrote:
> I am curious about what is happening with your iscsi configuration
> Is this a new iscsi config or something that has just cropped up ?
>
> We are using/have been using vmware for 5+ years with iscsi
> We are using the kernel iscsi vs tcmu
>

Do you mean you are using kernel target, not the ceph-iscsi/tcmu-runner
in user space, right ?

> We are running ALUA and all datastores are setup as RR
> We routinely reboot the iscsi gateways - during patching and updates 
and the storage migrates to and from all servers without issue
> We usually wait about 10 minutes before a gateway restart, so there 
is not an outage

>
> It has been extremely stable for us
>
> Thanks Joe
>
>
>
 Xiubo Li  12/13/2022 4:21 AM >>>
> On 13/12/2022 18:57, Stolte, Felix wrote:
>> Hi Xiubo,
>>
>> Thx for pointing me into the right direction. All involved esx host
>> seem to use the correct policy. I am going to detach the LUN on each
>> host one by one until i found the host causing the problem.
>>
>  From the logs it means the client was switching the path in turn.
>
> BTW, what's policy are you using ?
>
> Thanks
>
> - Xiubo
>
>> Regards Felix
>> 
-
>> 
-

>> Forschungszentrum Juelich GmbH
>> 52425 Juelich
>> Sitz der Gesellschaft: Juelich
>> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
>> Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
>> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
>> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
>> Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior
>> 
-
>> 
-

>>
>>> Am 12.12.2022 um 13:03 schrieb Xiubo Li :
>>>
>>> Hi Stolte,
>>>
>>> For the VMware config could you refer to :
>>> https://docs.ceph.com/en/latest/rbd/iscsi-initiator-esx/ 
 ?

>>>
>>> What's the "Path Selection Policy with ALUA" you are using ? The
>>> ceph-iscsi couldn't implement the real AA, so if you use the RR I
>>> think it will be like this.
>>>
>>> - Xiubo
>>>
>>> On 12/12/2022 17:45, Stolte, Felix wrote:
 Hi guys,

 we are using ceph-iscsi to provide block storage for Microsoft 
Exchange and vmware vsphere. Ceph docs state that you need to 
configure Windows iSCSI Initatior for fail-over-only but there is no 
such point for vmware. In my tcmu-runner logs on both ceph-iscsi 
gateways I see the following:


 2022-12-12 10:36:06.978 33789 [WARN] tcmu_notify_lock_lost:222 
rbd/mailbox.vmdk_junet_sata: Async lock drop. Old state 1
 2022-12-12 10:36:06.993 33789 [INFO] alua_implicit_transition:570 
rbd/mailbox.vmdk_junet_sata: Starting lock acquisition operation.
 2022-12-12 10:36:08.064 33789 [WARN] tcmu_rbd_lock:762 
rbd/mailbox.vmdk_junet_sata: Acquired exclusive lock.
 2022-12-12 10:36:09.067 33789 [WARN] tcmu_notify_lock_lost:222 
rbd/mailbox.vmdk_junet_sata: Async lock drop. Old state 1
 2022-12-12 10:36:09.071 33789 [INFO] alua_implicit_transition:570 
rbd/mailbox.vmdk_junet_sata: Starting lock acquisition operation.
 2022-12-12 10:36:10.109 33789 [WARN] tcmu_rbd_lock:762 
rbd/mailbox.vmdk_junet_sata: Acquired exclusive lock.
 2022-12-12 10:36:11.104 33789 [WARN] tcmu_notify_lock_lost:222 
rbd/mailbox.vmdk_junet_sata: Async lock drop. Old state 1
 2022-12-12 10:36:11.106 33789 [INFO] alua_implicit_transition:570 
rbd/mailbox.vmdk_junet_sata: Starting lock acquisition operation.


 At the same time there are these log entries in ceph.audit.logs:
 2022-12-12T10:36:06.731621+0100 mon.mon-k2-1 (mon.1) 3407851 : 
audit [INF] from='client.? 10.100.8.55:0/2392201639' 
entity='client.admin' cmd=[{"prefix": "osd blocklist", "blocklistop": 
"add", "addr": "10

 .100.8.56:0/1598475844"}]: dispatch
 2022-12-12T10:36:06.731913+0100 mon.mon-e2-1 (mon.0) 783726 : 
audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": 
"osd blocklist", "blocklistop": "add", "addr": 
"10.100.8.56:0/1598475844"}]

 : dispatch
 2022-12-12T10:36:06.905082+0100 mon.mon-e2-1 (mon.0) 783727 : 
audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": 
"osd blocklist", "blocklistop": "add", "addr": "10.100.8.56:0/1598475844"}

 ]': finished

 Can someone explaint to me, what is happening? Why are the 
gateways blacklisting each other? All involved daemons ar

[ceph-users] Re: ceph-iscsi lock ping pong

[Xiubo Li]

On 14/12/2022 16:32, Stolte, Felix wrote:
We have been using tgt for five years and switched to ceph-iscsi (LIO 
Framework) two months ago. We observed a massive performance boost. 
Can’t say though if the performance increase was only related to the 
different software or if our TGT configuration was not as could as it 
could have been. Personally i prefer the ceph-iscsi configuration, 
it’s way easier to setup and you can create targets, lun, etc. either 
via gwcli or ceph dashboard.


Years ago I knew there was one user had compared the performance between 
tgt and ceph-iscsi for ceph with their products, they also got the 
similar result with yours.


BRs

- Xiubo



Regards
Felix
-
-
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior
-
-


Am 13.12.2022 um 23:54 schrieb Joe Comeau :

I am curious about what is happening with your iscsi configuration
Is this a new iscsi config or something that has just cropped up ?
We are using/have been using vmware for 5+ years with iscsi
We are using the kernel iscsi vs tcmu
We are running ALUA and all datastores are setup as RR
We routinely reboot the iscsi gateways - during patching and updates 
and the storage migrates to and from all servers without issue
We usually wait about 10 minutes before a gateway restart, so there 
is not an outage

It has been extremely stable for us
Thanks Joe


>>> Xiubo Li  12/13/2022 4:21 AM >>>

On 13/12/2022 18:57, Stolte, Felix wrote:
> Hi Xiubo,
>
> Thx for pointing me into the right direction. All involved esx host
> seem to use the correct policy. I am going to detach the LUN on each
> host one by one until i found the host causing the problem.
>
From the logs it means the client was switching the path in turn.

BTW, what's policy are you using ?

Thanks

- Xiubo

> Regards Felix
> 
-
> 
-

> Forschungszentrum Juelich GmbH
> 52425 Juelich
> Sitz der Gesellschaft: Juelich
> Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
> Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
> Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
> Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
> Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior
> 
-
> 
-

>
>> Am 12.12.2022 um 13:03 schrieb Xiubo Li :
>>
>> Hi Stolte,
>>
>> For the VMware config could you refer to :
>> https://docs.ceph.com/en/latest/rbd/iscsi-initiator-esx/ 
 ?

>>
>> What's the "Path Selection Policy with ALUA" you are using ? The
>> ceph-iscsi couldn't implement the real AA, so if you use the RR I
>> think it will be like this.
>>
>> - Xiubo
>>
>> On 12/12/2022 17:45, Stolte, Felix wrote:
>>> Hi guys,
>>>
>>> we are using ceph-iscsi to provide block storage for Microsoft 
Exchange and vmware vsphere. Ceph docs state that you need to 
configure Windows iSCSI Initatior for fail-over-only but there is no 
such point for vmware. In my tcmu-runner logs on both ceph-iscsi 
gateways I see the following:

>>>
>>> 2022-12-12 10:36:06.978 33789 [WARN] tcmu_notify_lock_lost:222 
rbd/mailbox.vmdk_junet_sata: Async lock drop. Old state 1
>>> 2022-12-12 10:36:06.993 33789 [INFO] alua_implicit_transition:570 
rbd/mailbox.vmdk_junet_sata: Starting lock acquisition operation.
>>> 2022-12-12 10:36:08.064 33789 [WARN] tcmu_rbd_lock:762 
rbd/mailbox.vmdk_junet_sata: Acquired exclusive lock.
>>> 2022-12-12 10:36:09.067 33789 [WARN] tcmu_notify_lock_lost:222 
rbd/mailbox.vmdk_junet_sata: Async lock drop. Old state 1
>>> 2022-12-12 10:36:09.071 33789 [INFO] alua_implicit_transition:570 
rbd/mailbox.vmdk_junet_sata: Starting lock acquisition operation.
>>> 2022-12-12 10:36:10.109 33789 [WARN] tcmu_rbd_lock:762 
rbd/mailbox.vmdk_junet_sata: Acquired exclusive lock.
>>> 2022-12-12 10:36:11.104 33789 [WARN] tcmu_notify_lock_lost:222 
rbd/mailbox.vmdk_junet_sata: Async lock drop. Old state 1
>>> 2022-12-12 10:36:11.106 33789 [INFO] alua_impli

[ceph-users] max pool size (amount of data/number of OSDs)

[Christopher Durham]

Hi,
There are various articles, case studies, etc about large ceph clusters, 
storing 10s of PiB,with CERN being the largest cluster as far as I know.
Is there a largest pool capacity limit?  In other words, while you may have a 
30PiB cluster,is there a limit or recommendation as to max pool capacity. For 
example, in the 30PiB example,is there a limit or recommendation that says do 
not have a pool capacity of higher than 5iB, for 6pools in that cluster at a 
ttotal of 30PiB?

I know this would be contingent upon a variety of things, including, but not 
limited to network throughput, individual serversize (disk size and number, 
memory, compute). I am specifically talking about s3./rgw storage.

But is there a technical limit, or just a tested size, of a pool? Should I 
createdifferent pools when a given pool would otherwise reach a size capacity 
of Xor have N osds or PGs in it, when considering adding additional osds?
Thanks for any info
-Chris
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph-iscsi lock ping pong

[Xiubo Li]

On 14/12/2022 14:52, Stolte, Felix wrote:
Issue is resolved now. After verifying that all esx hosts are 
configured for MRU, i took a closer look on the paths on each host.


`gwcli` reported lun in question was owned by gateway A, but one esx 
host used the path to gateway B for I/O. I reconfigured that 
particular host and it’s now using the correct path to gateway A. Logs 
are clean now and I/O on that Datastore is back to normal.


Yeah.

When the exsi client sent IOs to gateway B, the gateway B will try to 
acquire the exclusive lock and then ceph will blocklist the current 
owner, which is gateway A, of it after succeeding.


This is why you were seeing the gateways were blocklisting each other.



This was probably caused by an outage of one of our gateways last week 
(the physical host, not the daemon), where the iSCSI Daemon didn’t 
shut down cleanly.


One last question though:

From my understanding the "Dynamic Discovery“ just creates the „Static 
Discovery“ Targets for all available gateways. Is it also responsible 
for telling the client, which path to use (aka which gateway is the 
owner of a LUN)?


In linux initiator I know the multipath will correctly configure the 
paths' priority by checking the configuration from gateways together 
with the multipath setting locally.


Not sure how the exsi will behave exactly.

BRs

- Xiubo


-
-
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior
-
-


Am 13.12.2022 um 13:21 schrieb Xiubo Li :


On 13/12/2022 18:57, Stolte, Felix wrote:

Hi Xiubo,

Thx for pointing me into the right direction. All involved esx host 
seem to use the correct policy. I am going to detach the LUN on each 
host one by one until i found the host causing the problem.



From the logs it means the client was switching the path in turn.

BTW, what's policy are you using ?

Thanks

- Xiubo


Regards Felix
-
-
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior
-
-


Am 12.12.2022 um 13:03 schrieb Xiubo Li :

Hi Stolte,

For the VMware config could you refer to : 
https://docs.ceph.com/en/latest/rbd/iscsi-initiator-esx/ ?


What's the "Path Selection Policy with ALUA" you are using ? The 
ceph-iscsi couldn't implement the real AA, so if you use the RR I 
think it will be like this.


- Xiubo

On 12/12/2022 17:45, Stolte, Felix wrote:

Hi guys,

we are using ceph-iscsi to provide block storage for Microsoft Exchange and 
vmware vsphere. Ceph docs state that you need to configure Windows iSCSI 
Initatior for fail-over-only but there is no such point for vmware. In my 
tcmu-runner logs on both ceph-iscsi gateways I see the following:

2022-12-12 10:36:06.978 33789 [WARN] tcmu_notify_lock_lost:222 
rbd/mailbox.vmdk_junet_sata: Async lock drop. Old state 1
2022-12-12 10:36:06.993 33789 [INFO] alua_implicit_transition:570 
rbd/mailbox.vmdk_junet_sata: Starting lock acquisition operation.
2022-12-12 10:36:08.064 33789 [WARN] tcmu_rbd_lock:762 
rbd/mailbox.vmdk_junet_sata: Acquired exclusive lock.
2022-12-12 10:36:09.067 33789 [WARN] tcmu_notify_lock_lost:222 
rbd/mailbox.vmdk_junet_sata: Async lock drop. Old state 1
2022-12-12 10:36:09.071 33789 [INFO] alua_implicit_transition:570 
rbd/mailbox.vmdk_junet_sata: Starting lock acquisition operation.
2022-12-12 10:36:10.109 33789 [WARN] tcmu_rbd_lock:762 
rbd/mailbox.vmdk_junet_sata: Acquired exclusive lock.
2022-12-12 10:36:11.104 33789 [WARN] tcmu_notify_lock_lost:222 
rbd/mailbox.vmdk_junet_sata: Async lock drop. Old state 1
2022-12-12 10:36:11.106 33789 [INFO] alua_implicit_transition:570 
rbd/mailbox.vmdk_junet_sata: Starting lock acquisition operation.

At the s

[ceph-users] Re: cephfs snap-mirror stalled

[Holger Naundorf]

On 15.12.22 14:06, Venky Shankar wrote:

Hi Holger,

(sorry for the late reply)

On Fri, Dec 9, 2022 at 6:22 PM Holger Naundorf  wrote:


As an update:
After the third restart now the mirror-daemon is running normal again -
only change to the restarts before was that during the restart
dbug_client was set to 20. (First restart was after ~48h of no data
movemnent to the receiver side and no changes, 2nd was with debug_mirror
set to 20 and the thisrd - maybe to quickly - after only ~12h of no data
movement with debug_mirror + _client set to 20). Currently I reset the
debug levels back down to reduce the load on the system disks.


Do you see ESTALE related messages in mirror daemon logs (with
debug_client: 20)? We have run into a couple of instances where the
mirror daemon would be stuck on a directory entry. The workaround for
that is to find the directory path (where the daemon is stuck) and
from another client (mount), list the entries in that directory (or at
times restarting the daemon works, as in your case).


Sounds like our bug - I do have the ESTALE messages with debug 20:

syslog.restart-mirror-client_debug_20.gz:Dec  7 10:05:30 ceph07 
bash[1740788]: debug 2022-12-07T09:05:30.299+ 7f05907a7700 20 
client.29690499 got ESTALE on tid 2258671 from mds.0
syslog.restart-mirror-client_debug_20.gz:Dec  7 10:05:30 ceph07 
bash[1740788]: debug 2022-12-07T09:05:30.299+ 7f05907a7700 20 
client.29690499 got ESTALE on tid 2258671 from mds.0
syslog.restart-mirror-client_debug_20.gz:Dec  7 10:05:30 ceph07 
bash[1740788]: debug 2022-12-07T09:05:30.299+ 7f05907a7700 20 
client.29690499 got ESTALE on tid 2258671 from mds.0

(...)

And when the mirror got stuck again after some time it mysteriously (at 
that time) started to work again while I was poking around in the 
logfiles and the system - and doing an 'ls' in the stuck dir was 
definitly included in the poking.




This will be fixed in the next pacific release (tracker:
https://tracker.ceph.com/issues/55935).


Next means the .11 release or the current .10 we have not yet upgraded to?

Thanks for the update.

Regards,
Holger Naundorf




Regards,
Holger Naundorf


On 07.12.22 15:53, Holger Naundorf wrote:

On 06.12.22 14:17, Venky Shankar wrote:

On Tue, Dec 6, 2022 at 6:34 PM Holger Naundorf
 wrote:




On 06.12.22 09:54, Venky Shankar wrote:

Hi Holger,

On Tue, Dec 6, 2022 at 1:42 PM Holger Naundorf
 wrote:


Hello,
we have set up a snap-mirror for a directory on one of our clusters -
running ceph version

ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific
(stable)

to get mirrorred our other cluster - running ceph version

ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific
(stable)

The initial setup went ok, when the first snapshot was created data
started to flow at a decent (for our HW) rate of 100-200MB/s. As the
directory contains  ~200TB this was expected to take some time -
but now
the process has stalled completely after ~100TB were mirrored and ~7d
running.

Up to now I do not have any hints why it has stopped - I do not see
any
error messages from the cephfs-mirror daemon. Can the small version
mismatch be a problem?

Any hints where to look to find out what has got stuck are welcome.


I'd look at the mirror daemon logs for any errors to start with. You
might want to crank up the log level for debugging (debug
cephfs_mirror=20).



Even on max debug I do not see anything which looks like an error - but
as this is the first time I try to dig into any cephfs-mirror logs I
might not notice (as long as it is not red and flashing).

The Log basically this type of sequence, repeating forever:

(...)
cephfs::mirror::MirrorWatcher handle_notify
cephfs::mirror::Mirror update_fs_mirrors
cephfs::mirror::Mirror schedule_mirror_update_task: scheduling fs mirror
update (0x556fe3a7f130) after 2 seconds
cephfs::mirror::Watcher handle_notify: notify_id=751516198184655,
handle=93939050205568, notifier_id=25504530
cephfs::mirror::MirrorWatcher handle_notify
cephfs::mirror::PeerReplayer(19361031-928d-4366-99bd-50df70d3adf1) run:
trying to pick from 1 directories
cephfs::mirror::PeerReplayer(19361031-928d-4366-99bd-50df70d3adf1)
pick_directory
cephfs::mirror::Watcher handle_notify: notify_id=751516198184656,
handle=93939050205568, notifier_id=25504530
cephfs::mirror::MirrorWatcher handle_notify
cephfs::mirror::Mirror update_fs_mirrors
cephfs::mirror::Mirror schedule_mirror_update_task: scheduling fs mirror
update (0x556fe3a7fc70) after 2 seconds
cephfs::mirror::Watcher handle_notify: notify_id=751516198184657,
handle=93939050205568, notifier_id=25504530
cephfs::mirror::MirrorWatcher handle_notify
(...)


Basically, the interesting bit is not captured since it probably
happened sometime back. Could you please set the following:

debug cephfs_mirror = 20
debug client = 20

and restart the mirror daemon? The daemon would start synchronizing
again. When synchronizing stalls, please share the daemon logs. If the
log