[ceph-users] Re: Radosgw multisite replication issues

[Mark Kogan]

hi Eli,

please check that the MS sync link network bandwidth in both directions is
suitable with a tool like *iperf3* and that connection number limits as
mentioned above are not imposing a problem (external firewalls &
iptables/nftables conntrack)

it could be worthwhile to record a short tcpdump and wireshark follow
tcp/http stream


if networking is not an issue, please check if there are errors on a
specific RGW out of the 3 in the endpoints and restart/analyze the specific
one,

if not issue specific to certain rgw, check that after the upgrade the CPU
utilization or memory consumption is not higher causing slow
response because of swapping for example (on OSD nodes also)

egreping `*latency=*` | /admin/log or endpoint addresses may show if the
issue is constant or sporadic and to which/all endpoint/s

also, depending on the amount of sync traffic objects and size), for the
sake of simplifying the diagnostic, it may be worth to temporarily change
the endpoints to 1x1 instead of 3x3 so that there is only 1 log file to
analyze on each side and direct the client traffic to a separate RGW that
is not a syc endpoint and has *rgw_run_sync_thread=0 *conf, this way the
sync endpoints RGW logs contain  only sync related operations


On Thu, May 11, 2023 at 5:45 PM Casey Bodley  wrote:

> On Tue, May 9, 2023 at 3:11 PM Tarrago, Eli (RIS-BCT)
>  wrote:
> >
> > East and West Clusters have been upgraded to quincy, 17.2.6.
> >
> > We are still seeing replication failures. Deep diving the logs, I found
> the following interesting items.
> >
> > What is the best way to continue to troubleshoot this?
>
> the curl timeouts make it look like a networking issue. can you
> reproduce these issues with normal s3 clients against the west zone
> endpoints?
>
> if it's not the network itself, it could also be that the remote
> radosgws have saturated their rgw_max_concurrent_requests, so are slow
> to start processing accepted connections. as you're probably aware,
> multisite replication sends a lot of requests to /admin/log to poll
> for changes. if the remote radosgw is slow to process those, this
> could be the result. there are two separate perf counters you might
> consult to check for this:
>
> on the remote (west) radosgws, there's a perf counter called "qactive"
> that you could query (either from the radosgw admin socket, or via
> 'ceph daemon perf') for comparison against the configured
> rgw_max_concurrent_requests
>
> on the local (east) radosgws, there's a set of perf counters under
> "data-sync-from-{zone}" that track polling errors and latency
>
> > What is the curl attempting to fetch, but failing to obtain?
> >
> > -
> > root@east01:~# radosgw-admin bucket sync --bucket=ceph-bucket
> --source-zone=rgw-west run
> > 2023-05-09T15:22:43.582+ 7f197d7fa700  0 WARNING: curl
> operation timed out, network average transfer speed less than 1024 Bytes
> per second during 300 seconds.
> > 2023-05-09T15:22:43.582+ 7f1a48dd9e40  0 data sync: ERROR:
> failed to fetch bucket index status
>
> this error would correspond to a request like "GET
> /admin/log/?type=bucket-instance&bucket-instance={instance id}&info",
> sent to one of the west zone endpoints (http://west01.example.net:8080
> etc). if you retry the command, you should be able to find such a
> request in one of the west zone's radosgw logs. if you raise 'debug
> rgw' level to 4 or more, that op would be logged as
> 'bucket_index_log_info'
>
> > 2023-05-09T15:22:43.582+ 7f1a48dd9e40  0
> RGW-SYNC:bucket[ceph-bucket:ddd66ab8-0417---.93706683.1:119<-ceph-bucket:ddd66ab8-0417---.93706683.93706683.1:119]:
> ERROR: init sync on bucket failed, retcode=-5
> > 2023-05-09T15:24:54.652+ 7f197d7fa700  0 WARNING: curl
> operation timed out, network average transfer speed less than 1024 Bytes
> per second during 300 seconds.
> > 2023-05-09T15:27:05.725+ 7f197d7fa700  0 WARNING: curl
> operation timed out, network average transfer speed less than 1024 Bytes
> per second during 300 seconds.
> > -
> >
> > radosgw-admin bucket sync --bucket=ceph-bucket-prd info
> >   realm 98e0e391- (rgw-blobs)
> >   zonegroup 0e0faf4e- (WestEastCeph)
> >zone ddd66ab8- (rgw-east)
> >  bucket :ceph-bucket[ddd66ab8-.93706683.1])
> >
> > source zone b2a4a31c-
> >  bucket :ceph-bucket[ddd66ab8-.93706683.1])
> > root@bctlpmultceph01:~# radosgw-admin bucket sync
> --bucket=ceph-bucket status
> >   realm 98e0e391- (rgw-blobs)
> >   zonegroup 0e0faf4e- (WestEastCeph)
> >zone ddd66ab8- (rgw-east)
> >  bucket :ceph-bucket[ddd66ab8.93706683.1])
> >
> > source zone b2a4a31c- (rgw-west)
> >   source bucket :ceph-bucket[ddd66ab8-.93706683.1])
> > full sync: 0/120 shards
> > 

[ceph-users] Re: mds dump inode crashes file system

[Frank Schilder]

Dear Xiubo and others.

>> I have never heard about that option until now. How do I check that and how 
>> to I disable it if necessary?
>> I'm in meetings pretty much all day and will try to send some more info 
>> later.
>
> $ mount|grep ceph

I get

MON-IPs:SRC on DST type ceph 
(rw,relatime,name=con-fs2-rit-pfile,secret=,noshare,acl,mds_namespace=con-fs2,_netdev)

so async dirop seems disabled.

> Yeah, the kclient just received a corrupted snaptrace from MDS.
> So the first thing is you need to fix the corrupted snaptrace issue in cephfs 
> and then continue.

Ooookaaa. I will take it as a compliment that you seem to assume I know how 
to do that. The documentation gives 0 hits. Could you please provide me with 
instructions of what to look for and/or what to do first?

> If possible you can parse the above corrupted snap message to check what 
> exactly corrupted.
> I haven't get a chance to do that.

Again, how would I do that? Is there some documentation and what should I 
expect?

> You seems didn't enable the 'osd blocklist' cephx auth cap for mon:

I can't find anything about an osd blocklist client auth cap in the 
documentation. Is this something that came after octopus? Our caps are as shown 
in the documentation for a ceph fs client 
(https://docs.ceph.com/en/octopus/cephfs/client-auth/), the one for mon is 
"allow r":

caps mds = "allow rw path=/shares"
caps mon = "allow r"
caps osd = "allow rw tag cephfs data=con-fs2"


> I checked that but by reading the code I couldn't get what had cause the MDS 
> crash.
> There seems something wrong corrupt the metadata in cephfs.

He wrote something about an invalid xattrib (empty value). It would be really 
helpful to get a clue how to proceed. I managed to dump the MDS cache with the 
critical inode in cache. Would this help with debugging? I also managed to get 
debug logs with debug_mds=20 during a crash caused by an "mds dump inode" 
command. Would this contain something interesting? I can also pull the rados 
objects out and can upload all of these files.

I managed to track the problem down to a specific folder with a few files (I'm 
not sure if this coincides with the snaptrace issue, we might have 2 issues 
here). I made a copy of the folder and checked that an "mds dump inode" for the 
copy does not crash the MDS. I then moved the folders for which this command 
causes a crash to a different location outside the mounts. Do you think this 
will help? I'm wondering if after taking our daily snapshot tomorrow we end up 
in the degraded situation again.

I really need instructions for how to check what is broken without an MDS crash 
and then how to fix it.

Thanks and best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Xiubo Li 
Sent: Friday, May 12, 2023 8:04 AM
To: Frank Schilder; ceph-users@ceph.io
Subject: Re: [ceph-users] Re: mds dump inode crashes file system


On 5/11/23 20:12, Frank Schilder wrote:

Dear Xiubo,

please see also my previous e-mail about the async dirop config.

I have a bit more log output from dmesg on the file server here: 
https://pastebin.com/9Y0EPgDD .

  1.
[Wed May 10 16:03:06 2023] ceph: corrupt snap message from mds1
  2.
[Wed May 10 16:03:06 2023] header: : 05 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 
  3.
[Wed May 10 16:03:06 2023] header: 0010: 12 03 7f 00 01 00 00 01 00 00 00 
00 00 00 00 00 
  4.
[Wed May 10 16:03:06 2023] header: 0020: 00 00 00 00 02 01 00 00 00 00 00 
00 00 01 00 00 
  5.
[Wed May 10 16:03:06 2023] header: 0030: 00 98 0d 60 93 ...`.
  6.
[Wed May 10 16:03:06 2023] front: : 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 
  7.
[Wed May 10 16:03:06 2023] front: 0010: 0c 00 00 00 88 00 00 00 d1 c0 71 38 
00 01 00 00 ..q8
  8.
[Wed May 10 16:03:06 2023] front: 0020: 22 c8 71 38 00 01 00 00 d7 c7 71 38 
00 01 00 00 ".q8..q8
  9.
[Wed May 10 16:03:06 2023] front: 0030: d9 c7 71 38 00 01 00 00 d4 c7 71 38 
00 01 00 00 ..q8..q8
  10.
[Wed May 10 16:03:06 2023] front: 0040: f1 c0 71 38 00 01 00 00 d4 c0 71 38 
00 01 00 00 ..q8..q8
  11.
[Wed May 10 16:03:06 2023] front: 0050: 20 c8 71 38 00 01 00 00 1d c8 71 38 
00 01 00 00 .q8..q8
  12.
[Wed May 10 16:03:06 2023] front: 0060: ec c0 71 38 00 01 00 00 d6 c0 71 38 
00 01 00 00 ..q8..q8
  13.
[Wed May 10 16:03:06 2023] front: 0070: ef c0 71 38 00 01 00 00 6a 11 2d 1a 
00 01 00 00 ..q8j.-.
  14.
[Wed May 10 16:03:06 2023] front: 0080: 01 00 00 00 00 00 00 00 01 00 00 00 
00 00 00 00 
  15.
[Wed May 10 16:03:06 2023] front: 0090: ee 01 00 00 00 00 00 00 01 00 00 00 
00 00 00 00 
  16.
[Wed May 10 16:03:06 2023] front: 00a0: 00 00 00 00 00 00 00 00 01 00 00 00 
00 00 00 00 
  17.
[Wed May 10 16:03:06 2023]

[ceph-users] Re: mds dump inode crashes file system

[Xiubo Li]

On 5/12/23 20:27, Frank Schilder wrote:

Dear Xiubo and others.


I have never heard about that option until now. How do I check that and how to 
I disable it if necessary?
I'm in meetings pretty much all day and will try to send some more info later.

$ mount|grep ceph

I get

MON-IPs:SRC on DST type ceph 
(rw,relatime,name=con-fs2-rit-pfile,secret=,noshare,acl,mds_namespace=con-fs2,_netdev)

so async dirop seems disabled.


Yeah.



Yeah, the kclient just received a corrupted snaptrace from MDS.
So the first thing is you need to fix the corrupted snaptrace issue in cephfs 
and then continue.

Ooookaaa. I will take it as a compliment that you seem to assume I know how 
to do that. The documentation gives 0 hits. Could you please provide me with 
instructions of what to look for and/or what to do first?


There is no doc about this as I know.


If possible you can parse the above corrupted snap message to check what 
exactly corrupted.
I haven't get a chance to do that.

Again, how would I do that? Is there some documentation and what should I 
expect?


Currently there is no easy way to do this as I know, last time I have 
parsed the corrupted binary data to the corresponding message manully.


And then we could know what exactly has happened for the snaptrace.



You seems didn't enable the 'osd blocklist' cephx auth cap for mon:

I can't find anything about an osd blocklist client auth cap in the documentation. Is 
this something that came after octopus? Our caps are as shown in the documentation for a 
ceph fs client (https://docs.ceph.com/en/octopus/cephfs/client-auth/), the one for mon is 
"allow r":

 caps mds = "allow rw path=/shares"
 caps mon = "allow r"
 caps osd = "allow rw tag cephfs data=con-fs2"
Yeah, it seems the 'osd blocklist' was disabled. As I remembered if 
enabled it should be something likes:


caps mon = "allow r, allow command \"osd blocklist\""




I checked that but by reading the code I couldn't get what had cause the MDS 
crash.
There seems something wrong corrupt the metadata in cephfs.

He wrote something about an invalid xattrib (empty value). It would be really helpful to 
get a clue how to proceed. I managed to dump the MDS cache with the critical inode in 
cache. Would this help with debugging? I also managed to get debug logs with debug_mds=20 
during a crash caused by an "mds dump inode" command. Would this contain 
something interesting? I can also pull the rados objects out and can upload all of these 
files.


Yeah, possibly. Where is the logs ?



I managed to track the problem down to a specific folder with a few files (I'm not sure 
if this coincides with the snaptrace issue, we might have 2 issues here). I made a copy 
of the folder and checked that an "mds dump inode" for the copy does not crash 
the MDS. I then moved the folders for which this command causes a crash to a different 
location outside the mounts. Do you think this will help? I'm wondering if after taking 
our daily snapshot tomorrow we end up in the degraded situation again.

I really need instructions for how to check what is broken without an MDS crash 
and then how to fix it.


Firstly we need to know where the corrupted metadata is.

I think the mds debug logs and the above corrupted snaptrace could help. 
Need to parse that corrupted binary data.


Thanks



Thanks and best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Xiubo Li 
Sent: Friday, May 12, 2023 8:04 AM
To: Frank Schilder; ceph-users@ceph.io
Subject: Re: [ceph-users] Re: mds dump inode crashes file system


On 5/11/23 20:12, Frank Schilder wrote:

Dear Xiubo,

please see also my previous e-mail about the async dirop config.

I have a bit more log output from dmesg on the file server here: 
https://pastebin.com/9Y0EPgDD .

   1.
[Wed May 10 16:03:06 2023] ceph: corrupt snap message from mds1
   2.
[Wed May 10 16:03:06 2023] header: : 05 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 00 
   3.
[Wed May 10 16:03:06 2023] header: 0010: 12 03 7f 00 01 00 00 01 00 00 00 
00 00 00 00 00 
   4.
[Wed May 10 16:03:06 2023] header: 0020: 00 00 00 00 02 01 00 00 00 00 00 
00 00 01 00 00 
   5.
[Wed May 10 16:03:06 2023] header: 0030: 00 98 0d 60 93 ...`.
   6.
[Wed May 10 16:03:06 2023] front: : 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00 
   7.
[Wed May 10 16:03:06 2023] front: 0010: 0c 00 00 00 88 00 00 00 d1 c0 71 38 
00 01 00 00 ..q8
   8.
[Wed May 10 16:03:06 2023] front: 0020: 22 c8 71 38 00 01 00 00 d7 c7 71 38 00 
01 00 00 ".q8..q8
   9.
[Wed May 10 16:03:06 2023] front: 0030: d9 c7 71 38 00 01 00 00 d4 c7 71 38 
00 01 00 00 ..q8..q8
   10.
[Wed May 10 16:03:06 2023] front: 0040: f1 c0 71 38 00 01 00 00 d4 c0 71 38 
00 01 00 00 ..q8..q8
   11.
[Wed May 10 16:03:06 2023] front: 0050

[ceph-users] Telemetry and Redmine sync

[Yaarit Hatuka]

Hi everyone,

Over this weekend we will run a sync between telemetry crashes and Redmine
tracker issues.
This might affect your inbox, depending on your Redmine email notification
setup. You can set up filters for these emails to skip your inbox.

Thanks,
Yaarit
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MDS "newly corrupt dentry" after patch version upgrade

[Janek Bevendorff]

If is thrown while decoding the file name, then somebody probably 
managed to store files with non-UTF-8 characters in the name. Although I 
don't really know how this can happen. Perhaps some OS quirk.


On 10/05/2023 22:33, Patrick Donnelly wrote:

Hi Janek,

All this indicates is that you have some files with binary keys  that
cannot be decoded as utf-8. Unfortunately, the rados python library
assumes that omap keys can be decoded this way. I have a ticket here:

https://tracker.ceph.com/issues/59716

I hope to have a fix soon.

On Thu, May 4, 2023 at 3:15 AM Janek Bevendorff
 wrote:

After running the tool for 11 hours straight, it exited with the
following exception:

Traceback (most recent call last):
File "/home/webis/first-damage.py", line 156, in 
  traverse(f, ioctx)
File "/home/webis/first-damage.py", line 84, in traverse
  for (dnk, val) in it:
File "rados.pyx", line 1389, in rados.OmapIterator.__next__
File "rados.pyx", line 318, in rados.decode_cstr
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 8:
invalid start byte

Does that mean that the last inode listed in the output file is corrupt?
Any way I can fix it?

The output file has 14 million lines. We have about 24.5 million objects
in the metadata pool.

Janek


On 03/05/2023 14:20, Patrick Donnelly wrote:

On Wed, May 3, 2023 at 4:33 AM Janek Bevendorff
 wrote:

Hi Patrick,


I'll try that tomorrow and let you know, thanks!

I was unable to reproduce the crash today. Even with
mds_abort_on_newly_corrupt_dentry set to true, all MDS booted up
correctly (though they took forever to rejoin with logs set to 20).

To me it looks like the issue has resolved itself overnight. I had run a
recursive scrub on the file system and another snapshot was taken, in
case any of those might have had an effect on this. It could also be the
case that the (supposedly) corrupt journal entry has simply been
committed now and hence doesn't trigger the assertion any more. Is there
any way I can verify this?

You can run:

https://github.com/ceph/ceph/blob/main/src/tools/cephfs/first-damage.py

Just do:

python3 first-damage.py --memo run.1 

No need to do any of the other steps if you just want a read-only check.


--

Bauhaus-Universität Weimar
Bauhausstr. 9a, R308
99423 Weimar, Germany

Phone: +49 3643 58 3577
www.webis.de




--

Bauhaus-Universität Weimar
Bauhausstr. 9a, R308
99423 Weimar, Germany

Phone: +49 3643 58 3577
www.webis.de
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Quincy Ceph-orchestrator and multipath SAS

[Deep Dish]

Hello,

I have a few hosts about to add into a cluster that have a multipath
storage config for SAS devices.Is this supported on Quincy, and how
would ceph-orchestrator and / or ceph-volume handle multipath storage?

Here's a snip of lsblk output of a host in question:

# lsblk

NAME  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS

...

sdc 8:32   0   9.1T  0 disk

└─mpathh  253:40   9.1T  0 mpath

sdd 8:48   0   9.1T  0 disk

└─mpathi  253:50   9.1T  0 mpath

sde 8:64   0   7.3T  0 disk

└─mpathj  253:60   7.3T  0 mpath

sdf 8:80   0   7.3T  0 disk

└─mpathl  253:70   7.3T  0 mpath

sdg 8:96   0   7.2T  0 disk

└─mpathk  253:80   7.2T  0 mpath

sdh 8:112  0   7.3T  0 disk

└─mpathe  253:90   7.3T  0 mpath

sdi 8:128  0   7.3T  0 disk

└─mpathg  253:10   0   7.3T  0 mpath

sdj 8:144  0   7.3T  0 disk

└─mpathf  253:11   0   7.3T  0 mpath

sdk 8:160  0   7.3T  0 disk

└─mpathc  253:12   0   7.3T  0 mpath

sdl 8:176  0   7.3T  0 disk

└─mpathd  253:13   0   7.3T  0 mpath

...
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] [Pacific] ceph orch device ls do not returns any HDD

[Patrick Begou]

Hi everyone

I'm new to CEPH, just a french 4 days training session with Octopus on 
VMs that convince me to build my first cluster.


At this time I have 4 old identical nodes for testing with 3 HDDs each, 
2 network interfaces and running Alma Linux8 (el8). I try to replay the 
training session but it fails, breaking the web interface because of 
some problems with podman 4.2 not compatible with Octopus.


So I try to deploy Pacific with cephadm tool on my first node (mostha1) 
(to enable testing also an upgrade later).


   dnf -y install
   
https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noarch.rpm

   monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }')
   cephadm bootstrap --mon-ip $monip --initial-dashboard-password x \
  --initial-dashboard-user admceph \
  --allow-fqdn-hostname --cluster-network 10.1.0.0/16

This was sucessfull.

But running "*c**eph orch device ls*" do not show any HDD even if I have 
/dev/sda (used by the OS), /dev/sdb and /dev/sdc


The web interface shows a row capacity which is an aggregate of the 
sizes of the 3 HDDs for the node.


I've also tried to reset /dev/sdb but cephadm do not see it:

   [ceph: root@mostha1 /]# ceph orch device zap
   mostha1.legi.grenoble-inp.fr /dev/sdb --force
   Error EINVAL: Device path '/dev/sdb' not found on host
   'mostha1.legi.grenoble-inp.fr'

On my first attempt with octopus, I was able to list the available HDD 
with this command line. Before moving to Pacific, the OS on this node 
has been reinstalled from scratch.


Any advices for a CEPH beginner ?

Thanks

Patrick
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [EXTERNAL] [Pacific] ceph orch device ls do not returns any HDD

[Beaman, Joshua]

I don’t quite understand why that zap would not work.  But, here’s where I’d 
start.


  1.  cephadm check-host
 *   Run this on each of your hosts to make sure cephadm, podman and all 
other prerequisites are installed and recognized
  2.  ceph orch ls
 *   This should show at least a mon, mgr, and osd spec deployed
  3.  ceph orch ls osd –export
 *   This will show the OSD placement service specifications that 
orchestrator uses to identify devices to deploy as OSDs
  4.  ceph orch host ls
 *   This will list the hosts that have been added to orchestrator’s 
inventory, and what labels are applied which correlate to the service placement 
labels
  5.  ceph log last cephadm
 *   This will show you what orchestrator has been trying to do, and how it 
may be failing

Also, it’s never un-helpful to have a look at “ceph -s” and “ceph health 
detail”, particularly for any people trying to help you without access to your 
systems.

Best of luck,
Josh Beaman

From: Patrick Begou 
Date: Friday, May 12, 2023 at 10:45 AM
To: ceph-users 
Subject: [EXTERNAL] [ceph-users] [Pacific] ceph orch device ls do not returns 
any HDD
Hi everyone

I'm new to CEPH, just a french 4 days training session with Octopus on
VMs that convince me to build my first cluster.

At this time I have 4 old identical nodes for testing with 3 HDDs each,
2 network interfaces and running Alma Linux8 (el8). I try to replay the
training session but it fails, breaking the web interface because of
some problems with podman 4.2 not compatible with Octopus.

So I try to deploy Pacific with cephadm tool on my first node (mostha1)
(to enable testing also an upgrade later).

dnf -y install

https://urldefense.com/v3/__https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noarch.rpm__;!!CQl3mcHX2A!H9cwNCJyKXYQ4BbGA3gwHHRitjOS4lBCZT9wlnBZ-8IDue0MvdcPD8Dnv5yQCZw_eA4BNDYaEq1eouKQcQO7HshgdUJ0SJ-EgLfaBGBmCQ$

monip=$(getent ahostsv4 mostha1 |head -n 1| awk '{ print $1 }')
cephadm bootstrap --mon-ip $monip --initial-dashboard-password x \
   --initial-dashboard-user admceph \
   --allow-fqdn-hostname --cluster-network 10.1.0.0/16

This was sucessfull.

But running "*c**eph orch device ls*" do not show any HDD even if I have
/dev/sda (used by the OS), /dev/sdb and /dev/sdc

The web interface shows a row capacity which is an aggregate of the
sizes of the 3 HDDs for the node.

I've also tried to reset /dev/sdb but cephadm do not see it:

[ceph: root@mostha1 /]# ceph orch device zap
mostha1.legi.grenoble-inp.fr /dev/sdb --force
Error EINVAL: Device path '/dev/sdb' not found on host
'mostha1.legi.grenoble-inp.fr'

On my first attempt with octopus, I was able to list the available HDD
with this command line. Before moving to Pacific, the OS on this node
has been reinstalled from scratch.

Any advices for a CEPH beginner ?

Thanks

Patrick
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [EXTERNAL] [Pacific] ceph orch device ls do not returns any HDD

[Patrick Begou]

Hi Joshua and thanks for this quick reply.

At this step I have only one node. I was checking what ceph was 
returning with different commands on this host before adding new hosts. 
Just to compare with my first Octopus install. As this hardware is for 
testing only, it remains easy for me to break everything and reinstall 
again.


[root@mostha1 ~]# cephadm check-host

   podman (/usr/bin/podman) version 4.2.0 is present
   systemctl is present
   lvcreate is present
   Unit chronyd.service is enabled and running
   Host looks OK

[ceph: root@mostha1 /]# ceph -s

  cluster:
    id: 4b7a6504-f0be-11ed-be1a-00266cf8869c
    health: HEALTH_WARN
    OSD count 0 < osd_pool_default_size 3

  services:
    mon: 1 daemons, quorum mostha1.legi.grenoble-inp.fr (age 5h)
    mgr: mostha1.legi.grenoble-inp.fr.hogwuz(active, since 5h)
    osd: 0 osds: 0 up, 0 in

  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:

[ceph: root@mostha1 /]# ceph orch ls

   NAME   PORTS RUNNING  REFRESHED  AGE  PLACEMENT
   alertmanager   ?:9093,9094  1/1  6m ago 6h   count:1
   crash   1/1  6m ago 6h   *
   grafana    ?:3000   1/1  6m ago 6h   count:1
   mgr 1/2  6m ago 6h   count:2
   mon 1/5  6m ago 6h   count:5
   node-exporter  ?:9100   1/1  6m ago 6h   *
   prometheus ?:9095   1/1  6m ago 6h count:1

[ceph: root@mostha1 /]# ceph orch ls osd -export

   No services reported

[ceph: root@mostha1 /]# ceph orch host ls

   HOST ADDR   LABELS  STATUS
   mostha1.legi.grenoble-inp.fr 194.254.66.34  _admin
   1 hosts in cluster

[ceph: root@mostha1 /]# ceph log last cephadm

   ...
   2023-05-12T15:19:58.754655+
   mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1876 : cephadm
   [INF] Zap device mostha1.legi.grenoble-inp.fr:/dev/sdb
   2023-05-12T15:19:58.756639+
   mgr.mostha1.legi.grenoble-inp.fr.hogwuz (mgr.44098) 1877 : cephadm
   [ERR] Device path '/dev/sdb' not found on host
   'mostha1.legi.grenoble-inp.fr'
   Traceback (most recent call last):
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125,
   in wrapper
    return OrchResult(f(*args, **kwargs))
  File "/usr/share/ceph/mgr/cephadm/module.py", line 2275, in
   zap_device
    f"Device path '{path}' not found on host '{host}'")
   orchestrator._interface.OrchestratorError: Device path '/dev/sdb'
   not found on host 'mostha1.legi.grenoble-inp.fr'
   

[ceph: root@mostha1 /]# ls -l /dev/sdb

   brw-rw 1 root disk 8, 16 May 12 15:16 /dev/sdb

[ceph: root@mostha1 /]# lsblk /dev/sdb

   NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
   sdb  8:16   1 465.8G  0 disk
   `-sdb1   8:17   1 465.8G  0 part

I have crated a full partition on /dev/sdb (for testing) and /dev/sdc 
has no partition table (removed).


But all seams fine with these commands.

Patrick

Le 12/05/2023 à 20:19, Beaman, Joshua a écrit :


I don’t quite understand why that zap would not work.  But, here’s 
where I’d start.


 1. cephadm check-host
 1. Run this on each of your hosts to make sure cephadm, podman
and all other prerequisites are installed and recognized
 2. ceph orch ls
 1. This should show at least a mon, mgr, and osd spec deployed
 3. ceph orch ls osd –export
 1. This will show the OSD placement service specifications that
orchestrator uses to identify devices to deploy as OSDs
 4. ceph orch host ls
 1. This will list the hosts that have been added to
orchestrator’s inventory, and what labels are applied which
correlate to the service placement labels
 5. ceph log last cephadm
 1. This will show you what orchestrator has been trying to do,
and how it may be failing

Also, it’s never un-helpful to have a look at “ceph -s” and “ceph 
health detail”, particularly for any people trying to help you without 
access to your systems.


Best of luck,

Josh Beaman

*From: *Patrick Begou 
*Date: *Friday, May 12, 2023 at 10:45 AM
*To: *ceph-users 
*Subject: *[EXTERNAL] [ceph-users] [Pacific] ceph orch device ls do 
not returns any HDD


Hi everyone

I'm new to CEPH, just a french 4 days training session with Octopus on
VMs that convince me to build my first cluster.

At this time I have 4 old identical nodes for testing with 3 HDDs each,
2 network interfaces and running Alma Linux8 (el8). I try to replay the
training session but it fails, breaking the web interface because of
some problems with podman 4.2 not compatible with Octopus.

So I try to deploy Pacific with cephadm tool on my first node (mostha1)
(to enable testing also an upgrade later).

    dnf -y install
https://urldefense.com/v3/__https://download.ceph.com/rpm-16.2.13/el8/noarch/cephadm-16.2.13-0.el8.noarch.rpm__;!!CQl3mcHX2A!H9cwNCJyKXYQ4BbGA3gwHHRitjOS4lBCZT9wln

[ceph-users] Re: [EXTERNAL] [Pacific] ceph orch device ls do not returns any HDD

[Beaman, Joshua]

The most significant point I see there, is you have no OSD service spec to tell 
orchestrator how to deploy OSDs.  The easiest fix for that would be “ceph orch 
apply osd --all-available-devices”
This will create a simple spec that should work for a test environment.  Most 
likely it will collocate the block, block.db, and WAL all on the same device.  
Not ideal for prod environments, but fine for practice and testing.

The other command I should have had you try is “cephadm ceph-volume inventory”. 
 That should show you the devices available for OSD deployment, and hopefully 
matches up to what your “lsblk” shows.  If you need to zap HDDs and 
orchestrator is still not seeing them, you can try “cephadm ceph-volume lvm zap 
/dev/sdb”

Thank you,
Josh Beaman

From: Patrick Begou 
Date: Friday, May 12, 2023 at 2:22 PM
To: Beaman, Joshua , ceph-users 
Subject: Re: [EXTERNAL] [ceph-users] [Pacific] ceph orch device ls do not 
returns any HDD
Hi Joshua and thanks for this quick reply.

At this step I have only one node. I was checking what ceph was returning with 
different commands on this host before adding new hosts. Just to compare with 
my first Octopus install. As this hardware is for testing only, it remains easy 
for me to break everything and reinstall again.

[root@mostha1 ~]# cephadm check-host
podman (/usr/bin/podman) version 4.2.0 is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
[ceph: root@mostha1 /]# ceph -s
  cluster:
id: 4b7a6504-f0be-11ed-be1a-00266cf8869c
health: HEALTH_WARN
OSD count 0 < osd_pool_default_size 3

  services:
mon: 1 daemons, quorum mostha1.legi.grenoble-inp.fr (age 5h)
mgr: mostha1.legi.grenoble-inp.fr.hogwuz(active, since 5h)
osd: 0 osds: 0 up, 0 in

  data:
pools:   0 pools, 0 pgs
objects: 0 objects, 0 B
usage:   0 B used, 0 B / 0 B avail
pgs:
[ceph: root@mostha1 /]# ceph orch ls
NAME   PORTSRUNNING  REFRESHED  AGE  PLACEMENT
alertmanager   ?:9093,9094  1/1  6m ago 6h   count:1
crash   1/1  6m ago 6h   *
grafana?:3000   1/1  6m ago 6h   count:1
mgr 1/2  6m ago 6h   count:2
mon 1/5  6m ago 6h   count:5
node-exporter  ?:9100   1/1  6m ago 6h   *
prometheus ?:9095   1/1  6m ago 6h   count:1
[ceph: root@mostha1 /]# ceph orch ls osd -export
No services reported
[ceph: root@mostha1 /]# ceph orch host ls
HOST  ADDR   LABELS  STATUS
mostha1.legi.grenoble-inp.fr  194.254.66.34  _admin
1 hosts in cluster
[ceph: root@mostha1 /]# ceph log last cephadm
...
2023-05-12T15:19:58.754655+ mgr.mostha1.legi.grenoble-inp.fr.hogwuz 
(mgr.44098) 1876 : cephadm [INF] Zap device 
mostha1.legi.grenoble-inp.fr:/dev/sdb
2023-05-12T15:19:58.756639+ mgr.mostha1.legi.grenoble-inp.fr.hogwuz 
(mgr.44098) 1877 : cephadm [ERR] Device path '/dev/sdb' not found on host 
'mostha1.legi.grenoble-inp.fr'
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, in wrapper
return OrchResult(f(*args, **kwargs))
  File "/usr/share/ceph/mgr/cephadm/module.py", line 2275, in zap_device
f"Device path '{path}' not found on host '{host}'")
orchestrator._interface.OrchestratorError: Device path '/dev/sdb' not found on 
host 'mostha1.legi.grenoble-inp.fr'

[ceph: root@mostha1 /]# ls -l /dev/sdb
brw-rw 1 root disk 8, 16 May 12 15:16 /dev/sdb
[ceph: root@mostha1 /]# lsblk /dev/sdb
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sdb  8:16   1 465.8G  0 disk
`-sdb1   8:17   1 465.8G  0 part
I have crated a full partition on /dev/sdb (for testing) and /dev/sdc has no 
partition table (removed).

But all seams fine with these commands.

Patrick

Le 12/05/2023 à 20:19, Beaman, Joshua a écrit :
I don’t quite understand why that zap would not work.  But, here’s where I’d 
start.


  1.  cephadm check-host

 *   Run this on each of your hosts to make sure cephadm, podman and all 
other prerequisites are installed and recognized

  1.  ceph orch ls

 *   This should show at least a mon, mgr, and osd spec deployed

  1.  ceph orch ls osd –export

 *   This will show the OSD placement service specifications that 
orchestrator uses to identify devices to deploy as OSDs

  1.  ceph orch host ls

 *   This will list the hosts that have been added to orchestrator’s 
inventory, and what labels are applied which correlate to the service placement 
labels

  1.  ceph log last cephadm

 *   This will show you what orchestrator has been trying to do, and how it 
may be failing

Also, it’s never un-helpful to have a look at “ceph -s” and “ceph health 
detail”, particularly for any people trying to help you without access to your 
systems.

Best of luck,
Josh Beaman

From: Patrick Begou 

Date: Friday, May 12, 2023