[ceph-users] Re: Ceph filesystem

[William Edwards]

> Op 20 dec. 2022 om 08:39 heeft akshay sharma  het 
> volgende geschreven:
> 
> Ceph fs Authorize cephfs client.user /.
> Sudo mount -t vm:6789,vm2:6789:/ /mnt/cephfs -o name=user, secret=***
> 
> Now, I'm able to copy files from the same machine.. basically copy file
> from home to /mnt/cephfs is working but when copying from remote machine
> using SFTP or SCP to /mnt/cephfs is not working.
> 
> Are we missing something here?

Yes, an explanation of ‘not working’.

> 
>> On Tue, Dec 20, 2022, 7:56 AM Xiubo Li  wrote:
>> 
>> 
>>> On 19/12/2022 21:19, akshay sharma wrote:
>>> Hi All,
>>> 
>>> I have three Virtual machines with a dedicated disk for ceph, ceph
>> cluster
>>> is up as shown below
>>> 
>>> user@ubuntu:~/ceph-deploy$ sudo ceph status
>>> 
>>>   cluster:
>>> 
>>> id: 06a014a8-d166-4add-a21d-24ed52dce5c0
>>> 
>>> health: HEALTH_WARN
>>> 
>>> mons are allowing insecure global_id reclaim
>>> 
>>> clock skew detected on mon.ubuntu36, mon.ubuntu68
>>> 
>>> 
>>> 
>>>   services:
>>> 
>>> mon: 3 daemons, quorum ubuntu35,ubuntu36,ubuntu68 (age 10m)
>>> 
>>> mgr: ubuntu68(active, since 4m)
>>> 
>>> mds: 1/1 daemons up
>>> 
>>> osd: 3 osds: 3 up (since 5m), 3 in (since 5m)
>>> 
>>> 
>>> 
>>>   data:
>>> 
>>> volumes: 1/1 healthy
>>> 
>>> pools:   3 pools, 41 pgs
>>> 
>>> objects: 22 objects, 2.3 KiB
>>> 
>>> usage:   16 MiB used, 150 GiB / 150 GiB avail
>>> 
>>> pgs: 41 active+clean
>>> 
>>> 
>>> 
>>>   progress:
>>> 
>>> 
>>> 
>>> Note: deployed ceph cluster using ceph-deploy utility ..version 2.1.0
>>> 
>>> 
>>> 
>>> Out those three virtual machine, two machines are being used a client
>> also,
>>> using ceph posix filesystem to store data to the cluster.
>>> 
>>> 
>>> 
>>> followed following commands.
>>> 
>>> 
>>> 
>>> Ran below command on the main machine, where all commands or ceph-deploy
>> is
>>> installed.
>>> 
>>> 
>>> sudo ceph auth get-or-create client.user mon 'allow r' mds 'allow r,
>> allow
>>> rw path=/home/cephfs' osd 'allow rw pool=cephfs_data' -o
>>> /etc/ceph/ceph.client.user.keyring
>>> 
>> As Robert mentioned the 'path=' here should be the relative path from
>> the root of the cephfs instead of your local fs.
>> 
>> 
>>> Ran these two on the client.
>>> 
>>> sudo mkdir /mnt/mycephfs $ sudo mount -t ceph
>>> ubuntu1:6789,ubuntu2:6789,ubuntu3:6789:/ /mnt/mycephfs -o
>>> name=user,secret=AQBxnDFdS5atIxAAV0rL9klnSxwy6EFpR/EFbg==
>>> 
>> And you just created the mds auth caps for "/home/cephfs" path, but you
>> were mounting the '/' path with that caps.
>> 
>> Thanks
>> 
>> - Xiubo
>> 
>>> 
>>> After this when we are trying to right to the mount path../mnt/mycephfs..
>>> it is giving permission denied.
>>> 
>>> 
>>> How can we resolve this?
>>> 
>>> 
>>> I tried disabling cephx, but still ceph-deploy mon create-inital is
>> failing
>>> as key mon not found?
>>> ___
>>> ceph-users mailing list -- ceph-users@ceph.io
>>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>> 
>> 
>> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
> 

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph filesystem

[Robert Sander]

On 20.12.22 08:38, akshay sharma wrote:


Now, I'm able to copy files from the same machine.. basically copy file
from home to /mnt/cephfs is working but when copying from remote machine
using SFTP or SCP to /mnt/cephfs is not working.


What account are you using when locally copying files?
What account are you using when doing the same via SCP?

What POSIX access rights do these accounts have in the filesystem?

Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin

https://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Amtsgericht Berlin-Charlottenburg - HRB 220009 B
Geschäftsführer: Peer Heinlein - Sitz: Berlin

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph filesystem

[Robert Sander]

On 20.12.22 10:21, akshay sharma wrote:

With account you mean the user?


If yes, then we are using different user..the mds auth is created with 
client.user..


This is the cephx key that is only used when mounting the filesystem.

>
while copying we are login as user- test but locally with

user test we are able to
  The remote machine is also with user test.

ls -lrth
Drwxr-xr-x 1 root root mycephfs

Mount is changing the permission to root/root.


With an Unix account "test" you usually cannot write into a directory 
that is owned by "root" and only writable by "root".


You will need to chown or chgrp and chmod directories and/or files if 
you want to change them. This is basic POSIC permissions management.


Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin

https://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Amtsgericht Berlin-Charlottenburg - HRB 220009 B
Geschäftsführer: Peer Heinlein - Sitz: Berlin

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] MDS: mclientcaps(revoke), pending pAsLsXsFsc issued pAsLsXsFsc

[Stolte, Felix]

Hi guys,

i stumbled about these log entries in my active MDS on a pacific (16.2.10) 
cluster:

2022-12-20T10:06:52.124+0100 7f11ab408700  0 log_channel(cluster) log [WRN] : 
client.1207771517 isn't responding to mclientcaps(revoke), ino 0x10017e84452 
pending pAsLsXsFsc issued pAsLsXsFsc, sent 62.934160 seconds ago
2022-12-20T10:07:52.122+0100 7f11ab408700  0 log_channel(cluster) log [WRN] : 
client.1207771517 isn't responding to mclientcaps(revoke), ino 0x10017e84452 
pending pAsLsXsFsc issued pAsLsXsFsc, sent 122.936458 seconds ago
2022-12-20T10:09:52.190+0100 7f11ab408700  0 log_channel(cluster) log [WRN] : 
client.1207771517 isn't responding to mclientcaps(revoke), ino 0x10017e84452 
pending pAsLsXsFsc issued pAsLsXsFsc, sent 243.003124 seconds ago
2022-12-20T10:13:52.143+0100 7f11ab408700  0 log_channel(cluster) log [WRN] : 
client.1207771517 isn't responding to mclientcaps(revoke), ino 0x10017e84452 
pending pAsLsXsFsc issued pAsLsXsFsc, sent 482.952970 seconds ago
2022-12-20T10:21:52.156+0100 7f11ab408700  0 log_channel(cluster) log [WRN] : 
client.1207771517 isn't responding to mclientcaps(revoke), ino 0x10017e84452 
pending pAsLsXsFsc issued pAsLsXsFsc, sent 962.968165 seconds ago
2022-12-20T10:37:52.274+0100 7f11ab408700  0 log_channel(cluster) log [WRN] : 
client.1207771517 isn't responding to mclientcaps(revoke), ino 0x10017e84452 
pending pAsLsXsFsc issued pAsLsXsFsc, sent 1923.087405 seconds ago
2022-12-20T11:09:52.510+0100 7f11ab408700  0 log_channel(cluster) log [WRN] : 
client.1207771517 isn't responding to mclientcaps(revoke), ino 0x10017e84452 
pending pAsLsXsFsc issued pAsLsXsFsc, sent 3843.324342 seconds ago


issued and pending caps are the same. I found this issue:

https://tracker.ceph.com/issues/39349

Looks to me that it was fixed for versions prior to pacific, but is happening 
in pacific again (at least to me). Client is using cephfs kernelmount on 5.4 
kernel.

Any suggestions?


Regards
Felix

-
-
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior
-
-

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephfs ceph.dir.rctime decrease

[Stolte, Felix]

Hi Dan,

thank you for your info. Anything i can do to help to resolve the issue? A 
reliable rctime would be really great for backup purposes (even it would need 
to fix the time of the files causing the future rctime).

Regards
Felix
-
-
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior
-
-

Am 19.12.2022 um 11:20 schrieb Dan van der Ster :

Hi,

Yes this is a known issue -- an mtime can be in the future, and an
rctime won't go backwards. There was an earlier attempt to allow
fixing the rctimes but this got stuck and needs effort to bring it up
to date: https://github.com/ceph/ceph/pull/37938

Cheers, dan

On Sun, Dec 18, 2022 at 12:23 PM Stolte, Felix  wrote:

Hi guys,

i want to use ceph.dir.rctime for backup purposes. Unfortunately there are some 
files in our filesystem which have a ctime of years in the future. This is 
reflected correctly by ceph.dir.rctime. I changed the the time of this files to 
now (just did a touch on the file), but rctime stays the same. I waited one day 
and remounted the filesytem, but the value of rctime stays the same.

It is possible to update the ceph.dir.rctime in some way or is it hard coded, 
that rctime will never be decreased?

Regards
Felix

-
-
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior
-
-

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: MDS: mclientcaps(revoke), pending pAsLsXsFsc issued pAsLsXsFsc

[Xiubo Li]

On 20/12/2022 18:34, Stolte, Felix wrote:

Hi guys,

i stumbled about these log entries in my active MDS on a pacific (16.2.10) 
cluster:

2022-12-20T10:06:52.124+0100 7f11ab408700  0 log_channel(cluster) log [WRN] : 
client.1207771517 isn't responding to mclientcaps(revoke), ino 0x10017e84452 
pending pAsLsXsFsc issued pAsLsXsFsc, sent 62.934160 seconds ago
2022-12-20T10:07:52.122+0100 7f11ab408700  0 log_channel(cluster) log [WRN] : 
client.1207771517 isn't responding to mclientcaps(revoke), ino 0x10017e84452 
pending pAsLsXsFsc issued pAsLsXsFsc, sent 122.936458 seconds ago
2022-12-20T10:09:52.190+0100 7f11ab408700  0 log_channel(cluster) log [WRN] : 
client.1207771517 isn't responding to mclientcaps(revoke), ino 0x10017e84452 
pending pAsLsXsFsc issued pAsLsXsFsc, sent 243.003124 seconds ago
2022-12-20T10:13:52.143+0100 7f11ab408700  0 log_channel(cluster) log [WRN] : 
client.1207771517 isn't responding to mclientcaps(revoke), ino 0x10017e84452 
pending pAsLsXsFsc issued pAsLsXsFsc, sent 482.952970 seconds ago
2022-12-20T10:21:52.156+0100 7f11ab408700  0 log_channel(cluster) log [WRN] : 
client.1207771517 isn't responding to mclientcaps(revoke), ino 0x10017e84452 
pending pAsLsXsFsc issued pAsLsXsFsc, sent 962.968165 seconds ago
2022-12-20T10:37:52.274+0100 7f11ab408700  0 log_channel(cluster) log [WRN] : 
client.1207771517 isn't responding to mclientcaps(revoke), ino 0x10017e84452 
pending pAsLsXsFsc issued pAsLsXsFsc, sent 1923.087405 seconds ago
2022-12-20T11:09:52.510+0100 7f11ab408700  0 log_channel(cluster) log [WRN] : 
client.1207771517 isn't responding to mclientcaps(revoke), ino 0x10017e84452 
pending pAsLsXsFsc issued pAsLsXsFsc, sent 3843.324342 seconds ago


issued and pending caps are the same. I found this issue:

https://tracker.ceph.com/issues/39349

Looks to me that it was fixed for versions prior to pacific, but is happening 
in pacific again (at least to me). Client is using cephfs kernelmount on 5.4 
kernel.

Any suggestions?


The above tracker didn't fix it thoroughly, I have a new tracker to fix 
this and please see: https://tracker.ceph.com/issues/57244.


Thanks,

- Xiubo





Regards
Felix

-
-
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior
-
-

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Possible auth bug in quincy 17.2.5 on Ubuntu jammy

[Eugen Block]

Hi,

I can't really confirm your observation, I have a test cluster  
(running on openSUSE Leap) upgraded from N to Q a few weeks ago  
(17.2.3) and this worked fine:


nautilus:~ # ceph auth get-or-create client.cinder mgr 'profile rbd'  
mon 'profile rbd' osd 'profile rbd pool=cinder'
nautilus:~ # ceph auth export client.cinder -o  
/etc/ceph/ceph.client.cinder.keyring

nautilus:~ # cat /etc/ceph/ceph.client.cinder.keyring
[client.cinder]
key = AQC1kaFj6YVJHhAAloN9PknqzrpQ83prgWGl7g==
caps mgr = "profile rbd"
caps mon = "profile rbd"
caps osd = "profile rbd pool=cinder"

nautilus:~ # rbd --id cinder -k /etc/ceph/ceph.client.cinder.keyring  
-p cinder create --size 1024 cinder/test2
nautilus:~ # rbd --id cinder -k /etc/ceph/ceph.client.cinder.keyring  
-p cinder ls

test2

Then I upgraded to 17.2.5 a few minutes ago but it still works for me.  
Are you running the commands from within the /etc/ceph directory? And  
have you tried to run the same command with the full keyring path  
('-k')? Although I would have expected a different error message:


nautilus:~ # rbd --id cinder -k ceph.client.cinder.keyring -p cinder  
create --size 1024 cinder/test3
2022-12-20T12:09:06.736+0100 7f9680aca380 -1 auth: unable to find a  
keyring on ceph.client.cinder.keyring: (2) No such file or directory
2022-12-20T12:09:06.736+0100 7f9680aca380 -1  
AuthRegistry(0x560cc61b5f18) no keyring found at  
ceph.client.cinder.keyring, disabling cephx
2022-12-20T12:09:06.740+0100 7f9680aca380 -1 auth: unable to find a  
keyring on ceph.client.cinder.keyring: (2) No such file or directory
2022-12-20T12:09:06.740+0100 7f9680aca380 -1  
AuthRegistry(0x7ffefe11dcc0) no keyring found at  
ceph.client.cinder.keyring, disabling cephx

rbd: couldn't connect to the cluster!


Zitat von J-P Methot :


Hi,

I've upgraded to the latest quincy release using cephadm on my test  
cluster (Ubuntu jammy) and I'm running in a very peculiar issue  
regarding user authentication:


-I have a pool called "cinder-replicated" for storing RBDs (application: RBD)

-I have a user called cinder with the following authorization caps :

client.cinder
    key: [redacted]
    caps: [mgr] profile rbd
    caps: [mon] profile rbd
    caps: [osd] profile rbd pool=cinder-replicated, profile rbd  
pool=nova-meta, profile rbd pool=glance-meta, profile rbd  
pool=cinder-erasure, profile rbd pool=cinder-meta


-If I use the command "rbd -p cinder-replicated --id cinder -k  
ceph.client.cinder.keyring ls" I get a list of RBDs in the pool, as  
you would expect


-If I use the command "rbd create --id cinder -k  
ceph.client.cinder.keyring --size 1024 cinder-replicated/test2", I  
get "rbd: create error: (22) Invalid argument"


-If I use the command "rbd create --size 1024  
cinder-replicated/test2" which uses the admin user and keyring by  
default, I have no problem creating the RBD.


The fact that it works with the admin user and not with the cinder  
user makes me believe that it's an authentication issue. A possible  
cause could be that my client is on version 17.2.0 and my cluster is  
on 17.2.5, but there doesn't seem to be official jammy packages for  
17.2.5 yet. Also, the release notes don't indicate any change to  
ceph auth.


--
Jean-Philippe Méthot
Senior Openstack system administrator
Administrateur système Openstack sénior
PlanetHoster inc.

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io




___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph filesystem

[akshay sharma]

Got it Robert.

I tired changing the permission of the mount point after Mounting with
cephfs it is not allowing..it is being changed to root/root.

The mount path /mnt/mycephfs..is test:test before mounting or we tried with
nobody nogroup also..but after mounting (mount -t) the permission is
getting changed to root/root.

On Tue, Dec 20, 2022, 2:59 PM Robert Sander 
wrote:

> On 20.12.22 10:21, akshay sharma wrote:
> > With account you mean the user?
>
> > If yes, then we are using different user..the mds auth is created with
> > client.user..
>
> This is the cephx key that is only used when mounting the filesystem.
>
>  >
> while copying we are login as user- test but locally with
> > user test we are able to
> >   The remote machine is also with user test.
> >
> > ls -lrth
> > Drwxr-xr-x 1 root root mycephfs
> >
> > Mount is changing the permission to root/root.
>
> With an Unix account "test" you usually cannot write into a directory
> that is owned by "root" and only writable by "root".
>
> You will need to chown or chgrp and chmod directories and/or files if
> you want to change them. This is basic POSIC permissions management.
>
> Regards
> --
> Robert Sander
> Heinlein Consulting GmbH
> Schwedter Str. 8/9b, 10119 Berlin
>
> https://www.heinlein-support.de
>
> Tel: 030 / 405051-43
> Fax: 030 / 405051-19
>
> Amtsgericht Berlin-Charlottenburg - HRB 220009 B
> Geschäftsführer: Peer Heinlein - Sitz: Berlin
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Cluster problem - Quncy

[Deep Dish]

Hello.



I have a few issues with my ceph cluster:



- RGWs have disappeared from management (console does not register any
RGWs) despite showing 4 services deployed and processes running;

- All object buckets not accessible / manageable;

- Console showing some of my pools are “updating” – its been like this for
a few days;



What was done:



- Expanded a 30 OSD / 3 node cluster to 60 OSDs across 6 nodes;

- Changed the failure domain of our crush rules from OSD to host;

- Increased PG and PGP of main data pools to reflect additional cluster
OSDs and maintain a ~ 100 PG / OSD target;

- Attempted to redeploy / re-create service for RGWs / remove SSL config;

- A single OSD with very high ECC errors was found and preemptively removed
from the cluster.



The cluster took a few days rebalancing itself, and impression was it would
be done by now.   It’s not healthy, and as per above, RGWs are no longer
manageable – not sure where to start troubleshooting this as I’ve never
encountered such a scenario before.



Cluster specs:

-   6 OSD nodes (10 OSDs each);

-   5 Monitors;

-   2 Managers;

-   5 MDS;

-   4 RGWs;

-   Quincy 17.2.5;

-
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 16.2.11 pacific QE validation status

[Casey Bodley]

thanks Yuri, rgw approved based on today's results from
https://pulpito.ceph.com/yuriw-2022-12-20_15:27:49-rgw-pacific_16.2.11_RC2-distro-default-smithi/

On Mon, Dec 19, 2022 at 12:08 PM Yuri Weinstein  wrote:

> If you look at the pacific 16.2.8 QE validation history (
> https://tracker.ceph.com/issues/55356), we had pacific-x, nautilus-x, and
> pacific-p2p all green with one exception (
> https://tracker.ceph.com/issues/51652)
>
> Now we see so many failures in this point release with references to old
> issues.
>
> Is there anything we can fix to make them less "red"?
>
> Thx
> YuriW
>
> On Thu, Dec 15, 2022 at 2:56 PM Laura Flores  wrote:
>
>> I reviewed the upgrade runs:
>>
>>
>> https://pulpito.ceph.com/yuriw-2022-12-13_15:57:57-upgrade:nautilus-x-pacific_16.2.11_RC-distro-default-smithi/
>>
>> https://pulpito.ceph.com/yuriw-2022-12-13_21:47:46-upgrade:nautilus-x-pacific_16.2.11_RC-distro-default-smithi/
>>
>> https://pulpito.ceph.com/yuriw-2022-12-13_15:58:18-upgrade:octopus-x-pacific_16.2.11_RC-distro-default-smithi/
>>
>> https://pulpito.ceph.com/yuriw-2022-12-14_15:41:10-upgrade:octopus-x-pacific_16.2.11_RC-distro-default-smithi/
>>
>> Failures:
>>   1. https://tracker.ceph.com/issues/50618 -- known bug assigned to
>> Ilya; assuming it's not a big deal since it's been around for over a year
>>
>> Details:
>>   1. qemu_xfstests_luks1 failed on xfstest 168 - Ceph - RBD
>>
>>
>>
>> https://pulpito.ceph.com/yuriw-2022-12-13_15:58:24-upgrade:pacific-p2p-pacific_16.2.11_RC-distro-default-smithi/
>>
>> https://pulpito.ceph.com/yuriw-2022-12-14_15:40:37-upgrade:pacific-p2p-pacific_16.2.11_RC-distro-default-smithi/
>>
>> Failures, unrelated:
>>   1. https://tracker.ceph.com/issues/58223 -- new failure reported by me
>> 7 days ago; seems infrastructure related and not regression-related
>>   2. https://tracker.ceph.com/issues/52590 -- closed by Casey; must not
>> be of importance
>>   3. https://tracker.ceph.com/issues/58289 -- new failure raised by me
>> today; seems related to other "wait_for_recovery" failures, which are
>> generally not cause for concern since they're so infrequent.
>>   4. https://tracker.ceph.com/issues/51652 -- known bug from over a year
>> ago
>>
>> Details;
>>   1. failure on `sudo fuser -v /var/lib/dpkg/lock-frontend` -
>> Infrastructure
>>   2. "[ FAILED ] CmpOmap.cmp_vals_u64_invalid_default" in
>> upgrade:pacific-p2p-pacific - Ceph - RGW
>>   3. "AssertionError: wait_for_recovery: failed before timeout expired"
>> from down pg in pacific-p2p-pacific - Ceph - RADOS
>>   4. heartbeat timeouts on filestore OSDs while deleting objects in
>> upgrade:pacific-p2p-pacific - Ceph - RADOS
>>
>> On Thu, Dec 15, 2022 at 4:34 PM Brad Hubbard  wrote:
>>
>>> On Fri, Dec 16, 2022 at 3:15 AM Yuri Weinstein 
>>> wrote:
>>> >
>>> > Details of this release are summarized here:
>>> >
>>> > https://tracker.ceph.com/issues/58257#note-1
>>> > Release Notes - TBD
>>> >
>>> > Seeking approvals for:
>>> >
>>> > rados - Neha (https://github.com/ceph/ceph/pull/49431 is still being
>>> > tested and will be merged soon)
>>> > rook - Sébastien Han
>>> > cephadm - Adam
>>> > dashboard - Ernesto
>>> > rgw - Casey (rwg will be rerun on the latest SHA1)
>>> > rbd - Ilya, Deepika
>>> > krbd - Ilya, Deepika
>>> > fs - Venky, Patrick
>>> > upgrade/nautilus-x (pacific) - Neha, Laura
>>> > upgrade/octopus-x (pacific) - Neha, Laura
>>> > upgrade/pacific-p2p - Neha - Neha, Laura
>>> > powercycle - Brad
>>>
>>> The failure here is due to fallout from the recent lab issues and was
>>> fixed in main by https://github.com/ceph/ceph/pull/49021 I'm waiting
>>> to see if there are plans to backport this to pacific and quincy since
>>> that will be needed.
>>>
>>> > ceph-volume - Guillaume, Adam K
>>> >
>>> > Thx
>>> > YuriW
>>> >
>>> > ___
>>> > Dev mailing list -- d...@ceph.io
>>> > To unsubscribe send an email to dev-le...@ceph.io
>>>
>>>
>>>
>>> --
>>> Cheers,
>>> Brad
>>>
>>> ___
>>> ceph-users mailing list -- ceph-users@ceph.io
>>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>>
>>
>>
>> --
>>
>> Laura Flores
>>
>> She/Her/Hers
>>
>> Software Engineer, Ceph Storage
>>
>> Red Hat Inc. 
>>
>> Chicago, IL
>>
>> lflo...@redhat.com
>> M: +17087388804
>> @RedHat    Red Hat
>>   Red Hat
>> 
>> 
>>
>> ___
> Dev mailing list -- d...@ceph.io
> To unsubscribe send an email to dev-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: 16.2.11 pacific QE validation status

[Neha Ojha]

On Tue, Dec 20, 2022 at 11:41 AM Casey Bodley  wrote:

> thanks Yuri, rgw approved based on today's results from
>
> https://pulpito.ceph.com/yuriw-2022-12-20_15:27:49-rgw-pacific_16.2.11_RC2-distro-default-smithi/
>
> On Mon, Dec 19, 2022 at 12:08 PM Yuri Weinstein 
> wrote:
>
> > If you look at the pacific 16.2.8 QE validation history (
> > https://tracker.ceph.com/issues/55356), we had pacific-x, nautilus-x,
> and
> > pacific-p2p all green with one exception (
> > https://tracker.ceph.com/issues/51652)
> >
> > Now we see so many failures in this point release with references to old
> > issues.
> >
> > Is there anything we can fix to make them less "red"?
> >
> > Thx
> > YuriW
> >
> > On Thu, Dec 15, 2022 at 2:56 PM Laura Flores  wrote:
> >
> >> I reviewed the upgrade runs:
> >>
> >>
> >>
> https://pulpito.ceph.com/yuriw-2022-12-13_15:57:57-upgrade:nautilus-x-pacific_16.2.11_RC-distro-default-smithi/
> >>
> >>
> https://pulpito.ceph.com/yuriw-2022-12-13_21:47:46-upgrade:nautilus-x-pacific_16.2.11_RC-distro-default-smithi/
> >>
> >>
> https://pulpito.ceph.com/yuriw-2022-12-13_15:58:18-upgrade:octopus-x-pacific_16.2.11_RC-distro-default-smithi/
> >>
> >>
> https://pulpito.ceph.com/yuriw-2022-12-14_15:41:10-upgrade:octopus-x-pacific_16.2.11_RC-distro-default-smithi/
> >>
> >> Failures:
> >>   1. https://tracker.ceph.com/issues/50618 -- known bug assigned to
> >> Ilya; assuming it's not a big deal since it's been around for over a
> year
> >>
> >> Details:
> >>   1. qemu_xfstests_luks1 failed on xfstest 168 - Ceph - RBD
> >>
> >>
> >>
> >>
> https://pulpito.ceph.com/yuriw-2022-12-13_15:58:24-upgrade:pacific-p2p-pacific_16.2.11_RC-distro-default-smithi/
> >>
> >>
> https://pulpito.ceph.com/yuriw-2022-12-14_15:40:37-upgrade:pacific-p2p-pacific_16.2.11_RC-distro-default-smithi/
> >>
> >> Failures, unrelated:
> >>   1. https://tracker.ceph.com/issues/58223 -- new failure reported by
> me
> >> 7 days ago; seems infrastructure related and not regression-related
> >>   2. https://tracker.ceph.com/issues/52590 -- closed by Casey; must not
> >> be of importance
> >>   3. https://tracker.ceph.com/issues/58289 -- new failure raised by me
> >> today; seems related to other "wait_for_recovery" failures, which are
> >> generally not cause for concern since they're so infrequent.
> >>   4. https://tracker.ceph.com/issues/51652 -- known bug from over a
> year
> >> ago
> >>
> >> Details;
> >>   1. failure on `sudo fuser -v /var/lib/dpkg/lock-frontend` -
> >> Infrastructure
> >>   2. "[ FAILED ] CmpOmap.cmp_vals_u64_invalid_default" in
> >> upgrade:pacific-p2p-pacific - Ceph - RGW
> >>   3. "AssertionError: wait_for_recovery: failed before timeout expired"
> >> from down pg in pacific-p2p-pacific - Ceph - RADOS
> >>   4. heartbeat timeouts on filestore OSDs while deleting objects in
> >> upgrade:pacific-p2p-pacific - Ceph - RADOS
> >>
> >> On Thu, Dec 15, 2022 at 4:34 PM Brad Hubbard 
> wrote:
> >>
> >>> On Fri, Dec 16, 2022 at 3:15 AM Yuri Weinstein 
> >>> wrote:
> >>> >
> >>> > Details of this release are summarized here:
> >>> >
> >>> > https://tracker.ceph.com/issues/58257#note-1
> >>> > Release Notes - TBD
> >>> >
> >>> > Seeking approvals for:
> >>> >
> >>> > rados - Neha (https://github.com/ceph/ceph/pull/49431 is still being
>

We are still waiting on test runs for
https://github.com/ceph/ceph/pull/49431 to go through.

Thanks,
Neha


> >>> > tested and will be merged soon)
> >>> > rook - Sébastien Han
> >>> > cephadm - Adam
> >>> > dashboard - Ernesto
> >>> > rgw - Casey (rwg will be rerun on the latest SHA1)
> >>> > rbd - Ilya, Deepika
> >>> > krbd - Ilya, Deepika
> >>> > fs - Venky, Patrick
> >>> > upgrade/nautilus-x (pacific) - Neha, Laura
> >>> > upgrade/octopus-x (pacific) - Neha, Laura
> >>> > upgrade/pacific-p2p - Neha - Neha, Laura
> >>> > powercycle - Brad
> >>>
> >>> The failure here is due to fallout from the recent lab issues and was
> >>> fixed in main by https://github.com/ceph/ceph/pull/49021 I'm waiting
> >>> to see if there are plans to backport this to pacific and quincy since
> >>> that will be needed.
> >>>
> >>> > ceph-volume - Guillaume, Adam K
> >>> >
> >>> > Thx
> >>> > YuriW
> >>> >
> >>> > ___
> >>> > Dev mailing list -- d...@ceph.io
> >>> > To unsubscribe send an email to dev-le...@ceph.io
> >>>
> >>>
> >>>
> >>> --
> >>> Cheers,
> >>> Brad
> >>>
> >>> ___
> >>> ceph-users mailing list -- ceph-users@ceph.io
> >>> To unsubscribe send an email to ceph-users-le...@ceph.io
> >>>
> >>
> >>
> >> --
> >>
> >> Laura Flores
> >>
> >> She/Her/Hers
> >>
> >> Software Engineer, Ceph Storage
> >>
> >> Red Hat Inc. 
> >>
> >> Chicago, IL
> >>
> >> lflo...@redhat.com
> >> M: +17087388804
> >> @RedHat    Red Hat
> >>   Red Hat
> >> 
> >> 
> >>
> >> 

[ceph-users] Empty /var/lib/ceph/osd/ceph-$osd after reboot

[Isaiah Tang Yue Shun]

Hi all,

>From what I understand, after creating an OSD using  "ceph-volume lvm create", 
>we will do a "ceph-volume lvm activate" so that the systemd is created.

However, I found that after rebooting a host, some OSDs in the host will have 
empty /var/lib/ceph/osd/ceph-$osd
And I am not able to recover from there. Am I missing any steps?

I am running Oracle Linux 8.7, pacific release, ol8.

Thanks in advance.

Regards,
Isaiah Tang.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io