[ceph-users] Re: Diskprediction_local mgr module removal - Call for feedback
Hi. From our point of view, it's important to keep disk failure prediction tool as part of Ceph, ideally as an MGR module. In environments with hundreds or thousands of disks, it's crucial to know whether, for example, a significant number of them are likely to fail within a month - which, in the best-case scenario, would mean performance degradation, and in the worst-case, data loss. Some have already responded to the deprecation of diskprediction by starting to develop their own solutions. For instance, just yesterday, Daniel Persson published a solution [1] on his website that addresses the same problem. Would it be possible to join forces and try to revive that module? [1] https://www.youtube.com/watch?v=Gr_GtC9dcMQ Thanks, Michal On 4/8/25 01:18, Yaarit Hatuka wrote: Hi everyone, On today's Ceph Steering Committee call we discussed the idea of removing the diskprediction_local mgr module, as the current prediction model is obsolete and not maintained. We would like to gather feedback from the community about the usage of this module, and find out if anyone is interested in maintaining it. Thanks, Yaarit ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io smime.p7s Description: S/MIME Cryptographic Signature ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph squid fresh install
These are your two Luminous clients:
---snip---
{
"name": "unknown.0",
"entity_name": "client.admin",
"addrs": {
"addrvec": [
{
"type": "none",
"addr": "172.27.254.7:0",
"nonce": 443842330
}
]
},
"socket_addr": {
"type": "none",
"addr": "172.27.254.7:0",
"nonce": 443842330
},
"con_type": "client",
"con_features": 3387146417253690110,
"con_features_hex": "2f018fb87aa4aafe",
"con_features_release": "luminous",
...
{
"name": "client.104098",
"entity_name": "client.admin",
"addrs": {
"addrvec": [
{
"type": "v1",
"addr": "172.27.254.6:0",
"nonce": 2027668300
}
]
},
"socket_addr": {
"type": "v1",
"addr": "172.27.254.6:0",
"nonce": 2027668300
},
"con_type": "client",
"con_features": 3387146417253690110,
"con_features_hex": "2f018fb87aa4aafe",
"con_features_release": "luminous",
---snip---
Zitat von quag...@bol.com.br:
Hi Eugen! Thanks a lot! I was able to find luminous connections,
but I still can't identify which client process. Here is the output:
Rafael.
──
De: "Eugen Block" Enviada: 2025/04/08 04:37:47 Para: ceph-users@ceph.io
Assunto: [ceph-users] Re: Ceph squid fresh install Hi, you can query
the MON sessions to identify your older clients with: ceph tell mon.
sessions It will show you the IP address, con_features_release (Luminous)
and a couple of other things. Zitat von Laura Flores : > Hi Rafael, >> I
would not force the min_compat_client to be reef when there are still >
luminous clients connected, as it is important for all clients to be >=Reef
to understand/encode the pg_upmap_primary feature in the osdmap. >> As
for checking which processes are still luminous, I am copying @Radoslaw >
Zarzynski who may be able to help more with that. >> Thanks, > Laura
Flores >> On Mon, Apr 7, 2025 at 11:30 AM quag...@bol.com.br > wrote:
Hi, >> I just did a new Ceph installation and would like to enable the
"read >> balancer". >> However, the documentation requires that the minimum
client version >> be reef. I checked this information through "ceph
features" and came across >> the situation of having 2 luminous clients. >>
# ceph features >> { >> "mon": [ >> { >> "features": "0x3f03cffd",
"release": "squid", >> "num": 2 >> } >> ], >> "mds": [ >> { >>
"features": "0x3f03cffd", >> "release": "squid", >> "num": 2 >> }
], >> "osd": [ >> { >> "features": "0x3f03cffd", >> "release":
"squid", >> "num": 38 >> } >> ], >> "client": [ >> { >> "features":
"0x2f018fb87aa4aafe", >> "release": "luminous", >> "num": 2 >> }, >> { >>
"features": "0x3f03cffd", >> "release": "squid", >> "num": 5 >> }
], >> "mgr": [ >> { >> "features": "0x3f03cffd", >> "release":
"squid", >> "num": 2 >> } >> ] >> } I tryed to configure the minimum
version to reef and received the >> following alert: >> # ceph osd
set-require-min-compat-client reef >> Error EPERM: cannot set
require_min_compat_client to reef: 2 connected >> client(s) look like
luminous (missing 0x8000); add >> --yes-i-really-mean-it to do it
anyway Is it ok do confirm anyway? >> Which processes are still as
luminous? Rafael. >> ___
ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an
email to ceph-users-le...@ceph.io >> -- >> Laura Flores >> She/Her/Hers
Software Engineer, Ceph Storage >>> Chicago, IL >> lflo...@ibm.com |
lflo...@redhat.com > M: +17087388804 >
___ > ceph-users mailing list
-- ceph-users@ceph.io > To unsubscribe send an email to
ceph-users-le...@ceph.io ___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email
to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph squid fresh install
> On Apr 8, 2025, at 9:13 AM, quag...@bol.com.br wrote:
>
> These 2 IPs are from the storage servers.
What is a “storage server”?
> There are no user processes running on them. It only has the operating system
> and ceph installed.
Nobody said anything about user processes.
>
>
> Rafael.
>
> De: "Eugen Block"
> Enviada: 2025/04/08 09:35:35
> Para: quag...@bol.com.br
> Cc: ceph-users@ceph.io
> Assunto: Re: [ceph-users] Ceph squid fresh install
>
> These are your two Luminous clients:
>
> ---snip---
> {
> "name": "unknown.0",
> "entity_name": "client.admin",
> "addrs": {
> "addrvec": [
> {
> "type": "none",
> "addr": "172.27.254.7:0",
> "nonce": 443842330
> }
> ]
> },
> "socket_addr": {
> "type": "none",
> "addr": "172.27.254.7:0",
> "nonce": 443842330
> },
> "con_type": "client",
> "con_features": 3387146417253690110,
> "con_features_hex": "2f018fb87aa4aafe",
> "con_features_release": "luminous",
> ...
>
> {
> "name": "client.104098",
> "entity_name": "client.admin",
> "addrs": {
> "addrvec": [
> {
> "type": "v1",
> "addr": "172.27.254.6:0",
> "nonce": 2027668300
> }
> ]
> },
> "socket_addr": {
> "type": "v1",
> "addr": "172.27.254.6:0",
> "nonce": 2027668300
> },
> "con_type": "client",
> "con_features": 3387146417253690110,
> "con_features_hex": "2f018fb87aa4aafe",
> "con_features_release": "luminous",
> ---snip---
>
> Zitat von quag...@bol.com.br:
>
> > Hi Eugen! Thanks a lot! I was able to find luminous connections,
> > but I still can't identify which client process. Here is the output:
> > Rafael.
> > ──
> > De: "Eugen Block" Enviada: 2025/04/08 04:37:47 Para: ceph-users@ceph.io
> > Assunto: [ceph-users] Re: Ceph squid fresh install Hi, you can query
> > the MON sessions to identify your older clients with: ceph tell mon.
> > sessions It will show you the IP address, con_features_release (Luminous)
> > and a couple of other things. Zitat von Laura Flores : > Hi Rafael, >> I
> > would not force the min_compat_client to be reef when there are still >
> > luminous clients connected, as it is important for all clients to be >=Reef
> >> to understand/encode the pg_upmap_primary feature in the osdmap. >> As
> > for checking which processes are still luminous, I am copying @Radoslaw >
> > Zarzynski who may be able to help more with that. >> Thanks, > Laura
> > Flores >> On Mon, Apr 7, 2025 at 11:30 AM quag...@bol.com.br > wrote:
> Hi, >> I just did a new Ceph installation and would like to enable the
> > "read >> balancer". >> However, the documentation requires that the minimum
> > client version >> be reef. I checked this information through "ceph
> > features" and came across >> the situation of having 2 luminous clients. >>
> > # ceph features >> { >> "mon": [ >> { >> "features": "0x3f03cffd",
> >>> "release": "squid", >> "num": 2 >> } >> ], >> "mds": [ >> { >>
> > "features": "0x3f03cffd", >> "release": "squid", >> "num": 2 >> }
> >>> ], >> "osd": [ >> { >> "features": "0x3f03cffd", >> "release":
> > "squid", >> "num": 38 >> } >> ], >> "client": [ >> { >> "features":
> > "0x2f018fb87aa4aafe", >> "release": "luminous", >> "num": 2 >> }, >> { >>
> > "features": "0x3f03cffd", >> "release": "squid", >> "num": 5 >> }
> >>> ], >> "mgr": [ >> { >> "features": "0x3f03cffd", >> "release":
> > "squid", >> "num": 2 >> } >> ] >> } I tryed to configure the minimum
> > version to reef and received the >> following alert: >> # ceph osd
> > set-require-min-compat-client reef >> Error EPERM: cannot set
> > require_min_compat_client to reef: 2 connected >> client(s) look like
> > luminous (missing 0x8000); add >> --yes-i-really-mean-it to do it
> > anyway Is it ok do confirm anyway? >> Which processes are still as
> > luminous? Rafael. >> ___
> >>> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an
> > email to ceph-users-le...@ceph.io >> -- >> Laura Flores >> She/Her/Hers
> >>> Software Engineer, Ceph Storage >>> Chicago, IL >> lflo...@ibm.com |
> > lflo...@redhat.com > M: +17087388804 >
> > ___ > ceph-users mailing list
> > -- ceph-users@ceph.io > To unsubscribe send an email to
> > ceph-users-le...@ceph.io ___
> > ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email
> > to ceph-users-le...@ceph.io
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Diskprediction_local mgr module removal - Call for feedback
> Hi. > > From our point of view, it's important to keep disk failure prediction tool > as part of Ceph, ideally as an MGR module > . In environments with hundreds or thousands of disks, it's crucial to know > whether, for example, a significant number of them are likely to fail within > a month - which, in the best-case scenario, would mean performance > degradation, and in the worst-case, data loss. Agreed, but the current implementation does not achieve this. > Some have already responded to the deprecation of diskprediction by starting > to develop their own solutions. For instance, just yesterday, Daniel Persson > published a solution [1] Videos and costumes aren’t solutions. > on his website that addresses the same problem. > > Would it be possible to join forces Are you volunteering to code? Can you sign up to be involved going forward? Ceph has suffered time and again from something being partly implemented, only to have the sole proponent wander away. > and try to revive that module? It really needs to be reimplemented in a less opaque way so that admins can contribute and tweak. No binary blobs. > > [1] https://www.youtube.com/watch?v=Gr_GtC9dcMQ > > Thanks, > Michal > > > On 4/8/25 01:18, Yaarit Hatuka wrote: >> Hi everyone, >> On today's Ceph Steering Committee call we discussed the idea of removing >> the diskprediction_local mgr module, as the current prediction model is >> obsolete and not maintained. >> We would like to gather feedback from the community about the usage of this >> module, and find out if anyone is interested in maintaining it. >> Thanks, >> Yaarit >> ___ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-le...@ceph.io > > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph squid fresh install
These 2 IPs are from the storage servers.
There are no user processes running on them. It only has the operating system and ceph installed.
Rafael.
De: "Eugen Block"
Enviada: 2025/04/08 09:35:35
Para: quag...@bol.com.br
Cc: ceph-users@ceph.io
Assunto: Re: [ceph-users] Ceph squid fresh install
These are your two Luminous clients:
---snip---
{
"name": "unknown.0",
"entity_name": "client.admin",
"addrs": {
"addrvec": [
{
"type": "none",
"addr": "172.27.254.7:0",
"nonce": 443842330
}
]
},
"socket_addr": {
"type": "none",
"addr": "172.27.254.7:0",
"nonce": 443842330
},
"con_type": "client",
"con_features": 3387146417253690110,
"con_features_hex": "2f018fb87aa4aafe",
"con_features_release": "luminous",
...
{
"name": "client.104098",
"entity_name": "client.admin",
"addrs": {
"addrvec": [
{
"type": "v1",
"addr": "172.27.254.6:0",
"nonce": 2027668300
}
]
},
"socket_addr": {
"type": "v1",
"addr": "172.27.254.6:0",
"nonce": 2027668300
},
"con_type": "client",
"con_features": 3387146417253690110,
"con_features_hex": "2f018fb87aa4aafe",
"con_features_release": "luminous",
---snip---
Zitat von quag...@bol.com.br:
> Hi Eugen! Thanks a lot! I was able to find luminous connections,
> but I still can't identify which client process. Here is the output:
> Rafael.
> ──
> De: "Eugen Block" Enviada: 2025/04/08 04:37:47 Para: ceph-users@ceph.io
> Assunto: [ceph-users] Re: Ceph squid fresh install Hi, you can query
> the MON sessions to identify your older clients with: ceph tell mon.
> sessions It will show you the IP address, con_features_release (Luminous)
> and a couple of other things. Zitat von Laura Flores : > Hi Rafael, >> I
> would not force the min_compat_client to be reef when there are still >
> luminous clients connected, as it is important for all clients to be >=Reef
>> to understand/encode the pg_upmap_primary feature in the osdmap. >> As
> for checking which processes are still luminous, I am copying @Radoslaw >
> Zarzynski who may be able to help more with that. >> Thanks, > Laura
> Flores >> On Mon, Apr 7, 2025 at 11:30 AM quag...@bol.com.br > wrote:
Hi, >> I just did a new Ceph installation and would like to enable the
> "read >> balancer". >> However, the documentation requires that the minimum
> client version >> be reef. I checked this information through "ceph
> features" and came across >> the situation of having 2 luminous clients. >>
> # ceph features >> { >> "mon": [ >> { >> "features": "0x3f03cffd",
>>> "release": "squid", >> "num": 2 >> } >> ], >> "mds": [ >> { >>
> "features": "0x3f03cffd", >> "release": "squid", >> "num": 2 >> }
>>> ], >> "osd": [ >> { >> "features": "0x3f03cffd", >> "release":
> "squid", >> "num": 38 >> } >> ], >> "client": [ >> { >> "features":
> "0x2f018fb87aa4aafe", >> "release": "luminous", >> "num": 2 >> }, >> { >>
> "features": "0x3f03cffd", >> "release": "squid", >> "num": 5 >> }
>>> ], >> "mgr": [ >> { >> "features": "0x3f03cffd", >> "release":
> "squid", >> "num": 2 >> } >> ] >> } I tryed to configure the minimum
> version to reef and received the >> following alert: >> # ceph osd
> set-require-min-compat-client reef >> Error EPERM: cannot set
> require_min_compat_client to reef: 2 connected >> client(s) look like
> luminous (missing 0x8000); add >> --yes-i-really-mean-it to do it
> anyway Is it ok do confirm anyway? >> Which processes are still as
> luminous? Rafael. >> ___
>>> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an
> email to ceph-users-le...@ceph.io >> -- >> Laura Flores >> She/Her/Hers
>>> Software Engineer, Ceph Storage >>> Chicago, IL >> lflo...@ibm.com |
> lflo...@redhat.com > M: +17087388804 >
> ___ > ceph-users mailing list
> -- ceph-users@ceph.io > To unsubscribe send an email to
> ceph-users-le...@ceph.io ___
> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email
> to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: NIH Datasets
> Maybe one of the many existing projects could be adapted, re-formed, > re-aimed. Or maybe they're all dead in the water because they failed > one of more of the above five bullets. Often the unstated sixth bullet: — Has a supportable architecture The ongoing viability of a system should not rely on someone constantly having to monitor and manage contracts with thousands of high-touch low-savvy micro providers. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph squid fresh install
What is the locally installed ceph version? Run 'ceph --version' to
get the local rpm/deb package version.
Zitat von quag...@bol.com.br:
These 2 IPs are from the storage servers.
There are no user processes running on them. It only has the operating
system and ceph installed.
Rafael.
-
DE: "Eugen Block"
ENVIADA: 2025/04/08 09:35:35
PARA: quag...@bol.com.br
CC: ceph-users@ceph.io
ASSUNTO: Re: [ceph-users] Ceph squid fresh install
These are your two Luminous clients:
---snip---
{
"name": "unknown.0",
"entity_name": "client.admin",
"addrs": {
"addrvec": [
{
"type": "none",
"addr": "172.27.254.7:0",
"nonce": 443842330
}
]
},
"socket_addr": {
"type": "none",
"addr": "172.27.254.7:0",
"nonce": 443842330
},
"con_type": "client",
"con_features": 3387146417253690110,
"con_features_hex": "2f018fb87aa4aafe",
"con_features_release": "luminous",
...
{
"name": "client.104098",
"entity_name": "client.admin",
"addrs": {
"addrvec": [
{
"type": "v1",
"addr": "172.27.254.6:0",
"nonce": 2027668300
}
]
},
"socket_addr": {
"type": "v1",
"addr": "172.27.254.6:0",
"nonce": 2027668300
},
"con_type": "client",
"con_features": 3387146417253690110,
"con_features_hex": "2f018fb87aa4aafe",
"con_features_release": "luminous",
---snip---
Zitat von quag...@bol.com.br:
Hi Eugen! Thanks a lot! I was able to find luminous connections,
but I still can't identify which client process. Here is the output:
Rafael.
──
De: "Eugen Block" Enviada: 2025/04/08 04:37:47 Para: ceph-users@ceph.io
Assunto: [ceph-users] Re: Ceph squid fresh install Hi, you can query
the MON sessions to identify your older clients with: ceph tell mon.
sessions It will show you the IP address, con_features_release (Luminous)
and a couple of other things. Zitat von Laura Flores : > Hi Rafael, >> I
would not force the min_compat_client to be reef when there are still >
luminous clients connected, as it is important for all clients to be
=Reef
to understand/encode the pg_upmap_primary feature in the osdmap. >> As
for checking which processes are still luminous, I am copying @Radoslaw >
Zarzynski who may be able to help more with that. >> Thanks, > Laura
Flores >> On Mon, Apr 7, 2025 at 11:30 AM quag...@bol.com.br > wrote:
Hi, >> I just did a new Ceph installation and would like to enable the
"read >> balancer". >> However, the documentation requires that the
minimum
client version >> be reef. I checked this information through "ceph
features" and came across >> the situation of having 2 luminous clients.
# ceph features >> { >> "mon": [ >> { >> "features":
"0x3f03cffd",
"release": "squid", >> "num": 2 >> } >> ], >> "mds": [ >> { >>
"features": "0x3f03cffd", >> "release": "squid", >> "num": 2 >> }
], >> "osd": [ >> { >> "features": "0x3f03cffd", >> "release":
"squid", >> "num": 38 >> } >> ], >> "client": [ >> { >> "features":
"0x2f018fb87aa4aafe", >> "release": "luminous", >> "num": 2 >> }, >> { >>
"features": "0x3f03cffd", >> "release": "squid", >> "num": 5 >> }
], >> "mgr": [ >> { >> "features": "0x3f03cffd", >> "release":
"squid", >> "num": 2 >> } >> ] >> } I tryed to configure the minimum
version to reef and received the >> following alert: >> # ceph osd
set-require-min-compat-client reef >> Error EPERM: cannot set
require_min_compat_client to reef: 2 connected >> client(s) look like
luminous (missing 0x8000); add >> --yes-i-really-mean-it to do it
anyway Is it ok do confirm anyway? >> Which processes are still as
luminous? Rafael. >> ___
ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an
email to ceph-users-le...@ceph.io >> -- >> Laura Flores >>
She/Her/Hers
Software Engineer, Ceph Storage >>> Chicago, IL >> lflo...@ibm.com |
lflo...@redhat.com > M: +17087388804 >
___ > ceph-users mailing list
-- ceph-users@ceph.io > To unsubscribe send an email to
ceph-users-le...@ceph.io ___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
email
to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph squid fresh install
___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph squid fresh install
Hey Rafael,
So these client mounts probably use the kernel mounts with an older kernel
version that gets identified as the luminous client.
You can try upgrading the kernel version and remounting.
Laimis J.
> On 8 Apr 2025, at 16:13, quag...@bol.com.br wrote:
>
> These 2 IPs are from the storage servers.
> There are no user processes running on them. It only has the operating system
> and ceph installed.
>
>
> Rafael.
>
> De: "Eugen Block"
> Enviada: 2025/04/08 09:35:35
> Para: quag...@bol.com.br
> Cc: ceph-users@ceph.io
> Assunto: Re: [ceph-users] Ceph squid fresh install
>
> These are your two Luminous clients:
>
> ---snip---
> {
> "name": "unknown.0",
> "entity_name": "client.admin",
> "addrs": {
> "addrvec": [
> {
> "type": "none",
> "addr": "172.27.254.7:0",
> "nonce": 443842330
> }
> ]
> },
> "socket_addr": {
> "type": "none",
> "addr": "172.27.254.7:0",
> "nonce": 443842330
> },
> "con_type": "client",
> "con_features": 3387146417253690110,
> "con_features_hex": "2f018fb87aa4aafe",
> "con_features_release": "luminous",
> ...
>
> {
> "name": "client.104098",
> "entity_name": "client.admin",
> "addrs": {
> "addrvec": [
> {
> "type": "v1",
> "addr": "172.27.254.6:0",
> "nonce": 2027668300
> }
> ]
> },
> "socket_addr": {
> "type": "v1",
> "addr": "172.27.254.6:0",
> "nonce": 2027668300
> },
> "con_type": "client",
> "con_features": 3387146417253690110,
> "con_features_hex": "2f018fb87aa4aafe",
> "con_features_release": "luminous",
> ---snip---
>
> Zitat von quag...@bol.com.br:
>
> > Hi Eugen! Thanks a lot! I was able to find luminous connections,
> > but I still can't identify which client process. Here is the output:
> > Rafael.
> > ──
> > De: "Eugen Block" Enviada: 2025/04/08 04:37:47 Para: ceph-users@ceph.io
> > Assunto: [ceph-users] Re: Ceph squid fresh install Hi, you can query
> > the MON sessions to identify your older clients with: ceph tell mon.
> > sessions It will show you the IP address, con_features_release (Luminous)
> > and a couple of other things. Zitat von Laura Flores : > Hi Rafael, >> I
> > would not force the min_compat_client to be reef when there are still >
> > luminous clients connected, as it is important for all clients to be >=Reef
> >> to understand/encode the pg_upmap_primary feature in the osdmap. >> As
> > for checking which processes are still luminous, I am copying @Radoslaw >
> > Zarzynski who may be able to help more with that. >> Thanks, > Laura
> > Flores >> On Mon, Apr 7, 2025 at 11:30 AM quag...@bol.com.br > wrote:
> Hi, >> I just did a new Ceph installation and would like to enable the
> > "read >> balancer". >> However, the documentation requires that the minimum
> > client version >> be reef. I checked this information through "ceph
> > features" and came across >> the situation of having 2 luminous clients. >>
> > # ceph features >> { >> "mon": [ >> { >> "features": "0x3f03cffd",
> >>> "release": "squid", >> "num": 2 >> } >> ], >> "mds": [ >> { >>
> > "features": "0x3f03cffd", >> "release": "squid", >> "num": 2 >> }
> >>> ], >> "osd": [ >> { >> "features": "0x3f03cffd", >> "release":
> > "squid", >> "num": 38 >> } >> ], >> "client": [ >> { >> "features":
> > "0x2f018fb87aa4aafe", >> "release": "luminous", >> "num": 2 >> }, >> { >>
> > "features": "0x3f03cffd", >> "release": "squid", >> "num": 5 >> }
> >>> ], >> "mgr": [ >> { >> "features": "0x3f03cffd", >> "release":
> > "squid", >> "num": 2 >> } >> ] >> } I tryed to configure the minimum
> > version to reef and received the >> following alert: >> # ceph osd
> > set-require-min-compat-client reef >> Error EPERM: cannot set
> > require_min_compat_client to reef: 2 connected >> client(s) look like
> > luminous (missing 0x8000); add >> --yes-i-really-mean-it to do it
> > anyway Is it ok do confirm anyway? >> Which processes are still as
> > luminous? Rafael. >> ___
> >>> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an
> > email to ceph-users-le...@ceph.io >> -- >> Laura Flores >> She/Her/Hers
> >>> Software Engineer, Ceph Storage >>> Chicago, IL >> lflo...@ibm.com |
> > lflo...@redhat.com > M: +17087388804 >
> > ___ > ceph-users mailing list
> > -- ceph-users@ceph.io > To unsubscribe send an email to
> > ceph-users-le...@ceph.io ___
> > ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email
> > to ceph-users-le...@ceph.io
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph squid fresh install
___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph squid fresh install
What is a “storage server”?
These are machines that only have the operating system and ceph installed.
De: "Anthony D'Atri"
Enviada: 2025/04/08 10:19:08
Para: quag...@bol.com.br
Cc: ebl...@nde.ag, ceph-users@ceph.io
Assunto: Re: [ceph-users] Ceph squid fresh install
> On Apr 8, 2025, at 9:13 AM, quag...@bol.com.br wrote:
>
> These 2 IPs are from the storage servers.
What is a “storage server”?
> There are no user processes running on them. It only has the operating system and ceph installed.
Nobody said anything about user processes.
>
>
> Rafael.
>
> De: "Eugen Block"
> Enviada: 2025/04/08 09:35:35
> Para: quag...@bol.com.br
> Cc: ceph-users@ceph.io
> Assunto: Re: [ceph-users] Ceph squid fresh install
>
> These are your two Luminous clients:
>
> ---snip---
> {
> "name": "unknown.0",
> "entity_name": "client.admin",
> "addrs": {
> "addrvec": [
> {
> "type": "none",
> "addr": "172.27.254.7:0",
> "nonce": 443842330
> }
> ]
> },
> "socket_addr": {
> "type": "none",
> "addr": "172.27.254.7:0",
> "nonce": 443842330
> },
> "con_type": "client",
> "con_features": 3387146417253690110,
> "con_features_hex": "2f018fb87aa4aafe",
> "con_features_release": "luminous",
> ...
>
> {
> "name": "client.104098",
> "entity_name": "client.admin",
> "addrs": {
> "addrvec": [
> {
> "type": "v1",
> "addr": "172.27.254.6:0",
> "nonce": 2027668300
> }
> ]
> },
> "socket_addr": {
> "type": "v1",
> "addr": "172.27.254.6:0",
> "nonce": 2027668300
> },
> "con_type": "client",
> "con_features": 3387146417253690110,
> "con_features_hex": "2f018fb87aa4aafe",
> "con_features_release": "luminous",
> ---snip---
>
> Zitat von quag...@bol.com.br:
>
> > Hi Eugen! Thanks a lot! I was able to find luminous connections,
> > but I still can't identify which client process. Here is the output:
> > Rafael.
> > ──
> > De: "Eugen Block" Enviada: 2025/04/08 04:37:47 Para: ceph-users@ceph.io
> > Assunto: [ceph-users] Re: Ceph squid fresh install Hi, you can query
> > the MON sessions to identify your older clients with: ceph tell mon.
> > sessions It will show you the IP address, con_features_release (Luminous)
> > and a couple of other things. Zitat von Laura Flores : > Hi Rafael, >> I
> > would not force the min_compat_client to be reef when there are still >
> > luminous clients connected, as it is important for all clients to be >=Reef
> >> to understand/encode the pg_upmap_primary feature in the osdmap. >> As
> > for checking which processes are still luminous, I am copying @Radoslaw >
> > Zarzynski who may be able to help more with that. >> Thanks, > Laura
> > Flores >> On Mon, Apr 7, 2025 at 11:30 AM quag...@bol.com.br > wrote:
> Hi, >> I just did a new Ceph installation and would like to enable the
> > "read >> balancer". >> However, the documentation requires that the minimum
> > client version >> be reef. I checked this information through "ceph
> > features" and came across >> the situation of having 2 luminous clients. >>
> > # ceph features >> { >> "mon": [ >> { >> "features": "0x3f03cffd",
> >>> "release": "squid", >> "num": 2 >> } >> ], >> "mds": [ >> { >>
> > "features": "0x3f03cffd", >> "release": "squid", >> "num": 2 >> }
> >>> ], >> "osd": [ >> { >> "features": "0x3f03cffd", >> "release":
> > "squid", >> "num": 38 >> } >> ], >> "client": [ >> { >> "features":
> > "0x2f018fb87aa4aafe", >> "release": "luminous", >> "num": 2 >> }, >> { >>
> > "features": "0x3f03cffd", >> "release": "squid", >> "num": 5 >> }
> >>> ], >> "mgr": [ >> { >> "features": "0x3f03cffd", >> "release":
> > "squid", >> "num": 2 >> } >> ] >> } I tryed to configure the minimum
> > version to reef and received the >> following alert: >> # ceph osd
> > set-require-min-compat-client reef >> Error EPERM: cannot set
> > require_min_compat_client to reef: 2 connected >> client(s) look like
> > luminous (missing 0x8000); add >> --yes-i-really-mean-it to do it
> > anyway Is it ok do confirm anyway? >> Which processes are still as
> > luminous? Rafael. >> ___
> >>> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an
> > email to ceph-users-le...@ceph.io >> -- >> Laura Flores >> She/Her/Hers
> >>> Software Engineer, Ceph Storage >>> Chicago, IL >> lflo...@ibm.com |
> > lflo...@redhat.com > M: +17087388804 >
> > ___ > ceph-users mailing list
> > -- ceph-users@ceph.io > To unsubscribe send an email to
> > ceph-users-le...@ceph.io ___
> > ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email
> > to ceph-users-le...@ceph.io
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list --
[ceph-users] Re: Ceph squid fresh install
___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: NIH Datasets
Hi Linas, Is the intent of purging of this data mainly due to just cost concerns? If the goal is purely preservation of data, the likely cheapest and least maintenance intensive way of doing this is a large scale tape archive. Such archives (purely based on a google search) exist at LLNL and OU, and there is a TAPAS service from SpectraLogic. I would imagine questions would arise about custody of the data, legal implications etc. The easiest is for the organization already hosting the data to just preserve it by archiving, and thereby claim a significant cost reduction. -- Alex Gorbachev On Sun, Apr 6, 2025 at 11:08 PM Linas Vepstas wrote: > OK what you will read below might sound insane but I am obliged to ask. > > There are 275 petabytes of NIH data at risk of being deleted. Cancer > research, medical data, HIPAA type stuff. Currently unclear where it's > located, how it's managed, who has access to what, but lets ignore > that for now. It's presumably splattered across data centers, cloud, > AWS, supercomputing labs, who knows. Everywhere. > > I'm talking to a biomed person in Australias that uses NCBI data > daily, she's in talks w/ Australian govt to copy and preserve the > datasets they use. Some multi-petabytes of stuff. I don't know. > > While bouncing around tech ideas, IPFS and Ceph came up. My experience > with IPFS is that it's not a serious contender for anything. My > experience with Ceph is that it's more-or-less A-list. > > OK. So here's the question: is it possible to (has anyone tried) set > up an internet-wide Ceph cluster? Ticking off the typical checkboxes > for "decentralized storage"? Stuff, like: internet connections need to > be encrypted. Connections go down, come back up. Slow. Sure, national > labs may have multi-terabit fiber, but little itty-bitty participants > trying to contribute a small collection of disks to a large pool might > only have a gigabit connection, of which maybe 10% is "usable". > Barely. So, a hostile networking environment. > > Is this like, totally insane, run away now, can't do that, it won't > work idea, or is there some glimmer of hope? > > Am I misunderstanding something about IPFS that merits taking a second > look at it? > > Is there any other way of getting scalable reliable "decentralized" > internet-wide storage? > > I mean, yes, of course, the conventional answer is that it could be > copied to AWS or some national lab or two somewhere in the EU or Aus > or UK or where-ever, That's the "obvious" answer. I'm looking for a > non-obvious answer, an IPFS-like thing, but one that actually works. > Could it work? > > -- Linas > > > -- > Patrick: Are they laughing at us? > Sponge Bob: No, Patrick, they are laughing next to us. > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: NIH Datasets
The intent is the US administration’s assault against science, Linas doesn’t *want* to do it, he wants to preserve for the hope of a better future. > On Apr 8, 2025, at 9:28 AM, Alex Gorbachev wrote: > > Hi Linas, > > Is the intent of purging of this data mainly due to just cost concerns? If > the goal is purely preservation of data, the likely cheapest and least > maintenance intensive way of doing this is a large scale tape archive. > Such archives (purely based on a google search) exist at LLNL and OU, and > there is a TAPAS service from SpectraLogic. > > I would imagine questions would arise about custody of the data, legal > implications etc. The easiest is for the organization already hosting the > data to just preserve it by archiving, and thereby claim a significant cost > reduction. > > -- > Alex Gorbachev > > > > > On Sun, Apr 6, 2025 at 11:08 PM Linas Vepstas > wrote: > >> OK what you will read below might sound insane but I am obliged to ask. >> >> There are 275 petabytes of NIH data at risk of being deleted. Cancer >> research, medical data, HIPAA type stuff. Currently unclear where it's >> located, how it's managed, who has access to what, but lets ignore >> that for now. It's presumably splattered across data centers, cloud, >> AWS, supercomputing labs, who knows. Everywhere. >> >> I'm talking to a biomed person in Australias that uses NCBI data >> daily, she's in talks w/ Australian govt to copy and preserve the >> datasets they use. Some multi-petabytes of stuff. I don't know. >> >> While bouncing around tech ideas, IPFS and Ceph came up. My experience >> with IPFS is that it's not a serious contender for anything. My >> experience with Ceph is that it's more-or-less A-list. >> >> OK. So here's the question: is it possible to (has anyone tried) set >> up an internet-wide Ceph cluster? Ticking off the typical checkboxes >> for "decentralized storage"? Stuff, like: internet connections need to >> be encrypted. Connections go down, come back up. Slow. Sure, national >> labs may have multi-terabit fiber, but little itty-bitty participants >> trying to contribute a small collection of disks to a large pool might >> only have a gigabit connection, of which maybe 10% is "usable". >> Barely. So, a hostile networking environment. >> >> Is this like, totally insane, run away now, can't do that, it won't >> work idea, or is there some glimmer of hope? >> >> Am I misunderstanding something about IPFS that merits taking a second >> look at it? >> >> Is there any other way of getting scalable reliable "decentralized" >> internet-wide storage? >> >> I mean, yes, of course, the conventional answer is that it could be >> copied to AWS or some national lab or two somewhere in the EU or Aus >> or UK or where-ever, That's the "obvious" answer. I'm looking for a >> non-obvious answer, an IPFS-like thing, but one that actually works. >> Could it work? >> >> -- Linas >> >> >> -- >> Patrick: Are they laughing at us? >> Sponge Bob: No, Patrick, they are laughing next to us. >> ___ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-le...@ceph.io >> > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] squid 19.2.2 hotfix QE validation status
This is a hotfix release to address only one issue: https://github.com/ceph/ceph/pull/62711 and has limited testing. Details of this release are summarized here: https://tracker.ceph.com/issues/70822#note-1 Release Notes - TBD LRC upgrade - excluded from this release Gibba upgrade - excluded from this release smoke - Adam E, Casey approved? rgw - Adam E, Casey approved? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: NIH Datasets
I don't think Linus is only concerned with public data, no. The United States Government has had in places for many years effective means of preserving their data. Some of those systems may be old and creaky, granted, and not always the most efficient, but they suffice. The problem is that the current administration and their unelected henchmen are running rampant over them. Firing people with critical knowledge, and ordering the destruction of carefully amassed data. So what we're looking at is more akin to wikileaks than to simple data archival. That is, for the good of the nation - and the world - that data should be preserved, not purged. And yes, that might include obtaining information by shady means because the administration would rather see some data completely expunged in the same way that the Taliban destroyed the Buddha statues. A Ceph-like system would definitely be a good model, but for the sake of reliability and accessibility, we'd want something that could endure nodes popping in and dropping out, and to lessen the chances of a full-out invasion, that data would best be anonymously replicated over many nodes. Ceph loves very large datasets, so it should be sufficient were we to keep all our eggs in one basket. but again, that data has enemies, so extra measures are needful that Ceph wasn't designed to handle. I think such a project is doable, but it should borrow from many other community service architectures as well. It's not something that is currently available off-the-shelf, but neither were a lot of the technologies we now depend on every day. On 4/8/25 13:26, Alex Gorbachev wrote: I was trying to analyze the original request, which seems to be something of the following set: - The goal is to archive a large amount of (presumably public) data on a community run globally sharded or distributed storage. - Can Ceph be used for this? Seems no, at least not in a sense of running lots of OSDs at different locations by different people loosely coupled into one global public data repo. Perhaps, there are some other ideas from people who have done this kind of thing. - Are there restrictions on obtaining the data? If it's public and accessible now, it should be able to be copied. If not, what are the restrictions on obtaining and copying the data? - Organization: how will the storage and maintenance of data be organized (and funded)? A foundation, a SETI-at-home like network, a blockchain (to preserve data veracity)? - Legal support? -- Alex Gorbachev On Tue, Apr 8, 2025 at 9:41 AM Anthony D'Atri wrote: The intent is the US administration’s assault against science, Linas doesn’t *want* to do it, he wants to preserve for the hope of a better future. On Apr 8, 2025, at 9:28 AM, Alex Gorbachev wrote: Hi Linas, Is the intent of purging of this data mainly due to just cost concerns? If the goal is purely preservation of data, the likely cheapest and least maintenance intensive way of doing this is a large scale tape archive. Such archives (purely based on a google search) exist at LLNL and OU, and there is a TAPAS service from SpectraLogic. I would imagine questions would arise about custody of the data, legal implications etc. The easiest is for the organization already hosting the data to just preserve it by archiving, and thereby claim a significant cost reduction. -- Alex Gorbachev On Sun, Apr 6, 2025 at 11:08 PM Linas Vepstas wrote: OK what you will read below might sound insane but I am obliged to ask. There are 275 petabytes of NIH data at risk of being deleted. Cancer research, medical data, HIPAA type stuff. Currently unclear where it's located, how it's managed, who has access to what, but lets ignore that for now. It's presumably splattered across data centers, cloud, AWS, supercomputing labs, who knows. Everywhere. I'm talking to a biomed person in Australias that uses NCBI data daily, she's in talks w/ Australian govt to copy and preserve the datasets they use. Some multi-petabytes of stuff. I don't know. While bouncing around tech ideas, IPFS and Ceph came up. My experience with IPFS is that it's not a serious contender for anything. My experience with Ceph is that it's more-or-less A-list. OK. So here's the question: is it possible to (has anyone tried) set up an internet-wide Ceph cluster? Ticking off the typical checkboxes for "decentralized storage"? Stuff, like: internet connections need to be encrypted. Connections go down, come back up. Slow. Sure, national labs may have multi-terabit fiber, but little itty-bitty participants trying to contribute a small collection of disks to a large pool might only have a gigabit connection, of which maybe 10% is "usable". Barely. So, a hostile networking environment. Is this like, totally insane, run away now, can't do that, it won't work idea, or is there some glimmer of hope? Am I misunderstanding something about IPFS that merits taking a second look at it? Is
[ceph-users] irc/slack/discord bridge failing
Hi all,
Our IRC/Slack/Discord bridge started failing since ~2 days ago. Slack
deprecated the use of legacy bots[1], so at the moment, IRC/Discord is the
only bridge working.
[1]
~~~
time="2025-04-08T18:09:30Z" level=error msg="Could not retrieve channels:
slack.SlackErrorResponse{Err:"legacy_custom_bots_deprecated",
ResponseMetadata:slack.ResponseMetadata{Cursor:"", Messages:[]string(nil),
Warnings:[]string(nil)}}" prefix=slack
~~~
I'll keep you posted.
Cheers!
--
Alvaro Soto
*Note: My work hours may not be your work hours. Please do not feel the
need to respond during a time that is not convenient for you.*
--
Great people talk about ideas,
ordinary people talk about things,
small people talk... about other people.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log
I think it's just hard-coded in the cephadm binary [0]:
_log_file_handler = {
'level': 'DEBUG',
'class': 'logging.handlers.WatchedFileHandler',
'formatter': 'cephadm',
'filename': '%s/cephadm.log' % LOG_DIR,
}
I haven't looked to deep yet if it can be overridden, but didn't find
anything on a quick search.
[0]
https://github.com/ceph/ceph/blob/46ba365307312980303de311cacbb49974dbc74f/src/cephadm/cephadmlib/logging.py#L77
Zitat von Alex :
Hi everyone.
My company has paid Ceph support.
The tech is saying:
"...cephadm package of ceph 5 is having bug. So It generate debug logs
even its set for "info" logs..."
I have two clusters running, one Ceph 5 and the other Ceph 6 (Quincy).
Both of them are sending "DEBUG -" messages to cephadm.log.
I checked everywhere I can think of and all the logs are set to INFO.
Is there really a bug? I can't find any mention of one anywhere on
the internet,
besides, it seems that if there really is a bug it hasn't been fixed
in version 6 either.
I tried setting " mgr/cephadm/log_to_cluster_level" and
"mgr/cephadm/log_level"
to WARN and noting changed.
/etc/ceph/ceph.conf has no mention of log level.
Any help would be appreciated,
Thanks!
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log
Interesting. So it's like that for everybody? Meaning cephadm.log logs debug messages. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph squid fresh install
___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph squid fresh install
___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph squid fresh install
___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log
Awesome. Wish I knew that before I spent half a day trying to overwrite it. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph squid fresh install
Some of your messages come across without newlines which makes them hard to
read.
Try
ceph tell mds.0 client ls | grep kernel_version
and
ceph daemon mon.whatever sessions
> On Apr 7, 2025, at 4:34 PM, quag...@bol.com.br wrote:
>
> Hi Anthony, Thanks for your reply. I dont have any client connected yet. It's
> a fresh install. I only made a CephFS mount point on each machine that I
> installed ceph. How can I identify these clients? Can you point me? Rafael.
> ── De:
> "Anthony D'Atri" Enviada: 2025/04/07 17:11:50 Para: quag...@bol.com.br
> Assunto: Re: [ceph-users] Ceph squid fresh install Clients, whatever your
> clients are. RBD attachments, CephFS mounts. I’m guessing you have clients
> mounting CephFS with a really old kernel. > On Apr 7, 2025, at 12:29 PM,
> quag...@bol.com.br wrote: > > Hi, > I just did a new Ceph installation and
> would like to enable the "read balancer". > However, the documentation
> requires that the minimum client version be reef. I checked this information
> through "ceph features" and came across the situation of having 2 luminous
> clients. > # ceph features > { > "mon": [ > { > "features":
> "0x3f03cffd", > "release": "squid", > "num": 2 > } > ], > "mds": [ >
> { > "features": "0x3f03cffd", > "release": "squid", > "num": 2 > } >
> ], > "osd": [ > { > "features": "0x3f03cffd", > "release": "squid", >
> "num": 38 > } > ], > "client": [ > { > "features": "0x2f018fb87aa4aafe", >
> "release": "luminous", > "num": 2 > }, > { > "features":
> "0x3f03cffd", > "release": "squid", > "num": 5 > } > ], > "mgr": [ >
> { > "features": "0x3f03cffd", > "release": "squid", > "num": 2 > } >
> ] > } > > I tryed to configure the minimum version to reef and received the
> following alert: > # ceph osd set-require-min-compat-client reef > Error
> EPERM: cannot set require_min_compat_client to reef: 2 connected client(s)
> look like luminous (missing 0x8000); add --yes-i-really-mean-it to do it
> anyway > > Is it ok do confirm anyway? > Which processes are still as
> luminous? > > Rafael. > ___ >
> ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email
> to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] RBD Block alignment 16k for Databases
Hi! Is Block alignment for Databases a thing with RBD? I heard something with 16K alignment ... If yes how can be checked what alignment a RBD got. And yes I need to run my Database in Ceph as I have no other storage with redundandency. Hints for optimizing a single Block device for a Database are very welcome. Greetings filip ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: squid 19.2.2 hotfix QE validation status
On 08/04/2025, Yuri Weinstein wrote: > This is a hotfix release to address only one issue: > https://github.com/ceph/ceph/pull/62711 and has limited testing. > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/70822#note-1 > > Release Notes - TBD > LRC upgrade - excluded from this release > Gibba upgrade - excluded from this release > > smoke - Adam E, Casey approved? Approved! > rgw - Adam E, Casey approved? Approved! ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph squid fresh install
Hi all,
This most likely boils down to the kernel version of connected clients. We
faced the same problem some time ago and just kicked away the client mds
connections with a follow up kernel update on the servers doing the mount.
I can check the version details later for more clarity.
Best,
Laimis J.
On Tue, Apr 8, 2025, 16:48 Anthony D'Atri wrote:
> So, OSD nodes that do not have any OSDs on them yet?
>
> > On Apr 8, 2025, at 9:41 AM, quag...@bol.com.br wrote:
> >
> > More complete description:
> >
> > 1-) I formatted and installed the operating system
> >
> > 2-) This is "ceph installed":
> >
> > curl --silent --remote-name --location
> https://download.ceph.com/rpm-19.2.1/el9/noarch/cephadm
> > chmod +x cephadm
> >
> > ./cephadm add-repo --release squid
> > ./cephadm install
> >
> > cephadm -v bootstrap --mon-ip 172.27.254.6 --cluster-network
> 172.28.254.0/24 --log-to-file
> >
> > cephadm install ceph-common
> >
> >
> > De: "Anthony D'Atri"
> > Enviada: 2025/04/08 10:35:22
> > Para: quag...@bol.com.br
> > Cc: ebl...@nde.ag, ceph-users@ceph.io
> > Assunto: Re: [ceph-users] Ceph squid fresh install
> >
> > What does “ceph installed” mean? I suspect that this description is not
> complete.
> >
> >>
> >> On Apr 8, 2025, at 9:21 AM, quag...@bol.com.br wrote:
> >>
> >> What is a “storage server”?
> >> These are machines that only have the operating system and
> ceph installed.
> >>
> >>
> >>
> >> De: "Anthony D'Atri"
> >> Enviada: 2025/04/08 10:19:08
> >> Para: quag...@bol.com.br
> >> Cc: ebl...@nde.ag, ceph-users@ceph.io
> >> Assunto: Re: [ceph-users] Ceph squid fresh install
> >>
> >>
> >>
> >> > On Apr 8, 2025, at 9:13 AM, quag...@bol.com.br wrote:
> >> >
> >> > These 2 IPs are from the storage servers.
> >>
> >> What is a “storage server”?
> >>
> >>
> >> > There are no user processes running on them. It only has the
> operating system and ceph installed.
> >>
> >> Nobody said anything about user processes.
> >>
> >> >
> >> >
> >> > Rafael.
> >> >
> >> > De: "Eugen Block"
> >> > Enviada: 2025/04/08 09:35:35
> >> > Para: quag...@bol.com.br
> >> > Cc: ceph-users@ceph.io
> >> > Assunto: Re: [ceph-users] Ceph squid fresh install
> >> >
> >> > These are your two Luminous clients:
> >> >
> >> > ---snip---
> >> > {
> >> > "name": "unknown.0",
> >> > "entity_name": "client.admin",
> >> > "addrs": {
> >> > "addrvec": [
> >> > {
> >> > "type": "none",
> >> > "addr": "172.27.254.7:0",
> >> > "nonce": 443842330
> >> > }
> >> > ]
> >> > },
> >> > "socket_addr": {
> >> > "type": "none",
> >> > "addr": "172.27.254.7:0",
> >> > "nonce": 443842330
> >> > },
> >> > "con_type": "client",
> >> > "con_features": 3387146417253690110,
> >> > "con_features_hex": "2f018fb87aa4aafe",
> >> > "con_features_release": "luminous",
> >> > ...
> >> >
> >> > {
> >> > "name": "client.104098",
> >> > "entity_name": "client.admin",
> >> > "addrs": {
> >> > "addrvec": [
> >> > {
> >> > "type": "v1",
> >> > "addr": "172.27.254.6:0",
> >> > "nonce": 2027668300
> >> > }
> >> > ]
> >> > },
> >> > "socket_addr": {
> >> > "type": "v1",
> >> > "addr": "172.27.254.6:0",
> >> > "nonce": 2027668300
> >> > },
> >> > "con_type": "client",
> >> > "con_features": 3387146417253690110,
> >> > "con_features_hex": "2f018fb87aa4aafe",
> >> > "con_features_release": "luminous",
> >> > ---snip---
> >> >
> >> > Zitat von quag...@bol.com.br:
> >> >
> >> > > Hi Eugen! Thanks a lot! I was able to find luminous connections,
> >> > > but I still can't identify which client process. Here is the output:
> >> > > Rafael.
> >> > > ──
> >> > > De: "Eugen Block" Enviada: 2025/04/08 04:37:47 Para:
> ceph-users@ceph.io
> >> > > Assunto: [ceph-users] Re: Ceph squid fresh install Hi, you can query
> >> > > the MON sessions to identify your older clients with: ceph tell mon.
> >> > > sessions It will show you the IP address, con_features_release
> (Luminous)
> >> > > and a couple of other things. Zitat von Laura Flores : > Hi Rafael,
> >> I
> >> > > would not force the min_compat_client to be reef when there are
> still >
> >> > > luminous clients connected, as it is important for all clients to
> be >=Reef
> >> > >> to understand/encode the pg_upmap_primary feature in the osdmap.
> >> As
> >> > > for checking which processes are still luminous, I am copying
> @Radoslaw >
> >> > > Zarzynski who may be able to help more with that. >> Thanks, > Laura
> >> > > Flores >> On Mon, Apr 7, 2025 at 11:30 AM quag...@bol.com.br >
> wrote:
> >> > Hi, >> I just did a new Ceph installation and would like to
> enable the
> >> > > "read >> balancer". >> However, the documentation requires that the
> minimum
> >> > > client version >> be reef. I checked this information through "ceph
> >> > > features" and came across >> the situation of having 2 luminous
> clients. >>
> >> > > # ceph features >> { >> "mon": [ >> { >> "features":
> "0x3f03cffd",
> >> > >>> "release": "squid", >> "num": 2 >> } >> ], >
[ceph-users] Re: NIH Datasets
If you have a line on the data, I have connections that can store it or I can consult pro bono on building a system to store it. However wan-ceph is not the answer here. On Sun, Apr 6, 2025, 11:08 PM Linas Vepstas wrote: > OK what you will read below might sound insane but I am obliged to ask. > > There are 275 petabytes of NIH data at risk of being deleted. Cancer > research, medical data, HIPAA type stuff. Currently unclear where it's > located, how it's managed, who has access to what, but lets ignore > that for now. It's presumably splattered across data centers, cloud, > AWS, supercomputing labs, who knows. Everywhere. > > I'm talking to a biomed person in Australias that uses NCBI data > daily, she's in talks w/ Australian govt to copy and preserve the > datasets they use. Some multi-petabytes of stuff. I don't know. > > While bouncing around tech ideas, IPFS and Ceph came up. My experience > with IPFS is that it's not a serious contender for anything. My > experience with Ceph is that it's more-or-less A-list. > > OK. So here's the question: is it possible to (has anyone tried) set > up an internet-wide Ceph cluster? Ticking off the typical checkboxes > for "decentralized storage"? Stuff, like: internet connections need to > be encrypted. Connections go down, come back up. Slow. Sure, national > labs may have multi-terabit fiber, but little itty-bitty participants > trying to contribute a small collection of disks to a large pool might > only have a gigabit connection, of which maybe 10% is "usable". > Barely. So, a hostile networking environment. > > Is this like, totally insane, run away now, can't do that, it won't > work idea, or is there some glimmer of hope? > > Am I misunderstanding something about IPFS that merits taking a second > look at it? > > Is there any other way of getting scalable reliable "decentralized" > internet-wide storage? > > I mean, yes, of course, the conventional answer is that it could be > copied to AWS or some national lab or two somewhere in the EU or Aus > or UK or where-ever, That's the "obvious" answer. I'm looking for a > non-obvious answer, an IPFS-like thing, but one that actually works. > Could it work? > > -- Linas > > > -- > Patrick: Are they laughing at us? > Sponge Bob: No, Patrick, they are laughing next to us. > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Diskprediction_local mgr module removal - Call for feedback
I’ve done the same as well. It doesn’t help that smartctl doesn’t even try to have consistent output for SATA, SAS, and NVMe drives, and especially that it doesn’t enforce uniform attribute labels. Yes, the fundamental problem is that SMART is mechanism without policy, and inconsistently implemented, but smartctl doesn’t help like it could. I’m working on a post processor for derived.h to at least debulk the madness, e.g. casting all of the SSD life remaining/used metrics into the same name, and adding a primitive to subtract from 100. If upstream won’t take that addition, I guess I’ll be forking smartmoxntools in toto. I had hopes for smartctl_exporter, but have given up on that project as too disorganized and contentious. So the last time I did this, I looked up the details of all 76 drive SKUs in my fleet and hardcoded every one into my collection script. Many systems cheap out and just use the overall pass/fail SMART health status attribute, which is very prone to reporting that clearly failing drives are just smurfy, and thus worthless. This is what BMCs seem to do, for example. Grown defects - on a spinner RMA or shred it if there are say more than 1 per 2TB. SATA downshift errors, though these can be HBA issues as well. UDMA/CRC errors - silently slow, but can be addressed by reseating in most cases, and using OEM carriers for SATA/SAS drives. Rate of reallocated blocks, alert if it isn’t quite slow Since Nautilus we are much less prone to grown errors then we used to be, the OSD will retry a failed write, so a remapped LBA will succeed the second time. There is IIRC a limit to how many of these are tolerated, but it does underscore the need to look deeper. Similarly one can alert on drives with high latency via primal query. Oh we were talking about the module. I vote to remove it. Note that the RH model is binary blobs as well as the ProphetStor model. Incomplete and impossible to maintain in the current state. It would be terrific to have a fully normalized and maintained / maintainable metric and prediction subsystem, but I haven’t seen anyone step up. It would be too much for me to do myself, especially without access to hardware, and I fear that without multiple people involved we won’t have continuity and it’ll lapse again. If at least one SWE with staying power steps up I’d be willing to entertain an idea for reimplementation eschewing opaque madness. anthonydatri@Mac models % pwd /Users/anthonydatri/git/ceph/src/pybind/mgr/diskprediction_local/models anthonydatri@Mac models % file redhat/* redhat/config.json: JSON data redhat/hgst_predictor.pkl:data redhat/hgst_scaler.pkl: data redhat/seagate_predictor.pkl: data redhat/seagate_scaler.pkl:data anthonydatri@Mac models % > On Apr 8, 2025, at 5:30 AM, Lukasz Borek wrote: > > +1 > > I wasn't aware that this module is obsolete and was trying to start it a > few weeks ago. > > We develop a home-made solution some time ago to monitor smart data from > both HDD (uncorrected errors, grown defect list) and SSD (WLC/TBW). But > keeping it up to date with non-unified disk models is a nightmare. > > Alert : "OSD.12 is going to fail. Replace it soon" before seeing SLOW_OPS > would be a game changer! > > Thanks! > > On Tue, 8 Apr 2025 at 10:00, Michal Strnad wrote: > >> Hi. >> >> From our point of view, it's important to keep disk failure prediction >> tool as part of Ceph, ideally as an MGR module. In environments with >> hundreds or thousands of disks, it's crucial to know whether, for >> example, a significant number of them are likely to fail within a month >> - which, in the best-case scenario, would mean performance degradation, >> and in the worst-case, data loss. >> >> Some have already responded to the deprecation of diskprediction by >> starting to develop their own solutions. For instance, just yesterday, >> Daniel Persson published a solution [1] on his website that addresses >> the same problem. >> >> Would it be possible to join forces and try to revive that module? >> >> [1] https://www.youtube.com/watch?v=Gr_GtC9dcMQ >> >> Thanks, >> Michal >> >> >> On 4/8/25 01:18, Yaarit Hatuka wrote: >>> Hi everyone, >>> >>> On today's Ceph Steering Committee call we discussed the idea of removing >>> the diskprediction_local mgr module, as the current prediction model is >>> obsolete and not maintained. >>> >>> We would like to gather feedback from the community about the usage of >> this >>> module, and find out if anyone is interested in maintaining it. >>> >>> Thanks, >>> Yaarit >>> ___ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-le...@ceph.io >> >> >> ___ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-le...@ceph.io >> > > > -- > Łukasz Borek > luk...@borek.org.pl >
[ceph-users] Re: Ceph squid fresh install
So, OSD nodes that do not have any OSDs on them yet?
> On Apr 8, 2025, at 9:41 AM, quag...@bol.com.br wrote:
>
> More complete description:
>
> 1-) I formatted and installed the operating system
>
> 2-) This is "ceph installed":
>
> curl --silent --remote-name --location
> https://download.ceph.com/rpm-19.2.1/el9/noarch/cephadm
> chmod +x cephadm
>
> ./cephadm add-repo --release squid
> ./cephadm install
>
> cephadm -v bootstrap --mon-ip 172.27.254.6 --cluster-network 172.28.254.0/24
> --log-to-file
>
> cephadm install ceph-common
>
>
> De: "Anthony D'Atri"
> Enviada: 2025/04/08 10:35:22
> Para: quag...@bol.com.br
> Cc: ebl...@nde.ag, ceph-users@ceph.io
> Assunto: Re: [ceph-users] Ceph squid fresh install
>
> What does “ceph installed” mean? I suspect that this description is not
> complete.
>
>>
>> On Apr 8, 2025, at 9:21 AM, quag...@bol.com.br wrote:
>>
>> What is a “storage server”?
>> These are machines that only have the operating system and ceph
>> installed.
>>
>>
>>
>> De: "Anthony D'Atri"
>> Enviada: 2025/04/08 10:19:08
>> Para: quag...@bol.com.br
>> Cc: ebl...@nde.ag, ceph-users@ceph.io
>> Assunto: Re: [ceph-users] Ceph squid fresh install
>>
>>
>>
>> > On Apr 8, 2025, at 9:13 AM, quag...@bol.com.br wrote:
>> >
>> > These 2 IPs are from the storage servers.
>>
>> What is a “storage server”?
>>
>>
>> > There are no user processes running on them. It only has the operating
>> > system and ceph installed.
>>
>> Nobody said anything about user processes.
>>
>> >
>> >
>> > Rafael.
>> >
>> > De: "Eugen Block"
>> > Enviada: 2025/04/08 09:35:35
>> > Para: quag...@bol.com.br
>> > Cc: ceph-users@ceph.io
>> > Assunto: Re: [ceph-users] Ceph squid fresh install
>> >
>> > These are your two Luminous clients:
>> >
>> > ---snip---
>> > {
>> > "name": "unknown.0",
>> > "entity_name": "client.admin",
>> > "addrs": {
>> > "addrvec": [
>> > {
>> > "type": "none",
>> > "addr": "172.27.254.7:0",
>> > "nonce": 443842330
>> > }
>> > ]
>> > },
>> > "socket_addr": {
>> > "type": "none",
>> > "addr": "172.27.254.7:0",
>> > "nonce": 443842330
>> > },
>> > "con_type": "client",
>> > "con_features": 3387146417253690110,
>> > "con_features_hex": "2f018fb87aa4aafe",
>> > "con_features_release": "luminous",
>> > ...
>> >
>> > {
>> > "name": "client.104098",
>> > "entity_name": "client.admin",
>> > "addrs": {
>> > "addrvec": [
>> > {
>> > "type": "v1",
>> > "addr": "172.27.254.6:0",
>> > "nonce": 2027668300
>> > }
>> > ]
>> > },
>> > "socket_addr": {
>> > "type": "v1",
>> > "addr": "172.27.254.6:0",
>> > "nonce": 2027668300
>> > },
>> > "con_type": "client",
>> > "con_features": 3387146417253690110,
>> > "con_features_hex": "2f018fb87aa4aafe",
>> > "con_features_release": "luminous",
>> > ---snip---
>> >
>> > Zitat von quag...@bol.com.br:
>> >
>> > > Hi Eugen! Thanks a lot! I was able to find luminous connections,
>> > > but I still can't identify which client process. Here is the output:
>> > > Rafael.
>> > > ──
>> > > De: "Eugen Block" Enviada: 2025/04/08 04:37:47 Para: ceph-users@ceph.io
>> > > Assunto: [ceph-users] Re: Ceph squid fresh install Hi, you can query
>> > > the MON sessions to identify your older clients with: ceph tell mon.
>> > > sessions It will show you the IP address, con_features_release (Luminous)
>> > > and a couple of other things. Zitat von Laura Flores : > Hi Rafael, >> I
>> > > would not force the min_compat_client to be reef when there are still >
>> > > luminous clients connected, as it is important for all clients to be
>> > > >=Reef
>> > >> to understand/encode the pg_upmap_primary feature in the osdmap. >> As
>> > > for checking which processes are still luminous, I am copying @Radoslaw >
>> > > Zarzynski who may be able to help more with that. >> Thanks, > Laura
>> > > Flores >> On Mon, Apr 7, 2025 at 11:30 AM quag...@bol.com.br > wrote:
>> > Hi, >> I just did a new Ceph installation and would like to enable the
>> > > "read >> balancer". >> However, the documentation requires that the
>> > > minimum
>> > > client version >> be reef. I checked this information through "ceph
>> > > features" and came across >> the situation of having 2 luminous clients.
>> > > >>
>> > > # ceph features >> { >> "mon": [ >> { >> "features":
>> > > "0x3f03cffd",
>> > >>> "release": "squid", >> "num": 2 >> } >> ], >> "mds": [ >> { >>
>> > > "features": "0x3f03cffd", >> "release": "squid", >> "num": 2 >> }
>> > >>> ], >> "osd": [ >> { >> "features": "0x3f03cffd", >> "release":
>> > > "squid", >> "num": 38 >> } >> ], >> "client": [ >> { >> "features":
>> > > "0x2f018fb87aa4aafe", >> "release": "luminous", >> "num": 2 >> }, >> { >>
>> > > "features": "0x3f03cffd", >> "release": "squid", >> "num": 5 >> }
>> > >>> ], >> "mgr": [ >> { >> "features": "0x3f03cffd", >> "release":
>> > > "squid", >> "num": 2 >> } >> ] >> } I tryed to configure the minimum
>> > > v
[ceph-users] Re: File immutability on CephFS ?
Hi Fabien, [cc Milind (who owns the tracker ticket)] On Tue, Apr 8, 2025 at 8:16 PM Fabien Sirjean wrote: > > Dear all, > > If I am to believe this issue [1], it seems that it is still not > possible to make files immutable (chattr +i) in cephfs. That's correct. > > Do you have any update on this matter ? Unfortunately, no. The tracker ticket hasn't been prioritized and no progress has been made. Will see what can be done to have it implemented. Thanks for the nudge on this. > > Thanks a lot for all the good work! > > Cheers, > > Fabien > > [1] : https://tracker.ceph.com/issues/10679 > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > -- Cheers, Venky ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Use cephfs when cephfs-data-scan cleanup is not finished
Hi Jiří, On Mon, Apr 7, 2025 at 8:36 AM Jiří Župka wrote: > > Dear conference, >I would like to ask if is secure use cephfs when cephfs-data-scan > cleanup is not > finished yet? You could interrupt the cleanup task and use the file system. BTW, you could also speed up the cleanup process by running it via multiple worker threads as with other data scan commands. > > Best regards, >Jiri > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > -- Cheers, Venky ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Diskprediction_local mgr module removal - Call for feedback
Hi, It's will be very nice, if this module will be removed. Everything that Ceph operator need can be covered via smartctl_exporter [1] Thanks, k [1] https://github.com/prometheus-community/smartctl_exporter Sent from my iPhone > On 8 Apr 2025, at 02:20, Yaarit Hatuka wrote: > > We would like to gather feedback from the community about the usage of this > module, and find out if anyone is interested in maintaining it. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log
I guess it's debatable as almost everything. ;-) One of the advantages is that you usually see immediately what is failing, you don't have to turn on debug first, retry the deployment or whatever again to reproduce. The file doesn't really grow to huge sizes (around 2 MB per day or so) and gets logrotated. Most people seem okay with it, at least I haven't seen many complaints. But of course, you're always welcome to propose a feature or PR. Zitat von Alex : Thanks Eugen! I think you're right since support had me grep for the same code. Seems crazy though that it's hardcoded doesn't it? I guess we can mod the Python file, but you'd think that wouldn't be necessary. Should we make a feature request or modify the code ourselves and make a pull request? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: NIH Datasets
I was trying to analyze the original request, which seems to be something of the following set: - The goal is to archive a large amount of (presumably public) data on a community run globally sharded or distributed storage. - Can Ceph be used for this? Seems no, at least not in a sense of running lots of OSDs at different locations by different people loosely coupled into one global public data repo. Perhaps, there are some other ideas from people who have done this kind of thing. - Are there restrictions on obtaining the data? If it's public and accessible now, it should be able to be copied. If not, what are the restrictions on obtaining and copying the data? - Organization: how will the storage and maintenance of data be organized (and funded)? A foundation, a SETI-at-home like network, a blockchain (to preserve data veracity)? - Legal support? -- Alex Gorbachev On Tue, Apr 8, 2025 at 9:41 AM Anthony D'Atri wrote: > The intent is the US administration’s assault against science, Linas > doesn’t *want* to do it, he wants to preserve for the hope of a better > future. > > > > On Apr 8, 2025, at 9:28 AM, Alex Gorbachev > wrote: > > > > Hi Linas, > > > > Is the intent of purging of this data mainly due to just cost concerns? > If > > the goal is purely preservation of data, the likely cheapest and least > > maintenance intensive way of doing this is a large scale tape archive. > > Such archives (purely based on a google search) exist at LLNL and OU, and > > there is a TAPAS service from SpectraLogic. > > > > I would imagine questions would arise about custody of the data, legal > > implications etc. The easiest is for the organization already hosting > the > > data to just preserve it by archiving, and thereby claim a significant > cost > > reduction. > > > > -- > > Alex Gorbachev > > > > > > > > > > On Sun, Apr 6, 2025 at 11:08 PM Linas Vepstas > > wrote: > > > >> OK what you will read below might sound insane but I am obliged to ask. > >> > >> There are 275 petabytes of NIH data at risk of being deleted. Cancer > >> research, medical data, HIPAA type stuff. Currently unclear where it's > >> located, how it's managed, who has access to what, but lets ignore > >> that for now. It's presumably splattered across data centers, cloud, > >> AWS, supercomputing labs, who knows. Everywhere. > >> > >> I'm talking to a biomed person in Australias that uses NCBI data > >> daily, she's in talks w/ Australian govt to copy and preserve the > >> datasets they use. Some multi-petabytes of stuff. I don't know. > >> > >> While bouncing around tech ideas, IPFS and Ceph came up. My experience > >> with IPFS is that it's not a serious contender for anything. My > >> experience with Ceph is that it's more-or-less A-list. > >> > >> OK. So here's the question: is it possible to (has anyone tried) set > >> up an internet-wide Ceph cluster? Ticking off the typical checkboxes > >> for "decentralized storage"? Stuff, like: internet connections need to > >> be encrypted. Connections go down, come back up. Slow. Sure, national > >> labs may have multi-terabit fiber, but little itty-bitty participants > >> trying to contribute a small collection of disks to a large pool might > >> only have a gigabit connection, of which maybe 10% is "usable". > >> Barely. So, a hostile networking environment. > >> > >> Is this like, totally insane, run away now, can't do that, it won't > >> work idea, or is there some glimmer of hope? > >> > >> Am I misunderstanding something about IPFS that merits taking a second > >> look at it? > >> > >> Is there any other way of getting scalable reliable "decentralized" > >> internet-wide storage? > >> > >> I mean, yes, of course, the conventional answer is that it could be > >> copied to AWS or some national lab or two somewhere in the EU or Aus > >> or UK or where-ever, That's the "obvious" answer. I'm looking for a > >> non-obvious answer, an IPFS-like thing, but one that actually works. > >> Could it work? > >> > >> -- Linas > >> > >> > >> -- > >> Patrick: Are they laughing at us? > >> Sponge Bob: No, Patrick, they are laughing next to us. > >> ___ > >> ceph-users mailing list -- ceph-users@ceph.io > >> To unsubscribe send an email to ceph-users-le...@ceph.io > >> > > ___ > > ceph-users mailing list -- ceph-users@ceph.io > > To unsubscribe send an email to ceph-users-le...@ceph.io > > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log
Can someone paste in here their copy of logrotate? The trick always with rotating logs is that the service writing to it needs to be restarted or told to stop writing so the file handle gets closed. Otherwise it stays open and the free disc space isn't recovered. ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log
It's the same across all versions I have used so far. Zitat von Alex : What about Pacific and Quincy? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log
What about Pacific and Quincy? ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Diskprediction_local mgr module removal - Call for feedback
Hi, Am 4/8/25 um 14:30 schrieb Anthony D'Atri: anthonydatri@Mac models % pwd /Users/anthonydatri/git/ceph/src/pybind/mgr/diskprediction_local/models anthonydatri@Mac models % file redhat/* redhat/config.json: JSON data redhat/hgst_predictor.pkl:data redhat/hgst_scaler.pkl: data redhat/seagate_predictor.pkl: data redhat/seagate_scaler.pkl:data anthonydatri@Mac models % These are Python pickle files from 2019 containing ML models made with a version of sklearn from 2019. It really looks like this part of the Ceph project is dead. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] File immutability on CephFS ?
Dear all, If I am to believe this issue [1], it seems that it is still not possible to make files immutable (chattr +i) in cephfs. Do you have any update on this matter ? Thanks a lot for all the good work! Cheers, Fabien [1] : https://tracker.ceph.com/issues/10679 ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph squid fresh install
What does “ceph installed” mean? I suspect that this description is not
complete.
> On Apr 8, 2025, at 9:21 AM, quag...@bol.com.br wrote:
>
> What is a “storage server”?
> These are machines that only have the operating system and ceph
> installed.
>
>
>
> De: "Anthony D'Atri"
> Enviada: 2025/04/08 10:19:08
> Para: quag...@bol.com.br
> Cc: ebl...@nde.ag, ceph-users@ceph.io
> Assunto: Re: [ceph-users] Ceph squid fresh install
>
>
>
> > On Apr 8, 2025, at 9:13 AM, quag...@bol.com.br wrote:
> >
> > These 2 IPs are from the storage servers.
>
> What is a “storage server”?
>
>
> > There are no user processes running on them. It only has the operating
> > system and ceph installed.
>
> Nobody said anything about user processes.
>
> >
> >
> > Rafael.
> >
> > De: "Eugen Block"
> > Enviada: 2025/04/08 09:35:35
> > Para: quag...@bol.com.br
> > Cc: ceph-users@ceph.io
> > Assunto: Re: [ceph-users] Ceph squid fresh install
> >
> > These are your two Luminous clients:
> >
> > ---snip---
> > {
> > "name": "unknown.0",
> > "entity_name": "client.admin",
> > "addrs": {
> > "addrvec": [
> > {
> > "type": "none",
> > "addr": "172.27.254.7:0",
> > "nonce": 443842330
> > }
> > ]
> > },
> > "socket_addr": {
> > "type": "none",
> > "addr": "172.27.254.7:0",
> > "nonce": 443842330
> > },
> > "con_type": "client",
> > "con_features": 3387146417253690110,
> > "con_features_hex": "2f018fb87aa4aafe",
> > "con_features_release": "luminous",
> > ...
> >
> > {
> > "name": "client.104098",
> > "entity_name": "client.admin",
> > "addrs": {
> > "addrvec": [
> > {
> > "type": "v1",
> > "addr": "172.27.254.6:0",
> > "nonce": 2027668300
> > }
> > ]
> > },
> > "socket_addr": {
> > "type": "v1",
> > "addr": "172.27.254.6:0",
> > "nonce": 2027668300
> > },
> > "con_type": "client",
> > "con_features": 3387146417253690110,
> > "con_features_hex": "2f018fb87aa4aafe",
> > "con_features_release": "luminous",
> > ---snip---
> >
> > Zitat von quag...@bol.com.br:
> >
> > > Hi Eugen! Thanks a lot! I was able to find luminous connections,
> > > but I still can't identify which client process. Here is the output:
> > > Rafael.
> > > ──
> > > De: "Eugen Block" Enviada: 2025/04/08 04:37:47 Para: ceph-users@ceph.io
> > > Assunto: [ceph-users] Re: Ceph squid fresh install Hi, you can query
> > > the MON sessions to identify your older clients with: ceph tell mon.
> > > sessions It will show you the IP address, con_features_release (Luminous)
> > > and a couple of other things. Zitat von Laura Flores : > Hi Rafael, >> I
> > > would not force the min_compat_client to be reef when there are still >
> > > luminous clients connected, as it is important for all clients to be
> > > >=Reef
> > >> to understand/encode the pg_upmap_primary feature in the osdmap. >> As
> > > for checking which processes are still luminous, I am copying @Radoslaw >
> > > Zarzynski who may be able to help more with that. >> Thanks, > Laura
> > > Flores >> On Mon, Apr 7, 2025 at 11:30 AM quag...@bol.com.br > wrote:
> > Hi, >> I just did a new Ceph installation and would like to enable the
> > > "read >> balancer". >> However, the documentation requires that the
> > > minimum
> > > client version >> be reef. I checked this information through "ceph
> > > features" and came across >> the situation of having 2 luminous clients.
> > > >>
> > > # ceph features >> { >> "mon": [ >> { >> "features": "0x3f03cffd",
> > >>> "release": "squid", >> "num": 2 >> } >> ], >> "mds": [ >> { >>
> > > "features": "0x3f03cffd", >> "release": "squid", >> "num": 2 >> }
> > >>> ], >> "osd": [ >> { >> "features": "0x3f03cffd", >> "release":
> > > "squid", >> "num": 38 >> } >> ], >> "client": [ >> { >> "features":
> > > "0x2f018fb87aa4aafe", >> "release": "luminous", >> "num": 2 >> }, >> { >>
> > > "features": "0x3f03cffd", >> "release": "squid", >> "num": 5 >> }
> > >>> ], >> "mgr": [ >> { >> "features": "0x3f03cffd", >> "release":
> > > "squid", >> "num": 2 >> } >> ] >> } I tryed to configure the minimum
> > > version to reef and received the >> following alert: >> # ceph osd
> > > set-require-min-compat-client reef >> Error EPERM: cannot set
> > > require_min_compat_client to reef: 2 connected >> client(s) look like
> > > luminous (missing 0x8000); add >> --yes-i-really-mean-it to do it
> > > anyway Is it ok do confirm anyway? >> Which processes are still as
> > > luminous? Rafael. >> ___
> > >>> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an
> > > email to ceph-users-le...@ceph.io >> -- >> Laura Flores >>
> > > She/Her/Hers
> > >>> Software Engineer, Ceph Storage >>> Chicago, IL >> lflo...@ibm.com |
> > > lflo...@redhat.com > M: +17087388804 >
> > > ___ > ceph-users mailing list
> > > -- ceph-users@ceph.io > To unsubscribe send an email
[ceph-users] Re: Cephadm upgrade from 16.2.15 -> 17.2.0
This seems to have worked to get the orch back up and put me back to 16.2.15.
Thank you. Debating on waiting for 18.2.5 to move forward.
-jeremy
> On Monday, Apr 07, 2025 at 1:26 AM, Eugen Block (mailto:ebl...@nde.ag)> wrote:
> Still no, just edit the unit.run file for the MGRs to use a different
> image. See Frédéric's instructions (now that I'm re-reading it,
> there's a little mistake with dots and hyphens):
>
> # Backup the unit.run file
> $ cp /var/lib/ceph/$(ceph fsid)/mgr.ceph01.eydqvm/unit.run{,.bak}
>
> # Change container image's signature. You can get the signature of the
> version you
> want to reach from https://quay.io/repository/ceph/ceph?tab=tags. It's
> in the URL of a
> version.
> $ sed -i
> 's/ceph@sha256:e40c19cd70e047d14d70f5ec3cf501da081395a670cd59ca881ff56119660c8f/ceph@sha256:d26c11e20773704382946e34f0d3d2c0b8bb0b7b37d9017faa9dc11a0196c7d9/g'
> /var/lib/ceph/$(ceph fsid)/mgr.ceph01.eydqvm/unit.run
>
> # Restart the container (systemctl daemon-reload not needed)
> $ systemctl restart ceph-$(ceph fsid)(a)mgr.ceph01.eydqvm.service
>
> # Run this command a few times and it should show the new version
> ceph orch ps --refresh --hostname ceph01 | grep mgr
>
> To get the image signature, you can also look into the other unit.run
> files, a version tag would also work.
>
> It depends on how often you need the orchestrator to maintain the
> cluster. If you have the time, you could wait a bit longer for other
> responses. If you need the orchestrator in the meantime, you can roll
> back the MGRs.
>
> https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/32APKOXKRAIZ7IDCNI25KVYFCCCF6RJG/
>
> Zitat von Jeremy Hansen :
>
> > Thank you. The only thing I’m unclear on is the rollback to pacific.
> >
> > Are you referring to
> >
> > > > >
> > > https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#manually-deploying-a-manager-daemon
> >
> > Thank you. I appreciate all the help. Should I wait for Adam to
> > comment? At the moment, the cluster is functioning enough to
> > maintain running vms, so if it’s wise to wait, I can do that.
> >
> > -jeremy
> >
> > > On Monday, Apr 07, 2025 at 12:23 AM, Eugen Block > > (mailto:ebl...@nde.ag)> wrote:
> > > I haven't tried it this way yet, and I had hoped that Adam would chime
> > > in, but my approach would be to remove this key (it's not present when
> > > no upgrade is in progress):
> > >
> > > ceph config-key rm mgr/cephadm/upgrade_state
> > >
> > > Then rollback the two newer MGRs to Pacific as described before. If
> > > they come up healthy, test if the orchestrator works properly first.
> > > For example, remove a node-exporter or crash or anything else
> > > uncritical and let it redeploy.
> > > If that works, try a staggered upgrade, starting with the MGRs only:
> > >
> > > ceph orch upgrade start --image --daemon-types mgr
> > >
> > > Since there's no need to go to Quincy, I suggest to upgrade to Reef
> > > 18.2.4 (or you wait until 18.2.5 is released, which should be very
> > > soon), so set the respective in the above command.
> > >
> > > If all three MGRs successfully upgrade, you can continue with the
> > > MONs, or with the entire rest.
> > >
> > > In production clusters, I usually do staggered upgrades, e. g. I limit
> > > the number of OSD daemons first just to see if they come up healthy,
> > > then I let it upgrade all other OSDs automatically.
> > >
> > > https://docs.ceph.com/en/latest/cephadm/upgrade/#staggered-upgrade
> > >
> > > Zitat von Jeremy Hansen :
> > >
> > > > Snipped some of the irrelevant logs to keep message size down.
> > > >
> > > > ceph config-key get mgr/cephadm/upgrade_state
> > > >
> > > > {"target_name": "quay.io/ceph/ceph:v17.2.0", "progress_id":
> > > > "e7e1a809-558d-43a7-842a-c6229fdc57af", "target_id":
> > > > "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9",
> > > > "target_digests":
> > > >
> > > ["quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a",
> > >
> > > "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667"],
> > > "target_version": "17.2.0", "fs_original_max_mds": null,
> > > "fs_original_allow_standby_replay": null, "error": null, "paused": false,
> > > "daemon_types": null, "hosts": null, "services": null, "total_count":
> > > null,
> > > "remaining_count":
> > > > null}
> > > >
> > > > What should I do next?
> > > >
> > > > Thank you!
> > > > -jeremy
> > > >
> > > > > On Sunday, Apr 06, 2025 at 1:38 AM, Eugen Block > > > > (mailto:ebl...@nde.ag)> wrote:
> > > > > Can you check if you have this config-key?
> > > > >
> > > > > ceph config-key get mgr/cephadm/upgrade_state
> > > > >
> > > > > If you reset the MGRs, it might be necessary to clear this key,
> > > > > otherwise you might end up in some inconsistency. Just to be sure.
> > > > >
> > > > > Zitat von Jeremy Hansen :
> > > > >
> > > > > > Thanks. I’m trying to be extra careful since this cluster is
> > > > > > actually in use
[ceph-users] Re: Cephadm flooding /var/log/ceph/cephadm.log
I mean this bit
'log_file': {
'level': 'DEBUG',
'class': 'logging.handlers.WatchedFileHandler',
'formatter': 'cephadm',
'filename': '%s/cephadm.log' % LOG_DIR,
}
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
