[ceph-users] How do I Talk to a Cash app representative to ask what amount I have to pay for $100?

[awalker97520]

In the event that you are going to make a transaction from your Cash app 
account that connected a credit card, at that point you have to pay a 3% charge 
expense. Along these lines, in the event that you have need to make a 
transaction of $100, at that point you have to add $103, to pay its charge 
expense as well. In the event that you have still any inquiries behind this 
idea, at that point proceed to How do I talk to a Cash app representative.

https://www.techjaadu.com/cash-app-support-number/
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] How Cash app customer service help us to report for cash?

[PEMLA landy]

Here, it is obligatory to ensure for all wages, even it is gotten or paid in 
cash. All the tolerant cash portions for any work are submitted here to record 
the pay and go to promise it on their administration charge archives. It is at 
least a point for the customers to follow trades similarly to avoid duped 
cases. In case of any issue, go to use our 
[url=https://www.pcmonks.net/blog/transfer-money-cash-app-to-bank-account/]Cash 
app customer service[/url] phone number.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Can Cash app customer service explain beyond what many would consider possible?

[PEMLA landy]

You can see beyond what many would consider possible which is depicted more or 
less. There is two spending limit, one is for the verified customer and other 
is for non-verified customers. If you are a non-verified customer, then, you 
can consume $250 dollar in any 7-days and get up to $1000 in any 30-days term. 
And, if you are a verified user, then you can send $7500 in a week and receive 
the unlimited amount. If any doubts, contact our Cash app customer service. 
https://www.pcmonks.net/blog/how-to-increase-cash-app-limit/
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Will Cash app customer service eradicate my account forever?

[PEMLA landy]

Generally no, until you didn't require, Cash app customer service can't 
eradicate your account for eternity in any scenario. Regardless, every so often 
happened that, the assistance bunch finds your account connected with any out 
of the ordinary activity. Thusly, in this circumstance, they can close your 
account on a passing reason. In case you have to get to it, then give all the 
docs to check your identity to use your account without any issue. 
https://www.pcmonks.net/blog/how-to-access-an-old-cash-app-account/
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: help me enable ceph iscsi gatewaty in ceph octopus

[Ricardo Marques]

Hi David,

I was able to configure iSCSI gateways on my local test environment using the 
following spec:

```
# tail -14 service_spec_gw.yml
---
service_type: iscsi
service_id: iscsi_service
placement:
hosts:
- 'node1'
- 'node2'
spec:
pool: rbd
trusted_ip_list: 10.20.94.201,10.20.94.202,10.20.94.203
api_port: 5000
api_user: admin1
api_password: admin2
api_secure: False

# ceph orch apply -i service_spec_gw.yml
```

You can use this spec as a starting point, but note that the pool must exist 
(in this case `rbd` pool), and you will need to adapt `hosts`,  
`trusted_ip_list`, etc...

You may also want to change `api_secure` to `True` and set `ssl_cert` and 
`ssh_key` accordingly.

Unfortunately, iSCSI deployment is not included in the documentation yet 
(coming soon).

[1] https://docs.ceph.com/docs/octopus/cephadm/install/


Ricardo Marques


From: David Thuong 
Sent: Wednesday, August 5, 2020 5:16 AM
To: ceph-users@ceph.io 
Subject: [ceph-users] help me enable ceph iscsi gatewaty in ceph octopus

Please help me enable ceph iscsi gatewaty in ceph octopus . when i install ceph 
complete . i see iscsi gateway not enable. please help me config it
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: help me enable ceph iscsi gatewaty in ceph octopus

[Sebastian Wagner]

hi David, hi Ricardo,

I think we first have to clarify, if that was actually a cephadm
deployment (and not ceph-ansible).

If you install Ceph using ceph-ansible, then please refer to the
ceph-ansible docs.

If we're actually talking about cephadm here (which is not clear to me):
iSCSI for cephadm will land in the next octopus release and at that
point we can add a proper documentation.



Hope that helps,

Sebastian

Am 05.08.20 um 11:11 schrieb Ricardo Marques:
> Hi David,
> 
> I was able to configure iSCSI gateways on my local test environment using the 
> following spec:
> 
> ```
> # tail -14 service_spec_gw.yml
> ---
> service_type: iscsi
> service_id: iscsi_service
> placement:
> hosts:
> - 'node1'
> - 'node2'
> spec:
> pool: rbd
> trusted_ip_list: 10.20.94.201,10.20.94.202,10.20.94.203
> api_port: 5000
> api_user: admin1
> api_password: admin2
> api_secure: False
> 
> # ceph orch apply -i service_spec_gw.yml
> ```
> 
> You can use this spec as a starting point, but note that the pool must exist 
> (in this case `rbd` pool), and you will need to adapt `hosts`,  
> `trusted_ip_list`, etc...
> 
> You may also want to change `api_secure` to `True` and set `ssl_cert` and 
> `ssh_key` accordingly.
> 
> Unfortunately, iSCSI deployment is not included in the documentation yet 
> (coming soon).
> 
> [1] https://docs.ceph.com/docs/octopus/cephadm/install/
> 
> 
> Ricardo Marques
> 
> 
> From: David Thuong 
> Sent: Wednesday, August 5, 2020 5:16 AM
> To: ceph-users@ceph.io 
> Subject: [ceph-users] help me enable ceph iscsi gatewaty in ceph octopus
> 
> Please help me enable ceph iscsi gatewaty in ceph octopus . when i install 
> ceph complete . i see iscsi gateway not enable. please help me config it
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
> 
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
> 

-- 
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg). Geschäftsführer: Felix Imendörffer



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: help me enable ceph iscsi gatewaty in ceph octopus

[Hoài Thương]

hello swagner,
Can you give me document , i use cephadm
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] How to get rid of HP printer driver is unavailable issue?

[sofiacarter120]

I’m working on the latest printing machine for the excellent printing 
performance. I am observing that my printing machine is performing poorly. My 
printing device is not printing the documents correctly. My printer is showing 
an error message such as 
[url=https://www.hpprintersupportpro.com/blog/hp-printer-in-error-state/https://www.hpprintersupportpro.com/blog/hp-printer-in-error-state/]printer
 in error state[/url] problem. I don’t have the permanent solutions for solving 
this error. I guess that the low ink can be the main reason of this technical 
error. I am trying to find out the cause of this technical glitch, but I am not 
sure the reasons. It can be very difficult to find out the exact causes. So 
anyone can recommend the easy troubleshooting resolutions to fix this error as 
soon as possible.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: block.db/block.wal device performance dropped after upgrade to 14.2.10

[Manuel Lausch]

Hello Vladimir,

I just tested this with a single node testcluster with 60 HDDs (3 of
them with bluestore without separate wal and db).

With the 14.2.10, I see on the bluestore OSDs a lot of read IOPs while
snaptrimming. With 14.2.9 this was not an issue. 

I wonder if this would explain the huge amount of slowops on my big
testcluster (44 Nodes 1056 OSDs) while snaptrimming. I
cannot test a downgrade there, because there are no packages of older
releases for CentOS 8 available.

Regards
Manuel

On Tue, 4 Aug 2020 13:22:34 +0300
Vladimir Prokofev  wrote:

> Here's some more insight into the issue.
> Looks like the load is triggered because of a snaptrim operation. We
> have a backup pool that serves as Openstack cinder-backup storage,
> performing snapshot backups every night. Old backups are also deleted
> every night, so snaptrim is initiated.
> This snaptrim increased load on the block.db devices after upgrade,
> and just kills one SSD's performance in particular. It serves as a
> block.db/wal device for one of the fatter backup pool OSDs which has
> more PGs placed there.
> This is a Kingston SSD, and we see this issue on other Kingston SSD
> journals too, Intel SSD journals are not that affected, though they
> too experience increased load.
> Nevertheless, there're now a lot of read IOPS on block.db devices
> after upgrade that were not there before.
> I wonder how 600 IOPS can destroy SSDs performance that hard.
> 
> вт, 4 авг. 2020 г. в 12:54, Vladimir Prokofev :
> 
> > Good day, cephers!
> >
> > We've recently upgraded our cluster from 14.2.8 to 14.2.10 release,
> > also performing full system packages upgrade(Ubuntu 18.04 LTS).
> > After that performance significantly dropped, main reason beeing
> > that journal SSDs are now have no merges, huge queues, and
> > increased latency. There's a few screenshots in attachments. This
> > is for an SSD journal that supports block.db/block.wal for 3
> > spinning OSDs, and it looks like this for all our SSD block.db/wal
> > devices across all nodes. Any ideas what may cause that? Maybe I've
> > missed something important in release notes?
> >  
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph does not recover from OSD restart

[Eric Smith]

You have a LOT of state transitions during your maintenance and I'm not really 
sure why (There are a lot of complaints about the network). There's are also a 
lot of "transitioning to Stray" after initial startup of an OSD. I'd say let 
your cluster heal first before you start doing a ton a maintenance so old PG 
maps can be trimmed. That's the best I can ascertain from the logs for now.

-Original Message-
From: Frank Schilder  
Sent: Tuesday, August 4, 2020 8:35 AM
To: Eric Smith ; ceph-users 
Subject: Re: Ceph does not recover from OSD restart

If with monitor log you mean the cluster log /var/log/ceph/ceph.log, I should 
have all of it. Please find a tgz-file here: 
https://linkprotect.cudasvc.com/url?a=https%3a%2f%2ffiles.dtu.dk%2fu%2ftFCEZJzQhH2mUIRk%2flogs.tgz%3fl&c=E,1,uqVWoKuvpNjjLYU1JO2De96Pz8ZN-UBmy7cFmI6RllcEJg1Nboe8wNTzEx0kJ4WGDxciAY2Mnq_jWNncInKPg-wSwWzu2kV-ZmWlJVb_O9P-At48cWcXTDI9&typo=1
 (valid 100 days).

Contents:

logs/ceph-2020-08-03.log  -  cluster log for the day of restart 
logs/ceph-osd.145.2020-08-03.log  -  log of "old" OSD trimmed to day of restart 
logs/ceph-osd.288.log  -  entire log of "new" OSD

Hope this helps.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Eric Smith 
Sent: 04 August 2020 14:15:11
To: Frank Schilder; ceph-users
Subject: RE: Ceph does not recover from OSD restart

Do you have any monitor / OSD logs from the maintenance when the issues 
occurred?


 Original message 
From: Frank Schilder 
Date: 8/4/20 8:07 AM (GMT-05:00)
To: Eric Smith , ceph-users 
Subject: Re: Ceph does not recover from OSD restart

Hi Eric,

thanks for the clarification, I did misunderstand you.

> You should not have to move OSDs in and out of the CRUSH tree however 
> in order to solve any data placement problems (This is the baffling part).

Exactly. Should I create a tracker issue? I think this is not hard to reproduce 
with a standard crush map where host-bucket=physical host and I would, in fact, 
expect that this scenario is part of the integration test.

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Eric Smith 
Sent: 04 August 2020 13:58:47
To: Frank Schilder; ceph-users
Subject: RE: Ceph does not recover from OSD restart

All seems in order in terms of your CRUSH layout. You can speed up the 
rebalancing / scale-out operations by increasing the osd_max_backfills on each 
OSD (Especially during off hours). The unnecessary degradation is not expected 
behavior with a cluster in HEALTH_OK status, but with backfill / rebalancing 
ongoing it's not unexpected. You should not have to move OSDs in and out of the 
CRUSH tree however in order to solve any data placement problems (This is the 
baffling part).

-Original Message-
From: Frank Schilder 
Sent: Tuesday, August 4, 2020 7:45 AM
To: Eric Smith ; ceph-users 
Subject: Re: Ceph does not recover from OSD restart

Hi Erik,

I added the disks and started the rebalancing. When I run into the issue, ca. 3 
days after start of rebalancing, it was about 25% done. The cluster does not go 
to HEALTH_OK before the rebalancing is finished, it shows the "xxx objects 
misplaced" warning. The OSD crush locations for the logical hosts are in 
ceph.conf, the OSDs come up in the proper crush bucket.

> All seems in order then

In what sense?

The rebalancing is still ongoing and usually a very long operation. This time I 
added only 9 disks, but we will almost triple the number of disks of a larger 
pool soon, which has 150 OSDs at the moment. I expect the rebalancing for this 
expansion to take months. Due to a memory leak, I need to restart OSDs 
regularly. Also, a host may restart or we might have a power outage during this 
window. In these situations, it will be a real pain if I have to play the crush 
move game with 300+ OSDs.

This unnecessary redundancy degradation on OSD restart cannot possibly be 
expected behaviour, or do I misunderstand something here?

Best regards,
=
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14


From: Eric Smith 
Sent: 04 August 2020 13:19:41
To: Frank Schilder; ceph-users
Subject: RE: Ceph does not recover from OSD restart

All seems in order then - when you ran into your maintenance issue, how long 
was if after you added the new OSDs and did Ceph ever get to HEALTH_OK so it 
could trim PG history? Also did the OSDs just start back up in the wrong place 
in the CRUSH tree?

-Original Message-
From: Frank Schilder 
Sent: Tuesday, August 4, 2020 7:10 AM
To: Eric Smith ; ceph-users 
Subject: Re: Ceph does not recover from OSD restart

Hi Eric,

> Have you adjusted the min_size for pool sr-rbd-data-one-hdd

Yes. For all EC pools located in datacenter ServerRoom, we currently set 
min_size=k=6, because we lack physical servers. Hosts ceph-21 and ceph-22

[ceph-users] Re: RGW versioned objects lost after Octopus 15.2.3 -> 15.2.4 upgrade

[Chris Palmer]

This is starting to look like a regression error in Octopus 15.2.4.

After cleaning things up by deleting all old versions, and deleting and 
recreating the bucket lifecycle policy (see below), I then let it run.
Each day a new version got created, dating back to 17 July (correct). 
Until this morning when we went over the 18-day non-current expiration: 
this morning all but the latest version of each object disappeared - 
which is what happened the midnight following after our 15.2.3->15.2.4 
upgrade.


So instead of a gently rolling 18-days of versions (on 15.2.3), we now 
build up to 18 days after which all non-current versions get deleted (on 
15.2.4).


Anyone come across versioning problems on 15.2.4?

Thanks, Chris

On 17/07/2020 09:11, Chris Palmer wrote:
This got worse this morning. An RGW daemon crashed at midnight with a 
segfault, and the backtrace hints that it was processing the 
expiration rule:


    "backtrace": [
    "(()+0x12730) [0x7f97b8c4e730]",
    "(()+0x15878a) [0x7f97b862378a]",
    "(std::__cxx11::basic_string, 
std::allocator >::compare(std::__cxx11::basic_stringstd::char_traits, std::allocator > const&) const+0x23) 
[0x7f97c25d3e43]",
    "(LCOpAction_DMExpiration::check(lc_op_ctx&, 
std::chrono::time_pointstd::chrono::duration > 
>*)+0x87) [0x7f97c283d127]",
    "(LCOpRule::process(rgw_bucket_dir_entry&, DoutPrefixProvider 
const*)+0x1b8) [0x7f97c281cbc8]",

    "(()+0x5b836d) [0x7f97c281d36d]",
    "(WorkQ::entry()+0x247) [0x7f97c28302d7]",
    "(()+0x7fa3) [0x7f97b8c43fa3]",
    "(clone()+0x3f) [0x7f97b85c44cf]"

One object version got removed when it should not have.

In an attempt to clean things up I have manually deleted all 
non-current versions, and removed and recreated the (same) lifecycle 
policy. I will also create a new test bucket with a similar policy and 
test that in parallel. We will see what happens tomorrow


Thanks, Chris


On 16/07/2020 08:22, Chris Palmer wrote:
I have an RGW bucket (backups) that is versioned. A nightly job 
creates a new version of a few objects. There is a lifecycle policy 
(see below) that keeps 18 days of versions. This has been working 
perfectly and has not been changed. Until I upgraded Octopus...


The nightly job creates separate log files, including a listing of 
the object versions. From these I can see that:


13/7  02:14   versions from 13/7 01:13 back to 24/6 01:17 (correct)

14/7  02:14   versions from 14/7 01:13 back to 25/6 01:14 (correct)

14/7  10:00   upgrade Octopus 15.2.3 -> 15.2.4

15/7  02:14   versions from 15/7 01:13 back to 25/6 01:14 (would have 
expected 25/6 to have expired)


16/7  02:14   versions from 16/7 01:13 back to 15/7 01:13 (now all 
pre-upgrade versions have wrongly disappeared)


It's not a big deal for me as they are only backups, providing it 
continues to work correctly from now on. However it may affect some 
other people  much more.


Any ideas on the root cause? And if it is likely to be stable again now?

Thanks, Chris

{
    "Rules": [
    {
    "Expiration": {
    "ExpiredObjectDeleteMarker": true
    },
    "ID": "Expiration & incomplete uploads",
    "Prefix": "",
    "Status": "Enabled",
    "NoncurrentVersionExpiration": {
    "NoncurrentDays": 18
    },
    "AbortIncompleteMultipartUpload": {
    "DaysAfterInitiation": 1
    }
    }
    ]
}

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RGW versioned objects lost after Octopus 15.2.3 -> 15.2.4 upgrade

[Dan van der Ster]

Hi Chris,

Is it possible that this is the correct behaviour for NoncurrentDays
expiration?

https://docs.aws.amazon.com/AmazonS3/latest/API/API_NoncurrentVersionExpiration.html

AFAIK, the xml to expire *only* the versions older than x days is somewhat
different:

https://clouddocs.web.cern.ch/object_store/s3cmd.html#object-expiration-with-s3cmd

.. Dan




On Wed, 5 Aug 2020, 14:52 Chris Palmer,  wrote:

> This is starting to look like a regression error in Octopus 15.2.4.
>
> After cleaning things up by deleting all old versions, and deleting and
> recreating the bucket lifecycle policy (see below), I then let it run.
> Each day a new version got created, dating back to 17 July (correct).
> Until this morning when we went over the 18-day non-current expiration:
> this morning all but the latest version of each object disappeared -
> which is what happened the midnight following after our 15.2.3->15.2.4
> upgrade.
>
> So instead of a gently rolling 18-days of versions (on 15.2.3), we now
> build up to 18 days after which all non-current versions get deleted (on
> 15.2.4).
>
> Anyone come across versioning problems on 15.2.4?
>
> Thanks, Chris
>
> On 17/07/2020 09:11, Chris Palmer wrote:
> > This got worse this morning. An RGW daemon crashed at midnight with a
> > segfault, and the backtrace hints that it was processing the
> > expiration rule:
> >
> > "backtrace": [
> > "(()+0x12730) [0x7f97b8c4e730]",
> > "(()+0x15878a) [0x7f97b862378a]",
> > "(std::__cxx11::basic_string,
> > std::allocator >::compare(std::__cxx11::basic_string > std::char_traits, std::allocator > const&) const+0x23)
> > [0x7f97c25d3e43]",
> > "(LCOpAction_DMExpiration::check(lc_op_ctx&,
> > std::chrono::time_point > std::chrono::duration >
> > >*)+0x87) [0x7f97c283d127]",
> > "(LCOpRule::process(rgw_bucket_dir_entry&, DoutPrefixProvider
> > const*)+0x1b8) [0x7f97c281cbc8]",
> > "(()+0x5b836d) [0x7f97c281d36d]",
> > "(WorkQ::entry()+0x247) [0x7f97c28302d7]",
> > "(()+0x7fa3) [0x7f97b8c43fa3]",
> > "(clone()+0x3f) [0x7f97b85c44cf]"
> >
> > One object version got removed when it should not have.
> >
> > In an attempt to clean things up I have manually deleted all
> > non-current versions, and removed and recreated the (same) lifecycle
> > policy. I will also create a new test bucket with a similar policy and
> > test that in parallel. We will see what happens tomorrow
> >
> > Thanks, Chris
> >
> >
> > On 16/07/2020 08:22, Chris Palmer wrote:
> >> I have an RGW bucket (backups) that is versioned. A nightly job
> >> creates a new version of a few objects. There is a lifecycle policy
> >> (see below) that keeps 18 days of versions. This has been working
> >> perfectly and has not been changed. Until I upgraded Octopus...
> >>
> >> The nightly job creates separate log files, including a listing of
> >> the object versions. From these I can see that:
> >>
> >> 13/7  02:14   versions from 13/7 01:13 back to 24/6 01:17 (correct)
> >>
> >> 14/7  02:14   versions from 14/7 01:13 back to 25/6 01:14 (correct)
> >>
> >> 14/7  10:00   upgrade Octopus 15.2.3 -> 15.2.4
> >>
> >> 15/7  02:14   versions from 15/7 01:13 back to 25/6 01:14 (would have
> >> expected 25/6 to have expired)
> >>
> >> 16/7  02:14   versions from 16/7 01:13 back to 15/7 01:13 (now all
> >> pre-upgrade versions have wrongly disappeared)
> >>
> >> It's not a big deal for me as they are only backups, providing it
> >> continues to work correctly from now on. However it may affect some
> >> other people  much more.
> >>
> >> Any ideas on the root cause? And if it is likely to be stable again now?
> >>
> >> Thanks, Chris
> >>
> >> {
> >> "Rules": [
> >> {
> >> "Expiration": {
> >> "ExpiredObjectDeleteMarker": true
> >> },
> >> "ID": "Expiration & incomplete uploads",
> >> "Prefix": "",
> >> "Status": "Enabled",
> >> "NoncurrentVersionExpiration": {
> >> "NoncurrentDays": 18
> >> },
> >> "AbortIncompleteMultipartUpload": {
> >> "DaysAfterInitiation": 1
> >> }
> >> }
> >> ]
> >> }
> >>
> >> ___
> >> ceph-users mailing list -- ceph-users@ceph.io
> >> To unsubscribe send an email to ceph-users-le...@ceph.io
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RGW versioned objects lost after Octopus 15.2.3 -> 15.2.4 upgrade

[Matt Benjamin]

Hi Chris,

There is new lifecycle processing logic backported to Octopus, it
looks like, in 15.2.3.  I'm looking at the non-current calculation to
see if it could incorrectly rely on a stale value (from an eralier
entry).

thanks,

Matt

On Wed, Aug 5, 2020 at 8:52 AM Chris Palmer  wrote:
>
> This is starting to look like a regression error in Octopus 15.2.4.
>
> After cleaning things up by deleting all old versions, and deleting and
> recreating the bucket lifecycle policy (see below), I then let it run.
> Each day a new version got created, dating back to 17 July (correct).
> Until this morning when we went over the 18-day non-current expiration:
> this morning all but the latest version of each object disappeared -
> which is what happened the midnight following after our 15.2.3->15.2.4
> upgrade.
>
> So instead of a gently rolling 18-days of versions (on 15.2.3), we now
> build up to 18 days after which all non-current versions get deleted (on
> 15.2.4).
>
> Anyone come across versioning problems on 15.2.4?
>
> Thanks, Chris
>
> On 17/07/2020 09:11, Chris Palmer wrote:
> > This got worse this morning. An RGW daemon crashed at midnight with a
> > segfault, and the backtrace hints that it was processing the
> > expiration rule:
> >
> > "backtrace": [
> > "(()+0x12730) [0x7f97b8c4e730]",
> > "(()+0x15878a) [0x7f97b862378a]",
> > "(std::__cxx11::basic_string,
> > std::allocator >::compare(std::__cxx11::basic_string > std::char_traits, std::allocator > const&) const+0x23)
> > [0x7f97c25d3e43]",
> > "(LCOpAction_DMExpiration::check(lc_op_ctx&,
> > std::chrono::time_point > std::chrono::duration >
> > >*)+0x87) [0x7f97c283d127]",
> > "(LCOpRule::process(rgw_bucket_dir_entry&, DoutPrefixProvider
> > const*)+0x1b8) [0x7f97c281cbc8]",
> > "(()+0x5b836d) [0x7f97c281d36d]",
> > "(WorkQ::entry()+0x247) [0x7f97c28302d7]",
> > "(()+0x7fa3) [0x7f97b8c43fa3]",
> > "(clone()+0x3f) [0x7f97b85c44cf]"
> >
> > One object version got removed when it should not have.
> >
> > In an attempt to clean things up I have manually deleted all
> > non-current versions, and removed and recreated the (same) lifecycle
> > policy. I will also create a new test bucket with a similar policy and
> > test that in parallel. We will see what happens tomorrow
> >
> > Thanks, Chris
> >
> >
> > On 16/07/2020 08:22, Chris Palmer wrote:
> >> I have an RGW bucket (backups) that is versioned. A nightly job
> >> creates a new version of a few objects. There is a lifecycle policy
> >> (see below) that keeps 18 days of versions. This has been working
> >> perfectly and has not been changed. Until I upgraded Octopus...
> >>
> >> The nightly job creates separate log files, including a listing of
> >> the object versions. From these I can see that:
> >>
> >> 13/7  02:14   versions from 13/7 01:13 back to 24/6 01:17 (correct)
> >>
> >> 14/7  02:14   versions from 14/7 01:13 back to 25/6 01:14 (correct)
> >>
> >> 14/7  10:00   upgrade Octopus 15.2.3 -> 15.2.4
> >>
> >> 15/7  02:14   versions from 15/7 01:13 back to 25/6 01:14 (would have
> >> expected 25/6 to have expired)
> >>
> >> 16/7  02:14   versions from 16/7 01:13 back to 15/7 01:13 (now all
> >> pre-upgrade versions have wrongly disappeared)
> >>
> >> It's not a big deal for me as they are only backups, providing it
> >> continues to work correctly from now on. However it may affect some
> >> other people  much more.
> >>
> >> Any ideas on the root cause? And if it is likely to be stable again now?
> >>
> >> Thanks, Chris
> >>
> >> {
> >> "Rules": [
> >> {
> >> "Expiration": {
> >> "ExpiredObjectDeleteMarker": true
> >> },
> >> "ID": "Expiration & incomplete uploads",
> >> "Prefix": "",
> >> "Status": "Enabled",
> >> "NoncurrentVersionExpiration": {
> >> "NoncurrentDays": 18
> >> },
> >> "AbortIncompleteMultipartUpload": {
> >> "DaysAfterInitiation": 1
> >> }
> >> }
> >> ]
> >> }
> >>
> >> ___
> >> ceph-users mailing list -- ceph-users@ceph.io
> >> To unsubscribe send an email to ceph-users-le...@ceph.io
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io



-- 

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RGW versioned objects lost after Octopus 15.2.3 -> 15.2.4 upgrade

[Chris Palmer]

Hi Dan

The second link you refer to is, I believe, version-agnostic. (And s3cmd 
is also version-unaware). So "an object" would expire after x days, 
rather than "non-current versions of an object".


I've re-read the first description several times, and remembering that 
it only applies to versioned buckets, still think that it is the correct 
xml to specify how long non-current versions are retained (which is what 
I want). It is also the exact same policy that I used without problem on 
15.2.3, and also one that I use on AWS S3 itself


Regards, Chris

On 05/08/2020 14:08, Dan van der Ster wrote:

Hi Chris,

Is it possible that this is the correct behaviour for NoncurrentDays 
expiration?


https://docs.aws.amazon.com/AmazonS3/latest/API/API_NoncurrentVersionExpiration.html

AFAIK, the xml to expire *only* the versions older than x days is 
somewhat different:


https://clouddocs.web.cern.ch/object_store/s3cmd.html#object-expiration-with-s3cmd

.. Dan




On Wed, 5 Aug 2020, 14:52 Chris Palmer, > wrote:


This is starting to look like a regression error in Octopus 15.2.4.

After cleaning things up by deleting all old versions, and
deleting and
recreating the bucket lifecycle policy (see below), I then let it run.
Each day a new version got created, dating back to 17 July (correct).
Until this morning when we went over the 18-day non-current
expiration:
this morning all but the latest version of each object disappeared -
which is what happened the midnight following after our
15.2.3->15.2.4
upgrade.

So instead of a gently rolling 18-days of versions (on 15.2.3), we
now
build up to 18 days after which all non-current versions get
deleted (on
15.2.4).

Anyone come across versioning problems on 15.2.4?

Thanks, Chris

On 17/07/2020 09:11, Chris Palmer wrote:
> This got worse this morning. An RGW daemon crashed at midnight
with a
> segfault, and the backtrace hints that it was processing the
> expiration rule:
>
>     "backtrace": [
>     "(()+0x12730) [0x7f97b8c4e730]",
>     "(()+0x15878a) [0x7f97b862378a]",
>     "(std::__cxx11::basic_string,
> std::allocator >::compare(std::__cxx11::basic_string std::char_traits, std::allocator > const&) const+0x23)
> [0x7f97c25d3e43]",
>     "(LCOpAction_DMExpiration::check(lc_op_ctx&,
> std::chrono::time_point std::chrono::duration >
> >*)+0x87) [0x7f97c283d127]",
>     "(LCOpRule::process(rgw_bucket_dir_entry&,
DoutPrefixProvider
> const*)+0x1b8) [0x7f97c281cbc8]",
>     "(()+0x5b836d) [0x7f97c281d36d]",
>     "(WorkQ::entry()+0x247) [0x7f97c28302d7]",
>     "(()+0x7fa3) [0x7f97b8c43fa3]",
>     "(clone()+0x3f) [0x7f97b85c44cf]"
>
> One object version got removed when it should not have.
>
> In an attempt to clean things up I have manually deleted all
> non-current versions, and removed and recreated the (same)
lifecycle
> policy. I will also create a new test bucket with a similar
policy and
> test that in parallel. We will see what happens tomorrow
>
> Thanks, Chris
>
>
> On 16/07/2020 08:22, Chris Palmer wrote:
>> I have an RGW bucket (backups) that is versioned. A nightly job
>> creates a new version of a few objects. There is a lifecycle
policy
>> (see below) that keeps 18 days of versions. This has been working
>> perfectly and has not been changed. Until I upgraded Octopus...
>>
>> The nightly job creates separate log files, including a listing of
>> the object versions. From these I can see that:
>>
>> 13/7  02:14   versions from 13/7 01:13 back to 24/6 01:17 (correct)
>>
>> 14/7  02:14   versions from 14/7 01:13 back to 25/6 01:14 (correct)
>>
>> 14/7  10:00   upgrade Octopus 15.2.3 -> 15.2.4
>>
>> 15/7  02:14   versions from 15/7 01:13 back to 25/6 01:14
(would have
>> expected 25/6 to have expired)
>>
>> 16/7  02:14   versions from 16/7 01:13 back to 15/7 01:13 (now all
>> pre-upgrade versions have wrongly disappeared)
>>
>> It's not a big deal for me as they are only backups, providing it
>> continues to work correctly from now on. However it may affect
some
>> other people  much more.
>>
>> Any ideas on the root cause? And if it is likely to be stable
again now?
>>
>> Thanks, Chris
>>
>> {
>>     "Rules": [
>>     {
>>     "Expiration": {
>>     "ExpiredObjectDeleteMarker": true
>>     },
>>     "ID": "Expiration & incomplete uploads",
>>     "Prefix": "",
>>     "Status": "Enabled",
>>     "NoncurrentVersionExpiration": {
>>     "NoncurrentDays": 18
>>     },
>>     "AbortIncompleteMultipartUpl

[ceph-users] Re: RGW versioned objects lost after Octopus 15.2.3 -> 15.2.4 upgrade

[Dan van der Ster]

Yeah my bad... That cern doc is different from what you're trying to
achieve.

.. dan


On Wed, 5 Aug 2020, 15:31 Chris Palmer,  wrote:

> Hi Dan
>
> The second link you refer to is, I believe, version-agnostic. (And s3cmd
> is also version-unaware). So "an object" would expire after x days, rather
> than "non-current versions of an object".
>
> I've re-read the first description several times, and remembering that it
> only applies to versioned buckets, still think that it is the correct xml
> to specify how long non-current versions are retained (which is what I
> want). It is also the exact same policy that I used without problem on
> 15.2.3, and also one that I use on AWS S3 itself
>
> Regards, Chris
>
> On 05/08/2020 14:08, Dan van der Ster wrote:
>
> Hi Chris,
>
> Is it possible that this is the correct behaviour for NoncurrentDays
> expiration?
>
>
> https://docs.aws.amazon.com/AmazonS3/latest/API/API_NoncurrentVersionExpiration.html
>
> AFAIK, the xml to expire *only* the versions older than x days is somewhat
> different:
>
>
> https://clouddocs.web.cern.ch/object_store/s3cmd.html#object-expiration-with-s3cmd
>
> .. Dan
>
>
>
>
> On Wed, 5 Aug 2020, 14:52 Chris Palmer, 
> wrote:
>
>> This is starting to look like a regression error in Octopus 15.2.4.
>>
>> After cleaning things up by deleting all old versions, and deleting and
>> recreating the bucket lifecycle policy (see below), I then let it run.
>> Each day a new version got created, dating back to 17 July (correct).
>> Until this morning when we went over the 18-day non-current expiration:
>> this morning all but the latest version of each object disappeared -
>> which is what happened the midnight following after our 15.2.3->15.2.4
>> upgrade.
>>
>> So instead of a gently rolling 18-days of versions (on 15.2.3), we now
>> build up to 18 days after which all non-current versions get deleted (on
>> 15.2.4).
>>
>> Anyone come across versioning problems on 15.2.4?
>>
>> Thanks, Chris
>>
>> On 17/07/2020 09:11, Chris Palmer wrote:
>> > This got worse this morning. An RGW daemon crashed at midnight with a
>> > segfault, and the backtrace hints that it was processing the
>> > expiration rule:
>> >
>> > "backtrace": [
>> > "(()+0x12730) [0x7f97b8c4e730]",
>> > "(()+0x15878a) [0x7f97b862378a]",
>> > "(std::__cxx11::basic_string,
>> > std::allocator >::compare(std::__cxx11::basic_string> > std::char_traits, std::allocator > const&) const+0x23)
>> > [0x7f97c25d3e43]",
>> > "(LCOpAction_DMExpiration::check(lc_op_ctx&,
>> > std::chrono::time_point> > std::chrono::duration >
>> > >*)+0x87) [0x7f97c283d127]",
>> > "(LCOpRule::process(rgw_bucket_dir_entry&, DoutPrefixProvider
>> > const*)+0x1b8) [0x7f97c281cbc8]",
>> > "(()+0x5b836d) [0x7f97c281d36d]",
>> > "(WorkQ::entry()+0x247) [0x7f97c28302d7]",
>> > "(()+0x7fa3) [0x7f97b8c43fa3]",
>> > "(clone()+0x3f) [0x7f97b85c44cf]"
>> >
>> > One object version got removed when it should not have.
>> >
>> > In an attempt to clean things up I have manually deleted all
>> > non-current versions, and removed and recreated the (same) lifecycle
>> > policy. I will also create a new test bucket with a similar policy and
>> > test that in parallel. We will see what happens tomorrow
>> >
>> > Thanks, Chris
>> >
>> >
>> > On 16/07/2020 08:22, Chris Palmer wrote:
>> >> I have an RGW bucket (backups) that is versioned. A nightly job
>> >> creates a new version of a few objects. There is a lifecycle policy
>> >> (see below) that keeps 18 days of versions. This has been working
>> >> perfectly and has not been changed. Until I upgraded Octopus...
>> >>
>> >> The nightly job creates separate log files, including a listing of
>> >> the object versions. From these I can see that:
>> >>
>> >> 13/7  02:14   versions from 13/7 01:13 back to 24/6 01:17 (correct)
>> >>
>> >> 14/7  02:14   versions from 14/7 01:13 back to 25/6 01:14 (correct)
>> >>
>> >> 14/7  10:00   upgrade Octopus 15.2.3 -> 15.2.4
>> >>
>> >> 15/7  02:14   versions from 15/7 01:13 back to 25/6 01:14 (would have
>> >> expected 25/6 to have expired)
>> >>
>> >> 16/7  02:14   versions from 16/7 01:13 back to 15/7 01:13 (now all
>> >> pre-upgrade versions have wrongly disappeared)
>> >>
>> >> It's not a big deal for me as they are only backups, providing it
>> >> continues to work correctly from now on. However it may affect some
>> >> other people  much more.
>> >>
>> >> Any ideas on the root cause? And if it is likely to be stable again
>> now?
>> >>
>> >> Thanks, Chris
>> >>
>> >> {
>> >> "Rules": [
>> >> {
>> >> "Expiration": {
>> >> "ExpiredObjectDeleteMarker": true
>> >> },
>> >> "ID": "Expiration & incomplete uploads",
>> >> "Prefix": "",
>> >> "Status": "Enabled",
>> >> "NoncurrentVersionExpiration": {
>> >> "NoncurrentDays": 18
>> >> },
>> >> 

[ceph-users] Re: help me enable ceph iscsi gatewaty in ceph octopus

[Sebastian Wagner]

Till iscsi is fully working in cephadm, you can install ceph-iscsi
manually as described here:

https://docs.ceph.com/docs/master/rbd/iscsi-target-cli/



Am 05.08.20 um 11:44 schrieb Hoài Thương:
> hello swagner,
> Can you give me document , i use cephadm

-- 
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
(HRB 36809, AG Nürnberg). Geschäftsführer: Felix Imendörffer



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RGW versioned objects lost after Octopus 15.2.3 -> 15.2.4 upgrade

[Matt Benjamin]

Hi Chris,

I've confirmed that the issue you're experiencing are addressed in
lifecycle commits that were required but missed during the backport to
Octopus.  I'll work with the backport team to address this quickly.

Thanks for providing the detailed reproducer information, it was very
helpful in identifying the issue.

Matt

On Wed, Aug 5, 2020 at 9:23 AM Matt Benjamin  wrote:
>
> Hi Chris,
>
> There is new lifecycle processing logic backported to Octopus, it
> looks like, in 15.2.3.  I'm looking at the non-current calculation to
> see if it could incorrectly rely on a stale value (from an eralier
> entry).
>
> thanks,
>
> Matt
>
> On Wed, Aug 5, 2020 at 8:52 AM Chris Palmer  
> wrote:
> >
> > This is starting to look like a regression error in Octopus 15.2.4.
> >
> > After cleaning things up by deleting all old versions, and deleting and
> > recreating the bucket lifecycle policy (see below), I then let it run.
> > Each day a new version got created, dating back to 17 July (correct).
> > Until this morning when we went over the 18-day non-current expiration:
> > this morning all but the latest version of each object disappeared -
> > which is what happened the midnight following after our 15.2.3->15.2.4
> > upgrade.
> >
> > So instead of a gently rolling 18-days of versions (on 15.2.3), we now
> > build up to 18 days after which all non-current versions get deleted (on
> > 15.2.4).
> >
> > Anyone come across versioning problems on 15.2.4?
> >
> > Thanks, Chris
> >
> > On 17/07/2020 09:11, Chris Palmer wrote:
> > > This got worse this morning. An RGW daemon crashed at midnight with a
> > > segfault, and the backtrace hints that it was processing the
> > > expiration rule:
> > >
> > > "backtrace": [
> > > "(()+0x12730) [0x7f97b8c4e730]",
> > > "(()+0x15878a) [0x7f97b862378a]",
> > > "(std::__cxx11::basic_string,
> > > std::allocator >::compare(std::__cxx11::basic_string > > std::char_traits, std::allocator > const&) const+0x23)
> > > [0x7f97c25d3e43]",
> > > "(LCOpAction_DMExpiration::check(lc_op_ctx&,
> > > std::chrono::time_point > > std::chrono::duration >
> > > >*)+0x87) [0x7f97c283d127]",
> > > "(LCOpRule::process(rgw_bucket_dir_entry&, DoutPrefixProvider
> > > const*)+0x1b8) [0x7f97c281cbc8]",
> > > "(()+0x5b836d) [0x7f97c281d36d]",
> > > "(WorkQ::entry()+0x247) [0x7f97c28302d7]",
> > > "(()+0x7fa3) [0x7f97b8c43fa3]",
> > > "(clone()+0x3f) [0x7f97b85c44cf]"
> > >
> > > One object version got removed when it should not have.
> > >
> > > In an attempt to clean things up I have manually deleted all
> > > non-current versions, and removed and recreated the (same) lifecycle
> > > policy. I will also create a new test bucket with a similar policy and
> > > test that in parallel. We will see what happens tomorrow
> > >
> > > Thanks, Chris
> > >
> > >
> > > On 16/07/2020 08:22, Chris Palmer wrote:
> > >> I have an RGW bucket (backups) that is versioned. A nightly job
> > >> creates a new version of a few objects. There is a lifecycle policy
> > >> (see below) that keeps 18 days of versions. This has been working
> > >> perfectly and has not been changed. Until I upgraded Octopus...
> > >>
> > >> The nightly job creates separate log files, including a listing of
> > >> the object versions. From these I can see that:
> > >>
> > >> 13/7  02:14   versions from 13/7 01:13 back to 24/6 01:17 (correct)
> > >>
> > >> 14/7  02:14   versions from 14/7 01:13 back to 25/6 01:14 (correct)
> > >>
> > >> 14/7  10:00   upgrade Octopus 15.2.3 -> 15.2.4
> > >>
> > >> 15/7  02:14   versions from 15/7 01:13 back to 25/6 01:14 (would have
> > >> expected 25/6 to have expired)
> > >>
> > >> 16/7  02:14   versions from 16/7 01:13 back to 15/7 01:13 (now all
> > >> pre-upgrade versions have wrongly disappeared)
> > >>
> > >> It's not a big deal for me as they are only backups, providing it
> > >> continues to work correctly from now on. However it may affect some
> > >> other people  much more.
> > >>
> > >> Any ideas on the root cause? And if it is likely to be stable again now?
> > >>
> > >> Thanks, Chris
> > >>
> > >> {
> > >> "Rules": [
> > >> {
> > >> "Expiration": {
> > >> "ExpiredObjectDeleteMarker": true
> > >> },
> > >> "ID": "Expiration & incomplete uploads",
> > >> "Prefix": "",
> > >> "Status": "Enabled",
> > >> "NoncurrentVersionExpiration": {
> > >> "NoncurrentDays": 18
> > >> },
> > >> "AbortIncompleteMultipartUpload": {
> > >> "DaysAfterInitiation": 1
> > >> }
> > >> }
> > >> ]
> > >> }
> > >>
> > >> ___
> > >> ceph-users mailing list -- ceph-users@ceph.io
> > >> To unsubscribe send an email to ceph-users-le...@ceph.io
> > > ___
> > > ceph-users mailing list -- ceph-users@c

[ceph-users] Abysmal performance in Ceph cluster

[Loschwitz,Martin Gerhard]

Folks,

we’re building a Ceph cluster based on HDDs with SSDs for WAL/DB files. We have 
four nodes with 8TB 
disks and two SSDs and four nodes with many small HDDs (1.4-2.7TB) and four 
SSDs for the journals. 
HDDs are configured as RAID 0 on the controllers with writethrough enabled. I 
am writing this e-mail as
we see absolutely catastrophic performance on the cluster (I am talking about 
anything between literally
no throughput for seconds to 200mb/s, wildly varying).

We’ve checked very single layer: network is far from being saturated (we have 
25gbit/s uplinks and we
can confirm that they deliver 25gbit/s). Using iostat, we can prove that during 
a „rados bench“ call neither
the SSDs nor the actual hard disks are anywhere near 100% disk utilization. 
Usually usage does not
exceed 55%.

Servers are Dell RX740d with PERC onboard controllers. 

We also have four SSD-only nodes. When benchmarking against a pool on these, I 
reliably get 400-500
mb/s when doing the same 64k size test that I ran against the HDD pool.

We’ve tried a number of things such as enabling the BBWC on the RAID 
controllers to no major success.
„ceph tell osd.X bench“ will show 55-100 iops for HDDs even though their 
journals are on SSDs. We
have also tried disabling the SSD’s write cache (the ones with the journals on 
them) to no success.

Any pointer to what we may haver overseen would be greatly appreciated. 

Best regards
Martin
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Remapped PGs

[David Orman]

Hi,

We see that we have 5 'remapped' PGs, but are unclear why/what to do about
it. We shifted some target ratios for the autobalancer and it resulted in
this state. When adjusting ratio, we noticed two OSDs go down, but we just
restarted the container for those OSDs with podman, and they came back up.
Here's status output:

###
root@ceph01:~# ceph status
INFO:cephadm:Inferring fsid x
INFO:cephadm:Inferring config x
INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15
  cluster:
id: 41bb9256-c3bf-11ea-85b9-9e07b0435492
health: HEALTH_OK

  services:
mon: 5 daemons, quorum ceph01,ceph04,ceph02,ceph03,ceph05 (age 2w)
mgr: ceph03.ytkuyr(active, since 2w), standbys: ceph01.aqkgbl,
ceph02.gcglcg, ceph04.smbdew, ceph05.yropto
osd: 168 osds: 168 up (since 2d), 168 in (since 2d); 5 remapped pgs

  data:
pools:   3 pools, 1057 pgs
objects: 18.00M objects, 69 TiB
usage:   119 TiB used, 2.0 PiB / 2.1 PiB avail
pgs: 1056 active+clean
 1active+clean+scrubbing+deep

  io:
client:   859 KiB/s rd, 212 MiB/s wr, 644 op/s rd, 391 op/s wr

root@ceph01:~#

###

When I look at ceph pg dump, I don't see any marked as remapped:

###
root@ceph01:~# ceph pg dump |grep remapped
INFO:cephadm:Inferring fsid x
INFO:cephadm:Inferring config x
INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15
dumped all
root@ceph01:~#
###

Any idea what might be going on/how to recover? All OSDs are up. Health is
'OK'. This is Ceph 15.2.4 deployed using Cephadm in containers, on Podman
2.0.3.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RGW versioned objects lost after Octopus 15.2.3 -> 15.2.4 upgrade

[Daniel Poelzleithner]

On 2020-08-05 15:23, Matt Benjamin wrote:

> There is new lifecycle processing logic backported to Octopus, it
> looks like, in 15.2.3.  I'm looking at the non-current calculation to
> see if it could incorrectly rely on a stale value (from an eralier
> entry).

So, you don't care about samever ?

Replacing a lifecyle processing logic in a patch version is, sorry to
say this, a nogo. At least, don't make the impression to use semantic
versioning then.

kind regards
 poelzi
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Change crush rule on pool

[jesper]

Hi

I would like to change the crush rule so data lands on ssd instead of hdd, can 
this be done on the fly and migration will just happen or do I need to do 
something to move data?

Jesper



Sent from myMail for iOS
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] module cephadm has failed

[Daniel Davidson]

I am having trouble getting rid of an error after creating a new ceph 
cluster.  The error is:


Module 'cephadm' has failed: auth get failed: failed to find 
client.crash.ceph-0 in keyring retval: -2


Checking the keyrings and disguising the keys:

# ceph auth ls
...
client.crash.ceph-0.data.igb.illinois.edu
    key: keyA
    caps: [mgr] profile crash
    caps: [mon] profile crash
...
mgr.ceph-0.uymvya
    key: keyB
    caps: [mds] allow *
    caps: [mon] profile mgr
    caps: [osd] allow *
...

Then on the node:

# cat mgr.ceph-0.uymvya/keyring
[mgr.ceph-0.uymvya]
    key = keyB
    caps mds = "allow *"
    caps mon = "profile mgr"
    caps osd = "allow *"

So the keys match where they should, but I am thinking that the domain 
missing off the mgr might be the issue.  This is keeping me from getting 
rados working since the cluster is showing a health issue.  It looks 
like this is an issue in my version of ceph (15.2.4)?  Can someone point 
me to the fix or at minimum tell me how to remove the mgr on ceph-0 from 
the config so that I can get the system healthy.


thanks,

Dan
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Change crush rule on pool

[Nathan Fish]

If you create the new rule, and set the pool in question to it, then
data movement will begin automatically. Be warned this might increase
your load for a long time.

On Wed, Aug 5, 2020 at 12:20 PM  wrote:
>
>
> Hi
>
> I would like to change the crush rule so data lands on ssd instead of hdd, 
> can this be done on the fly and migration will just happen or do I need to do 
> something to move data?
>
> Jesper
>
>
>
> Sent from myMail for iOS
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Quick interruptions in the Ceph cluster

[Gesiel Galvão Bernardes]

Hi,

I have been experiencing rapid outage events in the Ceph cluster. During
these events I receive messages from slow ops, OSD downs, but at the same
time it is operating. Magically everything is back to normal. These events
usually last about 2 minutes.
I couldn't find anything that could direct me to what causes these events,
can you help me?

I'm using Mimic (13.2.6) and CentOS7 on all nodes.
Below is "ceph -s" and log of when event occurs:

# ceph -s
  cluster:
id: 4ea72929-6f9e-453a-8cd5-bb0712f6b874
health: HEALTH_OK

  services:
mon: 2 daemons, quorum cmonitor,cmonitor2
mgr: cmonitor(active), standbys: cmonitor2
osd: 74 osds: 74 up, 74 in
tcmu-runner: 10 daemons active

  data:
pools:   7 pools, 3072 pgs
objects: 22.17 M objects, 83 TiB
usage:   225 TiB used, 203 TiB / 428 TiB avail
pgs: 3063 active+clean
 9active+clean+scrubbing+deep

==
Log of event:
2020-08-05 18:00:00.000179 [INF]  overall HEALTH_OK
2020-08-05 17:55:28.905024 [INF]  Cluster is now healthy
2020-08-05 17:55:28.904975 [INF]  Health check cleared: PG_DEGRADED (was:
Degraded data redundancy: 1/60350974 objects degraded (0.000%), 1 pg
degraded)
2020-08-05 17:55:27.746606 [WRN]  Health check update: Degraded data
redundancy: 1/60350974 objects degraded (0.000%), 1 pg degraded
(PG_DEGRADED)
2020-08-05 17:55:22.745820 [WRN]  Health check update: Degraded data
redundancy: 55/60350897 objects degraded (0.000%), 26 pgs degraded, 1 pg
undersized (PG_DEGRADED)
2020-08-05 17:55:17.744218 [WRN]  Health check update: Degraded data
redundancy: 123/60350666 objects degraded (0.000%), 63 pgs degraded
(PG_DEGRADED)
2020-08-05 17:55:12.743568 [WRN]  Health check update: Degraded data
redundancy: 192/60350660 objects degraded (0.000%), 88 pgs degraded
(PG_DEGRADED)
2020-08-05 17:55:07.741759 [WRN]  Health check update: Degraded data
redundancy: 290/60350737 objects degraded (0.000%), 117 pgs degraded
(PG_DEGRADED)
2020-08-05 17:55:02.737913 [WRN]  Health check update: Degraded data
redundancy: 299/60350764 objects degraded (0.000%), 119 pgs degraded
(PG_DEGRADED)
2020-08-05 17:54:57.736694 [WRN]  Health check update: Degraded data
redundancy: 299/60350746 objects degraded (0.000%), 119 pgs degraded
(PG_DEGRADED)
2020-08-05 17:54:52.736132 [WRN]  Health check update: Degraded data
redundancy: 299/60350731 objects degraded (0.000%), 119 pgs degraded
(PG_DEGRADED)
2020-08-05 17:54:47.735612 [WRN]  Health check update: Degraded data
redundancy: 299/60350689 objects degraded (0.000%), 119 pgs degraded
(PG_DEGRADED)
2020-08-05 17:54:42.734877 [WRN]  Health check update: Degraded data
redundancy: 301/60350677 objects degraded (0.000%), 120 pgs degraded
(PG_DEGRADED)
2020-08-05 17:54:38.210906 [INF]  Health check cleared: SLOW_OPS (was: 35
slow ops, oldest one blocked for 1954017 sec, daemons
[mon.cmonitor,mon.cmonitor2] have slow ops.)
2020-08-05 17:54:37.734218 [WRN]  Health check update: 35 slow ops, oldest
one blocked for 1954017 sec, daemons [mon.cmonitor,mon.cmonitor2] have slow
ops. (SLOW_OPS)
2020-08-05 17:54:37.734132 [WRN]  Health check update: Degraded data
redundancy: 380/60350611 objects degraded (0.001%), 154 pgs degraded
(PG_DEGRADED)
2020-08-05 17:54:34.171483 [INF]  Health check cleared: PG_AVAILABILITY
(was: Reduced data availability: 3 pgs inactive, 6 pgs peering)
2020-08-05 17:54:32.733499 [WRN]  Health check update: Degraded data
redundancy: 52121/60350544 objects degraded (0.086%), 211 pgs degraded
(PG_DEGRADED)
2020-08-05 17:54:27.080529 [WRN]  Monitor daemon marked osd.72 down, but it
is still running
2020-08-05 17:54:32.102889 [WRN]  Health check failed: 60 slow ops, oldest
one blocked for 1954017 sec, daemons
[osd.16,osd.22,osd.23,osd.27,osd.28,osd.29,osd.30,osd.35,osd.48,osd.5]...
have slow ops. (SLOW_OPS)
2020-08-05 17:54:32.102699 [WRN]  Health check update: Reduced data
availability: 3 pgs inactive, 6 pgs peering (PG_AVAILABILITY)
2020-08-05 17:54:27.951343 [INF]  osd.72 192.168.200.25:6844/64565 boot
2020-08-05 17:54:27.935996 [INF]  Health check cleared: OSD_DOWN (was: 1
osds down)
2020-08-05 17:54:27.732679 [WRN]  Health check update: Degraded data
redundancy: 1781748/60350443 objects degraded (2.952%), 381 pgs degraded
(PG_DEGRADED)
2020-08-05 17:54:26.916269 [WRN]  Health check update: Reduced data
availability: 5 pgs inactive, 10 pgs peering, 18 pgs incomplete
(PG_AVAILABILITY)
2020-08-05 17:54:19.712714 [WRN]  Monitor daemon marked osd.71 down, but it
is still running
2020-08-05 17:54:22.716043 [WRN]  Health check update: Degraded data
redundancy: 4057630/60350485 objects degraded (6.723%), 566 pgs degraded
(PG_DEGRADED)
2020-08-05 17:54:22.715939 [WRN]  Health check update: 1 osds down
(OSD_DOWN)
2020-08-05 17:54:14.042363 [WRN]  Monitor daemon marked osd.62 down, but it
is still running
2020-08-05 17:54:20.858582 [WRN]  Health check update: Reduced data
availability: 8 pgs inactive, 15 pgs peering

[ceph-users] Re: RGW versioned objects lost after Octopus 15.2.3 -> 15.2.4 upgrade

[Matt Benjamin]

The lifecycle changes in question do not change the semantics nor any
api of lifecycle.  The behavior change was a regression.

regards,

Matt

On Wed, Aug 5, 2020 at 12:12 PM Daniel Poelzleithner  wrote:
>
> On 2020-08-05 15:23, Matt Benjamin wrote:
>
> > There is new lifecycle processing logic backported to Octopus, it
> > looks like, in 15.2.3.  I'm looking at the non-current calculation to
> > see if it could incorrectly rely on a stale value (from an eralier
> > entry).
>
> So, you don't care about samever ?
>
> Replacing a lifecyle processing logic in a patch version is, sorry to
> say this, a nogo. At least, don't make the impression to use semantic
> versioning then.
>
> kind regards
>  poelzi
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>


-- 

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: help me enable ceph iscsi gatewaty in ceph octopus

[Sharad Mehrotra]

Sebastian et al:

How did you solve the "The first gateway
defined must be the local machine" issue that I asked about on another
thread?

I am deploying ceph-iscsi manually as described in the link that you sent
out (https://docs.ceph.com/docs/master/rbd/iscsi-target-cli/).
Thank you!


On Wed, Aug 5, 2020 at 2:37 AM Sebastian Wagner  wrote:

> hi David, hi Ricardo,
>
> I think we first have to clarify, if that was actually a cephadm
> deployment (and not ceph-ansible).
>
> If you install Ceph using ceph-ansible, then please refer to the
> ceph-ansible docs.
>
> If we're actually talking about cephadm here (which is not clear to me):
> iSCSI for cephadm will land in the next octopus release and at that
> point we can add a proper documentation.
>
>
>
> Hope that helps,
>
> Sebastian
>
> Am 05.08.20 um 11:11 schrieb Ricardo Marques:
> > Hi David,
> >
> > I was able to configure iSCSI gateways on my local test environment
> using the following spec:
> >
> > ```
> > # tail -14 service_spec_gw.yml
> > ---
> > service_type: iscsi
> > service_id: iscsi_service
> > placement:
> > hosts:
> > - 'node1'
> > - 'node2'
> > spec:
> > pool: rbd
> > trusted_ip_list: 10.20.94.201,10.20.94.202,10.20.94.203
> > api_port: 5000
> > api_user: admin1
> > api_password: admin2
> > api_secure: False
> >
> > # ceph orch apply -i service_spec_gw.yml
> > ```
> >
> > You can use this spec as a starting point, but note that the pool must
> exist (in this case `rbd` pool), and you will need to adapt `hosts`,
> `trusted_ip_list`, etc...
> >
> > You may also want to change `api_secure` to `True` and set `ssl_cert`
> and `ssh_key` accordingly.
> >
> > Unfortunately, iSCSI deployment is not included in the documentation yet
> (coming soon).
> >
> > [1] https://docs.ceph.com/docs/octopus/cephadm/install/
> >
> >
> > Ricardo Marques
> >
> > 
> > From: David Thuong 
> > Sent: Wednesday, August 5, 2020 5:16 AM
> > To: ceph-users@ceph.io 
> > Subject: [ceph-users] help me enable ceph iscsi gatewaty in ceph octopus
> >
> > Please help me enable ceph iscsi gatewaty in ceph octopus . when i
> install ceph complete . i see iscsi gateway not enable. please help me
> config it
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
>
> --
> SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg,
> Germany
> (HRB 36809, AG Nürnberg). Geschäftsführer: Felix Imendörffer
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Nautilus slow using "ceph tell osd.* bench"

[Jim Forde]

I have 2 clusters. Cluster 1 started at Hammer and has upgraded through the 
versions all the way to Nautilus 14.2.10 (Luminous to Nautilus in July 2020) . 
Cluster 2 started as Luminous and is now Nautilus 14.2.2 (Upgraded in September 
2019) The clusters are basically identical 5 OSD Nodes with 6 osd's per node. 
They are both using disk drives. No SSD's. Prior to upgrading Cluster 1 running 
"ceph tell osd.0 bench -f plain"
produced similar results across both clusters.
ceph tell osd.0 bench -f plain
bench: wrote 1 GiB in blocks of 4 MiB in 0.954819 sec at 1.0 GiB/sec 268 IOPS
Now cluster 1 results are terrible, about 25% from before the upgrade.
ceph tell osd.0 bench -f plain
bench: wrote 1 GiB in blocks of 4 MiB in 4.03434 sec at 254 MiB/sec 63 IOPS
Ceph -s shows HEALTH_OK. Dashboard looks good. 2 pools 
MON Dump
min_mon_release 14 (nautilus)
OSD Dump
require_min_compat_client luminous
min_compat_client jewel
require_osd_release nautilus

Not sure what is causing the slow performance. Ideas?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] librbd Image Watcher Errors

[Ml Ml]

Hello,

with cron i run backups with backurne
(https://github.com/JackSlateur/backurne) which is rbd based.

Sometimes i get those messages:

2020-08-05T18:42:18.915+0200 7fcdbd7fa700 -1 librbd::ImageWatcher:
0x7fcda400a6a0 image watch failed: 140521330717776, (107) Transport
endpoint is not connected
2020-08-05T18:42:18.915+0200 7fcdbd7fa700 -1 librbd::Watcher:
0x7fcda400a6a0 handle_error: handle=140521330717776: (107) Transport
endpoint is not connected
2020-08-05T18:45:13.496+0200 7fcdbd7fa700 -1 librbd::ImageWatcher:
0x7fcda400a6a0 image watch failed: 140521330717776, (110) Connection
timed out
2020-08-05T18:45:13.496+0200 7fcdbd7fa700 -1 librbd::Watcher:
0x7fcda400a6a0 handle_error: handle=140521330717776: (110) Connection
timed out
2020-08-05T18:58:01.408+0200 7fc66700 -1 librbd::ImageWatcher:
0x7fc65c00a770 image watch failed: 140490057987152, (107) Transport
endpoint is not connected
2020-08-05T18:58:01.408+0200 7fc66700 -1 librbd::Watcher:
0x7fc65c00a770 handle_error: handle=140490057987152: (107) Transport
endpoint is not connected

Any idea what this indicates and how it could be solved?

Thanks,
Michael
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: help me enable ceph iscsi gatewaty in ceph octopus

[Sharad Mehrotra]

Adding some additional context for my question below.

I am following the directions here:
https://docs.ceph.com/docs/master/rbd/iscsi-target-cli/, but am getting
stuck on step #3 of the "Configuring" section, similar to the issue
reported above that you worked on.

FYI, I installed my ceph-iscsi package manually using these directions:
https://docs.ceph.com/docs/master/rbd/iscsi-target-cli-manual-install, and
I am running CentOS 7.6 on the nodes where I am installing the iscsi
gateways.

Here is what happens when I try to create the first gateway:

/iscsi-target...-igw/gateways> create ceph-gw-1 10.0.201.66 skipchecks=true

*The first gateway defined must be the local machine*

/iscsi-target...-igw/gateways>


And here is my iscsi-gateway.cfg file.

[root@sv49 ceph]# cat iscsi-gateway.cfg

[config]

# Name of the Ceph storage cluster. A suitable Ceph configuration file
allowing

# access to the Ceph storage cluster from the gateway node is required, if
not

# colocated on an OSD node.

cluster_name = ceph


# Place a copy of the ceph cluster's admin keyring in the gateway's
/etc/ceph

# directory and reference the filename here

gateway_keyring = ceph.client.admin.keyring


# API settings.

# The API supports a number of options that allow you to tailor it to your

# local environment. If you want to run the API under https, you will need
to

# create cert/key files that are compatible for each iSCSI gateway node,
that is

# not locked to a specific node. SSL cert and key files *must* be called

# 'iscsi-gateway.crt' and 'iscsi-gateway.key' and placed in the
'/etc/ceph/' directory

# on *each* gateway node. With the SSL files in place, you can use
'api_secure = true'

# to switch to https mode.


# To support the API, the bear minimum settings are:

api_secure = false


# Additional API configuration options are as follows, defaults shown.

api_user = admin

api_password = admin

api_port = 5001

trusted_ip_list = 10.0.201.66, 10.0.201.67

Any help you can provide would be appreciated, thanks!


On Wed, Aug 5, 2020 at 11:38 AM Sharad Mehrotra 
wrote:

> Sebastian et al:
>
> How did you solve the "The first gateway
> defined must be the local machine" issue that I asked about on another
> thread?
>
> I am deploying ceph-iscsi manually as described in the link that you sent
> out (https://docs.ceph.com/docs/master/rbd/iscsi-target-cli/).
> Thank you!
>
>
> On Wed, Aug 5, 2020 at 2:37 AM Sebastian Wagner  wrote:
>
>> hi David, hi Ricardo,
>>
>> I think we first have to clarify, if that was actually a cephadm
>> deployment (and not ceph-ansible).
>>
>> If you install Ceph using ceph-ansible, then please refer to the
>> ceph-ansible docs.
>>
>> If we're actually talking about cephadm here (which is not clear to me):
>> iSCSI for cephadm will land in the next octopus release and at that
>> point we can add a proper documentation.
>>
>>
>>
>> Hope that helps,
>>
>> Sebastian
>>
>> Am 05.08.20 um 11:11 schrieb Ricardo Marques:
>> > Hi David,
>> >
>> > I was able to configure iSCSI gateways on my local test environment
>> using the following spec:
>> >
>> > ```
>> > # tail -14 service_spec_gw.yml
>> > ---
>> > service_type: iscsi
>> > service_id: iscsi_service
>> > placement:
>> > hosts:
>> > - 'node1'
>> > - 'node2'
>> > spec:
>> > pool: rbd
>> > trusted_ip_list: 10.20.94.201,10.20.94.202,10.20.94.203
>> > api_port: 5000
>> > api_user: admin1
>> > api_password: admin2
>> > api_secure: False
>> >
>> > # ceph orch apply -i service_spec_gw.yml
>> > ```
>> >
>> > You can use this spec as a starting point, but note that the pool must
>> exist (in this case `rbd` pool), and you will need to adapt `hosts`,
>> `trusted_ip_list`, etc...
>> >
>> > You may also want to change `api_secure` to `True` and set `ssl_cert`
>> and `ssh_key` accordingly.
>> >
>> > Unfortunately, iSCSI deployment is not included in the documentation
>> yet (coming soon).
>> >
>> > [1] https://docs.ceph.com/docs/octopus/cephadm/install/
>> >
>> >
>> > Ricardo Marques
>> >
>> > 
>> > From: David Thuong 
>> > Sent: Wednesday, August 5, 2020 5:16 AM
>> > To: ceph-users@ceph.io 
>> > Subject: [ceph-users] help me enable ceph iscsi gatewaty in ceph octopus
>> >
>> > Please help me enable ceph iscsi gatewaty in ceph octopus . when i
>> install ceph complete . i see iscsi gateway not enable. please help me
>> config it
>> > ___
>> > ceph-users mailing list -- ceph-users@ceph.io
>> > To unsubscribe send an email to ceph-users-le...@ceph.io
>> >
>> > ___
>> > ceph-users mailing list -- ceph-users@ceph.io
>> > To unsubscribe send an email to ceph-users-le...@ceph.io
>> >
>>
>> --
>> SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg,
>> Germany
>> (HRB 36809, AG Nürnberg). Geschäftsführer: Felix Imendörffer
>>
>> ___
>> ceph-users mailing 

[ceph-users] Re: Nautilus slow using "ceph tell osd.* bench"

[rainning]

Hi Jim, did you check system stat (e.g. iostat, top, etc.) on both osds when 
you ran osd bench? Those might be able to give you some clues. Moreover, did 
you compare both osds' configurations?





-- Original --
From:   
 "Jim Forde"

[ceph-users] Authorized UV Printer Supplier in India - PH UV Printer

[ela1789brown]

If you are really searching for the best and most reputed UV printer supplier 
in India then there is only one name i.e. PH UV Printer 
https://phuvprinter.com/. it is the only UV printer supplier having the wide 
range of best quality UV printers allows you to print custom design over your 
products. With the help of this best quality UV printing you can easily print 
custom design over your products to make them more unique. If you want to know 
more then visit the official website of PH UV Printer.

For more entertaining content visit: https://shayarikapitara.com/sorry-shayari/
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] made a huge mistake, seeking recovery advice (osd zapped)

[Peter Sarossy]

hey folks,

I was deploying a new set of NVMe cards into my cluster, and while getting
the new devices ready, it seems the device names got mixed up, and I
managed to to run "sgdisk --zap-all" and "dd if=/dev/zero of="/dev/sd"
bs=1M count=100" on some of the active devices.

I was adding new cards so I could migrate off the 2k+2m erasure coded setup
to a more redundant config, but in my mess up I ran the commands above on 3
of the 4 devices before the ceph status changed and I noticed the mistake.

I managed to restore the LVM partition table from backup but it seems to
not be enough to restart the OSD... I just need to recover one of the 3
drives to save all of my VM+Docker backing filesystem.

I'm running on Kubernetes with Rook, after restoring the partition table it
seems to be starting up ok, but then I get a stack trace and the container
goes into Error state: https://pastebin.com/5wk1bKy9

Any ideas how to fix this? Or somehow extract the data and put it back
together?


-- 
Cheers,
Peter Sarossy
Technical Program Manager
Data Center Data Security - Google LLC.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Does Cash App Customer Service request for SSN?

[mary smith]

Indeed, Cash App Customer Service requests your SSN to check your identity. It 
is fundamentally requested any government hostile to illegal tax avoidance and 
against tax avoidance. Here is a law, that makes it important to give your SSN. 
You don't have a need to enter your full SSN, enter just last four digits of 
it. Having such kind of warning permits them to affirm your identity, and by 
which they can easily catch dubious action. 
https://www.cash-app-customer-service.com/
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Get Cash App Customer Service to ask to for what reason is my cash not appearing?

[mary smith]

In the event that you are not seeing your cash into your Cash app balance, at 
that point you have to guarantee that, is there any telephone number or email 
address that is related to you. Now and again, it happens cash goes to another 
telephone number or email address which is lined up with you. In the event that 
you get any issue here, at that point go to converse with Cash App Customer 
Service. 
https://www.customercare-email.com/customer-service/cash-app.html
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Many scrub errors after update to 14.2.10

[Andreas Haupt]

Hi *,

after updating our CEPH cluster from 14.2.9 to 14.2.10 it accumulates
scrub errors on multiple osds:

[cephmon1] /root # ceph health detail
HEALTH_ERR 6 scrub errors; Possible data damage: 6 pgs inconsistent
OSD_SCRUB_ERRORS 6 scrub errors
PG_DAMAGED Possible data damage: 6 pgs inconsistent
pg 3.69 is active+clean+inconsistent, acting [59,65,61]
pg 3.73 is active+clean+inconsistent, acting [73,88,25]
pg 12.29 is active+clean+inconsistent, acting [55,92,42]
pg 12.38 is active+clean+inconsistent, acting [150,42,13]
pg 12.46 is active+clean+inconsistent, acting [55,18,84]
pg 12.75 is active+clean+inconsistent, acting [55,155,49]

They all can easily get repaired (ceph pg repair $pg) - but I wonder
what could be the source of the problem. The cluster started with
Luminous some years ago, was updated to Mimic, then Nautilus. Never
seen this before!

OSDs are a mixture of HDD/SSD, both are affected. All on Bluestore.

Any idea? Was there maybe a code change between 14.2.9 & 14.2.10 that
could explain this? Errors in syslog look like this:

Aug  5 19:21:21 krake08 ceph-osd: 2020-08-05 19:21:21.831 7fb6b2b9d700 -1 
log_channel(cluster) log [ERR] : 12.38 scrub : stat mismatch, got 74/74 
objects, 20/20 clones, 74/74 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 
0/0 whiteouts, 182904850/172877842 bytes, 0/0 manifest objects, 0/0 
hit_set_archive bytes.
Aug  5 19:21:21 krake08 ceph-osd: 2020-08-05 19:21:21.831 7fb6b2b9d700 -1 
log_channel(cluster) log [ERR] : 12.38 scrub 1 errors
Aug  6 08:28:44 krake08 ceph-osd: 2020-08-06 08:28:44.477 7fb6b2b9d700 -1 
log_channel(cluster) log [ERR] : 12.38 repair : stat mismatch, got 76/76 
objects, 22/22 clones, 76/76 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 
0/0 whiteouts, 183166994/173139986 bytes, 0/0 manifest objects, 0/0 
hit_set_archive bytes.
Aug  6 08:28:44 krake08 ceph-osd: 2020-08-06 08:28:44.477 7fb6b2b9d700 -1 
log_channel(cluster) log [ERR] : 12.38 repair 1 errors, 1 fixed

Thanks in advance,
Andreas
-- 
| Andreas Haupt| E-Mail: andreas.ha...@desy.de
|  DESY Zeuthen| WWW:http://www-zeuthen.desy.de/~ahaupt
|  Platanenallee 6 | Phone:  +49/33762/7-7359
|  D-15738 Zeuthen | Fax:+49/33762/7-7216



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io