[ceph-users] Re: PGs down

[Igor Fedotov]

Hi Jeremy,

good to know you managed to bring your OSDs up.

Have you been able to reweight them to 0 and migrate data out of these 
"broken" OSDs?


If so I suggest to redeploy them - the corruption is still in the DB and 
it might pop-up one day.


If not please do that first - you might still hit into issues along the 
process since DB is still corrupted. Disabling compaction just bypassed 
the reads from broken files - but this might happen again during 
different operations.



On 12/22/2020 7:17 AM, Jeremy Austin wrote:

Igor,

You're a bloomin' genius, as they say.

Disabling auto compaction allowed OSDs 11 and 12 to spin up/out. The 7 
down PGs recovered; there were a few unfound items previously which I 
went ahead and deleted, given that this is EC, revert not being an option.


HEALTH OK :)

I'm now intending to re-enable auto compaction. Should I also fsck the 
rest of the OSDs, or is the typical scrub/deep scrub cycle sufficient? 
(No PGs are behind on scrubbing, whereas they were during the degraded 
period.)


Please do not re-enable the compaction before you're ready to kill 
"broken" OSDs...


There is no much sense in fsck-ing other OSDs - this is unlikely to 
detect inconsistencies at PG level if any.


Hence it's [deep]-scrub which you might want to apply manually once 
reweight/rebalance is done. PGs located at "broken" OSDs are of the top 
priority to scrub.





Time will tell if I actually lost data, I guess.

On Mon, Dec 21, 2020 at 8:37 AM Igor Fedotov > wrote:


Hi Jeremy,

you might want to try RocksDB's disable_auto_compactions option
for that.

To adjust rocksdb's options one should  edit/insert
bluestore_rocksdb_options in ceph.conf.

E.g.

bluestore_rocksdb_options =

"disable_auto_compactions=true,compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824"


Please note bluestore specific defaults for rocksdb settings are
re-provided to make sure they aren't reset to rocksdb's ones.

Hope this helps.

Thanks,

Igor







On 12/21/2020 2:56 AM, Jeremy Austin wrote:



On Sun, Dec 20, 2020 at 2:25 PM Jeremy Austin mailto:jhaus...@gmail.com>> wrote:

Will attempt to disable compaction and report.


Not sure I'm doing this right. In [osd] section of ceph.conf, I added
periodic_compaction_seconds=0

and attempted to start the OSDs in question. Same error as
before. Am I setting compaction options correctly?


-- 
Jeremy Austin

jhaus...@gmail.com 




--
Jeremy Austin
jhaus...@gmail.com 

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] kvm vm cephfs mount hangs on osd node (something like umount -l available?) (help wanted going to production)

[Marc Roos]

I have a vm on a osd node (which can reach host and other nodes via the 
macvtap interface (used by the host and guest)). I just did a simple 
bonnie++ test and everything seems to be fine. Yesterday however the 
dovecot procces apparently caused problems (only using cephfs for an 
archive namespace, inbox is on rbd ssd, fs meta also on ssd)

How can I recover from such lock-up. If I have a similar situation with 
an nfs-ganesha mount, I have the option to do a umount -l, and clients 
recover quickly without any issues.

Having to reset the vm, is not really an option. What is best way to 
resolve this?



Ceph cluster: 14.2.11 (the vm has 14.2.16)

I have in my ceph.conf nothing special, these 2x in the mds section:

mds bal fragment size max = 12
# maybe for nfs-ganesha problems?
# http://docs.ceph.com/docs/master/cephfs/eviction/
#mds_session_blacklist_on_timeout = false
#mds_session_blacklist_on_evict = false
mds_cache_memory_limit = 17179860387


All running:
CentOS Linux release 7.9.2009 (Core)
Linux mail04 3.10.0-1160.6.1.el7.x86_64 #1 SMP Tue Nov 17 13:59:11 UTC 
2020 x86_64 x86_64 x86_64 GNU/Linux
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: kvm vm cephfs mount hangs on osd node (something like umount -l available?) (help wanted going to production)

[Marc Roos]

Just got this during bonnie test, trying to do an ls -l on the cephfs. I 
also have this kworker process constantly at 40% when doing this 
bonnie++ test.

[35281.101763] INFO: task bash:1169 blocked for more than 120 seconds.
[35281.102064] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[35281.102175] bashD a03fbfc9acc0 0  1169   1167 
0x0004
[35281.102181] Call Trace:
[35281.102275]  [] ? __schedule+0x3af/0x860
[35281.102285]  [] schedule+0x29/0x70
[35281.102296]  [] schedule_timeout+0x221/0x2d0
[35281.102332]  [] ? finish_wait+0x56/0x70
[35281.102342]  [] ? mutex_lock+0x12/0x2f
[35281.102381]  [] ? autofs4_wait+0x428/0x920
[35281.102386]  [] wait_for_completion+0xfd/0x140
[35281.102407]  [] ? wake_up_state+0x20/0x20
[35281.102422]  [] autofs4_expire_wait+0xab/0x160
[35281.102425]  [] do_expire_wait+0x1e0/0x210
[35281.102429]  [] autofs4_d_manage+0x73/0x1c0
[35281.102455]  [] follow_managed+0xba/0x310
[35281.102459]  [] lookup_fast+0x12d/0x230
[35281.102464]  [] path_lookupat+0x16d/0x8d0
[35281.102467]  [] ? do_last+0x66d/0x1340
[35281.102488]  [] ? __check_object_size+0x1ca/0x250
[35281.102499]  [] ? kmem_cache_alloc+0x35/0x1f0
[35281.102503]  [] ? getname_flags+0x4f/0x1a0
[35281.102507]  [] filename_lookup+0x2b/0xc0
[35281.102510]  [] user_path_at_empty+0x67/0xc0
[35281.102513]  [] user_path_at+0x11/0x20
[35281.102516]  [] vfs_fstatat+0x63/0xc0
[35281.102519]  [] SYSC_newstat+0x2e/0x60
[35281.102529]  [] ? 
system_call_after_swapgs+0xa2/0x13a
[35281.102533]  [] ? 
system_call_after_swapgs+0x96/0x13a
[35281.102536]  [] ? 
system_call_after_swapgs+0xa2/0x13a
[35281.102539]  [] ? 
system_call_after_swapgs+0x96/0x13a
[35281.102543]  [] ? 
system_call_after_swapgs+0xa2/0x13a
[35281.102546]  [] ? 
system_call_after_swapgs+0x96/0x13a
[35281.102549]  [] ? 
system_call_after_swapgs+0xa2/0x13a
[35281.102552]  [] ? 
system_call_after_swapgs+0x96/0x13a
[35281.102555]  [] ? 
system_call_after_swapgs+0xa2/0x13a
[35281.102558]  [] ? 
system_call_after_swapgs+0x96/0x13a
[35281.102561]  [] ? 
system_call_after_swapgs+0xa2/0x13a
[35281.102565]  [] SyS_newstat+0xe/0x10
[35281.102568]  [] system_call_fastpath+0x25/0x2a
[35281.102572]  [] ? 
system_call_after_swapgs+0xa2/0x13a
 

-Original Message-
To: ceph-users
Subject: [ceph-users] kvm vm cephfs mount hangs on osd node (something 
like umount -l available?) (help wanted going to production)



I have a vm on a osd node (which can reach host and other nodes via the 
macvtap interface (used by the host and guest)). I just did a simple 
bonnie++ test and everything seems to be fine. Yesterday however the
dovecot procces apparently caused problems (only using cephfs for an 
archive namespace, inbox is on rbd ssd, fs meta also on ssd)

How can I recover from such lock-up. If I have a similar situation with 
an nfs-ganesha mount, I have the option to do a umount -l, and clients 
recover quickly without any issues.

Having to reset the vm, is not really an option. What is best way to 
resolve this?



Ceph cluster: 14.2.11 (the vm has 14.2.16)

I have in my ceph.conf nothing special, these 2x in the mds section:

mds bal fragment size max = 12
# maybe for nfs-ganesha problems?
# http://docs.ceph.com/docs/master/cephfs/eviction/
#mds_session_blacklist_on_timeout = false
#mds_session_blacklist_on_evict = false
mds_cache_memory_limit = 17179860387


All running:
CentOS Linux release 7.9.2009 (Core)
Linux mail04 3.10.0-1160.6.1.el7.x86_64 #1 SMP Tue Nov 17 13:59:11 UTC 
2020 x86_64 x86_64 x86_64 GNU/Linux
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: kvm vm cephfs mount hangs on osd node (something like umount -l available?) (help wanted going to production)

[Marc Roos]

I can live-migrate the vm in this locked up state to a different host 
without any problems.


-Original Message-
To: ceph-users
Subject: [ceph-users] kvm vm cephfs mount hangs on osd node (something 
like umount -l available?) (help wanted going to production)



I have a vm on a osd node (which can reach host and other nodes via the 
macvtap interface (used by the host and guest)). I just did a simple 
bonnie++ test and everything seems to be fine. Yesterday however the
dovecot procces apparently caused problems (only using cephfs for an 
archive namespace, inbox is on rbd ssd, fs meta also on ssd)

How can I recover from such lock-up. If I have a similar situation with 
an nfs-ganesha mount, I have the option to do a umount -l, and clients 
recover quickly without any issues.

Having to reset the vm, is not really an option. What is best way to 
resolve this?



Ceph cluster: 14.2.11 (the vm has 14.2.16)

I have in my ceph.conf nothing special, these 2x in the mds section:

mds bal fragment size max = 12
# maybe for nfs-ganesha problems?
# http://docs.ceph.com/docs/master/cephfs/eviction/
#mds_session_blacklist_on_timeout = false
#mds_session_blacklist_on_evict = false
mds_cache_memory_limit = 17179860387


All running:
CentOS Linux release 7.9.2009 (Core)
Linux mail04 3.10.0-1160.6.1.el7.x86_64 #1 SMP Tue Nov 17 13:59:11 UTC 
2020 x86_64 x86_64 x86_64 GNU/Linux
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Can big data use Ceph?

[fantastic2085]

Can big data use Ceph?For example, can Hive Hbase Spark use Ceph?
https://github.com/ceph/cephfs-hadoop is no longer maintain?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Can big data use Ceph?

[Brian :]

Have a search for cern and ceph.

On Tuesday, December 22, 2020, fantastic2085  wrote:
> Can big data use Ceph?For example, can Hive Hbase Spark use Ceph?
> https://github.com/ceph/cephfs-hadoop is no longer maintain?
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Can big data use Ceph?

[Marc Roos]

 
I am not really familiar with spark, but I see it often used in 
combination with mesos. They currently implemented a csi solution that 
should enable access to ceph. I have been trying to get this to work[1]

I assume being able to scale tasks with distributed block devices or the 
cephfs would get you into the right direction not?

[1]
https://www.mail-archive.com/user@mesos.apache.org/msg10745.html


-Original Message-
From: fantastic2085 [mailto:fantastic2...@126.com] 
Sent: 22 December 2020 11:29
To: ceph-users@ceph.io
Subject: [ceph-users] Can big data use Ceph?

Can big data use Ceph?For example, can Hive Hbase Spark use Ceph?
https://github.com/ceph/cephfs-hadoop is no longer maintain?
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an 
email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: kvm vm cephfs mount hangs on osd node (something like umount -l available?) (help wanted going to production)

[Marc Roos]

Is there not some genius out there that can shed a ligth on this? ;) 
Currently I am not able to reproduce this. Thus it would be nice to have 
some procedure at hand that resolves stale cephfs mounts nicely.


-Original Message-
To: ceph-users
Subject: [ceph-users] kvm vm cephfs mount hangs on osd node (something 
like umount -l available?) (help wanted going to production)



I have a vm on a osd node (which can reach host and other nodes via the 
macvtap interface (used by the host and guest)). I just did a simple 
bonnie++ test and everything seems to be fine. Yesterday however the
dovecot procces apparently caused problems (only using cephfs for an 
archive namespace, inbox is on rbd ssd, fs meta also on ssd)

How can I recover from such lock-up. If I have a similar situation with 
an nfs-ganesha mount, I have the option to do a umount -l, and clients 
recover quickly without any issues.

Having to reset the vm, is not really an option. What is best way to 
resolve this?



Ceph cluster: 14.2.11 (the vm has 14.2.16)

I have in my ceph.conf nothing special, these 2x in the mds section:

mds bal fragment size max = 12
# maybe for nfs-ganesha problems?
# http://docs.ceph.com/docs/master/cephfs/eviction/
#mds_session_blacklist_on_timeout = false
#mds_session_blacklist_on_evict = false
mds_cache_memory_limit = 17179860387


All running:
CentOS Linux release 7.9.2009 (Core)
Linux mail04 3.10.0-1160.6.1.el7.x86_64 #1 SMP Tue Nov 17 13:59:11 UTC 
2020 x86_64 x86_64 x86_64 GNU/Linux
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Can big data use Ceph?

[Matt Benjamin]

Ceph RGW is frequently used as a backing store for Hadoop and Spark
(S3A connector).

Matt

On Tue, Dec 22, 2020 at 5:29 AM fantastic2085  wrote:
>
> Can big data use Ceph?For example, can Hive Hbase Spark use Ceph?
> https://github.com/ceph/cephfs-hadoop is no longer maintain?
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>


-- 

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: kvm vm cephfs mount hangs on osd node (something like umount -l available?) (help wanted going to production)

[Eugen Block]

Hi,

there have been several threads about hanging cephfs mounts, one quite  
long thread [1] describes a couple of debugging options but also  
mentions to avoid mounting cephfs on OSD nodes in a production  
environment.


Do you see blacklisted clients with 'ceph osd blacklist ls'? If the  
answer is yes try to unblock that client [2].
The same option ('umount -l') is available on a cephfs client, you can  
try that, too. Other options described in [1] are to execute an MDS  
failover, but sometimes a reboot of that VM is the only solution left.


Regards,
Eugen


[1]  
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-August/028719.html
[2]  
https://docs.ceph.com/en/latest/cephfs/eviction/#advanced-un-blocklisting-a-client



Zitat von Marc Roos :


Is there not some genius out there that can shed a ligth on this? ;)
Currently I am not able to reproduce this. Thus it would be nice to have
some procedure at hand that resolves stale cephfs mounts nicely.


-Original Message-
To: ceph-users
Subject: [ceph-users] kvm vm cephfs mount hangs on osd node (something
like umount -l available?) (help wanted going to production)



I have a vm on a osd node (which can reach host and other nodes via the
macvtap interface (used by the host and guest)). I just did a simple
bonnie++ test and everything seems to be fine. Yesterday however the
dovecot procces apparently caused problems (only using cephfs for an
archive namespace, inbox is on rbd ssd, fs meta also on ssd)

How can I recover from such lock-up. If I have a similar situation with
an nfs-ganesha mount, I have the option to do a umount -l, and clients
recover quickly without any issues.

Having to reset the vm, is not really an option. What is best way to
resolve this?



Ceph cluster: 14.2.11 (the vm has 14.2.16)

I have in my ceph.conf nothing special, these 2x in the mds section:

mds bal fragment size max = 12
# maybe for nfs-ganesha problems?
# http://docs.ceph.com/docs/master/cephfs/eviction/
#mds_session_blacklist_on_timeout = false
#mds_session_blacklist_on_evict = false
mds_cache_memory_limit = 17179860387


All running:
CentOS Linux release 7.9.2009 (Core)
Linux mail04 3.10.0-1160.6.1.el7.x86_64 #1 SMP Tue Nov 17 13:59:11 UTC
2020 x86_64 x86_64 x86_64 GNU/Linux
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: osd_pglog memory hoarding - another case

[Kalle Happonen]

For anybody facing similar issues, we wrote a blog post about everything we 
faced, and how we worked through it.

https://cloud.blog.csc.fi/2020/12/allas-november-2020-incident-details.html

Cheers,
Kalle

- Original Message -
> From: "Kalle Happonen" 
> To: "Dan van der Ster" , "ceph-users" 
> 
> Sent: Monday, 14 December, 2020 10:25:32
> Subject: [ceph-users] Re: osd_pglog memory hoarding - another case

> Hi all,
> Ok, so I have some updates on this.
> 
> We noticed that we had a bucket with tons of RGW garbage collection pending. 
> It
> was growing faster than we could clean it up.
> 
> We suspect this was because users tried to do "s3cmd sync" operations on SWIFT
> uploaded large files. This could logically cause issues as s3 and SWIFT
> calculate md5sums differently on large objects.
> 
> The following command showed the pending gc, and also shows which buckets are
> affected.
> 
> radosgw-admin gc list |grep oid >garbagecollectionlist.txt
> 
> Our total RGW GC backlog was up to ~40 M.
> 
> We stopped the main s3sync workflow which was affecting the GC growth. Then we
> started running more aggressive radosgw garbage collection.
> 
> This really helped with the memory use. It dropped a lot, and for now *knock 
> on
> wood* when the GC has been cleaned up, the memory has stayed at a more stable
> lower level.
> 
> So we hope we found the (or a) trigger for the problem.
> 
> Hopefully reveals another thread to pull for others debugging the same issue
> (and for us when we hit it again).
> 
> Cheers,
> Kalle
> 
> - Original Message -
>> From: "Dan van der Ster" 
>> To: "Kalle Happonen" 
>> Cc: "ceph-users" 
>> Sent: Tuesday, 1 December, 2020 16:53:50
>> Subject: Re: [ceph-users] Re: osd_pglog memory hoarding - another case
> 
>> Hi Kalle,
>> 
>> Thanks for the update. Unfortunately I haven't made any progress on
>> understanding the root cause of this issue.
>> (We are still tracking our mempools closely in grafana and in our case
>> they are no longer exploding like in the incident.)
>> 
>> Cheers, Dan
>> 
>> On Tue, Dec 1, 2020 at 3:49 PM Kalle Happonen  wrote:
>>>
>>> Quick update, restarting OSDs is not enough for us to compact the db. So we
>>> stop the osd
>>> ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-$osd compact
>>> start the osd
>>>
>>> It seems to fix the spillover. Until it grows again.
>>>
>>> Cheers,
>>> Kalle
>>>
>>> - Original Message -
>>> > From: "Kalle Happonen" 
>>> > To: "Dan van der Ster" 
>>> > Cc: "ceph-users" 
>>> > Sent: Tuesday, 1 December, 2020 15:09:37
>>> > Subject: [ceph-users] Re: osd_pglog memory hoarding - another case
>>>
>>> > Hi All,
>>> > back to this. Dan, it seems we're following exactly in your footsteps.
>>> >
>>> > We recovered from our large pg_log, and got the cluster running. A week 
>>> > after
>>> > our cluster was ok, we started seeing big memory increases again. I don't 
>>> > know
>>> > if we had buffer_anon issues before or if our big pg_logs were masking 
>>> > it. But
>>> > we started seeing bluefs spillover and buffer_anon growth.
>>> >
>>> > This led to whole other series of problems with OOM killing, which 
>>> > probably
>>> > resulted in mon node db growth which filled the disk, which  resulted in 
>>> > all
>>> > mons going down, and a bigger mess of bringing everything up.
>>> >
>>> > However. We're back. But I think we can confirm the buffer_anon growth, 
>>> > and
>>> > bluefs spillover.
>>> >
>>> > We now have a job that constatly writes 10k objects in a buckets and 
>>> > deletes
>>> > them.
>>> >
>>> > This may curb the memory growth, but I don't think it stops the problem. 
>>> > We're
>>> > just testing restarting OSDs and while it takes a while, it seems it may 
>>> > help.
>>> > Of course this is not the greatest fix in production.
>>> >
>>> > Has anybody gleaned any new information on this issue? Things to tweaks? 
>>> > Fixes
>>> > in the horizon? Other mitigations?
>>> >
>>> > Cheers,
>>> > Kalle
>>> >
>>> >
>>> > - Original Message -
>>> >> From: "Kalle Happonen" 
>>> >> To: "Dan van der Ster" 
>>> >> Cc: "ceph-users" 
>>> >> Sent: Thursday, 19 November, 2020 13:56:37
>>> >> Subject: [ceph-users] Re: osd_pglog memory hoarding - another case
>>> >
>>> >> Hello,
>>> >> I thought I'd post an update.
>>> >>
>>> >> Setting the pg_log size to 500, and running the offline trim operation
>>> >> sequentially on all OSDs seems to help. With our current setup, it takes 
>>> >> about
>>> >> 12-48h per node, depending on the pgs per osd. The PG amounts per OSD we 
>>> >> have
>>> >> are ~180-750, with a majority around 200, and some nodes consistently 
>>> >> have 500
>>> >> per OSD. The limiting factor of the recovery time seems to be our nvme, 
>>> >> which
>>> >> we use for rocksdb for the OSDs.
>>> >>
>>> >> We haven't fully recovered yet, we're working on it. Almost all our PGs 
>>> >> are back
>>> >> up, we still have ~40/18000 PGs down, but I think we'll get there. 
>>> >> Currently
>>> >> ~4

[ceph-users] Re: Data migration between clusters

[Kalle Happonen]

Hi Istvan,
I'm not sure it helps, but here's at least some pitfalls we faced when 
migrating radosgws between clusters.

https://cloud.blog.csc.fi/2019/12/ceph-object-storage-migraine-i-mean.html

Cheers,
Kalle

- Original Message -
> From: "Szabo, Istvan (Agoda)" 
> To: "ceph-users" 
> Sent: Thursday, 17 December, 2020 12:11:19
> Subject: [ceph-users] Data migration between clusters

> What is the easiest and best way to migrate bucket from an old cluster to a 
> new
> one?
> 
> Luminous to octopus not sure does it matter from the data perspective.
> 
> 
> This message is confidential and is for the sole use of the intended
> recipient(s). It may also be privileged or otherwise protected by copyright or
> other legal rules. If you have received it by mistake please let us know by
> reply email and delete it from your system. It is prohibited to copy this
> message or disclose its content to anyone. Any confidentiality or privilege is
> not waived or lost by any mistaken delivery or unauthorized disclosure of the
> message. All messages sent to and from Agoda may be monitored to ensure
> compliance with company policies, to protect the company's interests and to
> remove potential malware. Electronic messages may be intercepted, amended, 
> lost
> or deleted, or contain viruses.
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Debian repo for ceph-iscsi

[Chris Palmer]

Hi Joachim

Thanks for that pointer. I've pulled ceph-iscsi from there and and 
trying to get things going now on buster.


The problem I have at the moment though is with python3-rtslib-fb. That 
hasn't been backported to buster, and the latest in the main buster repo 
is 2.1.66, but ceph-iscsi requires 2.1.68 or later (which adds a 
"control" parameter that ceph needs).


Has anyone successfully solved or worked-around these dependencies on 
debian buster?


Thanks, Chris

On 18/12/2020 20:46, Joachim Kraftmayer wrote:

Hello Chris,
you are looking for this:

https://packages.debian.org/buster-backports/ceph-iscsi

Regards, Joachim


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Ceph rgw & dashboard problem

[Mika Saari]

Hi,

  Using Ceph Octopus installed with cephadm here. Version running currently
is 15.2.6. There are 3 machines running the cluster. Machine names are
introduced in /etc/hosts in long(FQDN) & short forms but ceph hostnames of
the servers are in short form (not sure if this affects anyway). rdb side
is working nicely, tested with a linux client.

  Trying to get object gateway to be visible in dashboard but getting error
when selecting "Object Gateway -> Daemons"
Error:
RGW REST API failed request with status code 403
(b'{"Code":"AccessDenied","RequestId":"tx00040-005fe20384-8ecbc'
b'-ou","HostId":"8ecbc-ou-default"}')

  What am I doing wrong here?

Thanks a lot,
 -Mika

  Procedure what I have done 
   1) ceph orch apply rgw default ou --placement="1 ceph1"
   2) radosgw-admin user create --uid=test --display-name=test
--access-key=test --secret-key=test
   3) radosgw-admin period update --rgw-realm=default --commit
   4) aws configure --profile=default
aws configure --profile=default
AWS Access Key ID [None]: test
AWS Secret Access Key [None]: test
Default region name [None]: default
Default output format [None]: json
5) aws s3 mb s3://test1 --endpoint-url http://ceph1
make_bucket: test1
5.1) radosgw-admin bucket list
[
"test1"
]
6) ceph dashboard --help | grep reset-rgw | awk '{print $2}' | xargs -n
1 ceph dashboard
Option RGW_API_ACCESS_KEY reset to default value ""
Option RGW_API_ADMIN_RESOURCE reset to default value "admin"
Option RGW_API_HOST reset to default value ""
Option RGW_API_PORT reset to default value "80"
Option RGW_API_SCHEME reset to default value "http"
Option RGW_API_SECRET_KEY reset to default value ""
Option RGW_API_SSL_VERIFY reset to default value "True"
Option RGW_API_USER_ID reset to default value ""
7) ceph dashboard set-rgw-api-user-id "test"
Option RGW_API_USER_ID updated
8) ceph dashboard set-rgw-api-access-key test
Option RGW_API_ACCESS_KEY updated
9) ceph dashboard set-rgw-api-secret-key test
Option RGW_API_SECRET_KEY updated
10) ceph mgr module disable dashboard
11) ceph mgr module enable dashboard
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: PGs down

[Jeremy Austin]

Hi Igor,

 I had taken the OSDs out already, so bringing them up allowed a full
rebalance to occur.

I verified that they were not exhibiting ATA or SMART-reportable errors,
wiped them and re-added.

I will deep scrub.

Thanks again!
Jeremy

On Mon, Dec 21, 2020 at 11:39 PM Igor Fedotov  wrote:

> Hi Jeremy,
>
> good to know you managed to bring your OSDs up.
>
> Have you been able to reweight them to 0 and migrate data out of  these
> "broken" OSDs?
>
> If so I suggest to redeploy them - the corruption is still in the DB and
> it might pop-up one day.
>
> If not please do that first - you might still hit into issues along the
> process since DB is still corrupted. Disabling compaction just bypassed the
> reads from broken files - but this might happen again during different
> operations.
>
>
> On 12/22/2020 7:17 AM, Jeremy Austin wrote:
>
> Igor,
>
> You're a bloomin' genius, as they say.
>
> Disabling auto compaction allowed OSDs 11 and 12 to spin up/out. The 7
> down PGs recovered; there were a few unfound items previously which I went
> ahead and deleted, given that this is EC, revert not being an option.
>
> HEALTH OK :)
>
> I'm now intending to re-enable auto compaction. Should I also fsck the
> rest of the OSDs, or is the typical scrub/deep scrub cycle sufficient? (No
> PGs are behind on scrubbing, whereas they were during the degraded period.)
>
> Please do not re-enable the compaction before you're ready to kill
> "broken" OSDs...
>
> There is no much sense in fsck-ing other OSDs - this is unlikely to detect
> inconsistencies at PG level if any.
>
> Hence it's [deep]-scrub which you might want to apply manually once
> reweight/rebalance is done. PGs located at "broken" OSDs are of the top
> priority to scrub.
>
>
>
> Time will tell if I actually lost data, I guess.
>
> On Mon, Dec 21, 2020 at 8:37 AM Igor Fedotov  wrote:
>
>> Hi Jeremy,
>>
>> you might want to try RocksDB's disable_auto_compactions option for that.
>>
>> To adjust rocksdb's options one should  edit/insert
>> bluestore_rocksdb_options in ceph.conf.
>>
>> E.g.
>>
>> bluestore_rocksdb_options =
>> "disable_auto_compactions=true,compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824"
>>
>>
>> Please note bluestore specific defaults for rocksdb settings are
>> re-provided to make sure they aren't reset to rocksdb's ones.
>>
>> Hope this helps.
>>
>> Thanks,
>>
>> Igor
>>
>>
>>
>>
>>
>>
>>
>> On 12/21/2020 2:56 AM, Jeremy Austin wrote:
>>
>>
>>
>> On Sun, Dec 20, 2020 at 2:25 PM Jeremy Austin  wrote:
>>
>>> Will attempt to disable compaction and report.
>>>
>>
>> Not sure I'm doing this right. In [osd] section of ceph.conf, I added
>> periodic_compaction_seconds=0
>>
>> and attempted to start the OSDs in question. Same error as before. Am I
>> setting compaction options correctly?
>>
>>
>> --
>> Jeremy Austin
>> jhaus...@gmail.com
>>
>>
>
> --
> Jeremy Austin
> jhaus...@gmail.com
>
> --
Jeremy Austin
jhaus...@gmail.com
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Failing OSD RocksDB Corrupt

[Ashley Merrick]

Hello,I had some faulty power cables on some OSD's in one server which caused 
lots of IO issues/disks appearing/disappearing.This has been corrected now, 2 
of the 10 OSD's are working, however 8 are failing to start due to what looks 
to be a corrupt DB.When running a ceph-bluestore-tool fsck I get the following 
output:rocksdb: [db/db_impl_open.cc:516] db.wal/002221.log: dropping 1302 
bytes; Corruption: missing start of fragmented 
record(2)2020-12-22T16:21:52.715+0100 7f7b6a1500c0 4 rocksdb: 
[db/db_impl.cc:389] Shutdown: canceling all background 
work2020-12-22T16:21:52.715+0100 7f7b6a1500c0 4 rocksdb: [db/db_impl.cc:563] 
Shutdown complete2020-12-22T16:21:52.715+0100 7f7b6a1500c0 -1 rocksdb: 
Corruption: missing start of fragmented record(2)2020-12-22T16:21:52.715+0100 
7f7b6a1500c0 -1 
bluestore(/var/lib/ceph/b1db6b36-0c4c-4bce-9cda-18834be0632d/osd.28) opendb 
erroring opening db:Trying to start the OSD leads to:ceph_abort_msg("Bad table 
magic number: expected 9863518390377041911, found 9372993859750765257 in 
db/002442.sst")It looks like the last write to these OSD's never fully 
completed, sadly as I was adding this new node to move from OSD to Host 
redundancy (EC Pool) I have 20% down PG's currently, is there anything I can do 
to remove the last entry in the DB or somehow clean up the rocksDB to get these 
OSD's atleast started? Understand may end up with some corrupted files.Thanks
 
Sent via MXlogin
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Debian repo for ceph-iscsi

[Chris Palmer]

Pulling the package python3-rtslib-fb_2.1.71-3_all.deb from bullseye and 
manually installing it on buster seems to have done the trick.


On 22/12/2020 13:20, Chris Palmer wrote:

Hi Joachim

Thanks for that pointer. I've pulled ceph-iscsi from there and and 
trying to get things going now on buster.


The problem I have at the moment though is with python3-rtslib-fb. 
That hasn't been backported to buster, and the latest in the main 
buster repo is 2.1.66, but ceph-iscsi requires 2.1.68 or later (which 
adds a "control" parameter that ceph needs).


Has anyone successfully solved or worked-around these dependencies on 
debian buster?


Thanks, Chris

On 18/12/2020 20:46, Joachim Kraftmayer wrote:

Hello Chris,
you are looking for this:

https://packages.debian.org/buster-backports/ceph-iscsi

Regards, Joachim


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: diskprediction_local fails with python3-sklearn 0.22.2

[Reed Dier]

I'm going to resurrect this thread in hopes that in the 6 months since, someone 
has found a solution?
After recently upgrading my mgr's to 20.04 and 15.2.8, the diskprediction_local 
module is failing for me in the exact same manner.

> $ dpkg -l | grep sklearn
> ii  python3-sklearn  0.22.2.post1+dfsg-5  
>   all  Python modules for machine learning and data mining - 
> Python 3
> ii  python3-sklearn-lib  0.22.2.post1+dfsg-5  
>   amd64low-level implementations and bindings for scikit-learn - 
> Python 3

> $ pip3 freeze | grep learn
> scikit-learn==0.22.2.post1

I see v0.24.0 in pypi , however I 
prefer to use apt packages where possible for easier management, and everything 
I've found points to 0.19.2 as the version required, which 0.22.2 would assume 
fulfill.

Thanks,
Reed

> On Jun 4, 2020, at 2:24 PM, Eric Dold  wrote:
> 
> Hello
> 
> the mgr module diskprediction_local fails under ubuntu 20.04 focal with
> python3-sklearn version 0.22.2
> Ceph version is 15.2.3
> 
> when the module is enabled i get the following error:
> 
>  File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 112, in
> serve
>self.predict_all_devices()
>  File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 279, in
> predict_all_devices
>result = self._predict_life_expentancy(devInfo['devid'])
>  File "/usr/share/ceph/mgr/diskprediction_local/module.py", line 222, in
> _predict_life_expentancy
>predicted_result = obj_predictor.predict(predict_datas)
>  File "/usr/share/ceph/mgr/diskprediction_local/predictor.py", line 457,
> in predict
>pred = clf.predict(ordered_data)
>  File "/usr/lib/python3/dist-packages/sklearn/svm/_base.py", line 585, in
> predict
>if self.break_ties and self.decision_function_shape == 'ovo':
> AttributeError: 'SVC' object has no attribute 'break_ties'
> 
> Best Regards
> Eric
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Larger number of OSDs, cheroot, cherrypy, limits + containers == broken

[Ken Dreyer]

Thanks David! A couple of things have happened since the last update.

The primary Fedora cheroot package maintainer updated cheroot from
8.5.0 to 8.5.1 in Rawhide. I've rebuilt this for el8 and put it into a
new repository here: https://fedorapeople.org/~ktdreyer/bz1907005/

There are a few more small cleanups I need to land in order to
reconcile the epel8 and master branches.

For rawhide (master):
https://src.fedoraproject.org/rpms/python-cheroot/pull-request/5
https://src.fedoraproject.org/rpms/python-cheroot/pull-request/7

For epel8:
https://src.fedoraproject.org/rpms/python-cheroot/pull-request/6
https://src.fedoraproject.org/rpms/python-cheroot/pull-request/8

Once those are merged, we can merge this final small diff to master:
https://src.fedoraproject.org/fork/ktdreyer/rpms/python-cheroot/c/34dcdbc3ae3c07bccc4d72b1f0defa95318d5c51
(this is the branch I used to build that fedorapeople.org RPM)

... and then merge master into epel8, and then build 8.5.1 in the main
build system (Koji) and ship to epel-testing.

- Ken

On Fri, Dec 11, 2020 at 5:38 AM David Orman  wrote:
>
> Hi Ken,
>
> This seems to have fixed that issue. It exposed another: 
> https://tracker.ceph.com/issues/39264 which is causing ceph-mgr to become 
> entirely unresponsive across the cluster, but cheroot seems to be ok.
>
> David
>
> On Wed, Dec 9, 2020 at 12:25 PM David Orman  wrote:
>>
>> Ken,
>>
>> We have rebuilt the container images of 15.2.7 with this RPM applied, and 
>> will be deploying it to a larger (504 OSD) cluster to test - this cluster 
>> had the issue previously until we disabled polling via Prometheus. We will 
>> update as soon as it's run for a day or two and we've been able to verify 
>> the mgr issues we saw no longer occur after extended polling via external 
>> and internal prometheus instances.
>>
>> Thank you again for the quick update, we'll let you know as soon as we have 
>> more feedback,
>> David
>>
>> On Tue, Dec 8, 2020 at 10:37 AM David Orman  wrote:
>>>
>>> Hi Ken,
>>>
>>> Thank you for the update! As per: 
>>> https://github.com/ceph/ceph-container/issues/1748
>>>
>>> We implemented the (dropping ulimit to 1024:4096 for mgr) suggested change 
>>> last night, and on our test cluster of 504 OSDs, being polled by the 
>>> internal prometheus and our external instance, the mgrs stopped responding 
>>> and dropped out of the cluster entirely. This is impacting not just 
>>> metrics, but the mgr itself. I think this is a high priority issue, as 
>>> metrics are critical for prod, but mgr itself seems to be impacted on a 
>>> moderately sized cluster.
>>>
>>> Respectfully,
>>> David Orman
>>>
>>> On Mon, Dec 7, 2020 at 1:50 PM Ken Dreyer  wrote:

 Thanks for bringing this up.

 We need to update Cheroot in Fedora and EPEL 8. I've opened
 https://src.fedoraproject.org/rpms/python-cheroot/pull-request/3 to
 get this into Fedora first.

 I've published an el8 RPM at
 https://fedorapeople.org/~ktdreyer/bz1868629/ for early testing. I can
 bring up a "hello world" cherrypy app with this, but I've not tested
 it with Ceph.

 - Ken

 On Mon, Dec 7, 2020 at 9:57 AM David Orman  wrote:
 >
 > Hi,
 >
 > We have a ceph 15.2.7 deployment using cephadm under podman w/ systemd.
 > We've run into what we believe is:
 >
 > https://github.com/ceph/ceph-container/issues/1748
 > https://tracker.ceph.com/issues/47875
 >
 > In our case, eventually the mgr container stops emitting output/logging. 
 > We
 > are polling with external prometheus clusters, which is likely what
 > triggers the issue, as it appears some amount of time after the container
 > is spawned.
 >
 > Unfortunately, setting limits in the systemd service file for the mgr
 > service on the host OS doesn't work, nor does modifying the unit.run file
 > which is used to start the container under podman to include the --ulimit
 > settings as suggested. Looking inside the container:
 >
 > lib/systemd/system/ceph-mgr@.service:LimitNOFILE=1048576
 >
 > This prevents us from deploying medium to large ceph clusters, so I would
 > argue it's a high priority bug that should not be closed, unless there 
 > is a
 > workaround that works until EPEL 8 contains the fixed version of cheroot
 > and the ceph containers include it.
 >
 > My understanding is this was fixed in cheroot 8.4.0:
 >
 > https://github.com/cherrypy/cheroot/issues/249
 > https://github.com/cherrypy/cheroot/pull/301
 >
 > Thank you in advance for any suggestions,
 > David
 > ___
 > ceph-users mailing list -- ceph-users@ceph.io
 > To unsubscribe send an email to ceph-users-le...@ceph.io
 >

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] after octopus cluster reinstall, rbd map fails with timeout

[Philip Brown]

More banging on my prototype cluster, and ran into an odd problem.

Used to be, when I create an rbd device, then try to map it, it would initially 
fail, saying I have to disable some features.
Then I just run the suggested disable line -- usually

  rbd feature disable poolname/rbdname  object-map fast-diff deep-flatten


and then I can map it fine.

but now after the latest cluster recreation, when I try to map, I just get

# rbd map testpool/zfs02
rbd: sysfs write failed
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (110) Connection timed out

and no errors in dmesg output

if I try to disable those features anyway, I get
librbd::Operations: one or more requested features are already disabled(22) 
Invalid argument

nothing in /var/log/ceph/cephadm.log either

Any suggestions?


--
Philip Brown| Sr. Linux System Administrator | Medata, Inc. 
5 Peters Canyon Rd Suite 250 
Irvine CA 92606 
Office 714.918.1310| Fax 714.918.1325 
pbr...@medata.com| www.medata.com
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io