[ceph-users] Re: How to enable lazyio under kcephfs?

[Dan van der Ster]

Hi,

client_force_lazyio only works for ceph-fuse and libcephfs:
https://github.com/ceph/ceph/pull/26976/files

You can use the ioctl to enable per file with the kernel mount, but
you might run into the same problem we did:
https://tracker.ceph.com/issues/44166

Please share if it works for you.

Cheers, Dan

On Tue, Jun 8, 2021 at 8:16 AM opengers  wrote:
>
> ceph: 14.2.x
> kernel: 4.15
>
> In cephfs, due to the need for cache consistency, When a client is
> executing buffer IO, another client will hang when reading and writing the
> same file
>
> It seems that lazyio can solve this problem, lazyio allows multiple clients
> to execute buffer IO at the same time(relax consistency), But I am not sure
> how to enable lazyio under the kernel mount, the test found that the
> "client_force_lazyio" parameter does not work
>
> My final requirement is to use lazyio to implement multiple clients to read
> and write the same file(buffer IO mode)
>
> Can someone explain how to enable lazyio under kcephfs, thanks
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: OT: How to Build a poor man's storage with ceph

[Janne Johansson]

Den tis 8 juni 2021 kl 11:39 skrev Ml Ml :

> Maybe combine 3x 10TB HDDs to a 30TB Raid0/striping Disk => which
> would speed up the performance, but have a bigger impact on a dying
> disk.

^^
This sounds like a very bad idea.

When this 30T monster fails, you will have to wait for 30TB to rebuild
elsewhere.
If you think waiting for 4-6-8TB is slow, 30 will be 5x worse.

-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: OT: How to Build a poor man's storage with ceph

[Christian Wuerdig]

Since you mention NextCloud it will probably be RWG deployment. ALso it's
not clear why 3 nodes? Is rack-space a premium?

Just to compare your suggestion:
3x24 (I guess 4U?) x 8TB with Replication = 576 TB raw storage + 192 TB
usable

Let's go 6x12 (2U) x 4TB with EC 3+2 = 288 TB raw storage + 172 TB usable.
Same rack-space, a little bit less usable storage, smaller drives (means
faster recovery). Put a SSD or NVME in each server for the Index pool. Plus
maybe 2 NVMEs for the WAL/DB

On Tue, 8 Jun 2021 at 21:39, Ml Ml  wrote:

> Hello List,
>
> i used to build 3 Node Cluster with spinning Rust and later with
> (Enterprise) SSDs.
> All i did was to buy a 19" Server with 10/12 Slots, plug in the Disks
> and i was done.
> The Requirements were just 10/15TB Disk usage (30-45TB Raw).
>
> Now i was asked if i could also build a cheap 200-500TB Cluster
> Storage, which should also scale. Just for Data Storage such as
> NextCloud/OwnCloud.
>
> Buying 3x 24 Slot Server with 8TB Enterprise SSDs ends up at about 3x
> 45k EUR = 135k EUR.
> Where the SSDs are 90% of the price. (about 1.700EUR per 8TB SSD)
>
> How do the "big boys" do this? Just throw money at it?
> Would a mix of OSD SSD Metadata + Spinning Rust do the job?
>
> My experience so far is that each time i had a crash/problem it was
> always such a pain to wait for the spinning rust.
>
> Do you have any experience/hints on this?
> Maybe combine 3x 10TB HDDs to a 30TB Raid0/striping Disk => which
> would speed up the performance, but have a bigger impact on a dying
> disk.
>
> My requirements are more or less low IO Traffic but loads of disk space
> usage.
>
> Any hints/ideas/links are welcome.
>
> Cheers,
> Michael
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] ceph buckets

[Rok Jaklič]

Hi,

I try to create buckets through rgw in following order:
- *bucket1* with *user1* with *access_key1* and *secret_key1*
- *bucket1* with *user2* with *access_key2* and *secret_key2*

when I try to create a second bucket1 with user2 I get *Error response code
BucketAlreadyExists.*

Why? Should not buckets relate only to users? Is this by design and is
there any particular reason that it follows this concept?

Regards,
Rok
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph buckets [EXT]

[Matthew Vernon]

Hi,

On 08/06/2021 11:37, Rok Jaklič wrote:


I try to create buckets through rgw in following order:
- *bucket1* with *user1* with *access_key1* and *secret_key1*
- *bucket1* with *user2* with *access_key2* and *secret_key2*

when I try to create a second bucket1 with user2 I get *Error response code
BucketAlreadyExists.*

Why? Should not buckets relate only to users? Is this by design and is
there any particular reason that it follows this concept?


By default, there is one bucket namespace (which makes sense if you're 
going with the common bucket name as hostname pattern); you can do 
multi-tenancy if you want, however:

https://docs.ceph.com/en/latest/radosgw/multitenancy/

Regards,

Matthew


--
The Wellcome Sanger Institute is operated by Genome Research 
Limited, a charity registered in England with number 1021457 and a 
company registered in England with number 2742969, whose registered 
office is 215 Euston Road, London, NW1 2BE. 
___

ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: OT: How to Build a poor man's storage with ceph

[Sebastian Knust]

Hi Michael,

On 08.06.21 11:38, Ml Ml wrote:

Now i was asked if i could also build a cheap 200-500TB Cluster
Storage, which should also scale. Just for Data Storage such as
NextCloud/OwnCloud.


With similar requirements (server primarily for Samba and NextCloud, 
some RBD use, very limited budget) I am using HDD for data and SSD for 
system and CephFS metadata.


Note that I am running NextCloud on CephFS storage. If you want to go 
with RGW/S3 as a storage backend instead, the following might not apply 
to your use case.


My nodes (bought end of 2020) are:
- 2U chassis with 12 3.5" SATA slots
- Intel Xeon Silver 4208
- 128 GB RAM
- 2 x 480 GB Samsung PM883 SSD
  -> 50 GB in MD-RAID1 for system
  -> 430 GB OSD (one per SSD)
- initially 6 x 14 TB Enterprise HDD
- 4 x 10 GBase-T (active/passive bonded, dedicated backend network)

Each node with this configuration cost about 4k EUR net at the end of 
2020. Due to increasing prices for storage, it will be a bit more 
expensive now. I am running five nodes now and have added a few more 
disks (ranging 8-14 TB), nearly filling up the nodes.


My experience so far:
- I had to throttle scrubbing (see below for details)
- For purely NextCloud and Samba performance is sufficient for a few 
hundred concurrent users with a handful of power users
- Migration of the mail server to this cluster was a disaster due to 
limited IOPS, had to add some more SSDs and place the mail server in an 
SSD-only pool.
- MDS needs a lot of memory for larger CephFS installs, I will move it 
to a dedicated server probably next year. 128 GB per node works but I 
would not recommend any less.
- Rebalancing takes an eternity (2-3 weeks), so make sure that your PG 
nums are okay from the start
- I have all but given up on snapshots with CephFS due to severe 
performance degradation with kernel client during backup


My scrubbing config looks like this:
osd_backfill_scan_max   16
osd_backfill_scan_min   4
osd_deep_scrub_interval 2592000.00
osd_deep_scrub_randomize_ratio  0.03
osd_recovery_max_active_hdd 1
osd_recovery_max_active_ssd 5
osd_recovery_sleep_hdd  0.05
osd_scrub_begin_hour18
osd_scrub_end_hour  7
osd_scrub_chunk_max 1
osd_scrub_chunk_min 1
osd_scrub_max_interval  2419200.00
osd_scrub_min_interval  172800.00
osd_scrub_sleep 0.10


My data is in a replicated pool with n=3 without compression. You might 
also consider EC and then want to aim for more nodes.


Cheers
Sebastian
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] Bucket creation on RGW Multisite env.

[Soumya Koduri]

On 6/8/21 4:59 PM, Szabo, Istvan (Agoda) wrote:

Yes, but with this the bucket contents will not be synced only. The bucket will 
be available everywhere just will be empty.



There is option to enable sync on the bucket(s) which will then be 
synced across all the configured zones (as per the group policy 
configured). Refer to the example I had mentioned.



Thanks,

Soumya



Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Soumya Koduri 
Sent: Monday, May 31, 2021 7:38 PM
To: mhnx ; Szabo, Istvan (Agoda) 

Cc: Ceph Users 
Subject: Re: [ceph-users] Re: [Suspicious newsletter] Bucket creation on RGW 
Multisite env.

On 5/31/21 3:02 PM, mhnx wrote:

Yes you're right. I have a Global sync rule in the zonegroup:
  "sync_from_all": "true",
  "sync_from": [],
  "redirect_zone": ""

If I need to stop/start the sync after creation I use the command:
radosgw-admin bucket sync enable/disable --bucket=$newbucket

I developed it but clients can create bucket and I have no control on
the bucket if it's created out of my program.

Because of that I'm looking something:
1- Do not sync buckets if its not enabled by me.
2- Sync all the other "sync started" buckets.


This can be achieved using Multisite sync-policy  [1]. Create a group policy which 
allows sync (but not enabled) across all the zones & buckets. And then create 
another policy at bucket(s) level to enable sync for only those particular ones. 
Refer to Example-3 in the same page [2].


Thanks,

Soumya


[1] https://docs.ceph.com/en/latest/radosgw/multisite-sync-policy/

[2]
https://docs.ceph.com/en/latest/radosgw/multisite-sync-policy/#example-3-mirror-a-specific-bucket






Szabo, Istvan (Agoda) , 31 May 2021 Pzt, 12:11
tarihinde şunu yazdı:

Bucket is created but if no sync rule set, the data will not be synced across.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: mhnx 
Sent: Monday, May 31, 2021 4:07 PM
To: Ceph Users 
Subject: [Suspicious newsletter] [ceph-users] Bucket creation on RGW Multisite 
env.

Hello.

I have a multisite RGW environment.
When I create a new bucket, the bucket is directly created on master and 
secondary.
If I don't want to sync a bucket, I need to stop sync after creation.
Is there any global option as "Do not sync directly, only start if I want to" ?
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
email to ceph-users-le...@ceph.io


This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.

___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an
email to ceph-users-le...@ceph.io



___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph buckets

[Janne Johansson]

Den tis 8 juni 2021 kl 12:38 skrev Rok Jaklič :
> Hi,
> I try to create buckets through rgw in following order:
> - *bucket1* with *user1* with *access_key1* and *secret_key1*
> - *bucket1* with *user2* with *access_key2* and *secret_key2*
>
> when I try to create a second bucket1 with user2 I get *Error response code
> BucketAlreadyExists.*
>
> Why? Should not buckets relate only to users? Is this by design and is
> there any particular reason that it follows this concept?

If you want to use
http(s)://bucketname.domain.name/ to access buckets it certainly helps if they
are uniquely named, no?

Apart from that, there is a mode for RGW with tenant/bucketname where
overlapping/duplicate bucket names are possible, but of course you
will not get simple DNS-names for published content in that case.

-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph buckets

[Rok Jaklič]

Which mode is that and where can I set it?

This one described in https://docs.ceph.com/en/latest/radosgw/multitenancy/
?

On Tue, Jun 8, 2021 at 2:24 PM Janne Johansson  wrote:

> Den tis 8 juni 2021 kl 12:38 skrev Rok Jaklič :
> > Hi,
> > I try to create buckets through rgw in following order:
> > - *bucket1* with *user1* with *access_key1* and *secret_key1*
> > - *bucket1* with *user2* with *access_key2* and *secret_key2*
> >
> > when I try to create a second bucket1 with user2 I get *Error response
> code
> > BucketAlreadyExists.*
> >
> > Why? Should not buckets relate only to users? Is this by design and is
> > there any particular reason that it follows this concept?
>
> If you want to use
> http(s)://bucketname.domain.name/ to access buckets it certainly helps if
> they
> are uniquely named, no?
>
> Apart from that, there is a mode for RGW with tenant/bucketname where
> overlapping/duplicate bucket names are possible, but of course you
> will not get simple DNS-names for published content in that case.
>
> --
> May the most significant bit of your life be positive.
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph buckets

[Janne Johansson]

Den tis 8 juni 2021 kl 14:31 skrev Rok Jaklič :
> Which mode is that and where can I set it?
> This one described in https://docs.ceph.com/en/latest/radosgw/multitenancy/ ?

Yes, the description says it all there, doesn't it?

>>
>> Apart from that, there is a mode for RGW with tenant/bucketname where
>> overlapping/duplicate bucket names are possible, but of course you
>> will not get simple DNS-names for published content in that case.



-- 
May the most significant bit of your life be positive.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Index pool hasn't been cleaned up and caused large omap, safe to delete the index file?

[Szabo, Istvan (Agoda)]

Hi,

In my multisite setup 1 big bucket has been deleted and seems like hasn't been 
cleaned up on one of the secondary site.
Is it safe to delete the 11 shard objects from the index pool which holding the 
omaps of that bucket files?

Also a quick question, is it a problem if we use like this?
Create a bucket which means create in all dc
Don't create any bucket sync the user upload different files in different dcs.

When a bucket deletion happens would this usage behavior cause issue that 
different files are in the buckets?
If yes, how to prevent this?

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---



This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] OT: How to Build a poor man's storage with ceph

[Ml Ml]

Hello List,

i used to build 3 Node Cluster with spinning Rust and later with
(Enterprise) SSDs.
All i did was to buy a 19" Server with 10/12 Slots, plug in the Disks
and i was done.
The Requirements were just 10/15TB Disk usage (30-45TB Raw).

Now i was asked if i could also build a cheap 200-500TB Cluster
Storage, which should also scale. Just for Data Storage such as
NextCloud/OwnCloud.

Buying 3x 24 Slot Server with 8TB Enterprise SSDs ends up at about 3x
45k EUR = 135k EUR.
Where the SSDs are 90% of the price. (about 1.700EUR per 8TB SSD)

How do the "big boys" do this? Just throw money at it?
Would a mix of OSD SSD Metadata + Spinning Rust do the job?

My experience so far is that each time i had a crash/problem it was
always such a pain to wait for the spinning rust.

Do you have any experience/hints on this?
Maybe combine 3x 10TB HDDs to a 30TB Raid0/striping Disk => which
would speed up the performance, but have a bigger impact on a dying
disk.

My requirements are more or less low IO Traffic but loads of disk space usage.

Any hints/ideas/links are welcome.

Cheers,
Michael
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: OT: How to Build a poor man's storage with ceph

[Eneko Lacunza]

Hi Michael,

El 8/6/21 a las 11:38, Ml Ml escribió:

Hello List,

i used to build 3 Node Cluster with spinning Rust and later with
(Enterprise) SSDs.
All i did was to buy a 19" Server with 10/12 Slots, plug in the Disks
and i was done.
The Requirements were just 10/15TB Disk usage (30-45TB Raw).

Now i was asked if i could also build a cheap 200-500TB Cluster
Storage, which should also scale. Just for Data Storage such as
NextCloud/OwnCloud.

Buying 3x 24 Slot Server with 8TB Enterprise SSDs ends up at about 3x
45k EUR = 135k EUR.
Where the SSDs are 90% of the price. (about 1.700EUR per 8TB SSD)

How do the "big boys" do this? Just throw money at it?
Would a mix of OSD SSD Metadata + Spinning Rust do the job?

My experience so far is that each time i had a crash/problem it was
always such a pain to wait for the spinning rust.

Do you have any experience/hints on this?
Maybe combine 3x 10TB HDDs to a 30TB Raid0/striping Disk => which
would speed up the performance, but have a bigger impact on a dying
disk.

My requirements are more or less low IO Traffic but loads of disk space usage.

Any hints/ideas/links are welcome.



You don't give details about RAM and CPU, but at 24xEnterprise SSD per 
server I think you'll be wasting much of those SSD's performance...


I suggest you consider a 4-6 server cluster, and SSD for wal + Spinning 
disks for storage. This will give you more redundancy for less money, 
and more peace of mind when a spinning disk fails.


Cheers

Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project

Tel. +34 943 569 206 | https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun

https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Index pool hasn't been cleaned up and caused large omap, safe to delete the index file?

[Szabo, Istvan (Agoda)]

Some more information:

HGK is the master, ASH and SGP is the secondary, let me show 1 shard in all DCs 
(FYI the bucket has been deleted which is relates to this bucket index).

HKG and ASH give back empty command output for this:

rados -p hkg or ash.rgw.buckets.index listomapvals 
.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.5411362.1.23219
However in SGP it's full with values like this:

ycsappfeed-images/area_id/572303/city_id/23053/country_id/35/datadate/2021-01-28/language_id/1/white_label_id/1/report_id/8/report#STD.jpg
value (333 bytes) :
  08 03 47 01 00 00 8a 00  00 00 79 63 73 61 70 70  |..G...ycsapp|
0010  66 65 65 64 2d 69 6d 61  67 65 73 2f 61 72 65 61  |feed-images/area|
0020  5f 69 64 2f 35 37 32 33  30 33 2f 63 69 74 79 5f  |_id/572303/city_|
0030  69 64 2f 32 33 30 35 33  2f 63 6f 75 6e 74 72 79  |id/23053/country|
0040  5f 69 64 2f 33 35 2f 64  61 74 61 64 61 74 65 2f  |_id/35/datadate/|
0050  32 30 32 31 2d 30 31 2d  32 38 2f 6c 61 6e 67 75  |2021-01-28/langu|
0060  61 67 65 5f 69 64 2f 31  2f 77 68 69 74 65 5f 6c  |age_id/1/white_l|
0070  61 62 65 6c 5f 69 64 2f  31 2f 72 65 70 6f 72 74  |abel_id/1/report|
0080  5f 69 64 2f 38 2f 72 65  70 6f 72 74 23 53 54 44  |_id/8/report#STD|
0090  2e 6a 70 67 2e 7f 32 00  00 00 00 00 01 07 03 61  |.jpg..2a|
00a0  00 00 00 01 1c 1d 00 00  00 00 00 00 33 4c 1a 60  |3L.`|
00b0  79 81 72 0b 20 00 00 00  65 31 35 66 65 37 32 32  |y.r. ...e15fe722|
00c0  64 62 31 65 39 30 65 36  37 32 63 61 65 35 65 64  |db1e90e672cae5ed|
00d0  39 39 39 66 61 65 65 36  03 00 00 00 70 69 78 03  |999faee6pix.|
00e0  00 00 00 70 69 78 09 00  00 00 69 6d 61 67 65 2f  |...piximage/|
00f0  6a 70 67 1c 1d 00 00 00  00 00 00 00 00 00 00 00  |jpg.|
0100  00 00 00 00 00 00 00 00  00 00 00 00 01 01 06 00  ||
0110  00 00 18 84 2e 7f 32 00  82 8e 02 20 00 00 00 5f  |..2 ..._|
0120  6b 6a 47 75 70 4d 30 66  76 4e 38 37 65 39 6c 34  |kjGupM0fvN87e9l4|
0130  42 68 70 4f 48 69 63 68  57 31 43 42 50 63 58 00  |BhpOHichW1CBPcX.|
0140  00 00 00 00 00 00 00 00  00 00 00 00 00   |.|
014d
The bucket with this id is not available anywhere:
9213182a-14ba-48ad-bde9-289a1c0c0de8.5411362.1

Why it is not cleaned up? How can I clean up the index pool with any 
unnecessary entries?

Thank you

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Szabo, Istvan (Agoda)  
Sent: Tuesday, June 8, 2021 3:56 PM
To: ceph-users 
Subject: [ceph-users] Index pool hasn't been cleaned up and caused large omap, 
safe to delete the index file?

Hi,

In my multisite setup 1 big bucket has been deleted and seems like hasn't been 
cleaned up on one of the secondary site.
Is it safe to delete the 11 shard objects from the index pool which holding the 
omaps of that bucket files?

Also a quick question, is it a problem if we use like this?
Create a bucket which means create in all dc Don't create any bucket sync the 
user upload different files in different dcs.

When a bucket deletion happens would this usage behavior cause issue that 
different files are in the buckets?
If yes, how to prevent this?

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---



This message is confidential and is for the sole use of the intended 
recipient(s). It may also be privileged or otherwise protected by copyright or 
other legal rules. If you have received it by mistake please let us know by 
reply email and delete it from your system. It is prohibited to copy this 
message or disclose its content to anyone. Any confidentiality or privilege is 
not waived or lost by any mistaken delivery or unauthorized disclosure of the 
message. All messages sent to and from Agoda may be monitored to ensure 
compliance with company policies, to protect the company's interests and to 
remove potential malware. Electronic messages may be intercepted, amended, lost 
or deleted, or contain viruses.
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [Suspicious newsletter] Bucket creation on RGW Multisite env.

[Szabo, Istvan (Agoda)]

Yes, but with this the bucket contents will not be synced only. The bucket will 
be available everywhere just will be empty.

Istvan Szabo
Senior Infrastructure Engineer
---
Agoda Services Co., Ltd.
e: istvan.sz...@agoda.com
---

-Original Message-
From: Soumya Koduri  
Sent: Monday, May 31, 2021 7:38 PM
To: mhnx ; Szabo, Istvan (Agoda) 

Cc: Ceph Users 
Subject: Re: [ceph-users] Re: [Suspicious newsletter] Bucket creation on RGW 
Multisite env.

On 5/31/21 3:02 PM, mhnx wrote:
> Yes you're right. I have a Global sync rule in the zonegroup:
>  "sync_from_all": "true",
>  "sync_from": [],
>  "redirect_zone": ""
>
> If I need to stop/start the sync after creation I use the command:
> radosgw-admin bucket sync enable/disable --bucket=$newbucket
>
> I developed it but clients can create bucket and I have no control on 
> the bucket if it's created out of my program.
>
> Because of that I'm looking something:
> 1- Do not sync buckets if its not enabled by me.
> 2- Sync all the other "sync started" buckets.


This can be achieved using Multisite sync-policy  [1]. Create a group policy 
which allows sync (but not enabled) across all the zones & buckets. And then 
create another policy at bucket(s) level to enable sync for only those 
particular ones. Refer to Example-3 in the same page [2].


Thanks,

Soumya


[1] https://docs.ceph.com/en/latest/radosgw/multisite-sync-policy/

[2]
https://docs.ceph.com/en/latest/radosgw/multisite-sync-policy/#example-3-mirror-a-specific-bucket

>
>
>
>
>
> Szabo, Istvan (Agoda) , 31 May 2021 Pzt, 12:11 
> tarihinde şunu yazdı:
>> Bucket is created but if no sync rule set, the data will not be synced 
>> across.
>>
>> Istvan Szabo
>> Senior Infrastructure Engineer
>> ---
>> Agoda Services Co., Ltd.
>> e: istvan.sz...@agoda.com
>> ---
>>
>> -Original Message-
>> From: mhnx 
>> Sent: Monday, May 31, 2021 4:07 PM
>> To: Ceph Users 
>> Subject: [Suspicious newsletter] [ceph-users] Bucket creation on RGW 
>> Multisite env.
>>
>> Hello.
>>
>> I have a multisite RGW environment.
>> When I create a new bucket, the bucket is directly created on master and 
>> secondary.
>> If I don't want to sync a bucket, I need to stop sync after creation.
>> Is there any global option as "Do not sync directly, only start if I want 
>> to" ?
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an 
>> email to ceph-users-le...@ceph.io
>>
>> 
>> This message is confidential and is for the sole use of the intended 
>> recipient(s). It may also be privileged or otherwise protected by copyright 
>> or other legal rules. If you have received it by mistake please let us know 
>> by reply email and delete it from your system. It is prohibited to copy this 
>> message or disclose its content to anyone. Any confidentiality or privilege 
>> is not waived or lost by any mistaken delivery or unauthorized disclosure of 
>> the message. All messages sent to and from Agoda may be monitored to ensure 
>> compliance with company policies, to protect the company's interests and to 
>> remove potential malware. Electronic messages may be intercepted, amended, 
>> lost or deleted, or contain viruses.
> ___
> ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an 
> email to ceph-users-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Announcing go-ceph v0.10.0

[John Mulligan]

I'm happy to announce another release of the go-ceph API 
bindings. This is a regular release following our every-two-months release 
cadence.

https://github.com/ceph/go-ceph/releases/tag/v0.10.0

Changes in the release are detailed in the link above.

The bindings aim to play a similar role to the "pybind" python bindings in the 
ceph tree but for the Go language. These API bindings require the use of cgo.  
There are already a few consumers of this library in the wild, including the 
ceph-csi project.

In addition to our regular release this week, we're also participating in this 
June's "Ceph Month" event with the "go-ceph get together" Birds-of-a-Feather 
session on Thursday June 10th at 10:10 Eastern time. It should be visible in 
the Ceph Community calendar [1]. If you can't make the BoF, questions, 
comments, bugs etc are best directed at our github issues 
tracker or github discussions forum.

[1] - https://ceph.io/contribute/#community-calendar

-- 
John Mulligan

phlogistonj...@asynchrono.us
jmulli...@redhat.com








___
Dev mailing list -- d...@ceph.io
To unsubscribe send an email to dev-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io


___
Dev mailing list -- d...@ceph.io
To unsubscribe send an email to dev-le...@ceph.io


___
Dev mailing list -- d...@ceph.io
To unsubscribe send an email to dev-le...@ceph.io


___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Mon crash when client mounts CephFS

[Phil Merricks]

Hey folks,

I have deployed a 3 node dev cluster using cephadm.  Deployment went
smoothly and all seems well.

If I try to mount a CephFS from a client node, 2/3 mons crash however.
I've begun picking through the logs to see what I can see, but so far
other than seeing the crash in the log itself, it's unclear what the cause
of the crash is.

Here's a log. .  You can see where the crash is
occurring around the line that begins with "Jun 08 18:56:04 okcomputer
podman[790987]:"

I would welcome any advice on either what the cause may be, or how I can
advance the analysis of what's wrong.

Best regards

Phil
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] DocuBetter Meeting -- 09 June 2021 1730 UTC

[John Zachary Dover]

A DocuBetter Meeting will be held on 09 June 2021 at 1730 UTC.

This is the monthly DocuBetter Meeting that is more convenient for
European and North American Ceph contributors than the other meeting,
which is convenient for people in Australia and Asia (and which is very
rarely attended).

Topics:
- cephadm docs rewrite (ongoing)
- ceph.io copy rewrite and information architecture restructure (ongoing)
- rgw manual install procedure (prospective)
- new ceph-docs mailing list

Bring your docs complaints and requests to this meeting.

Meeting: https://bluejeans.com/908675367
Etherpad: https://pad.ceph.com/p/Ceph_Documentation
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Mon crash when client mounts CephFS

[Ilya Dryomov]

On Tue, Jun 8, 2021 at 9:20 PM Phil Merricks  wrote:
>
> Hey folks,
>
> I have deployed a 3 node dev cluster using cephadm.  Deployment went
> smoothly and all seems well.
>
> If I try to mount a CephFS from a client node, 2/3 mons crash however.
> I've begun picking through the logs to see what I can see, but so far
> other than seeing the crash in the log itself, it's unclear what the cause
> of the crash is.
>
> Here's a log. .  You can see where the crash is
> occurring around the line that begins with "Jun 08 18:56:04 okcomputer
> podman[790987]:"

Hi Phil,

I assume you are mounting the kernel client, not ceph-fuse?  If so,
what is the kernel version on the client node?

ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0)
pacific (stable)
1: /lib64/libpthread.so.0(+0x12b20) [0x7fc36de86b20]
2: gsignal()
3: abort()
4: /lib64/libstdc++.so.6(+0x9009b) [0x7fc36d4a409b]
5: /lib64/libstdc++.so.6(+0x9653c) [0x7fc36d4aa53c]
6: /lib64/libstdc++.so.6(+0x96597) [0x7fc36d4aa597]
7: /lib64/libstdc++.so.6(+0x967f8) [0x7fc36d4aa7f8]
8: /lib64/libstdc++.so.6(+0x92045) [0x7fc36d4a6045]
9: /usr/bin/ceph-mon(+0x4d8da6) [0x563c51ad8da6]
10: (MDSMonitor::check_sub(Subscription*)+0x819) [0x563c51acf329]
11: (Monitor::handle_subscribe(boost::intrusive_ptr)+0xcd8)
[0x563c518c1258]
12: (Monitor::dispatch_op(boost::intrusive_ptr)+0x78d)
[0x563c518e72ed]
13: (Monitor::_ms_dispatch(Message*)+0x670) [0x563c518e8910]
14: (Dispatcher::ms_dispatch2(boost::intrusive_ptr
const&)+0x5c) [0x563c51916fdc]
15: (DispatchQueue::entry()+0x126a) [0x7fc3705c6b1a]
16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc370676b71]
17: /lib64/libpthread.so.0(+0x814a) [0x7fc36de7c14a]
18: clone()

Thanks,

Ilya
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Mon crash when client mounts CephFS

[Robert W. Eckert]

When I had issues with the monitors, it was access on the monitor folder under 
/var/lib/ceph//mon./store.db, make sure 
it is owned by the ceph user.

My issues originated from a hardware issue - the memory needed 1.3 v, but the 
mother board was only reading 1.2 (The memory had the issue, the firmware said 
1.2v required, the sticker on the side said 1.3).  So I had a script that 
copied the store across and fixed the permissions.

The other thing that helped a lot compared to the crash logs, was to edit the 
unit.run and remove  -rm parameter from the command.  That lets you see the 
podman logs using podman logs   it was  a bit more detailed.

When you do this, you will need to restore that afterwards, and clean up the 
'cid' and 'pid' files from /run/ceph-@mon..service-cid and 
/run/ceph-@mon..service-pid

My reference is from Redhat enterprise 8, so things may be a bit different on 
ubuntu.

If you get a message about the store.db files being off,  its easiest to stop 
the working node, copy them over , set the user id/group to ceph and start 
things up.  

Rob

-Original Message-
From: Phil Merricks  
Sent: Tuesday, June 8, 2021 3:18 PM
To: ceph-users 
Subject: [ceph-users] Mon crash when client mounts CephFS

Hey folks,

I have deployed a 3 node dev cluster using cephadm.  Deployment went smoothly 
and all seems well.

If I try to mount a CephFS from a client node, 2/3 mons crash however.
I've begun picking through the logs to see what I can see, but so far other 
than seeing the crash in the log itself, it's unclear what the cause of the 
crash is.

Here's a log. .  You can see where the crash is 
occurring around the line that begins with "Jun 08 18:56:04 okcomputer 
podman[790987]:"

I would welcome any advice on either what the cause may be, or how I can 
advance the analysis of what's wrong.

Best regards

Phil
___
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to 
ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] OSD bootstrap time

[Jan-Philipp Litza]

Hi everyone,

recently I'm noticing that starting OSDs for the first time takes ages
(like, more than an hour) before they are even picked up by the monitors
as "up" and start backfilling. I'm not entirely sure if this is a new
phenomenon or if it always was that way. Either way, I'd like to
understand why.

When I execute `ceph daemon osd.X status`, it says "state: preboot" and
I can see the "newest_map" increase slowly. Apparently, a new OSD
doesn't fetch the latest OSD map and gets to work, but instead fetches
hundreds of thousands of OSD maps from the mon, burning CPU while
parsing them.

I wasn't able to find any good documentation on the OSDMap, in
particular why its historical versions need to be kept and why the OSD
seemingly needs so many of them. Can anybody point me in the right
direction? Or is something wrong with my cluster?

Best regards,
Jan-Philipp Litza
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Ceph Ansible fails on check if monitor initial keyring already exists

[Jared Jacob]

I am running the Ceph ansible script to install ceph version Stable-6.0
(Pacific).

When running the sample yml file that was supplied by the github repo it
runs fine up until the "ceph-mon : check if monitor initial keyring already
exists" step. There it will hang for 30-40 minutes before failing.

>From my understanding ceph ansible should be creating this keyring and
using it for communication between monitors, so does anyone know why the
playbook would have a hard time with this step?

Thanks in advance!
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: OSD bootstrap time

[Richard Bade]

Hi Jan-Philipp,
I've noticed this a couple of times on Nautilus after doing some large
backfill operations. It seems the osd map doesn't clear properly after
the cluster returns to Health OK and builds up on the mons. I do a
"du" on the mon folder e.g. du -shx /var/lib/ceph/mon/ and this shows
several GB of data.
I give all my mgrs and mons a restart and after a few minutes I can
see this osd map data getting purged from the mons. After a while it
should be back to a few hundred MB (depending on cluster size).
This may not be the problem in your case, but an easy thing to try.
Note, if your cluster is being held in Warning or Error by something
this can also explain the osd maps not clearing. Make sure you get the
cluster back to health OK first.

Rich

On Wed, 9 Jun 2021 at 08:29, Jan-Philipp Litza  wrote:
>
> Hi everyone,
>
> recently I'm noticing that starting OSDs for the first time takes ages
> (like, more than an hour) before they are even picked up by the monitors
> as "up" and start backfilling. I'm not entirely sure if this is a new
> phenomenon or if it always was that way. Either way, I'd like to
> understand why.
>
> When I execute `ceph daemon osd.X status`, it says "state: preboot" and
> I can see the "newest_map" increase slowly. Apparently, a new OSD
> doesn't fetch the latest OSD map and gets to work, but instead fetches
> hundreds of thousands of OSD maps from the mon, burning CPU while
> parsing them.
>
> I wasn't able to find any good documentation on the OSDMap, in
> particular why its historical versions need to be kept and why the OSD
> seemingly needs so many of them. Can anybody point me in the right
> direction? Or is something wrong with my cluster?
>
> Best regards,
> Jan-Philipp Litza
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: ceph df: pool stored vs bytes_used -- raw or not?

[Konstantin Shalygin]

Stored==used was resolved for this cluster. Actually problem is what you was 
discover in previous year: zero's. Filestore lack of META counter - always 
zero. When I purged last drained OSD from cluster - statistics becomes to 
normal immediately




Thanks,
k

> On 20 May 2021, at 21:22, Dan van der Ster  wrote:
> 
> I can confirm that we still occasionally see stored==used even with 14.2.21, 
> but I didn't have time yet to debug the pattern behind the observations. I'll 
> let you know if we find anything useful.

___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: OSD bootstrap time

[Konstantin Shalygin]

Hi,

You mean new fresh deployed OSD's or old just restarted OSD's?


Thanks,
k

Sent from my iPhone

> On 8 Jun 2021, at 23:30, Jan-Philipp Litza  wrote:
> 
> recently I'm noticing that starting OSDs for the first time takes ages
> (like, more than an hour) before they are even picked up by the monitors
> as "up" and start backfilling. I'm not entirely sure if this is a new
> phenomenon or if it always was that way. Either way, I'd like to
> understand why.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io