[ceph-users] Can I create 8+2 Erasure coding pool on 5 node?

[by morphin]

Hello.

I have 5 node Cluster in A datacenter. Also I have same 5 node in B datacenter.
They're gonna be 10 node 8+2 EC cluster for backup but I need to add
the 5 node later.
I have to sync my S3 data with multisite on the 5 node cluster in A
datacenter and move
them to the B and add the other 5 node to the same cluster.

The question is: Can I create 8+2 ec pool on 5 node cluster and add
the 5 node later? How can I rebalance the data after that?
Or is there any better solution in my case? what should I do?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Can I create 8+2 Erasure coding pool on 5 node?

[by morphin]

Thank you for the answers.

But I don't have problem with setting 8+2. The problem is the expansion.

I need to move the 5 node with data in it and add 5 node later because
they're in different city. The goal I'm trying to reach is 8+2 (host
crush rule)
So I want to cut the data 10 pieces and put them in 5 node. After
adding the other 5 node I want to move 5 piece of the data to new
nodes and have 8+2 in the end.

Also the data is S3 and I shouldn't break RGW multisite to continue
later...  If I can not continue then the data will be garbage anyway.
If I use internet for sync the data, it will take 1 month or more. Its
easier to take 5 node in "B" and bring to "A" and create cluster, sync
the data and move back to B :)
Because of that I need to create 5 node cluster first with "8+2 EC",
sync the data, move the 5 node to B datacenter, add the other 5 nodes
later and rebalance all the data to reach 8+2 (host)
But I really don't know it will work. I'm using Replication this is
the first time dealing with EC setup.

BTW: Every node has 20x14TB Sas and 4x900GB SSD for RGW index. ( SSD's
are Replication 3)

Dan van der Ster , 25 Mar 2021 Per, 22:03
tarihinde şunu yazdı:
>
> Here's a crush ruleset for 8+2 that will choose 2 osds per host:
>
>
> rule cephfs_data_82 {
> id 4
> type erasure
> min_size 3
> max_size 10
> step set_chooseleaf_tries 5
> step set_choose_tries 100
> step take default class hdd
> step choose indep 5 type host
> step choose indep 2 type osd
> step emit
> }
>
>
>
> This is kind of useful because if you set min_size to 8, you could even lose 
> an entire host and stay online.
>
> Cheers, dan
>
>
>
>
>
> On Thu, Mar 25, 2021, 7:02 PM by morphin  wrote:
>>
>> Hello.
>>
>> I have 5 node Cluster in A datacenter. Also I have same 5 node in B 
>> datacenter.
>> They're gonna be 10 node 8+2 EC cluster for backup but I need to add
>> the 5 node later.
>> I have to sync my S3 data with multisite on the 5 node cluster in A
>> datacenter and move
>> them to the B and add the other 5 node to the same cluster.
>>
>> The question is: Can I create 8+2 ec pool on 5 node cluster and add
>> the 5 node later? How can I rebalance the data after that?
>> Or is there any better solution in my case? what should I do?
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Cluster suspends when Add Mon or stop and start after a while.

[by morphin]

Hello!

I have a cluster with Datacenter crushmap (A+B).(9+9 = 18 servers)
The cluster started with v12.2.0 Luminous 4 years ago.
All these years I upgraded the Cluster Luminous > Mimic > v14.2.16 Nautilus.
Now I have a weird issue. When I add a mon or shutdown a while and
start it up again, all the cluster suspends, ceph -s do not respond
and other two monitors starting election while booting mon is syncing.
(logs below)



2021-03-28 00:18:23.482 7fe2f3610700  1 mon.SRV-SB-1@1(electing) e9
handle_auth_request failed to assign global_id
2021-03-28 00:18:23.782 7fe2eee07700 -1 mon.SRV-SB-1@1(electing) e9
failed to get devid for : fallback method has serial ''but no model
2021-03-28 00:18:24.292 7fe2ede05700  1 mon.SRV-SB-1@1(electing) e9
handle_auth_request failed to assign global_id
2021-03-28 00:18:26.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9
get_health_metrics reporting 3919 slow ops, oldest is log(1 entries
from seq 2031 at 2021-03-28 00:08:41.094522)
2021-03-28 00:18:29.782 7fe2f160c700  1
mon.SRV-SB-1@1(electing).elector(7899) init, last seen epoch 7899,
mid-election, bumping
2021-03-28 00:18:29.812 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9
failed to get devid for : fallback method has serial ''but no model
2021-03-28 00:18:31.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9
get_health_metrics reporting 3951 slow ops, oldest is log(1 entries
from seq 2031 at 2021-03-28 00:08:41.094522)
2021-03-28 00:18:31.872 7fe2f3610700  1 mon.SRV-SB-1@1(electing) e9
handle_auth_request failed to assign global_id
2021-03-28 00:18:32.072 7fe2f3610700  1 mon.SRV-SB-1@1(electing) e9
handle_auth_request failed to assign global_id
2021-03-28 00:18:32.482 7fe2f3610700  1 mon.SRV-SB-1@1(electing) e9
handle_auth_request failed to assign global_id
2021-03-28 00:18:33.282 7fe2ede05700  1 mon.SRV-SB-1@1(electing) e9
handle_auth_request failed to assign global_id
2021-03-28 00:18:34.812 7fe2f160c700  1
mon.SRV-SB-1@1(electing).elector(7901) init, last seen epoch 7901,
mid-election, bumping
2021-03-28 00:18:34.842 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9
failed to get devid for : fallback method has serial ''but no model
2021-03-28 00:18:34.872 7fe2ede05700  1 mon.SRV-SB-1@1(electing) e9
handle_auth_request failed to assign global_id
2021-03-28 00:18:35.072 7fe2ede05700  1 mon.SRV-SB-1@1(electing) e9
handle_auth_request failed to assign global_id
2021-03-28 00:18:35.492 7fe2ede05700  1 mon.SRV-SB-1@1(electing) e9
handle_auth_request failed to assign global_id
2021-03-28 00:18:36.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9
get_health_metrics reporting 3989 slow ops, oldest is log(1 entries
from seq 2031 at 2021-03-28 00:08:41.094522)
2021-03-28 00:18:36.292 7fe2f2e0f700  1 mon.SRV-SB-1@1(electing) e9
handle_auth_request failed to assign global_id
2021-03-28 00:18:39.842 7fe2f160c700  1
mon.SRV-SB-1@1(electing).elector(7903) init, last seen epoch 7903,
mid-election, bumping
2021-03-28 00:18:39.872 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9
failed to get devid for : fallback method has serial ''but no model
2021-03-28 00:18:40.872 7fe2ede05700  1 mon.SRV-SB-1@1(electing) e9
handle_auth_request failed to assign global_id
2021-03-28 00:18:41.082 7fe2ede05700  1 mon.SRV-SB-1@1(electing) e9
handle_auth_request failed to assign global_id
2021-03-28 00:18:41.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9
get_health_metrics reporting 4027 slow ops, oldest is log(1 entries
from seq 2031 at 2021-03-28 00:08:41.094522)
2021-03-28 00:18:41.492 7fe2f3610700  1 mon.SRV-SB-1@1(electing) e9
handle_auth_request failed to assign global_id
2021-03-28 00:18:41.812 7fe2eee07700 -1 mon.SRV-SB-1@1(electing) e9
failed to get devid for : fallback method has serial ''but no model
2021-03-28 00:18:42.312 7fe2f3610700  1 mon.SRV-SB-1@1(electing) e9
handle_auth_request failed to assign global_id
2021-03-28 00:18:43.882 7fe2ede05700  1 mon.SRV-SB-1@1(electing) e9
handle_auth_request failed to assign global_id
2021-03-28 00:18:44.082 7fe2ede05700  1 mon.SRV-SB-1@1(electing) e9
handle_auth_request failed to assign global_id
2021-03-28 00:18:44.492 7fe2ede05700  1 mon.SRV-SB-1@1(electing) e9
handle_auth_request failed to assign global_id
2021-03-28 00:18:45.302 7fe2ede05700  1 mon.SRV-SB-1@1(electing) e9
handle_auth_request failed to assign global_id
2021-03-28 00:18:46.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9
get_health_metrics reporting 4062 slow ops, oldest is log(1 entries
from seq 2031 at 2021-03-28 00:08:41.094522)
2021-03-28 00:18:47.812 7fe2f160c700  1
mon.SRV-SB-1@1(electing).elector(7905) init, last seen epoch 7905,
mid-election, bumping
2021-03-28 00:18:47.842 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9
failed to get devid for : fallback method has serial ''but no model
2021-03-28 00:18:51.102 7fe2f160c700 -1 mon.SRV-SB-1@1(electing) e9
get_health_metrics reporting 4091 slow ops, oldest is log(1 entries
from seq 2031 at 2021-03-28 00:08:41.094522)
2021-03-28 00:18:52.842 7fe2f160c700  1
mon.SRV-SB-1@1(electing).elector(7907) ini

[ceph-users] RGW S3 user.rgw.olh.pending - Can not overwrite on 0 byte objects rgw sync leftovers.

[by morphin]

Hello. I was have one-way multisite S3 cluster and we've seen issues
with rgw-sync due to sharding problems and I've stopped the multisite
sync. This is not the topic just a knowledge about my story.
I have some leftover 0 byte objects in destination and I'm trying to
overwrite them with Rclone "path to path". But somehow I can not
overwrite these objects. If I delete with rclone or rados rm and do
rclone copy again, I got the result below. Rclone gives error but the
object is created again "0 byte" with pending attrs.  Why is this
happening?
I think somehow I need to clean these objects and copy from source
again but how?

What is "user.rgw.olh.pending" ?


[root@SRV1]# radosgw-admin --id radosgw.prod1 object stat
--bucket=mybucket
--object=images/2019/05/29/ad4ba79c-bb66-4ff6-847a-09a1e0cff49f
{
"name": "images/2019/05/29/ad4ba79c-bb66-4ff6-847a-09a1e0cff49f",
"size": 0,
"tag": "713li30rvcrjfwhctx894mj7vf1wa1a8",
"attrs": {
"user.rgw.manifest": "",
"user.rgw.olh.idtag": "v1m9jy4cjck38ptel09qebsbb10pe2af",
"user.rgw.olh.info": "\u0001\u0001�",
"user.rgw.olh.pending.606b04728gs23ecq11b3i3l1":
"\u0001\u0001\u0008",
"user.rgw.olh.pending.606b0472bfhdzxeb9wesd8t7":
"\u0001\u0001\u0008",
"user.rgw.olh.pending.606b0472fv06t1dob3vmo4da":
"\u0001\u0001\u0008",
"user.rgw.olh.pending.606b0472lql6c9o88rt211r9":
"\u0001\u0001\u0008",
"user.rgw.olh.ver": ""
}
}
[root@SRV1]# rados listxattr -p prod.rgw.buckets.data
c106b26b-xxx--xxx-dee3ca5c0968.121384004.3_images/2019/05/29/ad4ba79c-bb66-4ff6-847a-09a1e0cff49f
user.rgw.idtag
user.rgw.olh.idtag
user.rgw.olh.info
user.rgw.olh.ver
[root@SRV1]# rados -p prod.rgw.buckets.data stat
c106b26b-xxx--xxx-dee3ca5c0968.121384004.3_images/2019/05/29/ad4ba79c-bb66-4ff6-847a-09a1e0cff49f
prod.rgw.buckets.data/c106b26b-xxx--xxx-dee3ca5c0968.121384004.3_images/2019/05/29/ad4ba79c-bb66-4ff6-847a-09a1e0cff49f
mtime 2021-04-05 17:10:55.00, size 0
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] RGW: Corrupted Bucket index with nautilus 14.2.16

[by morphin]

Good morning.

I have a bucket and it has 50M object in it. The bucket created with
multisite sync and that is the masterzone and only zone now.
After a health check, I saw weird objects in pending attr state.
I've tried to remove them with "radosgw-admin object rm --bypas-gc"
but I coldn't delete them. After sending delete command I see object
attributes changing to pending state but it stays like that.
Also I've tried rados rm and the object data is deleted and can't see
with radosgw-admin stat. After that I tried to copy the object with
rclone "source to dest" But I can not write the object in same path!
Rclone gives error but the object is created anyway and I see 0 byte
object with rados and radosgw-admin again!
After that I've tried to copy the problematic object from
/samebucket/objectpath/theObject --> /samebucket/theObject and the 0
byte object copied with success and its not 0 byte anymore! It looks
like ok!

Then after all these I wonder if its an index issue?
I've run radosgw-admin bi list and I saw 29 entry for same object (I
guess all these retries left most of them behind) Also I run bi list
for a normal object and I see only 5 entry. (Versions).

There is bi list for problematic object:
https://paste.ubuntu.com/p/jRtH3cMC94/


I'm too scared of checking or re-creating with bucket index.
Is there anyone have seen the same issue?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] ERROR: read_key_entry() idx= 1000_ ret=-2

[by morphin]

Hello everyone!

I'm running nautilus 14.2.16 and I'm using RGW with Beast frontend.
I see this eror log in every SSD osd which is using for rgw index.
Can you please tell me what is the problem?

OSD LOG:
cls_rgw.cc:1102: ERROR: read_key_entry()
idx=�1000_matches/xdir/05/21/27260.jpg ret=-2
cls_rgw.cc:1102: ERROR: read_key_entry()
idx=�1000_matches/xdir/05/21/27253.jpg ret=-2


RADOSGW LOG:
2021-04-15 01:53:54.385 7f2e0f8e7700  1 beast: 0x55a4439f8710:
10.151.101.15 - - [2021-04-15 01:53:54.0.385327s] "HEAD
/xdir/04/13/704745.jpg HTTP/1.1" 200 0 - "aws-sdk-java/1.11.638
Linux/3.10.0-1062.12.1.el7.x86_64
Java_HotSpot(TM)_64-Bit_Server_VM/25.251-b08 java/1.8.0_251 groovy/2.4.3
vendor/Oracle_Corporation" -
2021-04-15 01:53:54.385 7f2d8b7df700  1 == starting new request
req=0x55a4439f8710 =
2021-04-15 01:53:54.405 7f2e008c9700  1 == req done req=0x55a43dbc6710
op status=0 http_status=204 latency=0.33s ==
2021-04-15 01:53:54.405 7f2e008c9700  1 beast: 0x55a43dbc6710:
10.151.101.15 - - [2021-04-15 01:53:54.0.405327s] "DELETE
/xdir/05/21/21586.gz HTTP/1.1" 204 0 - "aws-sdk-java/1.11.638
Linux/3.10.0-1062.12.1.el7.x86_64
Java_HotSpot(TM)_64-Bit_Server_VM/25.251-b08 java/1.8.0_251 groovy/2.4.3
vendor/Oracle_Corporation" -
2021-04-15 01:53:54.405 7f2d92fee700  1 == starting new request
req=0x55a43dbc6710 =
2021-04-15 01:53:54.405 7f2d92fee700  0 WARNING: couldn't find acl header
for object, generating default
2021-04-15 01:53:54.405 7f2d92fee700  1 == req done req=0x55a43dbc6710
op status=0 http_status=200 latency=0s ==
2021-04-15 01:53:54.405 7f2d92fee700  1 beast: 0x55a43dbc6710:
10.151.101.15 - - [2021-04-15 01:53:54.0.405327s] "HEAD
/xdir/2013/11/20/2a67508e-d7dd-4e0f-b959-d7575d5f65b1 HTTP/1.1" 200 0 -
"aws-sdk-java/1.11.638 Linux/3.10.0-1160.11.1.el7.x86_64
Java_HotSpot(TM)_64-Bit_Server_VM/25.281-b09 java/1.8.0_281 groovy/2.5.6
vendor/Oracle_Corporation" -


CEPH OSD DF
ID  CLASS WEIGHT   REWEIGHT SIZERAW USE DATAOMAPMETAAVAIL
%USE  VAR  PGS STATUS
 19   ssd  0.87320  1.0 894 GiB 436 GiB 101 GiB 332 GiB 2.5 GiB 458 GiB
48.75 1.84 115 up
208   ssd  0.87329  1.0 894 GiB 161 GiB  87 GiB  73 GiB 978 MiB 733 GiB
18.00 0.68 113 up
199   ssd  0.87320  1.0 894 GiB 272 GiB 106 GiB 163 GiB 2.4 GiB 623 GiB
30.37 1.14 123 up
202   ssd  0.87329  1.0 894 GiB 239 GiB  73 GiB 165 GiB 1.4 GiB 655 GiB
26.77 1.01 106 up
 39   ssd  0.87320  1.0 894 GiB 450 GiB  87 GiB 361 GiB 2.3 GiB 444 GiB
50.36 1.90 113 up
207   ssd  0.87329  1.0 894 GiB 204 GiB 100 GiB  98 GiB 6.0 GiB 691 GiB
22.76 0.86 118 up
 59   ssd  0.87320  1.0 894 GiB 372 GiB 107 GiB 263 GiB 3.0 GiB 522 GiB
41.64 1.57 122 up
203   ssd  0.87329  1.0 894 GiB 206 GiB  79 GiB 124 GiB 2.4 GiB 689 GiB
23.00 0.87 117 up
 79   ssd  0.87320  1.0 894 GiB 447 GiB 103 GiB 342 GiB 1.8 GiB 447 GiB
49.97 1.88 120 up
206   ssd  0.87329  1.0 894 GiB 200 GiB  81 GiB 119 GiB 1.0 GiB 694 GiB
22.38 0.84  94 up
 99   ssd  0.87320  1.0 894 GiB 333 GiB  87 GiB 244 GiB 2.0 GiB 562 GiB
37.19 1.40 106 up
205   ssd  0.87329  1.0 894 GiB 316 GiB  83 GiB 232 GiB 1.1 GiB 579 GiB
35.29 1.33 117 up
114   ssd  0.87329  1.0 894 GiB 256 GiB 100 GiB 154 GiB 1.7 GiB 638 GiB
28.61 1.08 113 up
200   ssd  0.87329  1.0 894 GiB 266 GiB 100 GiB 165 GiB 1.1 GiB 628 GiB
29.76 1.12 128 up
139   ssd  0.87320  1.0 894 GiB 234 GiB  79 GiB 153 GiB 1.7 GiB 660 GiB
26.14 0.98 104 up
204   ssd  0.87329  1.0 894 GiB 173 GiB 113 GiB  59 GiB 1.2 GiB 721 GiB
19.37 0.73 124 up
119   ssd  0.87329  1.0 894 GiB 248 GiB 108 GiB 139 GiB 1.9 GiB 646 GiB
27.76 1.05 130 up
159   ssd  0.87329  1.0 894 GiB 196 GiB  94 GiB  99 GiB 2.6 GiB 699 GiB
21.87 0.82 109 up
179   ssd  0.87329  1.0 894 GiB 427 GiB  81 GiB 341 GiB 4.7 GiB 467 GiB
47.73 1.80 114 up
201   ssd  0.87329  1.0 894 GiB 346 GiB 102 GiB 242 GiB 1.8 GiB 548 GiB
38.71 1.46 128 up

CEPH IOSTAT
+---+---+---+---+---+---+
|  Read | Write | Total | Read IOPS |Write
IOPS |Total IOPS |
+---+---+---+---+---+---+
| 329 MiB/s |  39 MiB/s | 368 MiB/s |109027 |
1646 |110673 |
| 329 MiB/s |  39 MiB/s | 368 MiB/s |109027 |
1646 |110673 |
| 331 MiB/s |  39 MiB/s | 371 MiB/s |114915 |
1631 |116547 |
| 331 MiB/s |  39 MiB/s | 371 MiB/s |114915 |
1631 |116547 |
| 308 MiB/s |  42 MiB/s | 350 MiB/s |108469 |
1635 |110104 |
| 308 MiB/s |  42 MiB/s | 350 MiB/s |108469 |
1635 |110104 |
| 291 MiB/s |  44 MiB/s | 335 MiB/s |105828 |
1687 |107516 |
___
ceph-users mailing list -- c

[ceph-users] Re: ERROR: read_key_entry() idx= 1000_ ret=-2

[by morphin]

More informations:

I have a overlimit bucket and the error belongs to this bucket.

fill_status=OVER 100%
objects_per_shard: 363472 (I use default 100K per shard)
num_shards: 750


I'm deleting objects from this bucket with absolute path and I dont use
dynamic bucket resharding due to multisite.
I've reviewed the code and I think I see the error because these objects
not exist on index.
Can anyone explain the code and the error please?

OSD LOG:
cls_rgw.cc:1102: ERROR: read_key_entry()
idx=�1000_matches/xdir/05/21/27260.jpg ret=-2

https://github.com/ceph/ceph/blob/master/src/cls/rgw/cls_rgw.cc


public:
BIVerObjEntry(cls_method_context_t& _hctx, const cls_rgw_obj_key& _key) :
hctx(_hctx), key(_key), initialized(false) {
}
int init(bool check_delete_marker = true) {
int ret = read_key_entry(hctx, key, &instance_idx, &instance_entry,
check_delete_marker && key.instance.empty()); /* this is potentially a
delete marker, for null objects we
keep separate instance entry for the delete markers */
if (ret < 0) {
CLS_LOG(0, "ERROR: read_key_entry() idx=%s ret=%d", instance_idx.c_str(),
ret);
return ret;
}
initialized = true;
CLS_LOG(20, "read instance_entry key.name=%s key.instance=%s flags=%d",
instance_entry.key.name.c_str(), instance_entry.key.instance.c_str(),
instance_entry.flags);
return 0;
}
rgw_bucket_dir_entry& get_dir_entry() {
return instance_entry;
}


by morphin , 15 Nis 2021 Per, 02:19 tarihinde
şunu yazdı:

> Hello everyone!
>
> I'm running nautilus 14.2.16 and I'm using RGW with Beast frontend.
> I see this eror log in every SSD osd which is using for rgw index.
> Can you please tell me what is the problem?
>
> OSD LOG:
> cls_rgw.cc:1102: ERROR: read_key_entry()
> idx=�1000_matches/xdir/05/21/27260.jpg ret=-2
> cls_rgw.cc:1102: ERROR: read_key_entry()
> idx=�1000_matches/xdir/05/21/27253.jpg ret=-2
>
>
> RADOSGW LOG:
> 2021-04-15 01:53:54.385 7f2e0f8e7700  1 beast: 0x55a4439f8710:
> 10.151.101.15 - - [2021-04-15 01:53:54.0.385327s] "HEAD
> /xdir/04/13/704745.jpg HTTP/1.1" 200 0 - "aws-sdk-java/1.11.638
> Linux/3.10.0-1062.12.1.el7.x86_64
> Java_HotSpot(TM)_64-Bit_Server_VM/25.251-b08 java/1.8.0_251 groovy/2.4.3
> vendor/Oracle_Corporation" -
> 2021-04-15 01:53:54.385 7f2d8b7df700  1 == starting new request
> req=0x55a4439f8710 =
> 2021-04-15 01:53:54.405 7f2e008c9700  1 == req done req=0x55a43dbc6710
> op status=0 http_status=204 latency=0.33s ==
> 2021-04-15 01:53:54.405 7f2e008c9700  1 beast: 0x55a43dbc6710:
> 10.151.101.15 - - [2021-04-15 01:53:54.0.405327s] "DELETE
> /xdir/05/21/21586.gz HTTP/1.1" 204 0 - "aws-sdk-java/1.11.638
> Linux/3.10.0-1062.12.1.el7.x86_64
> Java_HotSpot(TM)_64-Bit_Server_VM/25.251-b08 java/1.8.0_251 groovy/2.4.3
> vendor/Oracle_Corporation" -
> 2021-04-15 01:53:54.405 7f2d92fee700  1 == starting new request
> req=0x55a43dbc6710 =
> 2021-04-15 01:53:54.405 7f2d92fee700  0 WARNING: couldn't find acl header
> for object, generating default
> 2021-04-15 01:53:54.405 7f2d92fee700  1 == req done req=0x55a43dbc6710
> op status=0 http_status=200 latency=0s ==
> 2021-04-15 01:53:54.405 7f2d92fee700  1 beast: 0x55a43dbc6710:
> 10.151.101.15 - - [2021-04-15 01:53:54.0.405327s] "HEAD
> /xdir/2013/11/20/2a67508e-d7dd-4e0f-b959-d7575d5f65b1 HTTP/1.1" 200 0 -
> "aws-sdk-java/1.11.638 Linux/3.10.0-1160.11.1.el7.x86_64
> Java_HotSpot(TM)_64-Bit_Server_VM/25.281-b09 java/1.8.0_281 groovy/2.5.6
> vendor/Oracle_Corporation" -
>
>
> CEPH OSD DF
> ID  CLASS WEIGHT   REWEIGHT SIZERAW USE DATAOMAPMETAAVAIL
>   %USE  VAR  PGS STATUS
>  19   ssd  0.87320  1.0 894 GiB 436 GiB 101 GiB 332 GiB 2.5 GiB 458
> GiB 48.75 1.84 115 up
> 208   ssd  0.87329  1.0 894 GiB 161 GiB  87 GiB  73 GiB 978 MiB 733
> GiB 18.00 0.68 113 up
> 199   ssd  0.87320  1.0 894 GiB 272 GiB 106 GiB 163 GiB 2.4 GiB 623
> GiB 30.37 1.14 123 up
> 202   ssd  0.87329  1.0 894 GiB 239 GiB  73 GiB 165 GiB 1.4 GiB 655
> GiB 26.77 1.01 106 up
>  39   ssd  0.87320  1.0 894 GiB 450 GiB  87 GiB 361 GiB 2.3 GiB 444
> GiB 50.36 1.90 113 up
> 207   ssd  0.87329  1.0 894 GiB 204 GiB 100 GiB  98 GiB 6.0 GiB 691
> GiB 22.76 0.86 118 up
>  59   ssd  0.87320  1.0 894 GiB 372 GiB 107 GiB 263 GiB 3.0 GiB 522
> GiB 41.64 1.57 122 up
> 203   ssd  0.87329  1.0 894 GiB 206 GiB  79 GiB 124 GiB 2.4 GiB 689
> GiB 23.00 0.87 117 up
>  79   ssd  0.87320  1.0 894 GiB 447 GiB 103 GiB 342 GiB 1.8 GiB 447
> GiB 49.97 1.88 120 up
> 206   ssd  0.87329  1.0 894 GiB 200 GiB  81 GiB 119 GiB 1.0 GiB 694
> GiB 22.38 0.84  94 up
>  99   ssd  0.87320  1.0 894 GiB 333 GiB  87 GiB 244 GiB 2.0 GiB 562

[ceph-users] Re: Revisit Large OMAP Objects

[by morphin]

I've same issue and joined to the club.
Almost every deleted bucket is still there due to multisite. Also I've
removed secondary zone and stopped sync but these stale-instance's still
there.
Before adding new secondary zone I want to remove them. If you gonna run
anything let me know please.




 adresine sahip kullanıcı 14 Nis 2021 Çar, 21:20
tarihinde şunu yazdı:

> Casey;
>
> That makes sense, and I appreciate the explanation.
>
> If I were to shut down all uses of RGW, and wait for replication to catch
> up, would this then address most known issues with running this command in
> a multi-site environment?  Can I offline RADOSGW daemons as an added
> precaution?
>
> Thank you,
>
> Dominic L. Hilsbos, MBA
> Director – Information Technology
> Perform Air International Inc.
> dhils...@performair.com
> www.PerformAir.com
>
>
> -Original Message-
> From: Casey Bodley [mailto:cbod...@redhat.com]
> Sent: Wednesday, April 14, 2021 9:03 AM
> To: Dominic Hilsbos
> Cc: k0...@k0ste.ru; ceph-users@ceph.io
> Subject: Re: [ceph-users] Re: Revisit Large OMAP Objects
>
> On Wed, Apr 14, 2021 at 11:44 AM  wrote:
> >
> > Konstantin;
> >
> > Dynamic resharding is disabled in multisite environments.
> >
> > I believe you mean radosgw-admin reshard stale-instances rm.
> >
> > Documentation suggests this shouldn't be run in a multisite
> environment.  Does anyone know the reason for this?
>
> say there's a bucket with 10 objects in it, and that's been fully
> replicated to a secondary zone. if you want to remove the bucket, you
> delete its objects then delete the bucket
>
> when the bucket is deleted, rgw can't delete its bucket instance yet
> because the secondary zone may not be caught up with sync - it
> requires access to the bucket instance (and its index) to sync those
> last 10 object deletions
>
> so the risk with 'stales-instances rm' in multisite is that you might
> delete instances before other zones catch up, which can lead to
> orphaned objects
>
> >
> > Is it, in fact, safe, even in a multisite environment?
> >
> > Thank you,
> >
> > Dominic L. Hilsbos, MBA
> > Director – Information Technology
> > Perform Air International Inc.
> > dhils...@performair.com
> > www.PerformAir.com
> >
> >
> > -Original Message-
> > From: Konstantin Shalygin [mailto:k0...@k0ste.ru]
> > Sent: Wednesday, April 14, 2021 12:15 AM
> > To: Dominic Hilsbos
> > Cc: ceph-users@ceph.io
> > Subject: Re: [ceph-users] Revisit Large OMAP Objects
> >
> > Run reshard instances rm
> > And reshard your bucket by hand or leave dynamic resharding process to
> do this work
> >
> >
> > k
> >
> > Sent from my iPhone
> >
> > > On 13 Apr 2021, at 19:33, dhils...@performair.com wrote:
> > >
> > > All;
> > >
> > > We run 2 Nautilus clusters, with RADOSGW replication (14.2.11 -->
> 14.2.16).
> > >
> > > Initially our bucket grew very quickly, as I was loading old data into
> it and we quickly ran into Large OMAP Object warnings.
> > >
> > > I have since done a couple manual reshards, which has fixed the
> warning on the primary cluster.  I have never been able to get rid of the
> issue on the cluster with the replica.
> > >
> > > I prior conversation on this list led me to this command:
> > > radosgw-admin reshard stale-instances list --yes-i-really-mean-it
> > >
> > > The results of which look like this:
> > > [
> > >"nextcloud-ra:f91aeff8-a365-47b4-a1c8-928cd66134e8.185262.1",
> > >"nextcloud:f91aeff8-a365-47b4-a1c8-928cd66134e8.53761.6",
> > >"nextcloud:f91aeff8-a365-47b4-a1c8-928cd66134e8.53761.2",
> > >"nextcloud:f91aeff8-a365-47b4-a1c8-928cd66134e8.53761.5",
> > >"nextcloud:f91aeff8-a365-47b4-a1c8-928cd66134e8.53761.4",
> > >"nextcloud:f91aeff8-a365-47b4-a1c8-928cd66134e8.53761.3",
> > >"nextcloud:f91aeff8-a365-47b4-a1c8-928cd66134e8.53761.1",
> > >"3520ae821f974340afd018110c1065b8/OS
> Development:f91aeff8-a365-47b4-a1c8-928cd66134e8.4298264.1",
> > >
> "10dfdfadb7374ea1ba37bee1435d87ad/volumebackups:f91aeff8-a365-47b4-a1c8-928cd66134e8.4298264.2",
> > >"WorkOrder:f91aeff8-a365-47b4-a1c8-928cd66134e8.44130.1"
> > > ]
> > >
> > > I find this particularly interesting, as nextcloud-ra, /OS
> Development, /volumbackups, and WorkOrder buckets no longer exist.
> > >
> > > When I run:
> > > for obj in $(rados -p 300.rgw.buckets.index ls | grep
> f91aeff8-a365-47b4-a1c8-928cd66134e8.3512190.1);   do   printf "%-60s
> %7d\n" $obj $(rados -p 300.rgw.buckets.index listomapkeys $obj | wc -l);
>  done
> > >
> > > I get the expected 64 entries, with counts around 2 +/- 1000.
> > >
> > > Are the above listed stale instances ok to delete?  If so, how do I go
> about doing so?
> > >
> > > Thank you,
> > >
> > > Dominic L. Hilsbos, MBA
> > > Director - Information Technology
> > > Perform Air International Inc.
> > > dhils...@performair.com
> > > www.PerformAir.com
> > >
> > > ___
> > > ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to

[ceph-users] Radosgw - WARNING: couldn't find acl header for object, generating default

[by morphin]

Hello.

I've a RGW bucket (versioning=on). And there was objects like this:

radosgw-admin object stat --bucket=xdir
--object=f5492238-50cb-4bc2-93fa-424869018946
{
"name": "f5492238-50cb-4bc2-93fa-424869018946",
"size": 0,
"tag": "",
"attrs": {
"user.rgw.manifest": "",
"user.rgw.olh.idtag": "5rs3x0qh152tn0j865k8ybo9xqy92qjn",
"user.rgw.olh.info": "\u0001\u0001�",
"user.rgw.olh.pending.607c87b5pgo03tvm3sqt23i9":
"\u0001\u0001\u0008",
"user.rgw.olh.pending.607c87b5pyv13ugk3fadvxw7":
"\u0001\u0001\u0008",
"user.rgw.olh.pending.607c87b5qic02n0e54zsjkax":
"\u0001\u0001\u0008",
"user.rgw.olh.ver": "3"
}
}

I'm not sure but I suspect that these objects maybe leftover from
unfinished multisite sync.
I've removed the zone and made it master (because I have to for something
else) and I've created a newbucket with (versioning=off) and copied all
objects from old rgw bucket to a new rgw bucket with "rclone"

cmd: "rclone copy --files-from "object.list" old:bucket new:bucket
--no-traverse -vv --progress --fast-list --no-check-dest
--no-update-modtime"

Config:
[bucket]
type = s3
provider = Ceph
env_auth = false
acl = private
bucket =
access_key_id =
secret_access_key =
endpoint =

After the copy I've checked every object via "radosgw-admin object stat
object=$i" and there was still these problematic objects. I've tried to
overwrite these objects from the Backup-bucket at different cluster (these
objects are fine) and I tried to be sure everything written as expected via
object stat again and these pending object was gone and everything was ok.
All of them was OK!

After a while I started to see this warning in radosgw.log when the client
GET or HEAD the objects.

2021-04-19 11:37:50.230 7f2d917eb700  1 == starting new request
req=0x55a44414a710 =
2021-04-19 11:37:50.230 7f2d917eb700  0 WARNING: couldn't find acl header
for object, generating default
2021-04-19 11:37:50.230 7f2d917eb700  1 == req done req=0x55a44414a710
op status=0 http_status=200 latency=0s ==
2021-04-19 11:37:50.230 7f2d917eb700  1 beast: 0x55a44414a710: 10.10.10.1 -
- [2021-04-19 11:37:50.0.230489s] "HEAD
/xdir/f5492238-50cb-4bc2-93fa-424869018946 HTTP/1.1" 200 0 -
"aws-sdk-java/1.11.638 Linux/3.10.0-1160.11.1.el7.x86_64
Java_HotSpot(TM)_64-Bit_Server_VM/25.281-b09 java/1.8.0_281 groovy/2.5.6
vendor/Oracle_Corporation" -

These objects are problematic objects and somehow the overwritten objects
is gone and these objects left after the WARNING. (versioning is still =off
--> multisite = off --> the cluster alone in zonegroup and its masterzone)

But I've checked every overwritten object via "radosgw-admin object stat
object=$i" and there was all OK!!! How the hell they're gone and others
came back? HOW?
What should I do? Maybe "object rm" "log trim" and re-write again?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] BlueFS spillover detected (Nautilus 14.2.16)

[by morphin]

Hello.

I'm trying to fix a wrong cluster deployment (Nautilus 14.2.16)
Cluster usage is %40 EC pool with RGW

Every node has:
20 x OSD = TOSHIBA  MG08SCA16TEY 16.0TB
2 x DB = NVME PM1725b 1.6TB (linux mdadm raid1)

NVME usage always goes around %90-99.
With "iostat -xdh 1"
 r/s w/s rkB/s wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm
r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util Device
  168.00 3619.00  7.2M367.7M 0.00 90510.00   0.0%  96.2%
1.109.21  22.8643.8k   104.0k   0.25  96.0% nvme0c0n1
   19.00 3670.00  1.7M373.5M 0.00 90510.00   0.0%  96.1%
0.26   29.61  95.9989.7k   104.2k   0.27  98.0% nvme1c1n1

The problem is: BLUEFS_SPILLOVER BlueFS spillover detected on 120 OSD(s)
 osd.194 spilled over 42 GiB metadata from 'db' device (39 GiB used of
50 GiB) to slow device
 osd.195 spilled over 34 GiB metadata from 'db' device (40 GiB used of
50 GiB) to slow device
 osd.196 spilled over 28 GiB metadata from 'db' device (40 GiB used of
50 GiB) to slow device
 osd.197 spilled over 25 GiB metadata from 'db' device (41 GiB used of
50 GiB) to slow device
 osd.198 spilled over 30 GiB metadata from 'db' device (41 GiB used of
50 GiB) to slow device

Block and wal size:
bluestore_block_db_size = 53687091200
bluestore_block_wal_size = 0

nvme0n1
   259:20   1.5T  0 disk
└─md0
 9:00   1.5T  0 raid1
  ├─md0p1
   259:4050G  0 md
  ├─md0p2
   259:5050G  0 md
+n
  └─md0p20
  259:22   050G  0 md


How can I change the level up to 500MB --> 5GB --> 50GB ?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: BlueFS spillover detected (Nautilus 14.2.16)

[by morphin]

Thanks for the answer. It seems very easy.
I've never played with rocksdb options before. I always used default
and I think I need to play more with it but I couldn't find a good
config reference to understand at ceph side.
Can I use this guide instead?
https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide

This is the default options: "ceph config help bluestore_rocksdb_options"

bluestore_rocksdb_options - Rocksdb options
  (str, advanced)
  Default: 
compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2
  Can update at runtime: false


> * get you release options `ceph config help bluestore_rocksdb_options`
> * append `bluestore_rocksdb_options=536870912` to this list
> * set `ceph config set osd bluestore_rocksdb_options `

 Are you tried to say add these (below) options to the config?

- options.max_bytes_for_level_base = 536870912; // 512MB
- options.max_bytes_for_level_multiplier = 10;


At the link below I've found a tune but its for All-Flash cluster.

https://ceph.io/community/bluestore-default-vs-tuned-performance-comparison/
bluestore_rocksdb_options =
compression=kNoCompression,max_write_buffer_number=32,min_write_buffer_number_to_merge=2,recycle_log_file_num=32,compaction_style=kCompactionStyleLevel,write_buffer_size=67108864,target_file_size_base=67108864,max_background_compactions=31,level0_file_num_compaction_trigger=8,level0_slowdown_writes_trigger=32,level0_stop_writes_trigger=64,max_bytes_for_level_base=536870912,compaction_threads=32,max_bytes_for_level_multiplier=8,flusher_threads=8,compaction_readahead_size=2MB

Also I've 2 type OSD's. SSD's for RGW index(no external db) and HDD's
for EC pool with NVME cache. Do I need different options?






Konstantin Shalygin , 19 Nis 2021 Pzt, 20:16 tarihinde
şunu yazdı:
>
> You need to adjust max_bytes_for_level_base rocksdb option from 
> bluestore_rocksdb_options to 536870912
>
> * get you release options `ceph config help bluestore_rocksdb_options`
> * append `bluestore_rocksdb_options=536870912` to this list
> * set `ceph config set osd bluestore_rocksdb_options `
>
> Restart your OSD's.
>
>
> k
>
> On 19 Apr 2021, at 17:34, by morphin  wrote:
>
> How can I change the level up to 500MB --> 5GB --> 50GB ?
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: BlueFS spillover detected (Nautilus 14.2.16)

[by morphin]

There is a lot of bug-fix on RGW between 14.2.16 --> 19 and this is a
prod environment. I always follow few versions behind to minimize the
risk. Only for OSD improvement I'll not take the risk at RGW side.

It's better to play with rocksdb options.
Thanks for the advice.

Konstantin Shalygin , 19 Nis 2021 Pzt, 22:57 tarihinde
şunu yazdı:
>
> Multiplier is already == 10, it's not need to change, just a base. Or only 
> multiplier.
> I better suggest upgrade to 14.2.19 and use new bluestore policy to use extra 
> space for rocksdb levels (will be activated by default)
>
>
>
>
> k
>
> On 19 Apr 2021, at 21:09, by morphin  wrote:
>
> Are you tried to say add these (below) options to the config?
>
> - options.max_bytes_for_level_base = 536870912; // 512MB
> - options.max_bytes_for_level_multiplier = 10;
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] RGW objects has same marker and bucket id in different buckets.

[by morphin]

Hello.

I have a rgw s3 user and the user have 2 bucket.
I tried to copy objects from old.bucket to new.bucket with rclone. (in
the rgw client server)
After I checked the object with "radosgw-admin --bucket=new.bucket
object stat $i" and I saw old.bucket id and marker id also old bucket
name in the object stats.

Is rgw doing this for deduplication or is it a bug?
If it's not a bug then If I delete the old bucket what will happen to
these objects???
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RGW objects has same marker and bucket id in different buckets.

[by morphin]

Thanks for the answer.

I have 2 question:
1- If I use different user and a bucket what will happen?  Is this
design only for same user or user independent?
2- If I delete the Source bucket with radosgw-admin or via S3 delete,
what will happen to these objects?





Also I have more questions if you have time :)

Some objects are pending state due to zone-sync error. I've removed
the master zone and set secondary to master.
I still see the pending objects in the bucket. (below)

radosgw-admin --id radosgw.srv1 object stat --bucket=descript
--object=2020/01/17/1b819bd9-5036-4ca4-98f7-b0308e1e3017
{
"name": "2020/01/17/1b819bd9-5036-4ca4-98f7-b0308e1e3017",
"size": 0,
"tag": "",
"attrs": {
"user.rgw.manifest": "",
"user.rgw.olh.idtag": "ivlde1avu2l3lli6i349h62c0d79ao4u",
"user.rgw.olh.info": "\u0001\u0001�",
"user.rgw.olh.pending.607d4d5be0hh3lpzjd7vzt2j":
"\u0001\u0001\u0008",
"user.rgw.olh.pending.607d4d5c9uhlh9sf93j8lf7l":
"\u0001\u0001\u0008",
"user.rgw.olh.pending.607d4d5cpip1i8z8rytcnkqf":
"\u0001\u0001\u0008",
"user.rgw.olh.ver": "3"
}
}


I overwrite these objects with rclone from old zone bucket to new
created bucket on the same user at master zone.

After a while I noticed that I'm getting a warning for these objects
in rgw client log and the overwritten objects switching back to the
corrupted objects.

2021-04-22 10:27:55.445 7f2d85fd4700  0 WARNING: couldn't find acl
header for object, generating default
2021-04-22 10:27:55.445 7f2d85fd4700  1 == req done
req=0x55a441452710 op status=0 http_status=200 latency=0.022s
==
2021-04-22 10:27:55.445 7f2d85fd4700  1 beast: 0x55a441452710:
10.151.101.15 - - [2021-04-22 10:27:55.0.44549s] "GET
/descript/2020/01/17/1b819bd9-5036-4ca4-98f7-b0308e1e3017 HTTP/1.1"
200 0 - "aws-sdk-java/1.11.638 Linux/3.10.0-1160.11.1.el7.x86_64
Java_HotSpot(TM)_64-Bit_Server_VM/25.281-b09 java/1.8.0_281
groovy/2.5.6 vendor/Oracle_Corporation" -

Am I doing something wrong?
Also "sync error trim" does not work. How can I clean these errors and
these pending objects?

ceph version 14.2.16


Have a great day.
Regards.


Matt Benjamin , 22 Nis 2021 Per, 06:08 tarihinde
şunu yazdı:
>
> Hi Morphin,
>
> Yes, this is by design.  When an RGW object has tail chunks and is
> copied so as to duplicate an entire tail chunk, RGW causes the
> coincident chunk(s) to be shared.  Tail chunks are refcounted to avoid
> leaks.
>
> Matt
>
> On Wed, Apr 21, 2021 at 4:21 PM by morphin  wrote:
> >
> > Hello.
> >
> > I have a rgw s3 user and the user have 2 bucket.
> > I tried to copy objects from old.bucket to new.bucket with rclone. (in
> > the rgw client server)
> > After I checked the object with "radosgw-admin --bucket=new.bucket
> > object stat $i" and I saw old.bucket id and marker id also old bucket
> > name in the object stats.
> >
> > Is rgw doing this for deduplication or is it a bug?
> > If it's not a bug then If I delete the old bucket what will happen to
> > these objects???
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >
>
>
> --
>
> Matt Benjamin
> Red Hat, Inc.
> 315 West Huron Street, Suite 140A
> Ann Arbor, Michigan 48103
>
> http://www.redhat.com/en/technologies/storage
>
> tel.  734-821-5101
> fax.  734-769-8938
> cel.  734-216-5309
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: cephadm: how to create more than 1 rgw per host

[by morphin]

Hello.

Its easy. In ceph.conf copy the rgw fields and change 3 things.
1- name
2- log path name
3- client port.


After that feel free to start rgw service with systemctl. Check service
status and Tail the rgw log file. Try to read or write and check the logs.
If everything works as expected then you are ready to add the new service
to loadbalancer if you have one.



22 Nis 2021 Per 14:00 tarihinde i...@z1storage.com 
şunu yazdı:

> Does anyone know how to create more than 1 rgw per host? Surely it's not
> a rare configuration.
>
> On 2021/04/19 17:09, i...@z1storage.com wrote:
> >
> > Hi Sebastian,
> >
> > Thank you. Is there a way to create more than 1 rgw per host until
> > this new feature is released?
> >
> > On 2021/04/19 11:39, Sebastian Wagner wrote:
> >> Hi Ivan,
> >>
> >> this is a feature that is not yet released in Pacific. It seems the
> >> documentation is a bit ahead of time right now.
> >>
> >> Sebastian
> >>
> >> On Fri, Apr 16, 2021 at 10:58 PM i...@z1storage.com
> >>   >> > wrote:
> >>
> >> Hello,
> >>
> >> According to the documentation, there's count-per-host key to 'ceph
> >> orch', but it does not work for me:
> >>
> >> :~# ceph orch apply rgw z1 sa-1 --placement='label:rgw
> >> count-per-host:2'
> >> --port=8000 --dry-run
> >> Error EINVAL: Host and label are mutually exclusive
> >>
> >> Why it says anything about Host if I don't specify any hosts,
> >> just labels?
> >>
> >> ~# ceph orch host ls
> >> HOST  ADDR  LABELS   STATUS
> >> s101  s101  mon rgw
> >> s102  s102  mgr mon rgw
> >> s103  s103  mon rgw
> >> s104  s104  mgr mon rgw
> >> s105  s105  mgr mon rgw
> >> s106  s106  mon rgw
> >> s107  s107  mon rgw
> >>
> >> ___
> >> ceph-users mailing list -- ceph-users@ceph.io
> >> 
> >> To unsubscribe send an email to ceph-users-le...@ceph.io
> >> 
> >>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RGW objects has same marker and bucket id in different buckets.

[by morphin]

Its crystal clear now. Thank you so much for the answers.

The problem is I cant delete these pending objects with radosgw-admin
object rm $obj. Also sync error trim does nothing.
Due to this problem I've created a New bucket and copied these objects from
the main source (out of cluster) and also copied other files from
old.bucket because they was not on main source. Ive checked all the files
after the copy and they was all ok.

Somehow the copies started to change it self to these pending objects after
s3 get operations. I think somehow rgw reads the file and make current obj
to the sync errored obj. Only log is this:
WARNING: couldn't find acl header for object, generating default

I need to clear sync error logs and the pending objects. But can not do
both. Even deleting rados object directly does not change the situation.
When I try to copy after rados rm, the object turns into pending object
again.
Maybe deleting old periods will help. What do you think?


Its very Hard to explain. I tried my best.
Regards.




22 Nis 2021 Per 18:26 tarihinde Matt Benjamin  şunu
yazdı:

> Hi Morphin,
>
> On Thu, Apr 22, 2021 at 3:40 AM by morphin 
> wrote:
> >
> > Thanks for the answer.
> >
> > I have 2 question:
> > 1- If I use different user and a bucket what will happen?  Is this
> > design only for same user or user independent?
>
> It's user independent.
>
> > 2- If I delete the Source bucket with radosgw-admin or via S3 delete,
> > what will happen to these objects?
> >
>
> The refcount on each shared object will be decreased by 1.  If there
> are no more references, the objects will be garbage collected.
>
> >
> >
> > Also I have more questions if you have time :)
>
> I don't have the expertise in multisite replication to debug this.  I
> don't think the issue is related to RGW's copy-sharing.
>
> Matt
>
> >
> > Some objects are pending state due to zone-sync error. I've removed
> > the master zone and set secondary to master.
> > I still see the pending objects in the bucket. (below)
> >
> > radosgw-admin --id radosgw.srv1 object stat --bucket=descript
> > --object=2020/01/17/1b819bd9-5036-4ca4-98f7-b0308e1e3017
> > {
> > "name": "2020/01/17/1b819bd9-5036-4ca4-98f7-b0308e1e3017",
> > "size": 0,
> > "tag": "",
> > "attrs": {
> > "user.rgw.manifest": "",
> > "user.rgw.olh.idtag": "ivlde1avu2l3lli6i349h62c0d79ao4u",
> > "user.rgw.olh.info": "\u0001\u0001�",
> > "user.rgw.olh.pending.607d4d5be0hh3lpzjd7vzt2j":
> > "\u0001\u0001\u0008",
> > "user.rgw.olh.pending.607d4d5c9uhlh9sf93j8lf7l":
> > "\u0001\u0001\u0008",
> > "user.rgw.olh.pending.607d4d5cpip1i8z8rytcnkqf":
> > "\u0001\u0001\u0008",
> > "user.rgw.olh.ver": "3"
> > }
> > }
> >
> >
> > I overwrite these objects with rclone from old zone bucket to new
> > created bucket on the same user at master zone.
> >
> > After a while I noticed that I'm getting a warning for these objects
> > in rgw client log and the overwritten objects switching back to the
> > corrupted objects.
> >
> > 2021-04-22 10:27:55.445 7f2d85fd4700  0 WARNING: couldn't find acl
> > header for object, generating default
> > 2021-04-22 10:27:55.445 7f2d85fd4700  1 == req done
> > req=0x55a441452710 op status=0 http_status=200 latency=0.022s
> > ==
> > 2021-04-22 10:27:55.445 7f2d85fd4700  1 beast: 0x55a441452710:
> > 10.151.101.15 - - [2021-04-22 10:27:55.0.44549s] "GET
> > /descript/2020/01/17/1b819bd9-5036-4ca4-98f7-b0308e1e3017 HTTP/1.1"
> > 200 0 - "aws-sdk-java/1.11.638 Linux/3.10.0-1160.11.1.el7.x86_64
> > Java_HotSpot(TM)_64-Bit_Server_VM/25.281-b09 java/1.8.0_281
> > groovy/2.5.6 vendor/Oracle_Corporation" -
> >
> > Am I doing something wrong?
> > Also "sync error trim" does not work. How can I clean these errors and
> > these pending objects?
> >
> > ceph version 14.2.16
> >
> >
> > Have a great day.
> > Regards.
> >
> >
> > Matt Benjamin , 22 Nis 2021 Per, 06:08 tarihinde
> > şunu yazdı:
> > >
> > > Hi Morphin,
> > >
> > > Yes, this is by design.  When an RGW object has tail chunks and is
> > > copied so as to duplicate an entire tail chunk, RGW causes the
> > > coincident chunk(s) to be

[ceph-users] Re: [Suspicious newsletter] Re: cephadm: how to create more than 1 rgw per host

[by morphin]

Yes. I tried and used and had no problem.
On 5 node i have 2x rgw and sometimes for debuging I add new rgw and delete
after the test. Im using Nautilus and never tried on pacific but it should
work same way. Try on test Env first.


23 Nis 2021 Cum 04:51 tarihinde Szabo, Istvan (Agoda) <
istvan.sz...@agoda.com> şunu yazdı:

> Have you ever tried this? Did it work for you?
>
> Istvan Szabo
> Senior Infrastructure Engineer
> ---
> Agoda Services Co., Ltd.
> e: istvan.sz...@agoda.com
> ---
>
> On 2021. Apr 22., at 18:30, by morphin  wrote:
>
> Hello.
>
> Its easy. In ceph.conf copy the rgw fields and change 3 things.
> 1- name
> 2- log path name
> 3- client port.
>
>
> After that feel free to start rgw service with systemctl. Check service
> status and Tail the rgw log file. Try to read or write and check the logs.
> If everything works as expected then you are ready to add the new service
> to loadbalancer if you have one.
>
>
>
> 22 Nis 2021 Per 14:00 tarihinde i...@z1storage.com 
> şunu yazdı:
>
> Does anyone know how to create more than 1 rgw per host? Surely it's not
>
> a rare configuration.
>
>
> On 2021/04/19 17:09, i...@z1storage.com wrote:
>
>
> Hi Sebastian,
>
>
> Thank you. Is there a way to create more than 1 rgw per host until
>
> this new feature is released?
>
>
> On 2021/04/19 11:39, Sebastian Wagner wrote:
>
> Hi Ivan,
>
>
> this is a feature that is not yet released in Pacific. It seems the
>
> documentation is a bit ahead of time right now.
>
>
> Sebastian
>
>
> On Fri, Apr 16, 2021 at 10:58 PM i...@z1storage.com
>
> <mailto:i...@z1storage.com> 
> <mailto:i...@z1storage.com>> wrote:
>
>
>Hello,
>
>
>According to the documentation, there's count-per-host key to 'ceph
>
>orch', but it does not work for me:
>
>
>:~# ceph orch apply rgw z1 sa-1 --placement='label:rgw
>
>count-per-host:2'
>
>--port=8000 --dry-run
>
>Error EINVAL: Host and label are mutually exclusive
>
>
>Why it says anything about Host if I don't specify any hosts,
>
>just labels?
>
>
>~# ceph orch host ls
>
>HOST  ADDR  LABELS   STATUS
>
>s101  s101  mon rgw
>
>s102  s102  mgr mon rgw
>
>s103  s103  mon rgw
>
>s104  s104  mgr mon rgw
>
>s105  s105  mgr mon rgw
>
>s106  s106  mon rgw
>
>s107  s107  mon rgw
>
>
>___
>
>ceph-users mailing list -- ceph-users@ceph.io
>
><mailto:ceph-users@ceph.io>
>
>To unsubscribe send an email to ceph-users-le...@ceph.io
>
><mailto:ceph-users-le...@ceph.io>
>
>
> ___
>
> ceph-users mailing list -- ceph-users@ceph.io
>
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> --
> This message is confidential and is for the sole use of the intended
> recipient(s). It may also be privileged or otherwise protected by copyright
> or other legal rules. If you have received it by mistake please let us know
> by reply email and delete it from your system. It is prohibited to copy
> this message or disclose its content to anyone. Any confidentiality or
> privilege is not waived or lost by any mistaken delivery or unauthorized
> disclosure of the message. All messages sent to and from Agoda may be
> monitored to ensure compliance with company policies, to protect the
> company's interests and to remove potential malware. Electronic messages
> may be intercepted, amended, lost or deleted, or contain viruses.
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] RBD tuning for virtualization (all flash)

[by morphin]

Hello.

We're running 1000vm on 28 node with 6 ssd (no seperate db device) and
these vms are Mostly win10.

2 lvm osd Per 4tb device total 288osd and one RBD pool with 8192 PG.
Replication 3.

Ceph version : nautilus 14.2.16

I'm looking for all flash RBD tuning.
This is good test env and tomorrow gonna be prod. In vms I see 1800mb/s
read and 900mb/s write (qemu = write back)
With good tuning I believe this cluster will go further.
 Do you have any special suggestion?

Have a nice weekend.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RBD tuning for virtualization (all flash)

[by morphin]

2x10G for cluster + Public
2x10G for Users

lacp = 802.3ad

Smart Weblications GmbH , 26 Nis 2021 Pzt,
17:25 tarihinde şunu yazdı:
>
> Hi,
>
>
> Am 25.04.2021 um 03:58 schrieb by morphin:
> > Hello.
> >
> > We're running 1000vm on 28 node with 6 ssd (no seperate db device) and
> > these vms are Mostly win10.
> >
> > 2 lvm osd Per 4tb device total 288osd and one RBD pool with 8192 PG.
> > Replication 3.
> >
> > Ceph version : nautilus 14.2.16
> >
> > I'm looking for all flash RBD tuning.
> > This is good test env and tomorrow gonna be prod. In vms I see 1800mb/s
> > read and 900mb/s write (qemu = write back)
> > With good tuning I believe this cluster will go further.
> >   Do you have any special suggestion?
> >
>
> What is the network bandwidth?
>
>
> --
>
> Mit freundlichen Grüßen,
>
>
> Smart Weblications GmbH
> Martinsberger Str. 1
> D-95119 Naila
>
> fon.: +49 9282 9638 200
> fax.: +49 9282 9638 205
> 24/7: +49 900 144 000 00 - 0,99 EUR/Min*
> http://www.smart-weblications.de
>
> --
> Sitz der Gesellschaft: Naila
> Geschäftsführer: Florian Wiessner
> HRB-Nr.: HRB 3840 Amtsgericht Hof
> *aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] How can I get tail information a parted rados object

[by morphin]

Hello.

I'm trying to export objects from rados with rados get. Some objects
bigger than 4M and they have tails. Is there any easy way to get tail
information an object?

For example this is an object:
- c106b26b.3_Img/2017/12/im034113.jpg
These are the objet parts:
- 
c106b26b.3__multipart_Img/2017/12/im034113.jpg.2~fjrC5r_KCWMBat_4bFVtmBv9pxcVL-9.1
- 
c106b26b.3__shadow_Img/2017/12/im034113.jpg.2~fjrC5r_KCWMBat_4bFVtmBv9pxcVL-9.1_1
- 
c106b26b.3__shadow_Img/2017/12/im034113.jpg.2~fjrC5r_KCWMBat_4bFVtmBv9pxcVL-9.1_2
- 
c106b26b.3__multipart_Img/2017/12/im034113.jpg.2~fjrC5r_KCWMBat_4bFVtmBv9pxcVL-9.2

As you can see the object has 2 multipart and 2 shadow object.
This jpg only works when I get all the parts and make it one with the order.
order: "cat 9.1 9.1_1 9.1_2 9.2 > im034113.jpg"

I'm trying to write a code and the code gonna read objects from a list
and find all the parts, bring it together with the order...  But I
couldn't find a good way to get part information.

I followed the link https://www.programmersought.com/article/31497869978/
and I get the object manifest with getxattr and decode it with
"ceph-dencoder type RGWBucketEnt  decode dump_json"
But in the manifest I can not find a path to code it. It's not useful.
Is there any different place that I can take the part information an
object?

Or better! Is there any tool to export an object with its tails?

btw: these objects created by RGW using s3. RGW can not access these
files. Because of that I'm trying to export it from rados and send it
to different RGW.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: How can I get tail information a parted rados object

[by morphin]

Hi Rob.

I think I wasn't clear enough with the first mail.
I'm having issues with the RGW. radosgw-admin or s3 can not access
some objects in the bucket. These objects are exist in the "RADOS" and
I can export with "rados get -p $pooll $object".
But the problem ise 4M chunk and multiparts. I have to find all the
parts with the part number order and after export I need to bring all
the parts together.
I've write a code with golang and I took all the partition info from
rados attr "user.rgw.manifest" and some objects only have
"user.rgw.olh.info".
I can now decode these attrs with "ceph-dencoder  RGWOLHInfo  and
RGWObjManifest decode dump_json" and I can find the parts and order.

Now I can export anything in the rados.

I was looking a tool to do that for me but I think its not exist.
Because I couldn't find.
RGW has t many problems and Sharding issues with Multisite
setup! And the worst part is somehow RGW losing object records... It
could be Sharding, Multisite, Versioning or Lifecycle. I don't know
exactly why! But TO rescue these objects in the rados and write them
again with S3 I had to write special program for it.
I know ceph-dencoder for Developers but the documentation is poor for
RGW and I had to read code, understand, and write a new program
because I need a BRIDGE between RGW and the RADOS.

I think I should publish the program for the community. It can
"Directly rados download and upload to RGW or local" It can upload
1.5M object in 30 minute. (small files)
But  the program is designed for one job. It needs edit for public use.
If there is a option or program please let me know.
If there is no option and if it is a good idea to make one for the
community. I'm ready to publish and work on it.


Now I have Multisite sync error and I don't use these buckets anymore.
I changed the master zone and remove the secondary zone from
zonegroup. I dont have multisite anymore.
After that I migrate buckets to new one and I want to trim the sync
errors but "radosgw-admin error trim" not working.
What Should I do? I Really Really help on this! I'm using 14.2.16 nautilus.


Do you know anything about deleting old periods? What will the effect
on a cluster?



Rob Haverkamp , 3 May 2021 Pzt, 15:15 tarihinde
şunu yazdı:
>
> Hi Morphin,
>
> There are multiple ways you can do this.
>
> run a radosgw-admin bucket radoslist --bucket  write that 
> output to a file, grep all entries containing the object name ' 
> im034113.jpg', sort that list and download them.
> run a radosgw-admin object stat --bucket  --object  
> this will output a json document. With the information in the manifest key 
> you can find out what rados objects belong to the RGW object.
>
>
>
> Kind regards,
>
> Rob
> https://www.42on.com/
>
>
> 
> From: by morphin 
> Sent: Saturday, May 1, 2021 11:09 PM
> To: Ceph Users 
> Subject: [ceph-users] How can I get tail information a parted rados object
>
> Hello.
>
> I'm trying to export objects from rados with rados get. Some objects
> bigger than 4M and they have tails. Is there any easy way to get tail
> information an object?
>
> For example this is an object:
> - c106b26b.3_Img/2017/12/im034113.jpg
> These are the objet parts:
> - 
> c106b26b.3__multipart_Img/2017/12/im034113.jpg.2~fjrC5r_KCWMBat_4bFVtmBv9pxcVL-9.1
> - 
> c106b26b.3__shadow_Img/2017/12/im034113.jpg.2~fjrC5r_KCWMBat_4bFVtmBv9pxcVL-9.1_1
> - 
> c106b26b.3__shadow_Img/2017/12/im034113.jpg.2~fjrC5r_KCWMBat_4bFVtmBv9pxcVL-9.1_2
> - 
> c106b26b.3__multipart_Img/2017/12/im034113.jpg.2~fjrC5r_KCWMBat_4bFVtmBv9pxcVL-9.2
>
> As you can see the object has 2 multipart and 2 shadow object.
> This jpg only works when I get all the parts and make it one with the order.
> order: "cat 9.1 9.1_1 9.1_2 9.2 > im034113.jpg"
>
> I'm trying to write a code and the code gonna read objects from a list
> and find all the parts, bring it together with the order...  But I
> couldn't find a good way to get part information.
>
> I followed the link https://www.programmersought.com/article/31497869978/
> and I get the object manifest with getxattr and decode it with
> "ceph-dencoder type RGWBucketEnt  decode dump_json"
> But in the manifest I can not find a path to code it. It's not useful.
> Is there any different place that I can take the part information an
> object?
>
> Or better! Is there any tool to export an object with its tails?
>
> btw: these objects created by RGW using s3. RGW can not access these
> files. Because of that I'm trying to export it from rados and send it
> to different RGW.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] How to trim RGW sync errors

[by morphin]

Hello.

I was have multisite RGW (14.2.16 nautilus) setup and some of the
bucket couldn't finish bucket sync due to overfill buckets,
There was different needs and the sync started purpose of migration.
I made the secondary zone the master and removed the old master zone
from zonegroup.
Now I still have sync errors and sync error trim do not work.

radosgw-admin --id radosgw.srv1 sync error list  | grep name | wc -l
32000

Thats a lot of errors. Sync error trim does nothing.

When I run period update commit I saw sync status field has a lot of
records as below.
radosgw-admin --id radosgw.srv1 period update --commit
{
"id": "e5d30f8f",
"epoch": 7,
"predecessor_uuid": "1d0b7132",
"sync_status": [
"1_1611733356.499643_1448979853.1",
"1_1611225916.734727_865381974.1",
"1_1611648125.876993_1659659292.1",
"1_1608194415.061001_737663090.1",
"1_1605880458.143435_1259922694.1",
"1_1611225999.087089_1887995199.1",
"1_1586035175.626619_488028.1",
"",
"",
"1_1611057887.910246_973493243.1",
"1_1612180963.822684_807349060.1",
"",
"",
"1_1612180818.328001_807344892.1",
"1_1611058156.662721_1887884194.1",
"1_1611057588.159455_1887883796.1",
"1_1611647015.874625_1129837262.1",
"1_1586035175.602419_753756.1",
"",
"1_1606215091.912960_988474411.1",
"",
"1_1600418137.932356_1027064325.1",
"1_1609926537.036681_832230841.1",
"",
"",
"1_1611057624.857485_1658280806.1",
"1_1600419671.553723_365405366.1",
"",
"1_1611057662.014628_859134308.1",
"1_1611057665.933662_843443436.1",
"1_1605879154.805811_700811071.1",
"1_1602509494.904964_696294030.1",
"",
"1_1611057618.891024_1150752303.1",
"1_1611440831.055432_1458827253.1",
"1_1611451128.857514_806931659.1",
"",
"1_1611057597.877068_1785564634.1",
"1_1611057860.565465_1785564826.1",
"1_1585821684.950844_61616.1",
"",
"",
"",
"1_1601647994.988107_511440126.1",
"",
"1_1608194424.578834_777512349.1",
"1_1605879126.845904_958578574.1",
"",
"1_1590061636.162223_183644368.1",
"1_1609834839.884870_1076396513.1",
"",
"1_1612430017.546386_612493167.1",
"1_1605879158.230856_1635059634.1",
"",
"1_1612420115.322098_1468865033.1",
"1_1611057731.182423_817020944.1",
"1_1611225026.887795_806142997.1",
"1_1612188490.428048_1152864210.1",
"1_1612187913.914410_861646554.1",
"1_1609393942.952120_574675578.1",
"1_1611733086.223927_861322773.1",
"1_1605880394.928467_759903023.1",
"1_1600418082.175862_556536400.1",
"1_1605879150.320951_1210709666.1"
],
"period_map": {
"id": "e5d30f8f",
"zonegroups": [
{
"id": "667afef",
"name": "xy",
"api_name": "xy",
"is_master": "true",
"endpoints": [
"http://dns:80";
],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "fe8ee939",
"zones": [
{
"id": "fe8ee939",
"name": "prod",
"endpoints": [
 "http://dns:80";
   ],
"log_meta": "false",
"log_data": "false",
"bucket_index_max_shards": 101,
"read_only": "false",
"tier_type": "",
"sync_from_all": "false",
"sync_from": [],
"redirect_zone": ""
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": [],
"storage_classes": [
"STANDARD"
]
}
],
"default_placement": "default-placement",
"realm_id": "234837df"
}
],
"short_zone_ids": [
{
"key": "fe8ee939",
"val": 2970845644
}
]
},
"master_zonegroup": "667afefc",
"master_zone": "fe8ee939",
"period_config": {
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,

[ceph-users] OSD's still UP after power loss

[by morphin]

Hello

I have a weird problem on 3 node cluster. "Nautilus 14.2.9"
When I try power failure OSD's are not marking as DOWN and MDS do not
respond anymore.
If I manually set osd down then MDS becomes active again.

BTW: Only 2 node has OSD's. Third node is only for MON.

I've set mon_osd_down_out_interval = 0.3 in ceph.conf at global
section and restart all MON's but when I check it with "ceph daemon
mon.ID config show" I see mon_osd_down_out_interval: "600".  I didn't
get it why its still "600" and honestly I don't know even it has any
effect on my problem.

Where should I check?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: OSD's still UP after power loss

[by morphin]

I've figured out but I'm scared from the result.
The solution is "mon_osd_min_down_reporters = 1"
Due to "two node" cluster and "replicated 2" with "chooseleaf host"
the reporter count should be set to 1 but on a malfunction this could
be a serious problem.

Is there any better solution?

by morphin , 20 May 2021 Per, 22:04
tarihinde şunu yazdı:
>
> Hello
>
> I have a weird problem on 3 node cluster. "Nautilus 14.2.9"
> When I try power failure OSD's are not marking as DOWN and MDS do not
> respond anymore.
> If I manually set osd down then MDS becomes active again.
>
> BTW: Only 2 node has OSD's. Third node is only for MON.
>
> I've set mon_osd_down_out_interval = 0.3 in ceph.conf at global
> section and restart all MON's but when I check it with "ceph daemon
> mon.ID config show" I see mon_osd_down_out_interval: "600".  I didn't
> get it why its still "600" and honestly I don't know even it has any
> effect on my problem.
>
> Where should I check?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] SSD recommendations for RBD and VM's

[by morphin]

Hello.

I have virtualization env and I'm looking new SSD for HDD replacement.
What are the best Performance / Price SSDs in the market right now?
I'm looking 1TB, 512GB, 480GB, 256GB, 240GB.

Is there a SSD recommendation list for ceph?
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: SSD recommendations for RBD and VM's

[by morphin]

Hello Anthony.

I use Qemu and I don't need size.
I've 1000 vm and usually they're clones from the same rbd image. The
image is 30GB.
Right now I've 7TB Stored data. rep x3  = 20TB data. It's mostly read
intensive. Usage is stable and does not grow.
So I need I/O more than capacity. That's why I'm looking for 256-512GB SSD's.
I think right now 480-512GB is sweet spot for $ / GB. So 60PCS 512GB
will be enough. Actually 120PCS 256GB will be better but the price
goes up.
I have Dell R720-740 and I use SATA Intel DCS3700 for journal. I've
40PCS 100GB.  I'm gonna make them OSD as well.
7 years and DC S3700 still rocks. Not even one of them is dead.
The SSD must be Low price & High TBW life span. Rest is not important.


Anthony D'Atri , 30 May 2021 Paz, 02:26
tarihinde şunu yazdı:
>
> The choice depends on scale, your choice of chassis / form factor, budget, 
> workload and needs.
>
> The sizes you list seem awfully small.  Tell us more about your use-case.  
> OpenStack? Proxmox? QEMU? VMware? Converged? Dedicated ?
> —aad
>
>
> > On May 29, 2021, at 2:10 PM, by morphin  wrote:
> >
> > Hello.
> >
> > I have virtualization env and I'm looking new SSD for HDD replacement.
> > What are the best Performance / Price SSDs in the market right now?
> > I'm looking 1TB, 512GB, 480GB, 256GB, 240GB.
> >
> > Is there a SSD recommendation list for ceph?
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io