Hello,I have recreated my multisite lab form zone but I'm still stuck on the same bucket issue. Can someone with more experience tell me if this is valid multisite configuration?
zonegroup configuration
(all zones have configured the same system_key pair)
radosgw-admin zonegroup get --rgw-zonegroup poznan-1
{
"id": "73868eb9-fb8e-4ad8-bf31-37a16d2e6d14",
"name": "poznan-1",
"api_name": "poznan-1",
"is_master": true,
"endpoints": [
"http://10.0.9.120:8000";
],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "62ab5076-83f2-48ba-88f0-080aad9f271f",
"zones": [
{
"id": "02c9655c-0e61-4421-9df2-dd5b954bbe82",
"name": "zone-poznan-1-lodman",
"endpoints": [
"http://10.0.9.121:8000";
],
"log_meta": false,
"log_data": true,
"bucket_index_max_shards": 11,
"read_only": false,
"tier_type": "",
"sync_from_all": true,
"sync_from": [],
"redirect_zone": "",
"supported_features": [
"compress-encrypted",
"notification_v2",
"resharding"
]
},
{
"id": "62ab5076-83f2-48ba-88f0-080aad9f271f",
"name": "zone-poznan-1-pcss-bst",
"endpoints": [
"http://10.0.9.120:8000";
],
"log_meta": false,
"log_data": true,
"bucket_index_max_shards": 11,
"read_only": false,
"tier_type": "",
"sync_from_all": true,
"sync_from": [],
"redirect_zone": "",
"supported_features": [
"compress-encrypted",
"notification_v2",
"resharding"
]
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": [],
"storage_classes": [
"STANDARD"
]
}
],
"default_placement": "default-placement",
"realm_id": "4fa02715-aa1d-4163-8d7f-d4f52996d372",
"sync_policy": {
"groups": []
},
"enabled_features": [
"notification_v2",
"resharding"
]
}
gh-test-ceph-a ~ # radosgw-admin zonegroup get --rgw-zonegroup lodz-1
{
"id": "259ffb31-643b-4ae3-b248-1327f5ff0887",
"name": "lodz-1",
"api_name": "lodz-1",
"is_master": false,
"endpoints": [
"http://10.0.9.121:8001";
],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "3b79f8f4-869f-420f-b31a-b24ad0c72792",
"zones": [
{
"id": "3b79f8f4-869f-420f-b31a-b24ad0c72792",
"name": "zone-lodz-1-lodman",
"endpoints": [
"http://10.0.9.121:8001";
],
"log_meta": false,
"log_data": true,
"bucket_index_max_shards": 11,
"read_only": false,
"tier_type": "",
"sync_from_all": true,
"sync_from": [],
"redirect_zone": "",
"supported_features": [
"compress-encrypted",
"notification_v2",
"resharding"
]
},
{
"id": "7e5ca0ec-cb1d-4aaa-83ee-5af2061f5348",
"name": "zone-lodz-1-pcss-bst",
"endpoints": [
"http://10.0.9.120:8001";
],
"log_meta": false,
"log_data": true,
"bucket_index_max_shards": 11,
"read_only": false,
"tier_type": "",
"sync_from_all": true,
"sync_from": [],
"redirect_zone": "",
"supported_features": [
"compress-encrypted",
"notification_v2",
"resharding"
]
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": [],
"storage_classes": [
"STANDARD"
]
}
],
"default_placement": "default-placement",
"realm_id": "4fa02715-aa1d-4163-8d7f-d4f52996d372",
"sync_policy": {
"groups": []
},
"enabled_features": [
"notification_v2",
"resharding"
]
}
Creating bucket on zone-lodz-1-lodman zone causes 503 errors
s3cmd --host 10.0.9.121:8001 mb s3://lodz-1-test --region lodz-1
WARNING: Retrying failed request: / (503 (ServiceUnavailable))
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: / (503 (ServiceUnavailable))
WARNING: Waiting 6 sec...
WARNING: Retrying failed request: / (503 (ServiceUnavailable))
I see that zone zone-lodz-1-lodman is making some requests to master
zonezgroup but they are failing.
2025-06-05T11:04:55.196+0000 7febf0cac6c0 4 req 793098438972080696 0.000000000s ERROR: empty payload checksum mismatch, expected e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 got 477cc47e2b1886f364c3e7e343ce437e29a1c450895c96e7e4ea13768fd48244 2025-06-05T11:04:55.196+0000 7febf0cac6c0 20 req 793098438972080696 0.000000000s s3:create_bucket rgw::auth::s3::LocalEngine denied with reason=-2040 2025-06-05T11:04:55.196+0000 7febf0cac6c0 20 req 793098438972080696 0.000000000s s3:create_bucket rgw::auth::s3::AWSAuthStrategy denied with reason=-2040 2025-06-05T11:04:55.196+0000 7febf0cac6c0 5 req 793098438972080696 0.000000000s s3:create_bucket Failed the auth strategy, reason=-2040
2025-06-05T11:04:55.196+0000 7febf0cac6c0 10 failed to authorize request2025-06-05T11:04:55.196+0000 7febf0cac6c0 20 req 793098438972080696 0.000000000s op->ERRORHANDLER: err_no=-2040 new_err_no=-2040 2025-06-05T11:04:55.196+0000 7febf0cac6c0 10 req 793098438972080696 0.000000000s cache get: name=zone-poznan-1-pcss-bst.rgw.log++script.postrequest. : hit (negative entry) 2025-06-05T11:04:55.196+0000 7febf0cac6c0 2 req 793098438972080696 0.000000000s s3:create_bucket op status=0 2025-06-05T11:04:55.196+0000 7febf0cac6c0 2 req 793098438972080696 0.000000000s s3:create_bucket http status=400 2025-06-05T11:04:55.196+0000 7febf0cac6c0 1 ====== req done req=0x7feb8be483c0 op status=0 http_status=400 latency=0.000000000s ====== 2025-06-05T11:04:55.196+0000 7febf0cac6c0 1 beast: 0x7feb8be483c0: 10.0.9.121 - - [05/Jun/2025:11:04:55.196 +0000] "PUT /lodz-1-test/?rgwx-uid=test&rgwx-zonegroup=73868eb9-fb8e-4ad8-bf31-37a16d2e6d14 HTTP/1.1" 400 250 - - - latency=0.000000000s
2025-06-05T11:04:55.236+0000 7fecb56356c0 20 HTTP_ACCEPT=*/*2025-06-05T11:04:55.236+0000 7fecb56356c0 20 HTTP_AUTHORIZATION=AWS4-HMAC-SHA256 Credential=TWN7U7S3WCTGZ9PZSPJS/20250605/poznan-1/s3/aws4_request,SignedHeaders=date;host;x-amz-content-sha256;x-amz-date,Signature=63a881ab9f9b73ec1c06f722e09dee61bb7c5a1ab2223231f57dbad36537810f 2025-06-05T11:04:55.236+0000 7fecb56356c0 20 HTTP_DATE=Thu, 05 Jun 2025 11:04:55 +0000
2025-06-05T11:04:55.236+0000 7fecb56356c0 20 HTTP_HOST=10.0.9.120:8000 2025-06-05T11:04:55.236+0000 7fecb56356c0 20 HTTP_VERSION=1.1Creating bucket on zone-poznan-1-pcss-bst zone with lodz-1 region results in bucket in poznan-1 region. gh-test-ceph-a ~ # s3cmd --host 10.0.9.120:8000 mb s3://lodz-1-test --region lodz-1
Bucket 's3://lodz-1-test/' createdgh-test-ceph-a ~ # s3cmd --host 10.0.9.120:8000 info s3://lodz-1-test --region lodz-1
s3://lodz-1-test/ (bucket): Location: poznan-1 Payer: BucketOwner Ownership: none Versioning:none Expiration rule: none Block Public Access: none Policy: none CORS: none ACL: test: FULL_CONTROLIn that second scenario I see that rgw sees my lodz-1 region is specified in LocationConstraint but rgw ignores it and creates zone in wrong zonegroup.
2025-06-05T11:01:53.378+0000 7fec8cde46c0 20 req 3494031126505109504 0.000000000s s3:create_bucket create bucket input data=<CreateBucketConfiguration><LocationConstraint>lodz-1</LocationConstraint></CreateBucketConfiguration> 2025-06-05T11:01:53.378+0000 7fec8cde46c0 10 req 3494031126505109504 0.000000000s s3:create_bucket create bucket location constraint: lodz-1 2025-06-05T11:01:53.378+0000 7fec8cde46c0 20 req 3494031126505109504 0.000000000s s3:create_bucket using zonegroup-default placement target default-placement 2025-06-05T11:01:53.378+0000 7fec8cde46c0 10 req 3494031126505109504 0.000000000s s3:create_bucket cache get: name=zone-poznan-1-pcss-bst.rgw.meta+root+lodz-1-test : hit (negative entry) 2025-06-05T11:01:53.378+0000 7fec8cde46c0 10 req 3494031126505109504 0.000000000s s3:create_bucket user=test bucket=:lodz-1-test[]) 2025-06-05T11:01:53.395+0000 7fec8f5e96c0 10 req 3494031126505109504 0.016666736s s3:create_bucket cache put: name=zone-poznan-1-pcss-bst.rgw.meta+root+.bucket.meta.lodz-1-test:62ab5076-83f2-48ba-88f0-080aad9f271f.16467.3 info.flags=0x17 2025-06-05T11:01:53.395+0000 7fec8f5e96c0 10 req 3494031126505109504 0.016666736s s3:create_bucket adding zone-poznan-1-pcss-bst.rgw.meta+root+.bucket.meta.lodz-1-test:62ab5076-83f2-48ba-88f0-080aad9f271f.16467.3 to cache LRU end 2025-06-05T11:01:53.395+0000 7fec8f5e96c0 10 req 3494031126505109504 0.016666736s s3:create_bucket updating xattr: name=user.rgw.acl bl.length()=129
Am I encountering a bug or is me configuration faulty? Best regards Adam Prycki W dniu 3.06.2025 o 18:55, Adam Prycki pisze:
Hello, I have a question about ceph multisite theory. How to create buckets in non-master zonegroup? I'm trying to create crate a test enviroment with 2 regions. I've crated deployment which looks like this REALM region-1 (master zonegroup) region-1-zone-a (master zone) region-1-zone-b region-2 region-2-zone-a (master zone)The goal would be to have 2 regions, each with separate data and 2 active-active zones in each. Just after creating region-2-zone-a I've decided to create a bucket there in region-2 and upload something. But I cannot create a bucket in new region.Rgw allows you to create buckets only on master zone.If I try to use s3cmd to create bucket on region-1-zone-b endpoint I get error 503. (as expected)Trying to list region-1 bucket on region-2 zone gives me 301 (moved permanently) so region-2 has metadata and working authentication.But I cannot figure out how to make a bucket in region-2.s3cmd .s3cfg-region-2 mb s3://bucket gives me error 503 like if I were trying to create a bucket on non-master zone.Creating bucket on region-1 endpoint with s3cmd flags specifying LocationConstraint region-2 creates bucket in region-1 I see in region-1-zone-a logs that rgw sees region-2 in LocationConstraint argument but creates bucket in region-1I'm working on ceph 19.2.2 Best regards Adam Prycki _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
smime.p7s
Description: Kryptograficzna sygnatura S/MIME
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
