[ceph-users] s3cmd upload file successed but return This multipart completion is already in progress
Hi, cephers recently when using s3cmd to upload a large file, last POST request which meaned I have finished mutltipart upload.But in fact file upload success. some key points: 1.s3cmd send the POST request.but the server response spend 30s;why some times this POST request need 30s to finish, most of this POST request finished within 100ms - did it mean we face a performance bottleneck? we used ssd to store index pool. > ceph version is 10.2.11 > 42 nodes, each has ten 8T sata and two SSD, >.rgw.buckets.data 61 899T 96.5832644G 423943616 >.rgw.buckets.index 63 0 0 7025G 5472 2.s3cmd retry send POST request which do not wait for the first POST request 3.why the server return 500 when send multi same POST request, how about return 102? the request list is below: ## nginx1 10.200.59.145 - [16/Sep/2019:14:00:05 +0800] "hba3-backup-data0004-f.s3.nie.netease.com" "POST /yaz4nalmsgsam8kfk5doca/backup/_realm_data.tar.gz.0?uploads HTTP/1.1" 200 303 789 "-" "-" "-" 0.191 0.191 10.200.59.145 - [16/Sep/2019:14:00:05 +0800] "hba3-backup-data0004-f.s3.nie.netease.com" "PUT /yaz4nalmsgsam8kfk5doca/backup/_realm_data.tar.gz.0?partNumber=1&uploadId=2~v6s8KbNeMA7RaXS_ntcUPtfP1BmXVr1 HTTP/1.1" 200 0 15729218 "-" "-" "-" 0.511 0.383 ... 10.200.59.145 - [16/Sep/2019:14:01:07 +0800] "hba3-backup-data0004-f.s3.nie.netease.com" "PUT /yaz4nalmsgsam8kfk5doca/backup/_realm_data.tar.gz.0?partNumber=99&uploadId=2~v6s8KbNeMA7RaXS_ntcUPtfP1BmXVr1 HTTP/1.1" 200 0 15729218 "-" "-" "-" 0.511 0.383 ## nginx2 10.200.59.145 - [16/Sep/2019:14:01:09 +0800] "hba3-backup-data0004-f.s3.nie.netease.com" "PUT /yaz4nalmsgsam8kfk5doca/backup/_realm_data.tar.gz.0?partNumber=100&uploadId=2~v6s8KbNeMA7RaXS_ntcUPtfP1BmXVr1 HTTP/1.1" 200 0 15729220 "-" "-" "-" 2.430 2.298 ... 10.200.59.145 - [16/Sep/2019:14:03:43 +0800] "hba3-backup-data0004-f.s3.nie.netease.com" "PUT /yaz4nalmsgsam8kfk5doca/backup/_realm_data.tar.gz.0?partNumber=349&uploadId=2~v6s8KbNeMA7RaXS_ntcUPtfP1BmXVr1 HTTP/1.1" 200 0 1857220 "-" "-" "-" 0.160 0.155 10.200.59.145 - [16/Sep/2019:14:04:13 +0800] "hba3-backup-data0004-f.s3.nie.netease.com" "POST /yaz4nalmsgsam8kfk5doca/backup/_realm_data.tar.gz.0?uploadId=2~v6s8KbNeMA7RaXS_ntcUPtfP1BmXVr1 HTTP/1.1" 200 360 30520 "-" "-" "-" 30.225 30.223 ## nginx3 10.200.59.145 - [16/Sep/2019:14:03:56 +0800] "hba3-backup-data0004-f.s3.nie.netease.com" "POST /yaz4nalmsgsam8kfk5doca/backup/_realm_data.tar.gz.0?uploadId=2~v6s8KbNeMA7RaXS_ntcUPtfP1BmXVr1 HTTP/1.1" 500 291 30520 "-" "-" "-" 0.012 0.011 10.200.59.145 - [16/Sep/2019:14:04:02 +0800] "hba3-backup-data0004-f.s3.nie.netease.com" "POST /yaz4nalmsgsam8kfk5doca/backup/_realm_data.tar.gz.0?uploadId=2~v6s8KbNeMA7RaXS_ntcUPtfP1BmXVr1 HTTP/1.1" 500 291 30520 "-" "-" "-" 0.012 0.012 10.200.59.145 - [16/Sep/2019:14:04:11 +0800] "hba3-backup-data0004-f.s3.nie.netease.com" "POST /yaz4nalmsgsam8kfk5doca/backup/_realm_data.tar.gz.0?uploadId=2~v6s8KbNeMA7RaXS_ntcUPtfP1BmXVr1 HTTP/1.1" 500 291 30520 "-" "-" "-" 0.012 0.012 10.200.59.145 - [16/Sep/2019:14:04:23 +0800] "hba3-backup-data0004-f.s3.nie.netease.com" "POST /yaz4nalmsgsam8kfk5doca/backup/_realm_data.tar.gz.0?uploadId=2~v6s8KbNeMA7RaXS_ntcUPtfP1BmXVr1 HTTP/1.1" 500 291 30520 "-" "-" "-" 0.013 0.013 10.200.59.145 - [16/Sep/2019:14:04:38 +0800] "hba3-backup-data0004-f.s3.nie.netease.com" "POST /yaz4nalmsgsam8kfk5doca/backup/_realm_data.tar.gz.0?uploadId=2~v6s8KbNeMA7RaXS_ntcUPtfP1BmXVr1 HTTP/1.1" 500 291 30520 "-" "-" "-" 0.013 0.013 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Nautilus : ceph dashboard ssl not working
Hi Team, In ceph 14.2.2 , ceph dashboard does not have set-ssl-certificate . We are trying to enable ceph dashboard and while using the ssl certificate and key , it is not working . cn5.chn5au1c1.cdn ~# ceph dashboard set-ssl-certificate -i dashboard.crt no valid command found; 10 closest matches: dashboard set-grafana-update-dashboards dashboard reset-prometheus-api-host dashboard reset-ganesha-clusters-rados-pool-namespace dashboard set-grafana-api-username dashboard get-audit-api-log-payload dashboard get-grafana-api-password dashboard get-grafana-api-username dashboard set-rgw-api-access-key dashboard reset-rgw-api-host dashboard set-prometheus-api-host Error EINVAL: invalid command cn5.chn5au1c1.cdn ~# ceph -v ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable) How to set crt and key in this case. Thanks, Muthu ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] require_min_compat_client vs min_compat_client
On 9/14/19 4:24 AM, Alfred wrote: > Hi ceph users, > > > If I understand correctly the "min_compat_client" option in the OSD map > was replaced in Luminous with "require_min_compat_client". > > After upgrading a cluster to Luminous and setting > set-require-min-compat-client to jewel, the min_compat_client option > still shows as hammer. min_compat_client is the oldest version this cluster would be table to talk to. The other is the setting you enforce on the cluster. Wido > Is that expected and simply a relic or is min_compat_client still having > an effect on anything? > > Your feedback would be very much appreciated. > > Best regards, > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Help understanding EC object reads
Thanks for responding! It's good to hear that the primary OSD has some smarts when dealing with partial reads, and that seems to line up with what I was seeing, i.e. I would have expected drastically worse performance otherwise with our large object sizes and tiny block sizes. I'm am still seeing some performance degradation with the small block sizes, but I guess that is coming from the inefficiencies of lots of small requests (time spent queuing for the PG etc.), rather than anything related to EC. Cheers, Tom > -Original Message- > From: Gregory Farnum > Sent: 09 September 2019 23:25 > To: Byrne, Thomas (STFC,RAL,SC) > Cc: ceph-users > Subject: Re: [ceph-users] Help understanding EC object reads > > On Thu, Aug 29, 2019 at 4:57 AM Thomas Byrne - UKRI STFC > wrote: > > > > Hi all, > > > > I’m investigating an issue with our (non-Ceph) caching layers of our large > > EC > cluster. It seems to be turning users requests for whole objects into lots of > small byte range requests reaching the OSDs, but I’m not sure how > inefficient this behaviour is in reality. > > > > My limited understanding of an EC object partial read is that the entire > object is reconstructed on the primary OSD, and then the requested byte > range is sent to the client before the primary discards the reconstructed > object. > > Ah, it's not necessarily the entire object is reconstructed, but that any > stripes > covering the requested range are reconstructed. It's changed a bit over time > and there are some knobs controlling it, but I believe this is generally > efficient — if you ask for a byte range which simply lives on the primary, > it's > not going to talk to the other OSDs to provide that data. > > > > > Assuming this is correct, do multiple reads for different byte ranges of the > same object at effectively the same time result in the entire object being > reconstructed once for each request, or does the primary do something > clever and use the same reconstructed object for multiple requests before > discarding it? > > I'm pretty sure it's per-request; the EC pool code generally assumes you have > another cache on top of RADOS that deals with combining these requests. > There is a small cache in the OSD but IIRC it's just for keeping stuff > consistent > while writes are in progress. > -Greg > > > > > If I’m completely off the mark with what is going on under the hood here, a > nudge in the right direction would be appreciated! > > > > > > > > Cheers, > > > > Tom > > > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] require_min_compat_client vs min_compat_client
On 16/9/19 18:52, Wido den Hollander wrote: > > On 9/14/19 4:24 AM, Alfred wrote: >> Hi ceph users, >> >> >> If I understand correctly the "min_compat_client" option in the OSD map >> was replaced in Luminous with "require_min_compat_client". >> >> After upgrading a cluster to Luminous and setting >> set-require-min-compat-client to jewel, the min_compat_client option >> still shows as hammer. > > min_compat_client is the oldest version this cluster would be table to > talk to. The other is the setting you enforce on the cluster. Great, thank you very much for clarifying that. Cheers, > > Wido > >> Is that expected and simply a relic or is min_compat_client still having >> an effect on anything? >> >> Your feedback would be very much appreciated. >> >> Best regards, >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] increase pg_num error
So, my pool size has increased to a point where the autoscaler did suggest an increase of pg_num (from 100 to 512). Autoscaler mode is “on”, but no change happens.. ceph osd pool ls detail reports: … pool 10 'rbd1' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 100 pgp_num 100 pg_num_target 512 pgp_num_target 512 autoscale_mode on last_change 3007 flags hashpspool,selfmanaged_snaps,creating stripe_width 0 application rbd … For the pool in question, but no pg resizing activity. I also tried manually, and nothing happens. No errors in the logs either. Anybody has any thoughts? How can I change pg_num (and also pgp_num)? Thanks! George > On Sep 12, 2019, at 7:49 AM, Kyriazis, George > wrote: > > Hi Burkhard, > > I tried using the autoscaler, however it did not give a suggestion to resize > pg_num. Since my pg_num is not a power of 2, I wanted to fix that first, > manually, to only realize that it didn’t work. > > Because changing pg_num manually did not work, I am not convinced that the > autoscaler will work, either, when the time comes. The autoscaler pg_num > changes would follow the same execution path as manual changes, won’t they? > > Thanks! > > George > >> On Sep 12, 2019, at 4:37 AM, Burkhard Linke >> wrote: >> >> Hi, >> >> On 9/12/19 5:16 AM, Kyriazis, George wrote: >>> Ok, after all is settled, I tried changing pg_num again on my pool and it >>> still didn’t work: >>> >>> # ceph osd pool get rbd1 pg_num >>> pg_num: 100 >>> # ceph osd pool set rbd1 pg_num 128 >>> # ceph osd pool get rbd1 pg_num >>> pg_num: 100 >>> # ceph osd require-osd-release nautilus >>> # ceph osd pool set rbd1 pg_num 128 >>> # ceph osd pool get rbd1 pg_num >>> pg_num: 100 >>> # >>> >>> Suggestions, anybody? >> >> >> If you use the autoscaler, changing pg_num will actually change the >> pg_num_target setting. The autoscaler will then take care for gently >> increasing pg_num until it reaches the value of pg_num_target. >> >> >> Regards, >> >> Burkhard >> >> >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
