On Tuesday 13 March 2018 10:32 PM, Casey Bodley wrote:
On 03/10/2018 12:58 AM, Amardeep Singh wrote:
On Saturday 10 March 2018 02:01 AM, Casey Bodley wrote:
On 03/08/2018 07:16 AM, Amardeep Singh wrote:
Hi,
I am trying to configure server side encryption using Key
Management Service as per documentation
http://docs.ceph.com/docs/master/radosgw/encryption/
Configured Keystone/Barbican integration and its working, tested
using curl commands. After I configure RadosGW and use
boto.s3.connection from python or s3cmd client an error is thrown.
*
*/boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden//
//<?xml version="1.0"
encoding="UTF-8"?><Error><Code>AccessDenied</Code><Message>Failed
to retrieve the actual key, kms-keyid:
616b2ce2-053a-41e3-b51e-0ff53e33cf81</Message><BucketName>newbucket</BucketName><RequestId>tx000000000000000077750-005aa1274b-ac51-uk-west</RequestId><HostId>ac51-uk-west-uk</HostId></Error>//
/
In server side logs its getting the token and barbican is
authenticating the request then providing secret url, but unable to
serve key.
/
////22:10:03.940091 7f056f7eb700 15 ceph_armor ret=16
22:10:03.940111 7f056f7eb700 15
supplied_md5=eb1a3227cdc3fedbaec2fe38bf6c044a
22:10:03.940129 7f056f7eb700 20 reading from
uk-west.rgw.meta:root:.bucket.meta.newbucket:ee560b67-c330-4fd0-af50-aefff93735d2.4163.1
22:10:03.940138 7f056f7eb700 20 get_system_obj_state:
rctx=0x7f056f7e39f0
obj=uk-west.rgw.meta:root:.bucket.meta.newbucket:ee560b67-c330-4fd0-af50-aefff93735d2.4163.1
state=0x56540487a5a0 s->prefetch_data=0
22:10:03.940145 7f056f7eb700 10 cache get:
name=uk-west.rgw.meta+root+.bucket.meta.newbucket:ee560b67-c330-4fd0-af50-aefff93735d2.4163.1
: hit (requested=0x16, cached=0x17)
22:10:03.940152 7f056f7eb700 20 get_system_obj_state: s->obj_tag
was set empty
22:10:03.940155 7f056f7eb700 10 cache get:
name=uk-west.rgw.meta+root+.bucket.meta.newbucket:ee560b67-c330-4fd0-af50-aefff93735d2.4163.1
: hit (requested=0x11, cached=0x17)
22:10:03.944015 7f056f7eb700 20 bucket quota: max_objects=1638400
max_size=-1
22:10:03.944030 7f056f7eb700 20 bucket quota OK:
stats.num_objects=7 stats.size=50
22:10:03.944176 7f056f7eb700 20 Getting KMS encryption key for
key=616b2ce2-053a-41e3-b51e-0ff53e33cf81
22:10:03.944225 7f056f7eb700 20 Requesting secret from barbican
url=http://keyserver.rados:5000/v3/auth/tokens
22:10:03.944281 7f056f7eb700 20 sending request to
http://keyserver.rados:5000/v3/auth/tokens
* 22:10:04.405974 7f056f7eb700 20 sending request to
http://keyserver.rados:9311/v1/secrets/616b2ce2-053a-41e3-b51e-0ff53e33cf81*
* 22:10:05.519874 7f056f7eb700 5 Failed to retrieve secret from
barbican:616b2ce2-053a-41e3-b51e-0ff53e33cf81**
*/
It looks like this request is being rejected by barbican. Do you
have any logs on the barbican side that might show why?
Only get 2 lines in barbican logs, one shows warning.
22:10:08.255 807 WARNING barbican.api.controllers.secrets
[req-091413d2-9999-46e2-be5f-a3e68a480ac9
716dad1b8044459c99fea284dbfc47cc - - default default] Decrypted
secret 616b2ce2-053a-41e3-b51e-0ff53e33cf81 requested using
deprecated API call.
22:10:08.261 807 INFO barbican.api.middleware.context
[req-091413d2-9999-46e2-be5f-a3e68a480ac9
716dad1b8044459c99fea284dbfc47cc - - default default] Processed
request: 200 OK - GET
http://keyserver.rados:9311/v1/secrets/616b2ce2-053a-41e3-b51e-0ff53e33cf81
Okay, so barbican is returning 200 OK but radosgw is still converting
that to EACCES. I'm guessing that's happening in
request_key_from_barbican() here:
https://github.com/ceph/ceph/blob/master/src/rgw/rgw_crypt.cc#L779 -
is it possible the key in barbican is something other than AES256?
Yes - That was the issue. Its sorted now and I am able to encrypt the
documents. Thanks a lot.
Though I have now another issue because I am using Multisite setup with
one zone for data and second zone for metadata with elastic search tier.
http://docs.ceph.com/docs/master/radosgw/elastic-sync-module/
When document is encrypted the metadata is not pushed to elasticsearch
and if document is uploaded without encryption it works fine.
/2018-03-14 15:48:02.397490 7f0b4cbce700 20
cr:s=0x560a726c4000:op=0x560a7276e800:20RGWPutRESTResourceCRI15es_obj_metadataiE:
operate()
2018-03-14 15:48:02.397492 7f0b4cbce700 20
cr:s=0x560a726c4000:op=0x560a7276e800:20RGWPutRESTResourceCRI15es_obj_metadataiE:
operate()
*2018-03-14 15:48:02.397633 7f0b4cbce700 20 sending request to
http://192.168.95.60:9200/newbucket/object/ee560b67-c330-4fd0-af50-aefff93735d2.4163.1:testdocument:null*
2018-03-14 15:48:02.397653 7f0b4cbce700 20 register_request
mgr=0x560a720d5d58 req_data->id=1759, easy_handle=0x560a7348a000
2018-03-14 15:48:02.397666 7f0b4cbce700 20 run: stack=0x560a726c4000 is
io blocked
2018-03-14 15:48:02.397685 7f0b4b3cb700 20 link_request
req_data=0x560a727fae00 req_data->id=1758, easy_handle=0x560a733e6000
2018-03-14 15:48:02.397725 7f0b4b3cb700 20 link_request
req_data=0x560a72f31e00 req_data->id=1759, easy_handle=0x560a7348a000
2018-03-14 15:48:02.398609 7f0b4b3cb700 10 receive_http_header
2018-03-14 15:48:02.398631 7f0b4b3cb700 10 received header:HTTP/1.1 100
Continue
2018-03-14 15:48:02.398636 7f0b4b3cb700 10 received header:HTTP/1.1
2018-03-14 15:48:02.398638 7f0b4b3cb700 10 receive_http_header
2018-03-14 15:48:02.398639 7f0b4b3cb700 10 received header:
2018-03-14 15:48:02.398659 7f0b4b3cb700 10 receive_http_header
2018-03-14 15:48:02.398661 7f0b4b3cb700 10 received header:HTTP/1.1 100
Continue
2018-03-14 15:48:02.398662 7f0b4b3cb700 10 received header:HTTP/1.1
2018-03-14 15:48:02.398663 7f0b4b3cb700 10 receive_http_header
2018-03-14 15:48:02.398664 7f0b4b3cb700 10 received header:
2018-03-14 15:48:02.443530 7f0b4b3cb700 10 receive_http_header
*2018-03-14 15:48:02.443556 7f0b4b3cb700 10 received header:HTTP/1.1 400
Bad Request*
2018-03-14 15:48:02.443563 7f0b4b3cb700 10 receive_http_header
2018-03-14 15:48:02.443565 7f0b4b3cb700 10 received header:Warning: 299
Elasticsearch-5.6.2-57e20f3 "Content type detection for rest requests is
deprecated. Specify the content type using the [Content-Type] header."
"Wed, 14 Mar 2018 10:17:35 GMT"
2018-03-14 15:48:02.443574 7f0b4b3cb700 10 receive_http_header
2018-03-14 15:48:02.443575 7f0b4b3cb700 10 received header:content-type:
application/json; charset=UTF-8
2018-03-14 15:48:02.443588 7f0b4b3cb700 10 receive_http_header
2018-03-14 15:48:02.443591 7f0b4b3cb700 10 received
header:content-length: 374
2018-03-14 15:48:02.443594 7f0b4b3cb700 10 receive_http_header
2018-03-14 15:48:02.443595 7f0b4b3cb700 10 received header:
2018-03-14 15:48:02.443663 7f0b4cbce700 20
cr:s=0x560a732f4d20:op=0x560a72fa8000:20RGWPutRESTResourceCRI15es_obj_metadataiE:
operate()
2018-03-14 15:48:02.443675 7f0b4cbce700 5 failed to wait for op,
ret=-22: PUT
http://192.168.95.60:9200/newbucket/object/ee560b67-c330-4fd0-af50-aefff93735d2.4163.1:testdocument:null
2018-03-14 15:48:02.443754 7f0b4cbce700 20
cr:s=0x560a732f4d20:op=0x560a72fa8000:20RGWPutRESTResourceCRI15es_obj_metadataiE:
operate() returned r=-22
2018-03-14 15:48:02.443773 7f0b4cbce700 20
cr:s=0x560a732f4d20:op=0x560a7276c800:29RGWElasticHandleRemoteObjCBCR:
operate()
2018-03-14 15:48:02.443787 7f0b4cbce700 20
cr:s=0x560a732f4d20:op=0x560a7276c800:29RGWElasticHandleRemoteObjCBCR:
operate() returned r=-22
2018-03-14 15:48:02.443791 7f0b4cbce700 20
cr:s=0x560a732f4d20:op=0x560a72efb800:27RGWElasticHandleRemoteObjCR:
operate()
///Appreciate your help on this.
/*** 22:10:05.519901 7f056f7eb700 5 ERROR: failed to retrieve
actual key from key_id: 616b2ce2-053a-41e3-b51e-0ff53e33cf81*
22:10:05.519980 7f056f7eb700 2 req 387:1.581432:s3:PUT
/encrypted.txt:put_obj:completing
22:10:05.520187 7f056f7eb700 2 req 387:1.581640:s3:PUT
/encrypted.txt:put_obj:op status=-13
22:10:05.520193 7f056f7eb700 2 req 387:1.581645:s3:PUT
/encrypted.txt:put_obj:http status=403
22:10:05.520206 7f056f7eb700 1 ====== req done req=0x7f056f7e5190
op status=-13 http_status=403 ======
22:10:05.520225 7f056f7eb700 20 process_request() returned -13
22:10:05.520280 7f056f7eb700 1 civetweb: 0x5654042a1000:
192.168.100.200 - - [02/Mar/2018:22:10:03 +0530] "PUT
/encrypted.txt HTTP/1.1" 1 0 - Boto/2.38.0 Python/2.7.12
Linux/4.12.1-041201-generic
22:10:06.116527 7f056e7e9700 20 HTTP_ACCEPT=*/*/
The error thrown in from this line
https://github.com/ceph/ceph/blob/master/src/rgw/rgw_crypt.cc#L1063
I am unable to understand why its throwing the error.
In ceph.conf following settings are done.
[global]
rgw barbican url = http://keyserver.rados:9311
rgw keystone barbican user = rgwcrypt
rgw keystone barbican password = rgwpass
rgw keystone barbican project = service
rgw keystone barbican domain = default
rgw keystone url = http://keyserver.rados:5000
rgw keystone api version = 3
rgw crypt require ssl = false
Can someone help in figuring out what is missing.
Thanks,
Amar
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Amardeep Singh
IT Director
Direct: +91 124 4548389
Tel: +91 124 4548383 Ext- 1001
UK: +44 845 0047 142 Ext- 5003
TBS Website <http://www.techbluesoftware.co.in>
Techblue Software Pvt. Ltd
AIHP Tower, 249 G, 2nd Floor,
Udyog Vihar, Phase 4,
Gurugram- 122015 (Hr.)
www.techbluesoftware.co.in <http://www.techbluesoftware.co.in>
TBS Facebook
<https://www.facebook.com/pages/Techblue-Software-Limited/441777369284888>
TBS Twitter <https://twitter.com/TechbluSoftware> TBS Google+
<https://plus.google.com/+TechblueSoftwareCoIn> TBS Linked In
<https://www.linkedin.com/company/techblue-softwares-pvt-ltd>
TBS Branding <http://www.techbluesoftware.co.in>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Amardeep Singh
IT Director
Direct: +91 124 4548389
Tel: +91 124 4548383 Ext- 1001
UK: +44 845 0047 142 Ext- 5003
TBS Website <http://www.techbluesoftware.co.in>
Techblue Software Pvt. Ltd
AIHP Tower, 249 G, 2nd Floor,
Udyog Vihar, Phase 4,
Gurugram- 122015 (Hr.)
www.techbluesoftware.co.in <http://www.techbluesoftware.co.in>
TBS Facebook
<https://www.facebook.com/pages/Techblue-Software-Limited/441777369284888>
TBS Twitter <https://twitter.com/TechbluSoftware> TBS Google+
<https://plus.google.com/+TechblueSoftwareCoIn> TBS Linked In
<https://www.linkedin.com/company/techblue-softwares-pvt-ltd>
TBS Branding <http://www.techbluesoftware.co.in>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
