[ceph-users] Swift and Ceph

[alistair.whittle]

All,

I was hoping for some advice.   I have recently built a Ceph cluster on RHEL 
6.5 and have configured RGW.  I want to test Swift API access, and as a 
result have created a user, swift subuser and swift keys as per the output 
below:


1.   Create user


radosgw-admin user create --uid="testuser1" --display-name="Test User1"
{ "user_id": "testuser1",
  "display_name": "Test User1",
  "email": "",
  "suspended": 0,
  "max_buckets": 1000,
  "auid": 0,
  "subusers": [],
  "keys": [
{ "user": "testuser1",
  "access_key": "MJBEZLJ7BYG8XODXT71V",
  "secret_key": "tGnsm8JeEgPGAy1MGCKSVVoSIEs8iWNUOgiJ981p"}],
  "swift_keys": [],
  "caps": [],
  "op_mask": "read, write, delete",
  "default_placement": "",
  "placement_tags": [],
  "bucket_quota": { "enabled": false,
  "max_size_kb": -1,
  "max_objects": -1},
  "user_quota": { "enabled": false,
  "max_size_kb": -1,
  "max_objects": -1},
  "temp_url_keys": []}


2.   Create subuser.

radosgw-admin subuser create --uid=testuser1 --subuser=testuser1:swift 
--access=full
{ "user_id": "testuser1",
  "display_name": "Test User1",
  "email": "",
  "suspended": 0,
  "max_buckets": 1000,
  "auid": 0,
  "subusers": [
{ "id": "testuser1:swift",
  "permissions": "full-control"}],
  "keys": [
{ "user": "testuser1:swift",
  "access_key": "HX9Q30EJWCZG825AT7B0",
  "secret_key": ""},
{ "user": "testuser1",
  "access_key": "MJBEZLJ7BYG8XODXT71V",
  "secret_key": "tGnsm8JeEgPGAy1MGCKSVVoSIEs8iWNUOgiJ981p"}],
  "swift_keys": [],
  "caps": [],
  "op_mask": "read, write, delete",
  "default_placement": "",
  "placement_tags": [],
  "bucket_quota": { "enabled": false,
  "max_size_kb": -1,
  "max_objects": -1},
  "user_quota": { "enabled": false,
  "max_size_kb": -1,
  "max_objects": -1},
  "temp_url_keys": []}


3.   Create key

radosgw-admin key create --subuser=testuser1:swift --key-type=swift --gen-secret
{ "user_id": "testuser1",
  "display_name": "Test User1",
  "email": "",
  "suspended": 0,
  "max_buckets": 1000,
  "auid": 0,
  "subusers": [
{ "id": "testuser1:swift",
  "permissions": "full-control"}],
  "keys": [
{ "user": "testuser1:swift",
  "access_key": "HX9Q30EJWCZG825AT7B0",
  "secret_key": ""},
{ "user": "testuser1",
  "access_key": "MJBEZLJ7BYG8XODXT71V",
  "secret_key": "tGnsm8JeEgPGAy1MGCKSVVoSIEs8iWNUOgiJ981p"}],
  "swift_keys": [
{ "user": "testuser1:swift",
  "secret_key": "KpQCfPLstJhSMsR9qUzY9WfA1ebO4x7VRXkr1KSf"}],
  "caps": [],
  "op_mask": "read, write, delete",
  "default_placement": "",
  "placement_tags": [],
  "bucket_quota": { "enabled": false,
  "max_size_kb": -1,
  "max_objects": -1},
  "user_quota": { "enabled": false,
  "max_size_kb": -1,
  "max_objects": -1},
  "temp_url_keys": []}

When I try and do anything using the credentials above, I get "Account not 
found" errors as per the example below:

swift -A https:///auth/1.0 -U testuser1:swift -K 
"KpQCfPLstJhSMsR9qUzY9WfA1ebO4x7VRXkr1KSf" list

That's the first thing.

Secondly, when I follow the process above to create a second user "testuser2", 
the user and subuser is created, however, when I try and generate a swift key 
for it, I get the following error:

radosgw-admin key create --subuser=testuser2:swift --key-type=swift --gen-secret
could not create key: unable to add access key, unable to store user info
2015-04-23 15:42:38.897090 7f38e157d820  0 WARNING: can't store user info, 
swift id () already mapped to another user (testuser2)

This suggests there is something wrong with the users or the configuration of 
the gateway somewhere.   Can someone provide some advice on what might be 
wrong, or where I can look to find out.   I have gone through whatever log 
files I can and don't see anything of any use at the moment.

Any help appreciated.

Thanks

Alistair

___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Swift and Ceph

[alistair.whittle]

Can you explain this a bit more?   You mean try and create a second subuser for 
testuser1 or testuser2?

As an aside, I am running Ceph 0.80.7 as is packaged with ICE 1.2.2.  I believe 
that is the Firefly release.


-Original Message-
From: Yehuda Sadeh-Weinraub [mailto:yeh...@redhat.com] 
Sent: Thursday, April 23, 2015 6:18 PM
To: Whittle, Alistair: Investment Bank (LDN)
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Swift and Ceph

Sounds like you're hitting a known issue that was fixed a while back (although 
might not be fixed on the specific version you're running). Can you try 
creating a second subuser for the same user, see if that one works?

Yehuda

- Original Message -
> From: "alistair whittle" 
> To: ceph-users@lists.ceph.com
> Sent: Thursday, April 23, 2015 8:38:44 AM
> Subject: [ceph-users] Swift and Ceph
> 
> 
> 
> All,
> 
> 
> 
> I was hoping for some advice. I have recently built a Ceph cluster on 
> RHEL
> 6.5 and have configured RGW. I want to test Swift API access, and as a 
> result have created a user, swift subuser and swift keys as per the 
> output
> below:
> 
> 
> 
> 1. Create user
> 
> 
> 
> radosgw-admin user create --uid="testuser1" --display-name="Test User1"
> 
> { "user_id": "testuser1",
> 
> "display_name": "Test User1",
> 
> "email": "",
> 
> "suspended": 0,
> 
> "max_buckets": 1000,
> 
> "auid": 0,
> 
> "subusers": [],
> 
> "keys": [
> 
> { "user": "testuser1",
> 
> "access_key": "MJBEZLJ7BYG8XODXT71V",
> 
> "secret_key": "tGnsm8JeEgPGAy1MGCKSVVoSIEs8iWNUOgiJ981p"}],
> 
> "swift_keys": [],
> 
> "caps": [],
> 
> "op_mask": "read, write, delete",
> 
> "default_placement": "",
> 
> "placement_tags": [],
> 
> "bucket_quota": { "enabled": false,
> 
> "max_size_kb": -1,
> 
> "max_objects": -1},
> 
> "user_quota": { "enabled": false,
> 
> "max_size_kb": -1,
> 
> "max_objects": -1},
> 
> "temp_url_keys": []}
> 
> 
> 
> 2. Create subuser.
> 
> 
> 
> radosgw-admin subuser create --uid=testuser1 --subuser=testuser1:swift 
> --access=full
> 
> { "user_id": "testuser1",
> 
> "display_name": "Test User1",
> 
> "email": "",
> 
> "suspended": 0,
> 
> "max_buckets": 1000,
> 
> "auid": 0,
> 
> "subusers": [
> 
> { "id": "testuser1:swift",
> 
> "permissions": "full-control"}],
> 
> "keys": [
> 
> { "user": "testuser1:swift",
> 
> "access_key": "HX9Q30EJWCZG825AT7B0",
> 
> "secret_key": ""},
> 
> { "user": "testuser1",
> 
> "access_key": "MJBEZLJ7BYG8XODXT71V",
> 
> "secret_key": "tGnsm8JeEgPGAy1MGCKSVVoSIEs8iWNUOgiJ981p"}],
> 
> "swift_keys": [],
> 
> "caps": [],
> 
> "op_mask": "read, write, delete",
> 
> "default_placement": "",
> 
> "placement_tags": [],
> 
> "bucket_quota": { "enabled": false,
> 
> "max_size_kb": -1,
> 
> "max_objects": -1},
> 
> "user_quota": { "enabled": false,
> 
> "max_size_kb": -1,
> 
> "max_objects": -1},
> 
> "temp_url_keys": []}
> 
> 
> 
> 3. Create key
> 
> 
> 
> radosgw-admin key create --subuser=testuser1:swift --key-type=swift 
> --gen-secret
> 
> { "user_id": "testuser1",
> 
> "display_name": "Test User1",
> 
> "email": "",
> 
> "suspended": 0,
> 
> "max_buckets": 1000,
> 
> "auid": 0,
> 
> "subusers": [
> 
> { "id": "testuser1:swift",
> 
> "permissions": "full-control"}],
> 
> "keys": [
> 
> { "user": "testuser1:swift",
> 
> "access_key": "HX9Q30EJWCZG825AT7B0",
> 
> "secret_key": ""},
> 
> { "user": "testuser1",
> 
> "access_key": "MJBEZLJ7BYG8XODXT71V",
> 
> "secret_key": "tGnsm8JeEgPGAy1MGCKSVVoSIEs8iWNUOgiJ981p"}],
> 
> "swift_keys": [
> 
> { "user": "testuser1:swift",
> 
> "secret_key": "KpQCfPLstJhSMsR9qUzY9WfA1ebO4x7VRXkr1KSf"}],
> 
> "caps": [],
> 
> "op_mask": "read, write, delete",
> 
> "default_placement": "",
> 
> "placement_tags": [],
> 
> "bucket_quota": { "enabled": false,
> 
> "max_size_kb": -1,
> 
> "max_objects": -1},
> 
> "user_quota": { "enabled": false,
> 
> "max_size_kb": -1,
> 
> "max_objects": -1},
> 
> "temp_url_keys": []}
> 
> 
> 
> When I try and do anything using the credentials above, I get “Account 
> not found” errors as per the example below:
> 
> 
> 
> swift -A https:///auth/1.0 -U testuser1:swift -K 
> "KpQCfPLstJhSMsR9qUzY9WfA1ebO4x7VRXkr1KSf" list
> 
> 
> 
> That’s the first thing.
> 
> 
> 
> Secondly, when I follow the process above to create a second user 
> “testuser2”, the user and subuser is created, however, when I try and 
> generate a swift key for it, I get the following error:
> 
> 
> 
> radosgw-admin key create --subuser=testuser2:swift --key-type=swift 
> --gen-secret
> 
> could not create key: unable to add access key, unable to store user 
> info
> 
> 2015-04-23 15:42:38.897090 7f38e157d820 0 WARNING: can't store user 
> info, swift id () already mapped to another user (testuser2)
> 
> 
> 
> This suggests there is something wrong with the users or the 
> configuration of the gateway somewhere. Can someone provide some 
> advice on what might be wrong, or where I can look to find out. I have 
> gone through whatever log files I can and don’t see anything of any use at 
> the momen

[ceph-users] radosgw default.conf

[alistair.whittle]

I have had been trying to configure my radosgw-agent on RHEL 6.5, but after a 
recent reboot of the gateway node discovered that the file needed by the 
/etc/init.d/radosgw-agent script has disappeared 
(/etc/ceph/radosgw-agent/default.conf).   As a result, I can no longer start up 
the radosgw.

Has anybody seen this happen before?   Am I missing something?

Thanks

___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] FW: Install question

[alistair.whittle]

Hi,

I am trying to install Ceph on a Red Hat Linux server that does not have 
external access through which it can access the URL's and download the files 
needed.   The documentation is not clear (to me) on how to install the software 
under these circumstances.

Should I be downloading the source files and creating my own installable 
packages, or is there another easier way to do it?

Thanks
Alistair

___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] FW: Install question

[alistair.whittle]

Thanks.

I sadly have to do this manually, and have been going through the dependencies. 
 There are a LOT to work through, especially it seems around python.

I am getting the following error when trying to install one of the dependencies 
(python-babel):
error: Failed dependencies:
python(abi) = 2.4 is needed by python-babel-0.9.5-2.el5.noarch

I have done a lot of Googling around this and can’t find an rpm for this.   I 
also see a lot of related posts which suggest that this is more of a yum issue.

Has anyone seen this before?   Any ideas?

From: Abhay Sachan [mailto:abhay...@gmail.com]
Sent: Monday, October 07, 2013 7:10 PM
To: Whittle, Alistair: Investment Bank (LDN)
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] FW: Install question

Hi Alistair,
You can download the dumpling release rpms from this location: 
http://ceph.com/rpm-dumpling/rhel6/x86_64/
And cuttlefish from here: http://ceph.com/rpm-cuttlefish/rhel6/x86_64/

You can download ceph-deploy from here: 
http://ceph.com/rpm-dumpling/rhel6/noarch/
But from my personal experience, cuttlefish didn't work with RHEL 6.4, at least 
for me.
For all other dependencies(there are many :|), you need to download them 
manually(i used pkgs.org) and install them.
Let me know if you face some other problems in installation.

Thanks,
Abhay

On Mon, Oct 7, 2013 at 9:35 PM, 
mailto:alistair.whit...@barclays.com>> wrote:

Hi,

I am trying to install Ceph on a Red Hat Linux server that does not have 
external access through which it can access the URL’s and download the files 
needed.   The documentation is not clear (to me) on how to install the software 
under these circumstances.

Should I be downloading the source files and creating my own installable 
packages, or is there another easier way to do it?

Thanks
Alistair

___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer
 regarding market commentary from Barclays Sales and/or Trading, who are active 
market participants; and in respect of Barclays Research, including disclosures 
relating to specific issuers, please see http://publicresearch.barclays.com.

___

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] FW: Install question

[alistair.whittle]

Doh!

I’ve got python packages coming out of my ears.  Clearly downloaded the wrong 
one there.

That did the job.   Thanks.

From: Abhay Sachan [mailto:abhay...@gmail.com]
Sent: Tuesday, October 08, 2013 3:36 PM
To: Whittle, Alistair: Investment Bank (LDN)
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] FW: Install question

You need to install RHEL6 package not EL5, you can get it from here 
http://mirror.centos.org/centos/6/os/i386/Packages/python-babel-0.9.4-5.1.el6.noarch.rpm
-Abhay

On Tue, Oct 8, 2013 at 8:02 PM, 
mailto:alistair.whit...@barclays.com>> wrote:
Thanks.

I sadly have to do this manually, and have been going through the dependencies. 
 There are a LOT to work through, especially it seems around python.

I am getting the following error when trying to install one of the dependencies 
(python-babel):
error: Failed dependencies:
python(abi) = 2.4 is needed by python-babel-0.9.5-2.el5.noarch

I have done a lot of Googling around this and can’t find an rpm for this.   I 
also see a lot of related posts which suggest that this is more of a yum issue.

Has anyone seen this before?   Any ideas?

From: Abhay Sachan [mailto:abhay...@gmail.com]
Sent: Monday, October 07, 2013 7:10 PM
To: Whittle, Alistair: Investment Bank (LDN)
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] FW: Install question

Hi Alistair,
You can download the dumpling release rpms from this location: 
http://ceph.com/rpm-dumpling/rhel6/x86_64/
And cuttlefish from here: http://ceph.com/rpm-cuttlefish/rhel6/x86_64/

You can download ceph-deploy from here: 
http://ceph.com/rpm-dumpling/rhel6/noarch/
But from my personal experience, cuttlefish didn't work with RHEL 6.4, at least 
for me.
For all other dependencies(there are many :|), you need to download them 
manually(i used pkgs.org) and install them.
Let me know if you face some other problems in installation.

Thanks,
Abhay

On Mon, Oct 7, 2013 at 9:35 PM, 
mailto:alistair.whit...@barclays.com>> wrote:

Hi,

I am trying to install Ceph on a Red Hat Linux server that does not have 
external access through which it can access the URL’s and download the files 
needed.   The documentation is not clear (to me) on how to install the software 
under these circumstances.

Should I be downloading the source files and creating my own installable 
packages, or is there another easier way to do it?

Thanks
Alistair

___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer
 regarding market commentary from Barclays Sales and/or Trading, who are active 
market participants; and in respect of Barclays Research, including disclosures 
relating to specific issuers, please see http://publicresearch.barclays.com.

___

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer
 regarding market commentary from Barclays Sales and/or Trading, who are active 
market participants; and in respect of Barclays Research, including disclosures 
relating to specific issuers, please see http://publicresearch.barclays.com.

___


___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdiscla

[ceph-users] ceph-deploy install

[alistair.whittle]

Hello,

I have reached the stage on the install where I am running the ceph-deploy 
install command from the admin node to the server node.

I get the following output:

[ceph_deploy.install][DEBUG ] Installing stable version dumpling on cluster 
ceph hosts ldtdsr02se18
[ceph_deploy.install][DEBUG ] Detecting platform for host ldtdsr02se18 ...
[ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection without sudo
c...@ldtdsr02se18.etf.barcapetf.com's password:
[ceph_deploy.install][INFO  ] Distro info: RedHatEnterpriseServer 6.4 Santiago
[ldtdsr02se18][INFO  ] installing ceph on ldtdsr02se18
[ldtdsr02se18][INFO  ] Running command: su -c 'rpm --import 
"https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc";'
[ldtdsr02se18][ERROR ] Traceback (most recent call last):
[ldtdsr02se18][ERROR ]   File 
"/usr/lib/python2.6/site-packages/ceph_deploy/hosts/centos/install.py", line 
23, in install
[ldtdsr02se18][ERROR ]   File 
"/usr/lib/python2.6/site-packages/ceph_deploy/util/decorators.py", line 10, in 
inner
[ldtdsr02se18][ERROR ]   File 
"/usr/lib/python2.6/site-packages/ceph_deploy/util/wrappers.py", line 6, in 
remote_call
[ldtdsr02se18][ERROR ]   File "/usr/lib64/python2.6/subprocess.py", line 502, 
in check_call
[ldtdsr02se18][ERROR ] raise CalledProcessError(retcode, cmd)
[ldtdsr02se18][ERROR ] CalledProcessError: Command '['su -c \'rpm --import 
"https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc"\'']' 
returned non-zero exit status 1
[ldtdsr02se18][ERROR ] standard in must be a tty
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: su -c 'rpm 
--import "https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc";'

Given my hosts do not have access to the outside world, is there a way I can 
change it to look for this file locally?   I have a local copy which I managed 
to get yum to use

I did also run this import outside of this script in the hope that it would 
realise the key was already imported but that did not seem to help.

Any advice would be appreciated...

Alistair

___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph-deploy, sudo and proxies

[alistair.whittle]

I have an interesting problem I was hoping someone could help with.

My Red Hat servers are configured to use proxies to access the internet.   I 
have managed to successfully add the Ceph repo install ceph-deploy on the admin 
node and create the cluster.   All ceph nodes are no password sudo tested and I 
have made sure that the proxy settings are kept when trying an 'rpm' command 
using sudo.  All other preflight checks are completed with ceph being the 
default login user etc.

So, when I run the ceph-deploy install  command from the admin node, 
I get the following error:

ceph@ldtdsr02se17 PROD $ ceph-deploy install ldtdsr02se18
[ceph_deploy.cli][INFO  ] Invoked (1.2.7): /usr/bin/ceph-deploy install 
ldtdsr02se18
[ceph_deploy.install][DEBUG ] Installing stable version dumpling on cluster 
ceph hosts ldtdsr02se18
[ceph_deploy.install][DEBUG ] Detecting platform for host ldtdsr02se18 ...
[ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo
[ceph_deploy.install][INFO  ] Distro info: RedHatEnterpriseServer 6.4 Santiago
[ldtdsr02se18][INFO  ] installing ceph on ldtdsr02se18
[ldtdsr02se18][INFO  ] Running command: su -c 'rpm --import 
"https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc";'
[ldtdsr02se18][ERROR ] Traceback (most recent call last):
[ldtdsr02se18][ERROR ]   File 
"/usr/lib/python2.6/site-packages/ceph_deploy/hosts/centos/install.py", line 
23, in install
[ldtdsr02se18][ERROR ]   File 
"/usr/lib/python2.6/site-packages/ceph_deploy/util/decorators.py", line 10, in 
inner
[ldtdsr02se18][ERROR ]   File 
"/usr/lib/python2.6/site-packages/ceph_deploy/util/wrappers.py", line 6, in 
remote_call
[ldtdsr02se18][ERROR ]   File "/usr/lib64/python2.6/subprocess.py", line 502, 
in check_call
[ldtdsr02se18][ERROR ] raise CalledProcessError(retcode, cmd)
[ldtdsr02se18][ERROR ] CalledProcessError: Command '['su -c \'rpm --import 
"https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc"\'']' 
returned non-zero exit status 1
[ldtdsr02se18][ERROR ] curl: (7) couldn't connect to host
[ldtdsr02se18][ERROR ] error: 
https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc: import read 
failed(2).
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: su -c 'rpm 
--import "https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc";'

Note that it uses sudo as it should and then complains about not being able to 
connect.   When I run the exact same command on the ceph node itself as the 
"ceph" user, it works without any errors.   This implies that the 
authentication is in place between ceph and root, and the proxy settings are 
correct.   Yet, it fails to work when initiated from the admin node via 
ceph-deploy.

Any ideas what might be going on here?   I should add that I looked at the 
github page about using the "-no-adjust-repos" flag but my version of 
ceph-deploy says it is an invalid flag...

Please help
Alistair


___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph-deploy, sudo and proxies

[alistair.whittle]

Thanks, I already had the correct ceph-deply version, but had the flag in the 
wrong place.   
Solving that got me to the next problem...   I get the following error:

[ceph_deploy.cli][INFO  ] Invoked (1.2.7): /usr/bin/ceph-deploy install 
ldtdsr02se18 --no-adjust-repos
[ceph_deploy.install][DEBUG ] Installing stable version dumpling on cluster 
ceph hosts ldtdsr02se18
[ceph_deploy.install][DEBUG ] Detecting platform for host ldtdsr02se18 ...
[ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo
[ceph_deploy.install][INFO  ] Distro info: RedHatEnterpriseServer 6.4 Santiago
[ldtdsr02se18][INFO  ] installing ceph on ldtdsr02se18
[ldtdsr02se18][INFO  ] Running command: yum -y -q install ceph
[ldtdsr02se18][ERROR ] Traceback (most recent call last):
[ldtdsr02se18][ERROR ]   File 
"/usr/lib/python2.6/site-packages/ceph_deploy/hosts/centos/install.py", line 
57, in install
[ldtdsr02se18][ERROR ]   File 
"/usr/lib/python2.6/site-packages/ceph_deploy/util/decorators.py", line 10, in 
inner
[ldtdsr02se18][ERROR ]   File 
"/usr/lib/python2.6/site-packages/ceph_deploy/util/wrappers.py", line 6, in 
remote_call
[ldtdsr02se18][ERROR ]   File "/usr/lib64/python2.6/subprocess.py", line 502, 
in check_call
[ldtdsr02se18][ERROR ] raise CalledProcessError(retcode, cmd)
[ldtdsr02se18][ERROR ] CalledProcessError: Command '['yum', '-y', '-q', 
'install', 'ceph']' returned non-zero exit status 1
[ldtdsr02se18][ERROR ] Error: Nothing to do
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: yum -y -q 
install ceph

My ceph repo is pointing to http://ceph.com/rpm-dumpling/rhel6/noarch as per 
the documentation.  Is this correct, or have I missed something else?


-Original Message-
From: Alfredo Deza [mailto:alfredo.d...@inktank.com] 
Sent: Friday, October 25, 2013 6:53 PM
To: Whittle, Alistair: Investment Bank (LDN)
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Ceph-deploy, sudo and proxies

On Fri, Oct 25, 2013 at 1:25 PM,   wrote:
> I have an interesting problem I was hoping someone could help with.
>
>
>
> My Red Hat servers are configured to use proxies to access the internet.   I
> have managed to successfully add the Ceph repo install ceph-deploy on the
> admin node and create the cluster.   All ceph nodes are no password sudo
> tested and I have made sure that the proxy settings are kept when 
> trying an 'rpm' command using sudo.  All other preflight checks are 
> completed with ceph being the default login user etc.
>
>
>
> So, when I run the ceph-deploy install  command from the 
> admin node, I get the following error:
>
>
>
> ceph@ldtdsr02se17 PROD $ ceph-deploy install ldtdsr02se18
>
> [ceph_deploy.cli][INFO  ] Invoked (1.2.7): /usr/bin/ceph-deploy 
> install
> ldtdsr02se18
>
> [ceph_deploy.install][DEBUG ] Installing stable version dumpling on 
> cluster ceph hosts ldtdsr02se18
>
> [ceph_deploy.install][DEBUG ] Detecting platform for host ldtdsr02se18 ...
>
> [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with 
> sudo
>
> [ceph_deploy.install][INFO  ] Distro info: RedHatEnterpriseServer 6.4 
> Santiago
>
> [ldtdsr02se18][INFO  ] installing ceph on ldtdsr02se18
>
> [ldtdsr02se18][INFO  ] Running command: su -c 'rpm --import 
> "https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc";'
>
> [ldtdsr02se18][ERROR ] Traceback (most recent call last):
>
> [ldtdsr02se18][ERROR ]   File
> "/usr/lib/python2.6/site-packages/ceph_deploy/hosts/centos/install.py"
> , line 23, in install
>
> [ldtdsr02se18][ERROR ]   File
> "/usr/lib/python2.6/site-packages/ceph_deploy/util/decorators.py", 
> line 10, in inner
>
> [ldtdsr02se18][ERROR ]   File
> "/usr/lib/python2.6/site-packages/ceph_deploy/util/wrappers.py", line 
> 6, in remote_call
>
> [ldtdsr02se18][ERROR ]   File "/usr/lib64/python2.6/subprocess.py", line
> 502, in check_call
>
> [ldtdsr02se18][ERROR ] raise CalledProcessError(retcode, cmd)
>
> [ldtdsr02se18][ERROR ] CalledProcessError: Command '['su -c \'rpm 
> --import 
> "https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc"\'']'
> returned non-zero exit status 1
>
> [ldtdsr02se18][ERROR ] curl: (7) couldn't connect to host
>
> [ldtdsr02se18][ERROR ] error:
> https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc: 
> import read failed(2).
>
> [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: su -c 
> 'rpm --import 
> "https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc";'
>
>
>
> Note that it uses sudo as it should and then complains about not being able
> to connect.   When I run the exact same command on the ceph node itself as
> the "ceph" user, it works without any errors.   This implies that the
> authentication is in place between ceph and root, and the proxy settings are
> correct.   Yet, it fails to work when initiated from the admin node via
> ceph-deploy.
>
>
>
> Any ideas what might be going on here?   I should add that I looked at the
> github page about using the "-no-adjust-repos"

Re: [ceph-users] Ceph-deploy, sudo and proxies

[alistair.whittle]

I get "Error: Nothing to do" when doing this on the node itself with sudo.


-Original Message-
From: Alfredo Deza [mailto:alfredo.d...@inktank.com] 
Sent: Monday, October 28, 2013 12:12 PM
To: Whittle, Alistair: Investment Bank (LDN)
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Ceph-deploy, sudo and proxies

On Mon, Oct 28, 2013 at 7:33 AM,   wrote:
>
> Thanks, I already had the correct ceph-deply version, but had the flag in the 
> wrong place.
> Solving that got me to the next problem...   I get the following error:
>
> [ceph_deploy.cli][INFO  ] Invoked (1.2.7): /usr/bin/ceph-deploy 
> install ldtdsr02se18 --no-adjust-repos [ceph_deploy.install][DEBUG ] 
> Installing stable version dumpling on cluster ceph hosts ldtdsr02se18 
> [ceph_deploy.install][DEBUG ] Detecting platform for host ldtdsr02se18 ...
> [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with 
> sudo [ceph_deploy.install][INFO  ] Distro info: RedHatEnterpriseServer 
> 6.4 Santiago [ldtdsr02se18][INFO  ] installing ceph on ldtdsr02se18 
> [ldtdsr02se18][INFO  ] Running command: yum -y -q install ceph 
> [ldtdsr02se18][ERROR ] Traceback (most recent call last):
> [ldtdsr02se18][ERROR ]   File 
> "/usr/lib/python2.6/site-packages/ceph_deploy/hosts/centos/install.py", line 
> 57, in install
> [ldtdsr02se18][ERROR ]   File 
> "/usr/lib/python2.6/site-packages/ceph_deploy/util/decorators.py", line 10, 
> in inner
> [ldtdsr02se18][ERROR ]   File 
> "/usr/lib/python2.6/site-packages/ceph_deploy/util/wrappers.py", line 6, in 
> remote_call
> [ldtdsr02se18][ERROR ]   File "/usr/lib64/python2.6/subprocess.py", line 502, 
> in check_call
> [ldtdsr02se18][ERROR ] raise CalledProcessError(retcode, cmd)
> [ldtdsr02se18][ERROR ] CalledProcessError: Command '['yum', '-y', 
> '-q', 'install', 'ceph']' returned non-zero exit status 1 
> [ldtdsr02se18][ERROR ] Error: Nothing to do [ceph_deploy][ERROR ] 
> RuntimeError: Failed to execute command: yum -y -q install ceph
>
> My ceph repo is pointing to http://ceph.com/rpm-dumpling/rhel6/noarch as per 
> the documentation.  Is this correct, or have I missed something else?

That seems correct to me. When you try `yum -y -q install ceph` on that node, 
all is OK? Have you tried that?


>
>
> -Original Message-
> From: Alfredo Deza [mailto:alfredo.d...@inktank.com]
> Sent: Friday, October 25, 2013 6:53 PM
> To: Whittle, Alistair: Investment Bank (LDN)
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Ceph-deploy, sudo and proxies
>
> On Fri, Oct 25, 2013 at 1:25 PM,   wrote:
>> I have an interesting problem I was hoping someone could help with.
>>
>>
>>
>> My Red Hat servers are configured to use proxies to access the internet.   I
>> have managed to successfully add the Ceph repo install ceph-deploy on the
>> admin node and create the cluster.   All ceph nodes are no password sudo
>> tested and I have made sure that the proxy settings are kept when 
>> trying an 'rpm' command using sudo.  All other preflight checks are 
>> completed with ceph being the default login user etc.
>>
>>
>>
>> So, when I run the ceph-deploy install  command from the 
>> admin node, I get the following error:
>>
>>
>>
>> ceph@ldtdsr02se17 PROD $ ceph-deploy install ldtdsr02se18
>>
>> [ceph_deploy.cli][INFO  ] Invoked (1.2.7): /usr/bin/ceph-deploy 
>> install
>> ldtdsr02se18
>>
>> [ceph_deploy.install][DEBUG ] Installing stable version dumpling on 
>> cluster ceph hosts ldtdsr02se18
>>
>> [ceph_deploy.install][DEBUG ] Detecting platform for host ldtdsr02se18 ...
>>
>> [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with 
>> sudo
>>
>> [ceph_deploy.install][INFO  ] Distro info: RedHatEnterpriseServer 6.4 
>> Santiago
>>
>> [ldtdsr02se18][INFO  ] installing ceph on ldtdsr02se18
>>
>> [ldtdsr02se18][INFO  ] Running command: su -c 'rpm --import 
>> "https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc";'
>>
>> [ldtdsr02se18][ERROR ] Traceback (most recent call last):
>>
>> [ldtdsr02se18][ERROR ]   File
>> "/usr/lib/python2.6/site-packages/ceph_deploy/hosts/centos/install.py"
>> , line 23, in install
>>
>> [ldtdsr02se18][ERROR ]   File
>> "/usr/lib/python2.6/site-packages/ceph_deploy/util/decorators.py",
>> line 10, in inner
>>
>> [ldtdsr02se18][ERROR ]   File
>> "/usr/lib/python2.6/site-packages/ceph_deploy/util/wrappers.py", line 
>> 6, in remote_call
>>
>> [ldtdsr02se18][ERROR ]   File "/usr/lib64/python2.6/subprocess.py", line
>> 502, in check_call
>>
>> [ldtdsr02se18][ERROR ] raise CalledProcessError(retcode, cmd)
>>
>> [ldtdsr02se18][ERROR ] CalledProcessError: Command '['su -c \'rpm 
>> --import 
>> "https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc"\'']'
>> returned non-zero exit status 1
>>
>> [ldtdsr02se18][ERROR ] curl: (7) couldn't connect to host
>>
>> [ldtdsr02se18][ERROR ] error:
>> https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc:
>> import read failed(2).
>>
>> [ceph_deploy][ERROR ] RuntimeError: Failed to execu

Re: [ceph-users] Ceph-deploy, sudo and proxies

[alistair.whittle]

Yum tells me I have the following installed on the node:

ceph-deploy.noarch : Admin and deploy tool for Ceph
ceph-release.noarch : Ceph repository configuration

I think this means ceph is NOT already installed.  Interesting that ceph-deploy 
is on the node as well.   I only installed it on the admin node.   I assume 
this was done as an earlier step in the install process?


-Original Message-
From: Alfredo Deza [mailto:alfredo.d...@inktank.com] 
Sent: Monday, October 28, 2013 12:29 PM
To: Whittle, Alistair: Investment Bank (LDN)
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Ceph-deploy, sudo and proxies

On Mon, Oct 28, 2013 at 8:26 AM,   wrote:
>
>
> I get "Error: Nothing to do" when doing this on the node itself with sudo.

That may mean that it is already installed. Can you check if ceph is installed 
and that you can move forward with the rest of the process?

In this case, ceph-deploy would be OK to raise an error because YUM complained 
with a non-zero exit status.

>
>
> -Original Message-
> From: Alfredo Deza [mailto:alfredo.d...@inktank.com]
> Sent: Monday, October 28, 2013 12:12 PM
> To: Whittle, Alistair: Investment Bank (LDN)
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Ceph-deploy, sudo and proxies
>
> On Mon, Oct 28, 2013 at 7:33 AM,   wrote:
>>
>> Thanks, I already had the correct ceph-deply version, but had the flag in 
>> the wrong place.
>> Solving that got me to the next problem...   I get the following error:
>>
>> [ceph_deploy.cli][INFO  ] Invoked (1.2.7): /usr/bin/ceph-deploy 
>> install ldtdsr02se18 --no-adjust-repos [ceph_deploy.install][DEBUG ] 
>> Installing stable version dumpling on cluster ceph hosts ldtdsr02se18 
>> [ceph_deploy.install][DEBUG ] Detecting platform for host ldtdsr02se18 ...
>> [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with 
>> sudo [ceph_deploy.install][INFO  ] Distro info: 
>> RedHatEnterpriseServer
>> 6.4 Santiago [ldtdsr02se18][INFO  ] installing ceph on ldtdsr02se18 
>> [ldtdsr02se18][INFO  ] Running command: yum -y -q install ceph 
>> [ldtdsr02se18][ERROR ] Traceback (most recent call last):
>> [ldtdsr02se18][ERROR ]   File 
>> "/usr/lib/python2.6/site-packages/ceph_deploy/hosts/centos/install.py", line 
>> 57, in install
>> [ldtdsr02se18][ERROR ]   File 
>> "/usr/lib/python2.6/site-packages/ceph_deploy/util/decorators.py", line 10, 
>> in inner
>> [ldtdsr02se18][ERROR ]   File 
>> "/usr/lib/python2.6/site-packages/ceph_deploy/util/wrappers.py", line 6, in 
>> remote_call
>> [ldtdsr02se18][ERROR ]   File "/usr/lib64/python2.6/subprocess.py", line 
>> 502, in check_call
>> [ldtdsr02se18][ERROR ] raise CalledProcessError(retcode, cmd)
>> [ldtdsr02se18][ERROR ] CalledProcessError: Command '['yum', '-y', 
>> '-q', 'install', 'ceph']' returned non-zero exit status 1 
>> [ldtdsr02se18][ERROR ] Error: Nothing to do [ceph_deploy][ERROR ]
>> RuntimeError: Failed to execute command: yum -y -q install ceph
>>
>> My ceph repo is pointing to http://ceph.com/rpm-dumpling/rhel6/noarch as per 
>> the documentation.  Is this correct, or have I missed something else?
>
> That seems correct to me. When you try `yum -y -q install ceph` on that node, 
> all is OK? Have you tried that?
>
>
>>
>>
>> -Original Message-
>> From: Alfredo Deza [mailto:alfredo.d...@inktank.com]
>> Sent: Friday, October 25, 2013 6:53 PM
>> To: Whittle, Alistair: Investment Bank (LDN)
>> Cc: ceph-users@lists.ceph.com
>> Subject: Re: [ceph-users] Ceph-deploy, sudo and proxies
>>
>> On Fri, Oct 25, 2013 at 1:25 PM,   wrote:
>>> I have an interesting problem I was hoping someone could help with.
>>>
>>>
>>>
>>> My Red Hat servers are configured to use proxies to access the internet.   I
>>> have managed to successfully add the Ceph repo install ceph-deploy on the
>>> admin node and create the cluster.   All ceph nodes are no password sudo
>>> tested and I have made sure that the proxy settings are kept when 
>>> trying an 'rpm' command using sudo.  All other preflight checks are 
>>> completed with ceph being the default login user etc.
>>>
>>>
>>>
>>> So, when I run the ceph-deploy install  command from the 
>>> admin node, I get the following error:
>>>
>>>
>>>
>>> ceph@ldtdsr02se17 PROD $ ceph-deploy install ldtdsr02se18
>>>
>>> [ceph_deploy.cli][INFO  ] Invoked (1.2.7): /usr/bin/ceph-deploy 
>>> install
>>> ldtdsr02se18
>>>
>>> [ceph_deploy.install][DEBUG ] Installing stable version dumpling on 
>>> cluster ceph hosts ldtdsr02se18
>>>
>>> [ceph_deploy.install][DEBUG ] Detecting platform for host ldtdsr02se18 ...
>>>
>>> [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with 
>>> sudo
>>>
>>> [ceph_deploy.install][INFO  ] Distro info: RedHatEnterpriseServer 
>>> 6.4 Santiago
>>>
>>> [ldtdsr02se18][INFO  ] installing ceph on ldtdsr02se18
>>>
>>> [ldtdsr02se18][INFO  ] Running command: su -c 'rpm --import 
>>> "https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc";'
>>>
>>> [ldtdsr02se18][ERROR ] Traceback

Re: [ceph-users] Ceph-deploy, sudo and proxies

[alistair.whittle]

Sadly, this is already my second attempt on a "clean" build.

I have made more progress.   I altered my ceph repo to include the repos 
documented for a manual rpm build.   Ceph-deploy now finds the ceph package, 
but then got a number of yum dependency errors (mostly python related).   I 
sorted those out, and then tried the ceph-deploy from the admin node again.   
It failed trying to get the python-argparse package from the ceph repo (timed 
out).   I then went back to the node and ran the Ceph install from there.   
This time it worked without timing out with the argparse package.   I suppose 
this could be proxy related again.

Baby steps  Now to do it all again on the second node.


-Original Message-
From: Alfredo Deza [mailto:alfredo.d...@inktank.com] 
Sent: Monday, October 28, 2013 1:32 PM
To: Whittle, Alistair: Investment Bank (LDN)
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Ceph-deploy, sudo and proxies

On Mon, Oct 28, 2013 at 8:37 AM,   wrote:
> Yum tells me I have the following installed on the node:
>
> ceph-deploy.noarch : Admin and deploy tool for Ceph 
> ceph-release.noarch : Ceph repository configuration
>
> I think this means ceph is NOT already installed.  Interesting that 
> ceph-deploy is on the node as well.   I only installed it on the admin node.  
>  I assume this was done as an earlier step in the install process?

I believe that ceph-deploy comes with the installation process, yes.

Can you make sure that your repos have not been altered by ceph-deploy from the 
previous runs? At this point it looks like there has been a bunch of back and 
forth to attempt an install and that is causing the odd behavior. Even better 
would be to try this on a new/clean host and see what errors (if any) come up.

>
>
> -Original Message-
> From: Alfredo Deza [mailto:alfredo.d...@inktank.com]
> Sent: Monday, October 28, 2013 12:29 PM
> To: Whittle, Alistair: Investment Bank (LDN)
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Ceph-deploy, sudo and proxies
>
> On Mon, Oct 28, 2013 at 8:26 AM,   wrote:
>>
>>
>> I get "Error: Nothing to do" when doing this on the node itself with sudo.
>
> That may mean that it is already installed. Can you check if ceph is 
> installed and that you can move forward with the rest of the process?
>
> In this case, ceph-deploy would be OK to raise an error because YUM 
> complained with a non-zero exit status.
>
>>
>>
>> -Original Message-
>> From: Alfredo Deza [mailto:alfredo.d...@inktank.com]
>> Sent: Monday, October 28, 2013 12:12 PM
>> To: Whittle, Alistair: Investment Bank (LDN)
>> Cc: ceph-users@lists.ceph.com
>> Subject: Re: [ceph-users] Ceph-deploy, sudo and proxies
>>
>> On Mon, Oct 28, 2013 at 7:33 AM,   wrote:
>>>
>>> Thanks, I already had the correct ceph-deply version, but had the flag in 
>>> the wrong place.
>>> Solving that got me to the next problem...   I get the following error:
>>>
>>> [ceph_deploy.cli][INFO  ] Invoked (1.2.7): /usr/bin/ceph-deploy 
>>> install ldtdsr02se18 --no-adjust-repos [ceph_deploy.install][DEBUG ] 
>>> Installing stable version dumpling on cluster ceph hosts ldtdsr02se18 
>>> [ceph_deploy.install][DEBUG ] Detecting platform for host ldtdsr02se18 ...
>>> [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with 
>>> sudo [ceph_deploy.install][INFO  ] Distro info:
>>> RedHatEnterpriseServer
>>> 6.4 Santiago [ldtdsr02se18][INFO  ] installing ceph on ldtdsr02se18 
>>> [ldtdsr02se18][INFO  ] Running command: yum -y -q install ceph 
>>> [ldtdsr02se18][ERROR ] Traceback (most recent call last):
>>> [ldtdsr02se18][ERROR ]   File 
>>> "/usr/lib/python2.6/site-packages/ceph_deploy/hosts/centos/install.py", 
>>> line 57, in install
>>> [ldtdsr02se18][ERROR ]   File 
>>> "/usr/lib/python2.6/site-packages/ceph_deploy/util/decorators.py", line 10, 
>>> in inner
>>> [ldtdsr02se18][ERROR ]   File 
>>> "/usr/lib/python2.6/site-packages/ceph_deploy/util/wrappers.py", line 6, in 
>>> remote_call
>>> [ldtdsr02se18][ERROR ]   File "/usr/lib64/python2.6/subprocess.py", line 
>>> 502, in check_call
>>> [ldtdsr02se18][ERROR ] raise CalledProcessError(retcode, cmd)
>>> [ldtdsr02se18][ERROR ] CalledProcessError: Command '['yum', '-y', 
>>> '-q', 'install', 'ceph']' returned non-zero exit status 1 
>>> [ldtdsr02se18][ERROR ] Error: Nothing to do [ceph_deploy][ERROR ]
>>> RuntimeError: Failed to execute command: yum -y -q install ceph
>>>
>>> My ceph repo is pointing to http://ceph.com/rpm-dumpling/rhel6/noarch as 
>>> per the documentation.  Is this correct, or have I missed something else?
>>
>> That seems correct to me. When you try `yum -y -q install ceph` on that 
>> node, all is OK? Have you tried that?
>>
>>
>>>
>>>
>>> -Original Message-
>>> From: Alfredo Deza [mailto:alfredo.d...@inktank.com]
>>> Sent: Friday, October 25, 2013 6:53 PM
>>> To: Whittle, Alistair: Investment Bank (LDN)
>>> Cc: ceph-users@lists.ceph.com
>>> Subject: Re: [ceph-users] Ceph-deploy, sudo and p

[ceph-users] OSD activation issue

[alistair.whittle]

Hello all,

I am getting some issues when activating OSD's on my Red Hat 6.4 Ceph cluster.  
I am using the quick start mechanism so mounted a new xfs filesystem  and ran 
the "osd prepare" command.

The prepare seemed to be successful as per the log output below:

[ceph_deploy.cli][INFO  ] Invoked (1.2.7): /usr/bin/ceph-deploy osd prepare 
ldtdsr02se19:/osd2
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks ldtdsr02se19:/osd2:
[ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo
[ceph_deploy.osd][INFO  ] Distro info: RedHatEnterpriseServer 6.4 Santiago
[ceph_deploy.osd][DEBUG ] Deploying osd to ldtdsr02se19
[ldtdsr02se19][INFO  ] write cluster configuration to /etc/ceph/{cluster}.conf
[ldtdsr02se19][INFO  ] keyring file does not exist, creating one at: 
/var/lib/ceph/bootstrap-osd/ceph.keyring
[ldtdsr02se19][INFO  ] create mon keyring file
[ldtdsr02se19][INFO  ] Running command: udevadm trigger --subsystem-match=block 
--action=add
[ceph_deploy.osd][DEBUG ] Preparing host ldtdsr02se19 disk /osd2 journal None 
activate False
[ldtdsr02se19][INFO  ] Running command: ceph-disk-prepare --fs-type xfs 
--cluster ceph -- /osd2
[ceph_deploy.osd][DEBUG ] Host ldtdsr02se19 is now ready for osd use.

When I ran the activate, however, I got the following:

[ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo
[ceph_deploy.osd][INFO  ] Distro info: RedHatEnterpriseServer 6.4 Santiago
[ceph_deploy.osd][DEBUG ] activating host ldtdsr02se19 disk /osd2
[ceph_deploy.osd][DEBUG ] will use init type: sysvinit
[ldtdsr02se19][INFO  ] Running command: ceph-disk-activate --mark-init sysvinit 
--mount /osd2
[ldtdsr02se19][INFO  ] === osd.1 ===
[ldtdsr02se19][INFO  ] Starting Ceph osd.1 on ldtdsr02se19...
[ldtdsr02se19][INFO  ] starting osd.1 at :/0 osd_data /var/lib/ceph/osd/ceph-1 
/var/lib/ceph/osd/ceph-1/journal
[ldtdsr02se19][ERROR ] got latest monmap
[ldtdsr02se19][ERROR ] 2013-10-29 10:45:26.373347 7fbaa1b597a0 -1 journal 
FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio 
to force use of aio anyway
[ldtdsr02se19][ERROR ] 2013-10-29 10:45:26.403218 7fbaa1b597a0 -1 journal 
FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio 
to force use of aio anyway
[ldtdsr02se19][ERROR ] 2013-10-29 10:45:26.405363 7fbaa1b597a0 -1 
filestore(/osd2) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No 
such file or directory
[ldtdsr02se19][ERROR ] 2013-10-29 10:45:26.461080 7fbaa1b597a0 -1 created 
object store /osd2 journal /osd2/journal for osd.1 fsid 
148d95d1-a069-491d-8780-1bcbbefe624a
[ldtdsr02se19][ERROR ] 2013-10-29 10:45:26.461177 7fbaa1b597a0 -1 auth: error 
reading file: /osd2/keyring: can't open /osd2/keyring: (2) No such file or 
directory
[ldtdsr02se19][ERROR ] 2013-10-29 10:45:26.461306 7fbaa1b597a0 -1 created new 
key in keyring /osd2/keyring
[ldtdsr02se19][ERROR ] added key for osd.1
[ldtdsr02se19][ERROR ] create-or-move updating item name 'osd.1' weight 0.1 at 
location {host=ldtdsr02se19,root=default} to crush map

At this point, ceph-deploy just hung.  After waiting for many minutes, I had to 
kill it manually.   Looking at the OSD directory though, there is a lot of 
information that has been added there and when I do a "ceph health" it comes 
back as "HEALTH_OK".

Is it really OK?

Thanks


___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OSD activation issue

[alistair.whittle]

Thanks.   It does seem to be working ok and I can create / remove objects it 
seems without issues.

I am however having another problem.   In trying to add additional monitors to 
my cluster I am getting the following errors (note I did not see this when 
doing the first and currently only running monitor).   It seems to set it up 
fine, but then has problems starting it.

[ceph_deploy.cli][INFO  ] Invoked (1.2.7): /usr/bin/ceph-deploy mon create 
ldtdsr02se20
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ldtdsr02se20
[ceph_deploy.mon][DEBUG ] detecting platform for host ldtdsr02se20 ...
[ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo
[ceph_deploy.mon][INFO  ] distro info: RedHatEnterpriseServer 6.4 Santiago
[ldtdsr02se20][DEBUG ] determining if provided host has same hostname in remote
[ldtdsr02se20][DEBUG ] deploying mon to ldtdsr02se20
[ldtdsr02se20][DEBUG ] remote hostname: ldtdsr02se20
[ldtdsr02se20][INFO  ] write cluster configuration to /etc/ceph/{cluster}.conf
[ldtdsr02se20][INFO  ] creating path: /var/lib/ceph/mon/ceph-ldtdsr02se20
[ldtdsr02se20][DEBUG ] checking for done path: 
/var/lib/ceph/mon/ceph-ldtdsr02se20/done
[ldtdsr02se20][DEBUG ] done path does not exist: 
/var/lib/ceph/mon/ceph-ldtdsr02se20/done
[ldtdsr02se20][INFO  ] creating keyring file: 
/var/lib/ceph/tmp/ceph-ldtdsr02se20.mon.keyring
[ldtdsr02se20][INFO  ] create the monitor keyring file
[ldtdsr02se20][INFO  ] Running command: ceph-mon --cluster ceph --mkfs -i 
ldtdsr02se20 --keyring /var/lib/ceph/tmp/ceph-ldtdsr02se20.mon.keyring
[ldtdsr02se20][INFO  ] ceph-mon: set fsid to 
148d95d1-a069-491d-8780-1bcbbefe624a
[ldtdsr02se20][INFO  ] ceph-mon: created monfs at 
/var/lib/ceph/mon/ceph-ldtdsr02se20 for mon.ldtdsr02se20
[ldtdsr02se20][INFO  ] unlinking keyring file 
/var/lib/ceph/tmp/ceph-ldtdsr02se20.mon.keyring
[ldtdsr02se20][INFO  ] create a done file to avoid re-doing the mon deployment
[ldtdsr02se20][INFO  ] create the init path if it does not exist
[ldtdsr02se20][INFO  ] locating `service` executable...
[ldtdsr02se20][INFO  ] found `service` executable: /sbin/service
[ldtdsr02se20][INFO  ] Running command: sudo /sbin/service ceph -c 
/etc/ceph/ceph.conf start mon.ldtdsr02se20
[ldtdsr02se20][DEBUG ] === mon.ldtdsr02se20 ===
[ldtdsr02se20][DEBUG ] Starting Ceph mon.ldtdsr02se20 on ldtdsr02se20...
[ldtdsr02se20][DEBUG ] failed: 'ulimit -n 32768;  /usr/bin/ceph-mon -i 
ldtdsr02se20 --pid-file /var/run/ceph/mon.ldtdsr02se20.pid -c 
/etc/ceph/ceph.conf’'
[ldtdsr02se20][DEBUG ] Starting ceph-create-keys on ldtdsr02se20...
[ldtdsr02se20][WARNIN] No data was received after 7 seconds, disconnecting...
[ldtdsr02se20][INFO  ] Running command: sudo ceph --admin-daemon 
/var/run/ceph/ceph-mon.ldtdsr02se20.asok mon_status
[ldtdsr02se20][ERROR ] admin_socket: exception getting command descriptions: 
[Errno 2] No such file or directory
[ldtdsr02se20][WARNIN] monitor: mon.ldtdsr02se20, might not be running yet
[ldtdsr02se20][INFO  ] Running command: sudo ceph --admin-daemon 
/var/run/ceph/ceph-mon.ldtd
[ldtdsr02se20][ERROR ] admin_socket: exception getting command descriptions: 
[Errno 2] No such file or directory
[ldtdsr02se20][WARNIN] ldtdsr02se20 is not defined in `mon initial members`
[ldtdsr02se20][WARNIN] monitor ldtdsr02se20 does not exist in monmap
[ldtdsr02se20][WARNIN] neither `public_addr` nor `public_network` keys are 
defined for monit
[ldtdsr02se20][WARNIN] monitors may not be able to form quorum

From: Karan Singh [mailto:ksi...@csc.fi]
Sent: Tuesday, October 29, 2013 12:13 PM
To: Whittle, Alistair: Investment Bank (LDN)
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] OSD activation issue

Hello Alistar

I also faced exactly same issue with one of my  OSD , after OSD Activate , 
progress got hanged but finally OSD gets added in cluster with no problem.

My cluster is running without knows issues as of now. If this is a test setup , 
you can ignore this , but keep an eye on this.

Regards
Karan Singh


From: "alistair whittle" 
mailto:alistair.whit...@barclays.com>>
To: ceph-users@lists.ceph.com
Sent: Tuesday, 29 October, 2013 1:10:00 PM
Subject: [ceph-users] OSD activation issue

Hello all,

I am getting some issues when activating OSD’s on my Red Hat 6.4 Ceph cluster.  
I am using the quick start mechanism so mounted a new xfs filesystem  and ran 
the “osd prepare” command.

The prepare seemed to be successful as per the log output below:

[ceph_deploy.cli][INFO  ] Invoked (1.2.7): /usr/bin/ceph-deploy osd prepare 
ldtdsr02se19:/osd2
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks ldtdsr02se19:/osd2:
[ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo
[ceph_deploy.osd][INFO  ] Distro info: RedHatEnterpriseServer 6.4 Santiago
[ceph_deploy.osd][DEBUG ] Deploying osd to ldtdsr02se19
[ldtdsr02se19][INFO  ] write cluster configuration to /etc/ceph/{cluster}.conf
[ldtdsr02se19][INFO  ] keyring fi

Re: [ceph-users] ceph-deploy problems on CentOS-6.4

[alistair.whittle]

You also want to make sure that if you are using a proxy your proxy settings 
are maintained through sudo.

With my deployment I had to add a line to my sudoers file to specify that the 
https_proxy and http_proxy settings are maintained.  Didn't work otherwise.

Defaults env_keep += "http_proxy https_proxy"


From: ceph-users-boun...@lists.ceph.com 
[mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Gruher, Joseph R
Sent: Tuesday, October 29, 2013 4:55 PM
To: Trivedi, Narendra; ceph-users@lists.ceph.com
Subject: Re: [ceph-users] ceph-deploy problems on CentOS-6.4

If you are behind a proxy try configuring the wget proxy through /etc/wgetrc.  
I had a similar problem where I could complete wget commands manually but they 
would fail in ceph-deploy until I configured the wget proxy in that manner.

From: 
ceph-users-boun...@lists.ceph.com 
[mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Trivedi, Narendra
Sent: Tuesday, October 29, 2013 9:51 AM
To: ceph-users@lists.ceph.com
Subject: [ceph-users] ceph-deploy problems on CentOS-6.4

Hi All,

I am a newbie to ceph. I am installing ceph (dumpling release) using 
ceph-deploy (issued from my admin node) on one monitor and two OSD nodes 
running CentOS 6.4 (64-bit) using followed instructions in the link below:

http://ceph.com/docs/master/start/quick-ceph-deploy/

My setup looks exactly like the diagram. I followed pre-flight instructions 
exacty as outlined in the link below:

http://ceph.com/docs/master/start/quick-start-preflight/

The ceph-deploy takes forever and then throws up the following error:

2013-10-28 17:32:35,903 [ceph_deploy.cli][INFO  ] Invoked (1.2.7): 
/usr/bin/ceph-deploy new ceph-node1-mon-centos-6-4
2013-10-28 17:32:35,904 [ceph_deploy.new][DEBUG ] Creating new cluster named 
ceph
2013-10-28 17:32:35,904 [ceph_deploy.new][DEBUG ] Resolving host 
ceph-node1-mon-centos-6-4
2013-10-28 17:32:35,904 [ceph_deploy.new][DEBUG ] Monitor 
ceph-node1-mon-centos-6-4 at 10.12.0.70
2013-10-28 17:32:35,904 [ceph_deploy.new][DEBUG ] Monitor initial members are 
['ceph-node1-mon-centos-6-4']
2013-10-28 17:32:35,904 [ceph_deploy.new][DEBUG ] Monitor addrs are 
['10.12.0.70']
2013-10-28 17:32:35,905 [ceph_deploy.new][DEBUG ] Creating a random mon key...
2013-10-28 17:32:35,905 [ceph_deploy.new][DEBUG ] Writing initial config to 
ceph.conf...
2013-10-28 17:32:35,905 [ceph_deploy.new][DEBUG ] Writing monitor keyring to 
ceph.mon.keyring...
2013-10-28 17:33:10,287 [ceph_deploy.cli][INFO  ] Invoked (1.2.7): 
/usr/bin/ceph-deploy install ceph-node1-mon-centos-6-4 
ceph-node2-osd0-centos-6-4 ceph-admin-node-centos-6-4
2013-10-28 17:33:10,287 [ceph_deploy.install][DEBUG ] Installing stable version 
dumpling on cluster ceph hosts ceph-node1-mon-centos-6-4 
ceph-node2-osd0-centos-6-4 ceph-admin-node-centos-6-4
2013-10-28 17:33:10,288 [ceph_deploy.install][DEBUG ] Detecting platform for 
host ceph-node1-mon-centos-6-4 ...
2013-10-28 17:33:10,288 [ceph_deploy.sudo_pushy][DEBUG ] will use a remote 
connection without sudo
2013-10-28 17:33:10,626 [ceph_deploy.install][INFO  ] Distro info: CentOS 6.4 
Final
2013-10-28 17:33:10,626 [ceph-node1-mon-centos-6-4][INFO  ] installing ceph on 
ceph-node1-mon-centos-6-4
2013-10-28 17:33:10,633 [ceph-node1-mon-centos-6-4][INFO  ] adding EPEL 
repository
2013-10-28 17:33:10,633 [ceph-node1-mon-centos-6-4][INFO  ] Running command: 
wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
2013-10-28 19:20:35,893 [ceph-node1-mon-centos-6-4][ERROR ] Traceback (most 
recent call last):
2013-10-28 19:20:35,894 [ceph-node1-mon-centos-6-4][ERROR ]   File 
"/usr/lib/python2.6/site-packages/ceph_deploy/hosts/centos/install.py", line 
77, in install_epel
2013-10-28 19:20:35,899 [ceph-node1-mon-centos-6-4][ERROR ]   File 
"/usr/lib/python2.6/site-packages/ceph_deploy/util/decorators.py", line 10, in 
inner
2013-10-28 19:20:35,900 [ceph-node1-mon-centos-6-4][ERROR ]   File 
"/usr/lib/python2.6/site-packages/ceph_deploy/util/wrappers.py", line 6, in 
remote_call
2013-10-28 19:20:35,902 [ceph-node1-mon-centos-6-4][ERROR ]   File 
"/usr/lib64/python2.6/subprocess.py", line 502, in check_call
2013-10-28 19:20:35,903 [ceph-node1-mon-centos-6-4][ERROR ] raise 
CalledProcessError(retcode, cmd)
2013-10-28 19:20:35,904 [ceph-node1-mon-centos-6-4][ERROR ] CalledProcessError: 
Command '['wget', 
'http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm']' 
returned non-zero exit status 4
2013-10-28 19:20:35,911 [ceph-node1-mon-centos-6-4][ERROR ] --2013-10-28 
17:33:10--  
http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
2013-10-28 19:20:35,911 [ceph-node1-mon-centos-6-4][ERROR ] Resolving 
dl.fedoraproject.org... 209.132.181.25, 209.132.181.26, 209.132.181.27, ...
2013-10-28 19:20:35,912 [ceph-node1-mon-centos-6-4][ERROR ] Connecting to 
dl.fedoraproject.org|209.132.181.25|:

Re: [ceph-users] ceph-deploy problems on CentOS-6.4

[alistair.whittle]

From: Trivedi, Narendra [mailto:narendra.triv...@savvis.com]
Sent: Tuesday, October 29, 2013 5:33 PM
To: Whittle, Alistair: Investment Bank (LDN); joseph.r.gru...@intel.com; 
ceph-users@lists.ceph.com
Subject: RE: ceph-deploy problems on CentOS-6.4

Thanks a lot Joseph and Alistair... I have the following questions based on 
your inputs:


1)  Do I need to make changes to all the nodes or just the admin node? I 
guess all the nodes since ceph-deploy issues commands via ssh on all nodes...
AW::  I found I needed to make the changes to ALL nodes.It helps if they 
are pretty much identical in this regards prior to setting off.   Remember too 
that it's not just wget that needs the proxy settings.   You may want to check 
your yum proxy settings as well.   I went one step further and set both http 
and https proxy settings in my default user shell script (both ceph and root).


2)  The installation guide recommends using ceph user to communicate with 
other nodes. Do I login to the admin node as root user or ceph user?
AW::  Initially used root from the admin node, but found that to not work.   I 
eventually created a ceph user, logged in as that and did all my ceph related 
work as that user on all nodes.  The guide mentions not to run ceph-deploy as 
root or with sudo and I found to my frustration that is advice worth heeding.


From: alistair.whit...@barclays.com 
[mailto:alistair.whit...@barclays.com]
Sent: Tuesday, October 29, 2013 12:08 PM
To: joseph.r.gru...@intel.com; Trivedi, 
Narendra; ceph-users@lists.ceph.com
Subject: RE: ceph-deploy problems on CentOS-6.4

You also want to make sure that if you are using a proxy your proxy settings 
are maintained through sudo.

With my deployment I had to add a line to my sudoers file to specify that the 
https_proxy and http_proxy settings are maintained.  Didn't work otherwise.

Defaults env_keep += "http_proxy https_proxy"


From: 
ceph-users-boun...@lists.ceph.com 
[mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Gruher, Joseph R
Sent: Tuesday, October 29, 2013 4:55 PM
To: Trivedi, Narendra; 
ceph-users@lists.ceph.com
Subject: Re: [ceph-users] ceph-deploy problems on CentOS-6.4

If you are behind a proxy try configuring the wget proxy through /etc/wgetrc.  
I had a similar problem where I could complete wget commands manually but they 
would fail in ceph-deploy until I configured the wget proxy in that manner.

From: 
ceph-users-boun...@lists.ceph.com 
[mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Trivedi, Narendra
Sent: Tuesday, October 29, 2013 9:51 AM
To: ceph-users@lists.ceph.com
Subject: [ceph-users] ceph-deploy problems on CentOS-6.4

Hi All,

I am a newbie to ceph. I am installing ceph (dumpling release) using 
ceph-deploy (issued from my admin node) on one monitor and two OSD nodes 
running CentOS 6.4 (64-bit) using followed instructions in the link below:

http://ceph.com/docs/master/start/quick-ceph-deploy/

My setup looks exactly like the diagram. I followed pre-flight instructions 
exacty as outlined in the link below:

http://ceph.com/docs/master/start/quick-start-preflight/

The ceph-deploy takes forever and then throws up the following error:

2013-10-28 17:32:35,903 [ceph_deploy.cli][INFO  ] Invoked (1.2.7): 
/usr/bin/ceph-deploy new ceph-node1-mon-centos-6-4
2013-10-28 17:32:35,904 [ceph_deploy.new][DEBUG ] Creating new cluster named 
ceph
2013-10-28 17:32:35,904 [ceph_deploy.new][DEBUG ] Resolving host 
ceph-node1-mon-centos-6-4
2013-10-28 17:32:35,904 [ceph_deploy.new][DEBUG ] Monitor 
ceph-node1-mon-centos-6-4 at 10.12.0.70
2013-10-28 17:32:35,904 [ceph_deploy.new][DEBUG ] Monitor initial members are 
['ceph-node1-mon-centos-6-4']
2013-10-28 17:32:35,904 [ceph_deploy.new][DEBUG ] Monitor addrs are 
['10.12.0.70']
2013-10-28 17:32:35,905 [ceph_deploy.new][DEBUG ] Creating a random mon key...
2013-10-28 17:32:35,905 [ceph_deploy.new][DEBUG ] Writing initial config to 
ceph.conf...
2013-10-28 17:32:35,905 [ceph_deploy.new][DEBUG ] Writing monitor keyring to 
ceph.mon.keyring...
2013-10-28 17:33:10,287 [ceph_deploy.cli][INFO  ] Invoked (1.2.7): 
/usr/bin/ceph-deploy install ceph-node1-mon-centos-6-4 
ceph-node2-osd0-centos-6-4 ceph-admin-node-centos-6-4
2013-10-28 17:33:10,287 [ceph_deploy.install][DEBUG ] Installing stable version 
dumpling on cluster ceph hosts ceph-node1-mon-centos-6-4 
ceph-node2-osd0-centos-6-4 ceph-admin-node-centos-6-4
2013-10-28 17:33:10,288 [ceph_deploy.install][DEBUG ] Detecting platform for 
host ceph-node1-mon-centos-6-4 ...
2013-10-28 17:33:10,288 [ceph_deploy.sudo_pushy][DEBUG ] will use a remote 
connection without sudo
2013-10-28 17:33:10,626 [ceph_deploy.install][INFO  ] Distro info: CentOS 6.4 
Final
2013-10-28 17:33:10,626 [

[ceph-users] Ceph monitor problems

[alistair.whittle]

All,

My ceph cluster is failing to respond to calls for health status etc.  It 
simply hangs at each command and then tells me there is an error connecting to 
the cluster.   I assume that it is because I tried to add a second monitor 
which seemed to hang during the "hunting" phase.  I eventually had to kill the 
tasks which I think resulted in my monmap becoming corrupt.   I have tried to 
access it from the only remaining working monitor, but have not had much joy 
with that.

I have also tried to follow the docs re removing monitors from unhealthy 
clusters, but have as yet had no luck with getting the stable monmap 
extracted

When I run any ceph command from the remaining monitor host I get the following 
error:

2013-10-30 14:19:01.822660 7f16080f2700 -1 monclient(hunting): ERROR: missing 
keyring, cannot use cephx for authentication
2013-10-30 14:19:01.822743 7f16080f2700  0 librados: client.admin 
initialization error (2) No such file or directory
Error connecting to cluster: ObjectNotFound

It's the hunting part which implies that something is still stuck from when I 
tried to add the second monitor.

Any ideas on how I can fix the monmap so the remaining monitor can be used 
again?

Thanks.


___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph monitor problems

[alistair.whittle]

Thanks Karan.

I have checked and there are no iptables chains/rules  configured.   Any other 
ideas?

From: Karan Singh [mailto:ksi...@csc.fi]
Sent: Wednesday, October 30, 2013 2:49 PM
To: Whittle, Alistair: Investment Bank (LDN)
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Ceph monitor problems

Hi AW

Did you check on firewall ( iptables ) between the nodes. If this is a test 
cluster disable iptables and try.

Regards
Karan Singh


From: "alistair whittle" 
mailto:alistair.whit...@barclays.com>>
To: ceph-users@lists.ceph.com
Sent: Wednesday, 30 October, 2013 4:20:27 PM
Subject: [ceph-users] Ceph monitor problems

All,

My ceph cluster is failing to respond to calls for health status etc.  It 
simply hangs at each command and then tells me there is an error connecting to 
the cluster.   I assume that it is because I tried to add a second monitor 
which seemed to hang during the “hunting” phase.  I eventually had to kill the 
tasks which I think resulted in my monmap becoming corrupt.   I have tried to 
access it from the only remaining working monitor, but have not had much joy 
with that.

I have also tried to follow the docs re removing monitors from unhealthy 
clusters, but have as yet had no luck with getting the stable monmap 
extracted

When I run any ceph command from the remaining monitor host I get the following 
error:

2013-10-30 14:19:01.822660 7f16080f2700 -1 monclient(hunting): ERROR: missing 
keyring, cannot use cephx for authentication
2013-10-30 14:19:01.822743 7f16080f2700  0 librados: client.admin 
initialization error (2) No such file or directory
Error connecting to cluster: ObjectNotFound

It’s the hunting part which implies that something is still stuck from when I 
tried to add the second monitor.

Any ideas on how I can fix the monmap so the remaining monitor can be used 
again?

Thanks.


___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer
 regarding market commentary from Barclays Sales and/or Trading, who are active 
market participants; and in respect of Barclays Research, including disclosures 
relating to specific issuers, please see http://publicresearch.barclays.com.

___

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph monitor problems

[alistair.whittle]

Never mind, problem solved.   Cluster is now healthy again.   ☺

Tried the recovery process again and managed to remove the hung monitor.   Not 
sure why it didn’t work last time, but hey.

From: Karan Singh [mailto:ksi...@csc.fi]
Sent: Wednesday, October 30, 2013 2:49 PM
To: Whittle, Alistair: Investment Bank (LDN)
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Ceph monitor problems

Hi AW

Did you check on firewall ( iptables ) between the nodes. If this is a test 
cluster disable iptables and try.

Regards
Karan Singh


From: "alistair whittle" 
mailto:alistair.whit...@barclays.com>>
To: ceph-users@lists.ceph.com
Sent: Wednesday, 30 October, 2013 4:20:27 PM
Subject: [ceph-users] Ceph monitor problems

All,

My ceph cluster is failing to respond to calls for health status etc.  It 
simply hangs at each command and then tells me there is an error connecting to 
the cluster.   I assume that it is because I tried to add a second monitor 
which seemed to hang during the “hunting” phase.  I eventually had to kill the 
tasks which I think resulted in my monmap becoming corrupt.   I have tried to 
access it from the only remaining working monitor, but have not had much joy 
with that.

I have also tried to follow the docs re removing monitors from unhealthy 
clusters, but have as yet had no luck with getting the stable monmap 
extracted

When I run any ceph command from the remaining monitor host I get the following 
error:

2013-10-30 14:19:01.822660 7f16080f2700 -1 monclient(hunting): ERROR: missing 
keyring, cannot use cephx for authentication
2013-10-30 14:19:01.822743 7f16080f2700  0 librados: client.admin 
initialization error (2) No such file or directory
Error connecting to cluster: ObjectNotFound

It’s the hunting part which implies that something is still stuck from when I 
tried to add the second monitor.

Any ideas on how I can fix the monmap so the remaining monitor can be used 
again?

Thanks.


___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer
 regarding market commentary from Barclays Sales and/or Trading, who are active 
market participants; and in respect of Barclays Research, including disclosures 
relating to specific issuers, please see http://publicresearch.barclays.com.

___

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Red Hat clients

[alistair.whittle]

Now that my ceph cluster seems to be happy and stable, I have been looking at 
different ways of using it.   Object, block and file.

Object is relatively easy and I will use different ones to test with Ceph.

When I look at block, I'm getting the impression from a lot of Googling that 
deploying clients on Red Hat to connect to a Ceph cluster can be complex.   As 
I understand it, the rbd module is not currently in the Red Hat kernel (and I 
am not allowed to make changes to our standard kernel as is suggested in places 
as a possible solution).  Does this mean I can't connect a Red Hat machine to 
Ceph as a block client?


___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Activating OSDs takes forever

[alistair.whittle]

I got these exact same error messages on my RHEL build.   It did however seem 
that my OSD's were correctly built and brought online.   Never got to the 
bottom of these errors though.

There were others on this forum that had similar issues, and theirs too seemed 
to be working successfully despite the apparent problems during activation.

From: ceph-users-boun...@lists.ceph.com 
[mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Trivedi, Narendra
Sent: Wednesday, October 30, 2013 8:58 PM
To: ceph-users@lists.ceph.com
Subject: [ceph-users] Activating OSDs takes forever

Hi All,

I had a pretty good run until I issued a command to activate OSDs. Now I am 
back with some more problems:(. My setup is exactly like the one in the 
official ceph documentation:

http://ceph.com/docs/master/start/quick-ceph-deploy/

That means, I am just using node2:/tmp/osd0 and node3:/tmp/osd1 as OSDs. I am 
running 64-bit CentOS-6-4.

I was able to prepare OSDs fine but when I am activating them it just takes 
forver (15 mins and counting):

[ceph@ceph-admin-node-centos-6-4 mycluster]$ ceph-deploy osd activate 
ceph-node2-osd0-centos-6-4:/tmp/osd0 ceph-node3-osd1-centos-6-4:/tmp/osd1
[ceph_deploy.cli][INFO  ] Invoked (1.2.7): /usr/bin/ceph-deploy osd activate 
ceph-node2-osd0-centos-6-4:/tmp/osd0 ceph-node3-osd1-centos-6-4:/tmp/osd1
[ceph_deploy.osd][DEBUG ] Activating cluster ceph disks 
ceph-node2-osd0-centos-6-4:/tmp/osd0: ceph-node3-osd1-centos-6-4:/tmp/osd1:
[ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection with sudo
[ceph_deploy.osd][INFO  ] Distro info: CentOS 6.4 Final
[ceph_deploy.osd][DEBUG ] activating host ceph-node2-osd0-centos-6-4 disk 
/tmp/osd0
[ceph_deploy.osd][DEBUG ] will use init type: sysvinit
[ceph-node2-osd0-centos-6-4][INFO  ] Running command: ceph-disk-activate 
--mark-init sysvinit --mount /tmp/osd0
[ceph-node2-osd0-centos-6-4][INFO  ] === osd.0 ===
[ceph-node2-osd0-centos-6-4][INFO  ] Starting Ceph osd.0 on 
ceph-node2-osd0-centos-6-4...
[ceph-node2-osd0-centos-6-4][INFO  ] starting osd.0 at :/0 osd_data 
/var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
[ceph-node2-osd0-centos-6-4][ERROR ] got latest monmap
[ceph-node2-osd0-centos-6-4][ERROR ] 2013-10-30 15:34:54.059954 7faffbd537a0 -1 
journal FileJournal::_open: disabling aio for non-block journal.  Use 
journal_force_aio to force use of aio anyway
[ceph-node2-osd0-centos-6-4][ERROR ] 2013-10-30 15:34:54.121705 7faffbd537a0 -1 
journal FileJournal::_open: disabling aio for non-block journal.  Use 
journal_force_aio to force use of aio anyway
[ceph-node2-osd0-centos-6-4][ERROR ] 2013-10-30 15:34:54.122260 7faffbd537a0 -1 
filestore(/tmp/osd0) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) 
No such file or directory
[ceph-node2-osd0-centos-6-4][ERROR ] 2013-10-30 15:34:54.204635 7faffbd537a0 -1 
created object store /tmp/osd0 journal /tmp/osd0/journal for osd.0 fsid 
ec8c48d5-3889-433b-bf68-558f2eb39a8c
[ceph-node2-osd0-centos-6-4][ERROR ] 2013-10-30 15:34:54.204714 7faffbd537a0 -1 
auth: error reading file: /tmp/osd0/keyring: can't open /tmp/osd0/keyring: (2) 
No such file or directory
[ceph-node2-osd0-centos-6-4][ERROR ] 2013-10-30 15:34:54.204938 7faffbd537a0 -1 
created new key in keyring /tmp/osd0/keyring
[ceph-node2-osd0-centos-6-4][ERROR ] added key for osd.0
[ceph-node2-osd0-centos-6-4][ERROR ] create-or-move updating item name 'osd.0' 
weight 0 at location {host=ceph-node2-osd0-centos-6-4,root=default} to crush map

Why is ceph-deploy throwing bunch of errors when all I did was to create 
directories under /tmp on both nodes and issue a command to prepare them 
without any problems. I mean I can always delete the /tmp/osd0-1 and re-issue 
the commands again but I am wondering if anyone else has been this before?

Thanks a lot in advance!
Narendra



This message contains information which may be confidential and/or privileged. 
Unless you are the intended recipient (or authorized to receive for the 
intended recipient), you may not read, use, copy or disclose to anyone the 
message or any information contained in the message. If you have received the 
message in error, please advise the sender by reply e-mail and delete the 
message and any attachment(s) thereto without retaining any copies.

___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please s

[ceph-users] radosgw configuration problems - Swift access

[alistair.whittle]

All,

I have configured a rados gateway as per the Dumpling quick instructions on a 
Red Hat 6 server.  The idea is to use Swift API to access my cluster via this 
interface.

Have configured FastCGI, httpd, as per the guides, did all the user 
creations/authtool commands for the Swift user/subuser etc and it all "seemed" 
to go through smoothly.
When I created the user I got back the following results:

{ "user_id": "greg",
  "display_name": "",
  "email": "",
  "suspended": 0,
  "max_buckets": 1000,
  "auid": 0,
  "subusers": [
{ "id": "greg:swift",
  "permissions": "full-control"}],
  "keys": [
{ "user": "greg",
  "access_key": "B776L21S7ACY97B3VDLE",
  "secret_key": "1V4BMEixut1SgzbPsKmt3SfqHhOgxMjkbINF4bMy"}],
  "swift_keys": [
{ "user": "greg:swift",
  "secret_key": "a3FOvDaCHNotU93jIJaUE0Qyzc6CPLH5By5Km7p6"}],
  "caps": [],
  "op_mask": "read, write, delete",
  "default_placement": "",
  "placement_tags": []}

At least a week or so passed between doing the above configuration and then 
trying to connect via Swift.   There were no changes done to the Ceph 
configuration during this time.

Now, when trying to access the cluster using Swift, we get an "account not 
found" error.   This is using the following example command listed in the ceph 
docs:

swift -V 1.0 -A http://fqdn_of_gateway/auth -U greg:swift -K 
a3FOvDaCHNotU93jIJaUE0Qyzc6CPLH5By5Km7p6 post test

When doing a debug version of this command it adds the fact that it can't find 
the /auth directory prior to saying "Account not found".

As part of debugging this, I tried to run the "radosgw-admin user check" 
command to confirm the configuration.   It gives the following error:

"2013-11-08 12:55:06.228905 7f8dd9d32820  0 WARNING: cannot read region map"

I then tried to add a new user using the same process as before, and it gave 
the same error.

My questions are this:

-  Has anyone seen this before?

-  Why could I create the initial user above, and suddenly I can't 
create anymore, or indeed view this one's details?

-  Is there a step I have missed to create a region, because the 
version of the doc I read did not (as far as I recall) include anything about 
it.  I have looked at the current docs, but they have been updated to include 
the "Emperor" stuff which is multi-region and I can't find the dumpling 
specific stuff anymore to be sure.

Any help would be appreciated.

Thanks


___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph-deploy changing ceph.conf file permissions

[alistair.whittle]

All,

I have a situation on my RHEL 6.4 cluster that seems to be caused by 
ceph-deploy changing the file permissions on /etc/ceph/ceph.conf after running 
a command such as "ceph-deploy mon create node2 node3".  The idea was to create 
additional monitors for a healthy 3 node cluster that already has a functioning 
default monitor, 3 OSD's each and a single MDS.

This command failed, and I'm not going to cover that here, but another outcome 
was that after running that example command trying to run the "ceph health" or 
"ceph status" command on either the two node results in an error about failing 
to initialize cluster.

A huge amount of research proved that this failed because ceph.conf only had rw 
permissions for "root", and because the ceph commands are run as user "ceph" 
they failed because they could not read ceph.conf.   Changing the file 
permissions on that file resolved this issue.  I should point out that the file 
had full read/write permissions for ceph prior to running the ceph-deploy 
command.

The time stamp of the latest change to the ceph.conf file coincides with when 
the ceph-deploy command above was run.

I am running the latest version of ceph-deploy (1.3.4-0) on the Emperor version 
of Ceph.

I did check the ceph tracker for similar situations, but have not come across 
anything so would appreciate some feedback on this.

Thanks
Alistair


___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Adding mon manually

[alistair.whittle]

All,

Having failed to successfully and new monitors using ceph-deploy, I tried the 
documented manual approach.

The platform:
OS:  RHEL 6.4
Ceph:  Emperor
Ceph-deploy:  1.3.4-0

When following the procedure on an existing node in a working cluster that has 
an existing single monitor configured as part of the quick start procedure 
through ceph-deploy, I get the following error when running the following set 
of commands on a second node:

#sudo mkdir /var/lib/ceph/mon/ceph-ldtdsr00559
#ceph auth get mon. -o /home/ceph/tmp/monkey
# ceph mon getmap -o /home/ceph/tmp/monmap
# sudo ceph-mon -i ldtdsr00559 --mkfs --monmap /home/ceph/tmp/monmap 
--keyring /home/ceph/tmp/monkey
ceph-mon: set fsid to 74fdf8eb-fa3a-47ea-afde-593894c86cac
ceph-mon: created monfs at /var/lib/ceph/mon/ceph-ldtdsr00559 for 
mon.ldtdsr00559
# ceph mon add ldtdsr00559 10.123.4.72:6789
2014-01-22 17:27:41.626839 7f388f5fe700  0 monclient: hunting for new mon

And there it sits.   Sat for many minutes with no further output.  Eventually I 
had to kill the process manually.

While this command was being run, and subsequently after killing the process, 
the remaining nodes are now timing out when running "ceph health" or "ceph 
status" commands.   This tells me that the previous command has gotten stuck 
somewhere as has resulted in the remaining monitor not being available.

Any ideas what is going on here?

Thanks
Alistair


___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Adding mon manually

[alistair.whittle]

Once I figure out how to get my cluster healthy again after the monitor problem 
discussed below, then I will try ceph-deploy again and send you the output.

I have been trying to re-inject the last healthy monmap into all the nodes, 
however this has proved unsuccessful thus far.

-Original Message-
From: Alfredo Deza [mailto:alfredo.d...@inktank.com] 
Sent: Wednesday, January 22, 2014 8:15 PM
To: Whittle, Alistair: Investment Bank (LDN)
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Adding mon manually

On Wed, Jan 22, 2014 at 12:47 PM,   wrote:
> All,
>
>
>
> Having failed to successfully and new monitors using ceph-deploy, I 
> tried the documented manual approach.
>
Would you be able to share why/how it didn't work? Maybe some logs or output 
would be great so that we can continue to improve the tool
>
>
> The platform:
>
> OS:  RHEL 6.4
>
> Ceph:  Emperor
>
> Ceph-deploy:  1.3.4-0
>
>
>
> When following the procedure on an existing node in a working cluster 
> that has an existing single monitor configured as part of the quick 
> start procedure through ceph-deploy, I get the following error when 
> running the following set of commands on a second node:
>
>
>
> #sudo mkdir /var/lib/ceph/mon/ceph-ldtdsr00559
>
> #ceph auth get mon. -o /home/ceph/tmp/monkey
>
> # ceph mon getmap -o /home/ceph/tmp/monmap
>
> # sudo ceph-mon -i ldtdsr00559 --mkfs --monmap 
> /home/ceph/tmp/monmap --keyring /home/ceph/tmp/monkey
>
> ceph-mon: set fsid to 74fdf8eb-fa3a-47ea-afde-593894c86cac
>
> ceph-mon: created monfs at /var/lib/ceph/mon/ceph-ldtdsr00559 for
> mon.ldtdsr00559
>
> # ceph mon add ldtdsr00559 10.123.4.72:6789
>
> 2014-01-22 17:27:41.626839 7f388f5fe700  0 monclient: hunting for new 
> mon
>
>
>
> And there it sits.   Sat for many minutes with no further output.
> Eventually I had to kill the process manually.
>
>
>
> While this command was being run, and subsequently after killing the 
> process, the remaining nodes are now timing out when running "ceph health"
> or "ceph status" commands.   This tells me that the previous command has
> gotten stuck somewhere as has resulted in the remaining monitor not 
> being available.
>
>
>
> Any ideas what is going on here?
>
>
>
> Thanks
>
> Alistair
>
>
>
> ___
>
> This message is for information purposes only, it is not a 
> recommendation, advice, offer or solicitation to buy or sell a product 
> or service nor an official confirmation of any transaction. It is 
> directed at persons who are professionals and is not intended for 
> retail customer use. Intended for recipient only. This message is subject to 
> the terms at:
> www.barclays.com/emaildisclaimer.
>
> For important disclosures, please see:
> www.barclays.com/salesandtradingdisclaimer regarding market commentary 
> from Barclays Sales and/or Trading, who are active market 
> participants; and in respect of Barclays Research, including 
> disclosures relating to specific issuers, please see 
> http://publicresearch.barclays.com.
>
> ___
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph-deploy admin command

[alistair.whittle]

Hi,

Having messed up my last RHEL6 Ceph cluster rather spectacularly I decided to 
build again from scratch using the latest versions of the various packages.   
Not having proxy access does make it more of a pfaff to deploy Ceph, but I will 
admit the latest version of ceph-deploy is a vast improvement to earlier 
versions.

That said, I did notice something.

After going through the OSD prepare/activate process with ceph-deploy the last 
step in the "quick start" is to push the admin configuration to all the nodes 
using the "ceph-deploy admin ..." command.
Once complete the quick start documentation states that you need to check the 
file permissions on ceph.client.admin.keyring along with a sudo command to 
change the permissions to allow read access.

Running the "ceph health" after this did not work and I soon discovered I had 
to run the same file permission command on file /etc/ceph/ceph.conf on each 
node as well.   Only then did the "ceph health" command work and report back 
with a "HEALTH_OK".

On the admin node itself, the configuration files prior to running the 
"ceph-deploy admin" command is:

-rw-r--r-- 1 ceph ceph72 Jan 27 18:34 ceph.bootstrap-mds.keyring
-rw-r--r-- 1 ceph ceph72 Jan 27 18:34 ceph.bootstrap-osd.keyring
-rw-r--r-- 1 ceph ceph64 Jan 27 18:34 ceph.client.admin.keyring
-rw-r--r-- 1 ceph ceph   237 Jan 27 17:49 ceph.conf
-rw-r--r-- 1 ceph ceph 54181 Jan 27 18:45 ceph.log
-rw-r--r-- 1 ceph ceph73 Jan 27 17:49 ceph.mon.keyring

After running "ceph-deploy admin" The file permissions on the nodes in 
/etc/ceph were as follows:

-rw--- 1 root root  64 Jan 27 18:45 ceph.client.admin.keyring
-rw--- 1 root root 237 Jan 27 18:45 ceph.conf
-rwxr-xr-x 1 root root  92 Dec 20 22:47 rbdmap

The exception to this is the admin node itself.   The ceph-deploy admin command 
resulted in the following file permissions on the admin node in /etc/ceph

-rw-r--r-- 1 root root  64 Jan 27 19:13 ceph.client.admin.keyring
-rw-r--r-- 1 root root 237 Jan 27 19:13 ceph.conf

There seems to be inconsistency here somewhere.   Is this expected behavior?

Cheers
Alistair


___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] RBD module - RHEL 6.4

[alistair.whittle]

In researching the creation of a block device to my Ceph object store, I see 
there are two options for getting hold of the RBD module in RHEL (until RedHat 
includes the necessary modules by default).   Please correct me if I'm wrong.


-  Use an EPEL repo

We don't have external internet access from our production servers for security 
reasons unless such access is severely restricted and carefully managed.   
Using an external repo for this is not an option I would like to pursue.


-  Use a version of the kernel that has these modules compiled it in

We will not be able to deploy anything other than a fully supported RedHat 
kernel, which I believe this would not be.  It would also have to be certified 
and standardized by our Unix team before it could go anywhere near production 
and this will be nigh on impossible.

So a number of questions:

-  Is there a way to easily get the required modules without using the 
external EPEL repo?  Can one download just the necessary ones and install as 
required?  Are there implications in doing this?

-  Do we know when the modules will be available by default within 
RedHat?   We will be moving to RHEL 6.5 soon.

-  Is there a third party mechanism other than EPEL that can be used 
instead of relying on the Linux kernel for this connectivity?

Thanks

___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Starting radosgw - RHEL6.4

[alistair.whittle]

Hi,

I am busy configuring the object gateway and have reached the point where I 
need to start the radosgw.   The documentation calls for the following command:

/etc/init.d/radosgw start

On my deployment, radosgw seems to be located in /usr/bin and trying to start 
it from there results in the following error:

radosgw: must specify 'rgw socket path' or 'rgw port' to run as a daemon
2014-01-30 17:14:37.315927 7ff9eec4a820 -1 WARNING: libcurl doesn't support 
curl_multi_wait()
2014-01-30 17:14:37.315931 7ff9eec4a820 -1 WARNING: cross zone / region 
transfer performance may be affected

Have I missed something somewhere?

Thanks


___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Starting radosgw - RHEL6.4

[alistair.whittle]

I have all this information in my ceph.conf file.

My question was more about the executable itself and the fact that it differs 
from what I can see documented.  It is not in the location mentioned in the 
docs (/usr/bin instead of /etc/init.d) and seems to want some parameters which 
is also not specified in the docs at the URL you mention below.

I'm trying to bring the gateway up, but am not entirely sure how to do it.



-Original Message-
From: Derek Yarnell [mailto:de...@umiacs.umd.edu] 
Sent: Thursday, January 30, 2014 11:06 PM
To: Whittle, Alistair: Investment Bank (LDN); ceph-users@lists.ceph.com
Cc: de...@umiacs.umd.edu
Subject: Re: [ceph-users] Starting radosgw - RHEL6.4



On 1/30/14, 12:16 PM, alistair.whit...@barclays.com wrote:
> radosgw: must specify 'rgw socket path' or 'rgw port' to run as a 
> daemon
> 

This is defined in your /etc/ceph/ceph.conf.  For example if the host you are 
running the radosgw is named 'myhostname' then your ceph.conf should have a 
section like this, including where you stashed the keyring.  If you are running 
with mod_fastcgi you very well may need the rgw_print_continue line.

[client.radosgw.myhostname]
keyring = /etc/ceph/keyring.radosgw.myhostname
rgw_print_continue = false
host = myhostname
rgw_socket_path = /tmp/radosgw.sock

See http://ceph.com/docs/master/radosgw/config/ for more information.

--
Derek T. Yarnell
University of Maryland
Institute for Advanced Computer Studies
___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Starting radosgw - RHEL6.4

[alistair.whittle]

Found the problem, I think.   

The executable IS in /etc/init.d, but it called 'ceph-radosgw', not 'radosgw' 
as per the documentation.



-Original Message-
From: Whittle, Alistair: Investment Bank (LDN) 
Sent: Friday, January 31, 2014 3:01 PM
To: 'Derek Yarnell'; ceph-users@lists.ceph.com
Subject: RE: [ceph-users] Starting radosgw - RHEL6.4

I have all this information in my ceph.conf file.

My question was more about the executable itself and the fact that it differs 
from what I can see documented.  It is not in the location mentioned in the 
docs (/usr/bin instead of /etc/init.d) and seems to want some parameters which 
is also not specified in the docs at the URL you mention below.

I'm trying to bring the gateway up, but am not entirely sure how to do it.



-Original Message-
From: Derek Yarnell [mailto:de...@umiacs.umd.edu] 
Sent: Thursday, January 30, 2014 11:06 PM
To: Whittle, Alistair: Investment Bank (LDN); ceph-users@lists.ceph.com
Cc: de...@umiacs.umd.edu
Subject: Re: [ceph-users] Starting radosgw - RHEL6.4



On 1/30/14, 12:16 PM, alistair.whit...@barclays.com wrote:
> radosgw: must specify 'rgw socket path' or 'rgw port' to run as a 
> daemon
> 

This is defined in your /etc/ceph/ceph.conf.  For example if the host you are 
running the radosgw is named 'myhostname' then your ceph.conf should have a 
section like this, including where you stashed the keyring.  If you are running 
with mod_fastcgi you very well may need the rgw_print_continue line.

[client.radosgw.myhostname]
keyring = /etc/ceph/keyring.radosgw.myhostname
rgw_print_continue = false
host = myhostname
rgw_socket_path = /tmp/radosgw.sock

See http://ceph.com/docs/master/radosgw/config/ for more information.

--
Derek T. Yarnell
University of Maryland
Institute for Advanced Computer Studies
___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Radosgw / Chunked transfer / RHEL / Swift

[alistair.whittle]

All,

My radosgw seems to be working, generally, however I have been experiencing 
problems when trying to connect to it from CTERA via OpenStack Swift.

I get the following errors:

[client 10.125.190.59] chunked Transfer-Encoding forbidden: 
/swift/v1/Ctera_ceph01/fileMaps/1266/bad10d636c9373fd24001cad5d8eecacc4283e3e

After some research on the Ceph list archive and Google, the answers seem to 
point to the version of fcgi I am running and the need to run the Ceph specific 
version.   Well, I am running the fcgi version that is included with the 
Emperor code for RHEL - fcgi-2.4.0-10.el6.x86_64

More research seems to suggest that this may be related to a bug that was 
raised quite some time ago (http://tracker.ceph.com/issues/3297) and I am 
wondering if there is something wrong with my fastcgi configuration or my HTTPD 
configuration.

I am running the following config:

OS:  RHEL 6.4
Ceph: Emperor release 0.72
Httpd:  2.2.15-26.el6
FastCGI: As above, taken from the Ceph downloads 
(http://ceph.com/rpm/el6/x86_64/)

Any ideas?

Thanks

Alistair


___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Radosgw / Chunked transfer / RHEL / Swift

[alistair.whittle]

Does anybody have any advice for me on the issue discussed below?

From: ceph-users-boun...@lists.ceph.com 
[mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Whittle, Alistair: 
Investment Bank (LDN)
Sent: Monday, February 10, 2014 3:44 PM
To: ceph-users@lists.ceph.com
Subject: [ceph-users] Radosgw / Chunked transfer / RHEL / Swift

All,

My radosgw seems to be working, generally, however I have been experiencing 
problems when trying to connect to it from CTERA via OpenStack Swift.

I get the following errors:

[client 10.125.190.59] chunked Transfer-Encoding forbidden: 
/swift/v1/Ctera_ceph01/fileMaps/1266/bad10d636c9373fd24001cad5d8eecacc4283e3e

After some research on the Ceph list archive and Google, the answers seem to 
point to the version of fcgi I am running and the need to run the Ceph specific 
version.   Well, I am running the fcgi version that is included with the 
Emperor code for RHEL - fcgi-2.4.0-10.el6.x86_64

More research seems to suggest that this may be related to a bug that was 
raised quite some time ago (http://tracker.ceph.com/issues/3297) and I am 
wondering if there is something wrong with my fastcgi configuration or my HTTPD 
configuration.

I am running the following config:

OS:  RHEL 6.4
Ceph: Emperor release 0.72
Httpd:  2.2.15-26.el6
FastCGI: As above, taken from the Ceph downloads 
(http://ceph.com/rpm/el6/x86_64/)

Any ideas?

Thanks

Alistair


___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer
 regarding market commentary from Barclays Sales and/or Trading, who are active 
market participants; and in respect of Barclays Research, including disclosures 
relating to specific issuers, please see http://publicresearch.barclays.com.

___

___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Radosgw / Chunked transfer / RHEL / Swift

[alistair.whittle]

Thanks Greg.   I seem to have missed that distinction when doing the original 
gateway install using that very web page.   Hopefully this fixes the problem.


-Original Message-
From: Gregory Farnum [mailto:g...@inktank.com] 
Sent: Wednesday, February 12, 2014 5:14 PM
To: Whittle, Alistair: Investment Bank (LDN)
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Radosgw / Chunked transfer / RHEL / Swift

If you look at http://ceph.com/docs/master/install/install-ceph-gateway,
the fcgi package you have installed is labeled as "W/Out 100
Continue". You need to add a separate "gitbuilder" repository if you
want it with 100-continue (supporting chunked transfers); it's all
described on that page.
-Greg
Software Engineer #42 @ http://inktank.com | http://ceph.com


On Mon, Feb 10, 2014 at 7:43 AM,   wrote:
> All,
>
>
>
> My radosgw seems to be working, generally, however I have been experiencing
> problems when trying to connect to it from CTERA via OpenStack Swift.
>
>
>
> I get the following errors:
>
>
>
> [client 10.125.190.59] chunked Transfer-Encoding forbidden:
> /swift/v1/Ctera_ceph01/fileMaps/1266/bad10d636c9373fd24001cad5d8eecacc4283e3e
>
>
>
> After some research on the Ceph list archive and Google, the answers seem to
> point to the version of fcgi I am running and the need to run the Ceph
> specific version.   Well, I am running the fcgi version that is included
> with the Emperor code for RHEL - fcgi-2.4.0-10.el6.x86_64
>
>
>
> More research seems to suggest that this may be related to a bug that was
> raised quite some time ago (http://tracker.ceph.com/issues/3297) and I am
> wondering if there is something wrong with my fastcgi configuration or my
> HTTPD configuration.
>
>
>
> I am running the following config:
>
>
>
> OS:  RHEL 6.4
>
> Ceph: Emperor release 0.72
>
> Httpd:  2.2.15-26.el6
>
> FastCGI: As above, taken from the Ceph downloads
> (http://ceph.com/rpm/el6/x86_64/)
>
>
>
> Any ideas?
>
>
>
> Thanks
>
>
>
> Alistair
>
>
>
> ___
>
> This message is for information purposes only, it is not a recommendation,
> advice, offer or solicitation to buy or sell a product or service nor an
> official confirmation of any transaction. It is directed at persons who are
> professionals and is not intended for retail customer use. Intended for
> recipient only. This message is subject to the terms at:
> www.barclays.com/emaildisclaimer.
>
> For important disclosures, please see:
> www.barclays.com/salesandtradingdisclaimer regarding market commentary from
> Barclays Sales and/or Trading, who are active market participants; and in
> respect of Barclays Research, including disclosures relating to specific
> issuers, please see http://publicresearch.barclays.com.
>
> ___
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___

This message is for information purposes only, it is not a recommendation, 
advice, offer or solicitation to buy or sell a product or service nor an 
official confirmation of any transaction. It is directed at persons who are 
professionals and is not intended for retail customer use. Intended for 
recipient only. This message is subject to the terms at: 
www.barclays.com/emaildisclaimer.

For important disclosures, please see: 
www.barclays.com/salesandtradingdisclaimer regarding market commentary from 
Barclays Sales and/or Trading, who are active market participants; and in 
respect of Barclays Research, including disclosures relating to specific 
issuers, please see http://publicresearch.barclays.com.

___
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com