[ceph-users] IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable
Hi all,
approaching ceph today for the first time, so apologize for the basic
questions I promise I will do all my homework :-)
Following the documentation "storage cluster quick start" I am soon
stuck with the issue below while creating a first mon:
ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] failed: 'ulimit -n 32768; /usr/bin/ceph-mon -i
ceph-node1 --pid-file /var/run/ceph/mon.ceph-node1.pid -c
/etc/ceph/ceph.conf '
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph-node1][ERROR ] admin_socket: exception getting command
descriptions: [Errno 2] No such file or directory
[ceph-node1][WARNIN] monitor: mon.ceph-node1, might not be running yet
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph-node1][ERROR ] admin_socket: exception getting command
descriptions: [Errno 2] No such file or directory
[ceph-node1][WARNIN] monitor ceph-node1 does not exist in monmap
[ceph-node1][WARNIN] neither `public_addr` nor `public_network` keys are
defined for monitors
[ceph-node1][WARNIN] monitors may not be able to form quorum
It seems ceph-mon does not exit with success, in fact:
ceph-node1 # sudo /usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
2013-12-05 10:06:27.429602 7fe06baf9780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 35662
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:27.436295 7fe06baf9780 -1 failed to create new leveldb
store
ceph@ceph-node1:/var/run/ceph> sudo /usr/bin/ceph-mon -i ceph-node1
--pid-file /var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:43.961756 7fd71d27c780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 36216
2013-12-05 10:06:43.964079 7fd71d27c780 -1 failed to create new leveldb
store
Thanks,
Jan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable
Hello Jan
I faced similar kind of errors and these are really annoying. I tried this and
worked for me.
1. Your ceph-node1 is now a monitor node, but it cannot form quorum.
2. check monitor logs from ceph-node1 on /var/lib/ceph directory. This will
give you more strength.
3. You might need to manually add monitor in cluster.
Below is my solution ( you should change it as per your hostname and IP etc)
login on monitor node and execute commands like below
ceph mon add ceph-mon2 192.168.1.33:6789
ceph-mon -i ceph-mon2 --public-addr 192.168.1.33:6789
service ceph status
service ceph restart
ps -ef | grep ceph
In case ceph service is not showing as running in service ceph status command ,
however ps -ef | grep ceph , shows monitor running , kill the process manually
and restart ceph services , monitor now should start working
# service ceph status
# ps -ef | grep ceph
# kill -9 6554
### Finally you would see your ceph cluster is healthy with all the OSD and
monitors UP and Running
[root@ceph-mon1 ~]# ceph status
cluster 0ff473d9-0670-42a3-89ff-81bbfb2e676a
health HEALTH_OK
monmap e3: 3 mons at
{ceph-mon1=192.168.1.38:6789/0,ceph-mon2=192.168.1.33:6789/0,ceph-mon3=192.168.1.31:6789/0},
election epoch 10, quorum 0,1,2 ceph-mon1,ceph-mon2,ceph-mon3
osdmap e97: 8 osds: 8 up, 8 in
pgmap v246: 192 pgs: 192 active+clean; 0 bytes data, 2352 MB used, 3509 GB
/ 3512 GB avail
mdsmap e1: 0/0/1 up
[root@ceph-mon1 ~]#
All the best :-)
Many Thanks
Karan Singh
- Original Message -
From: "Jan Kalcic"
To: ceph-users@lists.ceph.com
Sent: Thursday, 5 December, 2013 11:16:32 AM
Subject: [ceph-users] IO error: lock
/var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily
unavailable
Hi all,
approaching ceph today for the first time, so apologize for the basic
questions I promise I will do all my homework :-)
Following the documentation "storage cluster quick start" I am soon
stuck with the issue below while creating a first mon:
ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] failed: 'ulimit -n 32768; /usr/bin/ceph-mon -i
ceph-node1 --pid-file /var/run/ceph/mon.ceph-node1.pid -c
/etc/ceph/ceph.conf '
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph-node1][ERROR ] admin_socket: exception getting command
descriptions: [Errno 2] No such file or directory
[ceph-node1][WARNIN] monitor: mon.ceph-node1, might not be running yet
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
[ceph-node1][ERROR ] admin_socket: exception getting command
descriptions: [Errno 2] No such file or directory
[ceph-node1][WARNIN] monitor ceph-node1 does not exist in monmap
[ceph-node1][WARNIN] neither `public_addr` nor `public_network` keys are
defined for monitors
[ceph-node1][WARNIN] monitors may not be able to form quorum
It seems ceph-mon does not exit with success, in fact:
ceph-node1 # sudo /usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
2013-12-05 10:06:27.429602 7fe06baf9780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 35662
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unava
[ceph-users] pool size 1 RBD distribution
hi ceph,
just for testing (on emperor 0.72.1) I created two OSD’s on a single server,
resized the pool to a replication factor of one, and created 200 PG’s for that
pool:
# ceph osd dump
...
pool 4 'rbd' rep size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num
200 pgp_num 200 last_change 64 owner 18446744073709551615
crush_ruleset 0 is - IMHO - stating that data should be distributed on the OSD
level (step chooseleaf firstn 0 type osd):
# rules
rule data {
ruleset 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type osd
step emit
}
Now I do an rbd import of an RBD Image (which is 1G in size), and I would
expect that RBD image to stripe across the two OSD’s. Well, this is just not
happening, everything sits on OSD2 (osd1 and osd0 have been removed in the mean
time, they have been part of the first test):
# df -h
...
/dev/vdc1 2.0G 905M 1.2G 45% /var/lib/ceph/osd/ceph-2
/dev/vdb1 2.0G 37M 2.0G 2% /var/lib/ceph/osd/ceph-3
# ceph -w
cluster 6db7c956-cfbb-437a-88b6-78e1c9e68c80
health HEALTH_OK
monmap e1: 1 mons at {ceph-node1=XXX:6789/0}, election epoch 1, quorum 0
ceph-node1
osdmap e65: 2 osds: 2 up, 2 in
pgmap v187: 200 pgs, 5 pools, 868 MB data, 220 objects
941 MB used, 3132 MB / 4073 MB avail
200 active+clean
2013-12-05 09:46:43.210312 mon.0 [INF] pgmap v187: 200 pgs: 200 active+clean;
868 MB data, 941 MB used, 3132 MB / 4073 MB avail
Any hints are more than welcome, this is for sure not a real life scenario, but
it really confuses my understanding of ceph.
Wolfgang
--
http://www.wogri.at
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Journal, SSD and OS
2013/12/4 Simon Leinen : > I think this is a fine configuration - you won't be writing to the root > partition too much, outside journals. We also put journals on the same > SSDs as root partitions (not that we're very ambitious about > performance...). Do you suggest a RAID1 for the OS partitions on SSDs ? Is this safe or a RAID1 will decrease SSD life? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Error in starting ceph
Installed ceph-emperor using apt-get in ubuntu 12.04 by following the steps given in installation part of ceph-doc website. http://ceph.com/docs/master/install/get-packages/ http://ceph.com/docs/master/install/install-storage-cluster/ But get error when this command is run : service ceph -a start service ceph -a start === mon.a === Starting Ceph mon.a on ems...already running === mds.a === Starting Ceph mds.a on ems...already running === osd.0 === Mounting ext4 on ems:/var/lib/ceph/osd/ceph-0 Error ENOENT: osd.0 does not exist. create it before updating the crush map failed: 'timeout 10 /usr/bin/ceph --name=osd.0 --keyring=/var/lib/ceph/osd/ceph-0/keyring osd crush create-or-move -- 0 0.10root=default host=hostname' "Error ENOENT: osd.0 does not exist. create it before updating the crush map" this was warning in dumpling but in emperor its been converted as error. Please let me know the steps to solve the problem Thanks, Sahana ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable
Hi Karan,
On 12/05/2013 10:31 AM, Karan Singh wrote:
Hello Jan
I faced similar kind of errors and these are really annoying. I tried this and
worked for me.
Glad to know I am not alone :-) , though this sounds like a not really
robust procedure...
1. Your ceph-node1 is now a monitor node, but it cannot form quorum.
2. check monitor logs from ceph-node1 on /var/lib/ceph directory. This will
give you more strength.
3. You might need to manually add monitor in cluster.
Below is my solution ( you should change it as per your hostname and IP etc)
login on monitor node and execute commands like below
ceph mon add ceph-mon2 192.168.1.33:6789
ceph-mon -i ceph-mon2 --public-addr 192.168.1.33:6789
service ceph status
service ceph restart
ps -ef | grep ceph
In case ceph service is not showing as running in service ceph status command ,
however ps -ef | grep ceph , shows monitor running , kill the process manually
and restart ceph services , monitor now should start working
# service ceph status
# ps -ef | grep ceph
# kill -9 6554
I have played around with your suggestions but I could not solve.. I
removed the existing directory in /var/lib/ceph/mon for the mon node and
it seems I could go a step ahead:
ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] done path does not exist:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][INFO ] creating keyring file:
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create the monitor keyring file
[ceph-node1][INFO ] Running command: sudo ceph-mon --cluster ceph
--mkfs -i ceph-node1 --keyring /var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] ceph-mon: mon.noname-a 172.17.43.11:6789/0 is
local, renaming to mon.ceph-node1
[ceph-node1][DEBUG ] ceph-mon: set fsid to
18203301-f70b-40f1-b93e-fde4731f3bee
[ceph-node1][DEBUG ] ceph-mon: created monfs at
/var/lib/ceph/mon/ceph-ceph-node1 for mon.ceph-node1
[ceph-node1][INFO ] unlinking keyring file
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
But then the command is pending and nothing happens. If I kill it and
then restart it with /etc/init.d/ceph start it is the same:
ceph-node1 # /etc/init.d/ceph start
=== mon.ceph-node1 ===
Starting Ceph mon.ceph-node1 on ceph-node1...
Anything else?
Thanks,
Jan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Error in starting ceph
Hi Sahana, Did you already create any osd? With the osd prepare and activate command? Best regards Enviado desde mi Personal Samsung GT-i8190L Original message From: Sahana Date: 05/12/2013 07:26 (GMT-03:00) To: ceph-us...@ceph.com Subject: [ceph-users] Error in starting ceph Installed ceph-emperor using apt-get in ubuntu 12.04 by following the steps given in installation part of ceph-doc website. http://ceph.com/docs/master/install/get-packages/ http://ceph.com/docs/master/install/install-storage-cluster/ But get error when this command is run : service ceph -a start service ceph -a start === mon.a === Starting Ceph mon.a on ems...already running === mds.a === Starting Ceph mds.a on ems...already running === osd.0 === Mounting ext4 on ems:/var/lib/ceph/osd/ceph-0 Error ENOENT: osd.0 does not exist. create it before updating the crush map failed: 'timeout 10 /usr/bin/ceph --name=osd.0 --keyring=/var/lib/ceph/osd/ceph-0/keyring osd crush create-or-move -- 0 0.10 root=default host=hostname ' "Error ENOENT: osd.0 does not exist. create it before updating the crush map" this was warning in dumpling but in emperor its been converted as error. Please let me know the steps to solve the problem Thanks, Sahana ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable
what ceph status and ceph mon_status output says , did you checked logs after
this anything interesting there ?
Many Thanks
Karan Singh
- Original Message -
From: "Jan Kalcic"
To: "Karan Singh"
Cc: ceph-users@lists.ceph.com
Sent: Thursday, 5 December, 2013 12:58:33 PM
Subject: Re: [ceph-users] IO error: lock
/var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily
unavailable
Hi Karan,
On 12/05/2013 10:31 AM, Karan Singh wrote:
> Hello Jan
>
> I faced similar kind of errors and these are really annoying. I tried this
> and worked for me.
Glad to know I am not alone :-) , though this sounds like a not really
robust procedure...
>
> 1. Your ceph-node1 is now a monitor node, but it cannot form quorum.
> 2. check monitor logs from ceph-node1 on /var/lib/ceph directory. This will
> give you more strength.
> 3. You might need to manually add monitor in cluster.
>
>
>Below is my solution ( you should change it as per your hostname and IP
> etc)
>
>
>
> login on monitor node and execute commands like below
> ceph mon add ceph-mon2 192.168.1.33:6789
> ceph-mon -i ceph-mon2 --public-addr 192.168.1.33:6789
> service ceph status
> service ceph restart
> ps -ef | grep ceph
>
> In case ceph service is not showing as running in service ceph status command
> , however ps -ef | grep ceph , shows monitor running , kill the process
> manually and restart ceph services , monitor now should start working
>
> # service ceph status
> # ps -ef | grep ceph
> # kill -9 6554
>
I have played around with your suggestions but I could not solve.. I
removed the existing directory in /var/lib/ceph/mon for the mon node and
it seems I could go a step ahead:
ceph-admin # ceph-deploy mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][DEBUG ] done path does not exist:
/var/lib/ceph/mon/ceph-ceph-node1/done
[ceph-node1][INFO ] creating keyring file:
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create the monitor keyring file
[ceph-node1][INFO ] Running command: sudo ceph-mon --cluster ceph
--mkfs -i ceph-node1 --keyring /var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] ceph-mon: mon.noname-a 172.17.43.11:6789/0 is
local, renaming to mon.ceph-node1
[ceph-node1][DEBUG ] ceph-mon: set fsid to
18203301-f70b-40f1-b93e-fde4731f3bee
[ceph-node1][DEBUG ] ceph-mon: created monfs at
/var/lib/ceph/mon/ceph-ceph-node1 for mon.ceph-node1
[ceph-node1][INFO ] unlinking keyring file
/var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
[ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-node1][DEBUG ] create the init path if it does not exist
[ceph-node1][DEBUG ] locating the `service` executable...
[ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
start mon.ceph-node1
[ceph-node1][DEBUG ] === mon.ceph-node1 ===
[ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
[ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
[ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
[ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
--admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
But then the command is pending and nothing happens. If I kill it and
then restart it with /etc/init.d/ceph start it is the same:
ceph-node1 # /etc/init.d/ceph start
=== mon.ceph-node1 ===
Starting Ceph mon.ceph-node1 on ceph-node1...
Anything else?
Thanks,
Jan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Error in starting ceph
ems is a remote machine? Did you set up the corresponding directories: /var/lib/ceph/osd/ceph-0, and called mkcephfs before? You can also try starting osd manually by 'ceph-osd -i 0 -c /etc/ceph/ceph.conf', then 'pgrep ceph-osd' to see if they are there, then 'ceph -s' to check the health. On 2013/12/5 18:26, Sahana wrote: Installed ceph-emperor using apt-get in ubuntu 12.04 by following the steps given in installation part of ceph-doc website. http://ceph.com/docs/master/install/get-packages/ http://ceph.com/docs/master/install/install-storage-cluster/ But get error when this command is run : service ceph -a start service ceph -a start === mon.a === Starting Ceph mon.a on ems...already running === mds.a === Starting Ceph mds.a on ems...already running === osd.0 === Mounting ext4 on ems:/var/lib/ceph/osd/ceph-0 Error ENOENT: osd.0 does not exist. create it before updating the crush map failed: 'timeout 10 /usr/bin/ceph --name=osd.0 --keyring=/var/lib/ceph/osd/ceph-0/keyring osd crush create-or-move -- 0 0.10root=default host=hostname' "Error ENOENT: osd.0 does not exist. create it before updating the crush map" this was warning in dumpling but in emperor its been converted as error. Please let me know the steps to solve the problem Thanks, Sahana ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] pgs incomplete
Hi All, I found 6 pgs incomplete while "ceph health detail" after 3 osds down, but after i manage to start again all 3 osds, only 1 left incomplete pg. root@:~# ceph health detail | grep 4.7d pg 4.7d is stuck inactive for 306404.577611, current state incomplete, last acting [6,0] pg 4.7d is stuck unclean for 306404.577947, current state incomplete, last acting [6,0] pg 4.7d is incomplete, acting [6,0] does anyone know how to fix/repair this "incomplete" state ? i have tried to add more osd, done almost all suggestion i found in google, but no luck. and also i can't start any instance in the openstack. Thanks, RzK ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Openstack--instance-boot-from-ceph-volume:: error could not open disk image rbd
Hello Everyone
Trying to boot from ceph volume using bolg
http://www.sebastien-han.fr/blog/2012/06/10/introducing-ceph-to-openstack/ and
http://docs.openstack.org/user-guide/content/boot_from_volume.html
Need help for this error.
=
Logs from /var/log/libvirt/qemu ::
=
qemu-kvm: -drive
file=rbd:ceph-volumes/volume-dd315dda-b22a-4cf8-8b77-7c2b2f163155:id=volumes:key=AQC804xS8HzFJxAAD/zzQ8LMzq9wDLq/5a472g==:auth_supported=cephx\;none:mon_host=192.168.1.31\:6789\;192.168.1.33\:6789\;192.168.1.38\:6789,if=none,id=drive-virtio-disk0,format=raw,serial=dd315dda-b22a-4cf8-8b77-7c2b2f163155,cache=none:
could not open disk image
rbd:ceph-volumes/volume-dd315dda-b22a-4cf8-8b77-7c2b2f163155:id=volumes:key=AQC804xS8HzFJxAAD/zzQ8LMzq9wDLq/5a472g==:auth_supported=cephx\;none:mon_host=192.168.1.31\:6789\;192.168.1.33\:6789\;192.168.1.38\:6789:
No such file or directory
2013-12-05 12:42:29.544+: shutting down
I tried below steps needed for booting from volume
I am using qemu-kvm provided by ceph repository that have RBD support . Using
this i am able to create RBD volumes and its working good.
[root@rdo qemu(keystone_admin)]# rbd list ceph-volumes
volume-0e2bfced-be6a-44ec-a3ca-22c771c66cdc
volume-10cc0855-652a-4a9b-baa1-80bc86dc12ac
volume-5e09ce7b-b874-4c24-97ae-341a305841d5
volume-9671edaa-62c8-4f98-a36c-d6e59612141b
volume-a75d435b-ebce-4013-9c68-8df5d2c917e1
volume-dd315dda-b22a-4cf8-8b77-7c2b2f163155
[root@rdo qemu(keystone_admin)]#
http://docs.openstack.org/user-guide/content/boot_from_volume.html
[root@rdo nova(keystone_admin)]# nova image-list
+--+-+++
| ID | Name | Status | Server |
+--+-+++
| f61edc8d-c9a1-4ff4-b4fc-c8128bd1a10b | Ubuntu 12.04 cloudimg amd64 | ACTIVE |
|
| fcc07414-bbb3-4473-a8df-523664c8c9df | ceph-glance-image | ACTIVE | |
| be62a5bf-879f-4d1f-846c-fdef960224ff | precise-cloudimg.raw | ACTIVE | |
| 3c2db0ad-8d1e-400d-ba13-a506448f2a8e | precise-server-cloudimg | ACTIVE | |
+--+-+++
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]# cinder list
+--+---+---+--+--+--+--+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--+---+---+--+--+--+--+
| 0e2bfced-be6a-44ec-a3ca-22c771c66cdc | in-use | nova-vol_1 | 2 | None | false
| 9d3c327f-1893-40ff-8a82-16fad9ce6d91 |
| 10cc0855-652a-4a9b-baa1-80bc86dc12ac | available | ceph-vol1 | 5 |
ceph-storage | false | |
| 5e09ce7b-b874-4c24-97ae-341a305841d5 | available | precise-cloudimg_vol_boot
| 5 | None | true | |
| 9671edaa-62c8-4f98-a36c-d6e59612141b | available | boot_from_volume | 20 |
None | false | |
| a75d435b-ebce-4013-9c68-8df5d2c917e1 | available | dashboard-volume | 1 |
ceph-storage | false | |
+--+---+---+--+--+--+--+
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]# cinder create --image-id
be62a5bf-879f-4d1f-846c-fdef960224ff --display-name new_boot_from_vol 10
+-+--+
| Property | Value |
+-+--+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| created_at | 2013-12-05T12:34:45.241324 |
| display_description | None |
| display_name | new_boot_from_vol |
| id | dd315dda-b22a-4cf8-8b77-7c2b2f163155 |
| image_id | be62a5bf-879f-4d1f-846c-fdef960224ff |
| metadata | {} |
| size | 10 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| volume_type | None |
+-+--+
[root@rdo nova(keystone_admin)]#
[root@rdo nova(keystone_admin)]# cinder list
+--+---+---+--+--+--+--+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--+---+---+--+-
Re: [ceph-users] pool size 1 RBD distribution
On 12/05/2013 10:52 AM, Wolfgang Hennerbichler wrote:
hi ceph,
just for testing (on emperor 0.72.1) I created two OSD’s on a single server,
resized the pool to a replication factor of one, and created 200 PG’s for that
pool:
# ceph osd dump
...
pool 4 'rbd' rep size 1 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num
200 pgp_num 200 last_change 64 owner 18446744073709551615
crush_ruleset 0 is - IMHO - stating that data should be distributed on the OSD
level (step chooseleaf firstn 0 type osd):
# rules
rule data {
ruleset 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type osd
step emit
}
Now I do an rbd import of an RBD Image (which is 1G in size), and I would
expect that RBD image to stripe across the two OSD’s. Well, this is just not
happening, everything sits on OSD2 (osd1 and osd0 have been removed in the mean
time, they have been part of the first test):
Could you run this against your crushmap?
$ ceph osd getcrushmap -o crushmap
$ crushtool --test -i crushmap --num-rep 1 --rule 0 --show-statistics
I tried that locally and gave me a result like:
rule 0 (data), x = 0..1023, numrep = 1..1
CRUSH rule 0 x 0 [0]
..
..
CRUSH rule 0 x 1019 [1]
CRUSH rule 0 x 1020 [0]
CRUSH rule 0 x 1021 [1]
CRUSH rule 0 x 1022 [1]
CRUSH rule 0 x 1023 [0]
rule 0 (data) num_rep 1 result size == 1: 1024/1024
My plain-text crushmap is attached and there you see it works. So I'm
curious to what output you have?
Wido
# df -h
...
/dev/vdc1 2.0G 905M 1.2G 45% /var/lib/ceph/osd/ceph-2
/dev/vdb1 2.0G 37M 2.0G 2% /var/lib/ceph/osd/ceph-3
# ceph -w
cluster 6db7c956-cfbb-437a-88b6-78e1c9e68c80
health HEALTH_OK
monmap e1: 1 mons at {ceph-node1=XXX:6789/0}, election epoch 1, quorum 0
ceph-node1
osdmap e65: 2 osds: 2 up, 2 in
pgmap v187: 200 pgs, 5 pools, 868 MB data, 220 objects
941 MB used, 3132 MB / 4073 MB avail
200 active+clean
2013-12-05 09:46:43.210312 mon.0 [INF] pgmap v187: 200 pgs: 200 active+clean;
868 MB data, 941 MB used, 3132 MB / 4073 MB avail
Any hints are more than welcome, this is for sure not a real life scenario, but
it really confuses my understanding of ceph.
Wolfgang
--
Wido den Hollander
42on B.V.
Phone: +31 (0)20 700 9902
Skype: contact42on
# begin crush map
# devices
device 0 osd.0
device 1 osd.1
# types
type 0 osd
type 1 host
type 2 rack
type 3 row
type 4 room
type 5 datacenter
type 6 root
# buckets
host wido-laptop {
id -2 # do not change unnecessarily
# weight 2.000
alg straw
hash 0 # rjenkins1
item osd.0 weight 1.000
item osd.1 weight 1.000
}
rack unknownrack {
id -3 # do not change unnecessarily
# weight 2.000
alg straw
hash 0 # rjenkins1
item wido-laptop weight 2.000
}
root default {
id -1 # do not change unnecessarily
# weight 2.000
alg straw
hash 0 # rjenkins1
item unknownrack weight 2.000
}
# rules
rule data {
ruleset 0
type replicated
min_size 1
max_size 10
step take default
step choose firstn 0 type osd
step emit
}
rule metadata {
ruleset 1
type replicated
min_size 1
max_size 10
step take default
step choose firstn 0 type osd
step emit
}
rule rbd {
ruleset 2
type replicated
min_size 1
max_size 10
step take default
step choose firstn 0 type osd
step emit
}
# end crush map
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] pool size 1 RBD distribution
> On 12/05/2013 10:52 AM, Wolfgang Hennerbichler wrote: >> Now I do an rbd import of an RBD Image (which is 1G in size), and I would >> expect that RBD image to stripe across the two OSD’s. Well, this is just not >> happening, everything sits on OSD2 (osd1 and osd0 have been removed in the >> mean time, they have been part of the first test): > > Could you run this against your crushmap? > > $ ceph osd getcrushmap -o crushmap > $ crushtool --test -i crushmap --num-rep 1 --rule 0 --show-statistics # crushtool --test -i crushmap --num-rep 1 --rule 0 --show-statistics rule 0 (data), x = 0..1023, numrep = 1..1 CRUSH rule 0 x 0 [2] CRUSH rule 0 x 1 [2] CRUSH rule 0 x 2 [2] CRUSH rule 0 x 3 [2] CRUSH rule 0 x 4 [2] … CRUSH rule 0 x 1022 [2] CRUSH rule 0 x 1023 [2] rule 0 (data) num_rep 1 result size == 1: 1024/1024 > I tried that locally and gave me a result like: > > rule 0 (data), x = 0..1023, numrep = 1..1 > CRUSH rule 0 x 0 [0] > .. > .. > CRUSH rule 0 x 1019 [1] > CRUSH rule 0 x 1020 [0] > CRUSH rule 0 x 1021 [1] > CRUSH rule 0 x 1022 [1] > CRUSH rule 0 x 1023 [0] > rule 0 (data) num_rep 1 result size == 1: 1024/1024 > > My plain-text crushmap is attached and there you see it works. So I'm curious > to what output you have? I don’t know how to interpret my output. I have a [2] at all the lines of my crushtest. after running a diff between your and my crushmap, it turned out that this differed: - step choose firstn 0 type osd + step chooseleaf firstn 0 type osd that didn’t make a difference after adjusting, but having the OSD’s in my crushmap with a weight of 0 did. I reweighted them, and now things are balancing out. thanks a lot. > Wido Wolfgang ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph-deploy 1.2.2 vs fedora 19
Perfect, that worked very well. Thanks a lot. Another question: Using http://ceph.com/howto/deploying-ceph-with-ceph-deploy/ as a guide to set up my test-cluster I now have a working cluster with 12 osd's in and up. I've create a client, a 10gb rbd volume, mounted it, written data all good. Looking at my ceph.conf it seems it's using all defaults: [root@ceph02 ~]# cat /etc/ceph/ceph.conf [global] fsid = f865694c-7a50-46a9-9550-f6b160c00313 mon_initial_members = ceph02, ceph03, ceph04 mon_host = 10.130.21.33,10.130.21.34,10.130.21.42 auth_supported = cephx osd_journal_size = 1024 filestore_xattr_use_omap = true Is there any way to dump the default running config to the config-file so I can start tinkering around? "ceph --show-config" seems to show all parameters running, but I don't see any mentioning of the monitors/osd's at all? Last question: I had a too low number of pgs, which caused a health warning. Since I type a lot faster than I think sometimes I adjusted the value to 128 (ceph osd pool set rbd pg_num 128), which turned out to be too high, but I cant seem to be able to tune it down again. How do I achieve this? Thanks a lot in advance! Kind regards Jonas -Original Message- From: Alfredo Deza [mailto:alfredo.d...@inktank.com] Sent: Sunday, December 01, 2013 6:30 PM To: Jonas Andersson Cc: ceph-us...@ceph.com Subject: Re: [ceph-users] ceph-deploy 1.2.2 vs fedora 19 On Thu, Nov 28, 2013 at 8:25 AM, Jonas Andersson wrote: > Hi all, > > > > I am seeing some weirdness when trying to deploy Ceph Emperor on > fedora 19 using ceph-deploy. Problem occurs when trying to install > ceph-deploy, and seems to point to the version of pushy in your repository: > > Since ceph-deploy version 1.3 there is no longer a requirement on pushy. You should update to the latest version (currently at 1.3.3) > > [root@ceph02 ~]# yum install ceph-deploy > > Loaded plugins: priorities, protectbase > > imc-default > | 1.1 kB 00:00:00 > > imc-shared > | 1.1 kB 00:00:00 > > imc-systemimages > | 1.1 kB 00:00:00 > > imc-systemimages-shared > | 1.1 kB 00:00:00 > > 45 packages excluded due to repository priority protections > > 0 packages excluded due to repository protections > > Resolving Dependencies > > --> Running transaction check > > ---> Package ceph-deploy.noarch 0:1.2.2-0 will be installed > > --> Processing Dependency: python-pushy >= 0.5.3 for package: > ceph-deploy-1.2.2-0.noarch > > --> Processing Dependency: pushy >= 0.5.3 for package: > ceph-deploy-1.2.2-0.noarch > > --> Processing Dependency: or for package: ceph-deploy-1.2.2-0.noarch > > --> Processing Dependency: gdisk for package: > --> ceph-deploy-1.2.2-0.noarch > > --> Running transaction check > > ---> Package ceph-deploy.noarch 0:1.2.2-0 will be installed > > --> Processing Dependency: python-pushy >= 0.5.3 for package: > ceph-deploy-1.2.2-0.noarch > > --> Processing Dependency: or for package: ceph-deploy-1.2.2-0.noarch > > ---> Package gdisk.x86_64 0:0.8.8-1.fc19 will be installed > > --> Processing Dependency: libicuuc.so.50()(64bit) for package: > gdisk-0.8.8-1.fc19.x86_64 > > --> Processing Dependency: libicuio.so.50()(64bit) for package: > gdisk-0.8.8-1.fc19.x86_64 > > ---> Package pushy.noarch 0:0.5.3-1 will be installed > > --> Running transaction check > > ---> Package ceph-deploy.noarch 0:1.2.2-0 will be installed > > --> Processing Dependency: python-pushy >= 0.5.3 for package: > ceph-deploy-1.2.2-0.noarch > > --> Processing Dependency: or for package: ceph-deploy-1.2.2-0.noarch > > ---> Package libicu.x86_64 0:50.1.2-9.fc19 will be installed > > --> Finished Dependency Resolution > > Error: Package: ceph-deploy-1.2.2-0.noarch (ceph-extras-noarch) > >Requires: python-pushy >= 0.5.3 > >Available: python-pushy-0.5.1-6.1.noarch > (ceph-extras-noarch) > >python-pushy = 0.5.1-6.1 > > Error: Package: ceph-deploy-1.2.2-0.noarch (ceph-extras-noarch) > >Requires: or > > You could try using --skip-broken to work around the problem > > You could try running: rpm -Va --nofiles -nodigest > > > > To work around this I tried to use pip to install pushy 0.5.3: > > [root@ceph02 pushy-master]# pip install pushy > > Downloading/unpacking pushy > > Downloading pushy-0.5.3.zip (48kB): 48kB downloaded > > Running setup.py egg_info for package pushy > > > > Installing collected packages: pushy > > Running setup.py install for pushy > > > > Successfully installed pushy > > Cleaning up... > > > > Verifying: > > > > [root@ceph02 ~]# pip list | grep pushy > > pushy (0.5.3) > > > > However the installer does not seem to notice that pushy is there, and > it fails on the same dependency with the same error. > > > > Any clue what I'm doing wrong here? > > > > Kind regards > > > > Jonas > > > > > The information in this e-mail is intended only for the person or > entity to which it is addressed. > > It may contain confidential and /or privileged material. If s
Re: [ceph-users] IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable
On Thu, Dec 5, 2013 at 7:12 AM, Karan Singh wrote:
> what ceph status and ceph mon_status output says , did you checked logs
> after this anything interesting there ?
>
> Many Thanks
> Karan Singh
>
>
> - Original Message -
> From: "Jan Kalcic"
> To: "Karan Singh"
> Cc: ceph-users@lists.ceph.com
> Sent: Thursday, 5 December, 2013 12:58:33 PM
> Subject: Re: [ceph-users] IO error: lock
> /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily
> unavailable
>
> Hi Karan,
>
> On 12/05/2013 10:31 AM, Karan Singh wrote:
>> Hello Jan
>>
>> I faced similar kind of errors and these are really annoying. I tried this
>> and worked for me.
>
> Glad to know I am not alone :-) , though this sounds like a not really
> robust procedure...
>>
>> 1. Your ceph-node1 is now a monitor node, but it cannot form quorum.
>> 2. check monitor logs from ceph-node1 on /var/lib/ceph directory. This
>> will give you more strength.
>> 3. You might need to manually add monitor in cluster.
>>
>>
>>Below is my solution ( you should change it as per your hostname and IP
>> etc)
>>
>>
>>
>> login on monitor node and execute commands like below
>> ceph mon add ceph-mon2 192.168.1.33:6789
>> ceph-mon -i ceph-mon2 --public-addr 192.168.1.33:6789
>> service ceph status
>> service ceph restart
>> ps -ef | grep ceph
>>
>> In case ceph service is not showing as running in service ceph status
>> command , however ps -ef | grep ceph , shows monitor running , kill the
>> process manually and restart ceph services , monitor now should start working
>>
>> # service ceph status
>> # ps -ef | grep ceph
>> # kill -9 6554
>>
>
> I have played around with your suggestions but I could not solve.. I
> removed the existing directory in /var/lib/ceph/mon for the mon node and
> it seems I could go a step ahead:
>
>
> ceph-admin # ceph-deploy mon create ceph-node1
> [ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy mon
> create ceph-node1
> [ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
> [ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
> [ceph-node1][DEBUG ] connected to host: ceph-node1
> [ceph-node1][DEBUG ] detect platform information from remote host
> [ceph-node1][DEBUG ] detect machine type
> [ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
> x86_64
> [ceph-node1][DEBUG ] determining if provided host has same hostname in
> remote
> [ceph-node1][DEBUG ] get remote short hostname
> [ceph-node1][DEBUG ] deploying mon to ceph-node1
> [ceph-node1][DEBUG ] get remote short hostname
> [ceph-node1][DEBUG ] remote hostname: ceph-node1
> [ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
> [ceph-node1][DEBUG ] create the mon path if it does not exist
> [ceph-node1][DEBUG ] checking for done path:
> /var/lib/ceph/mon/ceph-ceph-node1/done
> [ceph-node1][DEBUG ] done path does not exist:
> /var/lib/ceph/mon/ceph-ceph-node1/done
> [ceph-node1][INFO ] creating keyring file:
> /var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
> [ceph-node1][DEBUG ] create the monitor keyring file
> [ceph-node1][INFO ] Running command: sudo ceph-mon --cluster ceph
> --mkfs -i ceph-node1 --keyring /var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
> [ceph-node1][DEBUG ] ceph-mon: mon.noname-a 172.17.43.11:6789/0 is
> local, renaming to mon.ceph-node1
> [ceph-node1][DEBUG ] ceph-mon: set fsid to
> 18203301-f70b-40f1-b93e-fde4731f3bee
> [ceph-node1][DEBUG ] ceph-mon: created monfs at
> /var/lib/ceph/mon/ceph-ceph-node1 for mon.ceph-node1
> [ceph-node1][INFO ] unlinking keyring file
> /var/lib/ceph/tmp/ceph-ceph-node1.mon.keyring
> [ceph-node1][DEBUG ] create a done file to avoid re-doing the mon deployment
> [ceph-node1][DEBUG ] create the init path if it does not exist
> [ceph-node1][DEBUG ] locating the `service` executable...
> [ceph-node1][INFO ] Running command: sudo rcceph -c /etc/ceph/ceph.conf
> start mon.ceph-node1
> [ceph-node1][DEBUG ] === mon.ceph-node1 ===
> [ceph-node1][DEBUG ] Starting Ceph mon.ceph-node1 on ceph-node1...
> [ceph-node1][DEBUG ] Starting ceph-create-keys on ceph-node1...
> [ceph-node1][WARNIN] No data was received after 7 seconds, disconnecting...
> [ceph-node1][INFO ] Running command: sudo ceph --cluster=ceph
> --admin-daemon /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
>
> But then the command is pending and nothing happens. If I kill it and
> then restart it with /etc/init.d/ceph start it is the same:
>
> ceph-node1 # /etc/init.d/ceph start
> === mon.ceph-node1 ===
> Starting Ceph mon.ceph-node1 on ceph-node1...
>
> Anything else?
Do you have iptables enabled? If so, either open the ports for the
monitors to be able to talk or disable it and try again
>
> Thanks,
> Jan
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-us
Re: [ceph-users] Journal, SSD and OS
Hi guys, I won’t do a RAID 1 with SSDs since they both write the same data. Thus, they are more likely to “almost” die at the same time. What I will try to do instead is to use both disk in JBOD mode or (degraded RAID0). Then I will create a tiny root partition for the OS. Then I’ll still have something like /dev/sda2 and /dev/sdb2 and then I can take advantage of the 2 disks independently. The good thing with that is that you can balance your journals across both SSDs. From a performance perspective this is really good. The bad thing as always is that if you loose a SSD you loose all the journals attached to it. Cheers. Sébastien Han Cloud Engineer "Always give 100%. Unless you're giving blood.” Phone: +33 (0)1 49 70 99 72 Mail: sebastien@enovance.com Address : 10, rue de la Victoire - 75009 Paris Web : www.enovance.com - Twitter : @enovance On 05 Dec 2013, at 10:53, Gandalf Corvotempesta wrote: > 2013/12/4 Simon Leinen : >> I think this is a fine configuration - you won't be writing to the root >> partition too much, outside journals. We also put journals on the same >> SSDs as root partitions (not that we're very ambitious about >> performance...). > > Do you suggest a RAID1 for the OS partitions on SSDs ? Is this safe or > a RAID1 will decrease SSD life? > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com signature.asc Description: Message signed with OpenPGP using GPGMail ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph-deploy 1.2.2 vs fedora 19
On Thu, Dec 5, 2013 at 9:18 AM, Jonas Andersson wrote: > Perfect, that worked very well. Thanks a lot. > > Another question: > > Using http://ceph.com/howto/deploying-ceph-with-ceph-deploy/ as a guide to > set up my test-cluster I now have a working cluster with 12 osd's in and up. > I've create a client, a 10gb rbd volume, mounted it, written data all good. > > Looking at my ceph.conf it seems it's using all defaults: > [root@ceph02 ~]# cat /etc/ceph/ceph.conf > [global] > fsid = f865694c-7a50-46a9-9550-f6b160c00313 > mon_initial_members = ceph02, ceph03, ceph04 > mon_host = 10.130.21.33,10.130.21.34,10.130.21.42 > auth_supported = cephx > osd_journal_size = 1024 > filestore_xattr_use_omap = true > > Is there any way to dump the default running config to the config-file so I > can start tinkering around? What do you mean by dump? You can backup that copy which is the one that ceph-deploy uses and use a new one and push it to your nodes and try that way "ceph --show-config" seems to show all parameters running, but I don't see any mentioning of the monitors/osd's at all? > > Last question: > I had a too low number of pgs, which caused a health warning. Since I type a > lot faster than I think sometimes I adjusted the value to 128 (ceph osd pool > set rbd pg_num 128), which turned out to be too high, but I cant seem to be > able to tune it down again. How do I achieve this? > > Thanks a lot in advance! > > Kind regards > > Jonas > > -Original Message- > From: Alfredo Deza [mailto:alfredo.d...@inktank.com] > Sent: Sunday, December 01, 2013 6:30 PM > To: Jonas Andersson > Cc: ceph-us...@ceph.com > Subject: Re: [ceph-users] ceph-deploy 1.2.2 vs fedora 19 > > On Thu, Nov 28, 2013 at 8:25 AM, Jonas Andersson > wrote: >> Hi all, >> >> >> >> I am seeing some weirdness when trying to deploy Ceph Emperor on >> fedora 19 using ceph-deploy. Problem occurs when trying to install >> ceph-deploy, and seems to point to the version of pushy in your repository: >> >> > > Since ceph-deploy version 1.3 there is no longer a requirement on pushy. You > should update to the latest version (currently at 1.3.3) > >> >> [root@ceph02 ~]# yum install ceph-deploy >> >> Loaded plugins: priorities, protectbase >> >> imc-default >> | 1.1 kB 00:00:00 >> >> imc-shared >> | 1.1 kB 00:00:00 >> >> imc-systemimages >> | 1.1 kB 00:00:00 >> >> imc-systemimages-shared >> | 1.1 kB 00:00:00 >> >> 45 packages excluded due to repository priority protections >> >> 0 packages excluded due to repository protections >> >> Resolving Dependencies >> >> --> Running transaction check >> >> ---> Package ceph-deploy.noarch 0:1.2.2-0 will be installed >> >> --> Processing Dependency: python-pushy >= 0.5.3 for package: >> ceph-deploy-1.2.2-0.noarch >> >> --> Processing Dependency: pushy >= 0.5.3 for package: >> ceph-deploy-1.2.2-0.noarch >> >> --> Processing Dependency: or for package: ceph-deploy-1.2.2-0.noarch >> >> --> Processing Dependency: gdisk for package: >> --> ceph-deploy-1.2.2-0.noarch >> >> --> Running transaction check >> >> ---> Package ceph-deploy.noarch 0:1.2.2-0 will be installed >> >> --> Processing Dependency: python-pushy >= 0.5.3 for package: >> ceph-deploy-1.2.2-0.noarch >> >> --> Processing Dependency: or for package: ceph-deploy-1.2.2-0.noarch >> >> ---> Package gdisk.x86_64 0:0.8.8-1.fc19 will be installed >> >> --> Processing Dependency: libicuuc.so.50()(64bit) for package: >> gdisk-0.8.8-1.fc19.x86_64 >> >> --> Processing Dependency: libicuio.so.50()(64bit) for package: >> gdisk-0.8.8-1.fc19.x86_64 >> >> ---> Package pushy.noarch 0:0.5.3-1 will be installed >> >> --> Running transaction check >> >> ---> Package ceph-deploy.noarch 0:1.2.2-0 will be installed >> >> --> Processing Dependency: python-pushy >= 0.5.3 for package: >> ceph-deploy-1.2.2-0.noarch >> >> --> Processing Dependency: or for package: ceph-deploy-1.2.2-0.noarch >> >> ---> Package libicu.x86_64 0:50.1.2-9.fc19 will be installed >> >> --> Finished Dependency Resolution >> >> Error: Package: ceph-deploy-1.2.2-0.noarch (ceph-extras-noarch) >> >>Requires: python-pushy >= 0.5.3 >> >>Available: python-pushy-0.5.1-6.1.noarch >> (ceph-extras-noarch) >> >>python-pushy = 0.5.1-6.1 >> >> Error: Package: ceph-deploy-1.2.2-0.noarch (ceph-extras-noarch) >> >>Requires: or >> >> You could try using --skip-broken to work around the problem >> >> You could try running: rpm -Va --nofiles -nodigest >> >> >> >> To work around this I tried to use pip to install pushy 0.5.3: >> >> [root@ceph02 pushy-master]# pip install pushy >> >> Downloading/unpacking pushy >> >> Downloading pushy-0.5.3.zip (48kB): 48kB downloaded >> >> Running setup.py egg_info for package pushy >> >> >> >> Installing collected packages: pushy >> >> Running setup.py install for pushy >> >> >> >> Successfully installed pushy >> >> Cleaning up... >> >> >> >> Verifying: >> >> >> >> [root@ceph02 ~]# pip list | grep pushy >> >> pushy (0.5.3) >> >
Re: [ceph-users] ceph-deploy 1.2.2 vs fedora 19
I mean, I have OSD's and MON's running now, but I see no mention of them in the current config file (/etc/ceph/ceph.conf) so backing that file up would not allow me to see where monitors/objectstores/journals where placed. Is there a nifty command that allows me to push these defaults to something that can be used as a config file that allows me to see how it was setup once I am done with my tests? I want to be able to do performance tests that that I can attach to individual configs which allows me to revert to the best config found for the ultimate performance once I have it tuned and compared the data? -Original Message- From: Alfredo Deza [mailto:alfredo.d...@inktank.com] Sent: Thursday, December 05, 2013 3:34 PM To: Jonas Andersson Cc: ceph-us...@ceph.com Subject: Re: [ceph-users] ceph-deploy 1.2.2 vs fedora 19 On Thu, Dec 5, 2013 at 9:18 AM, Jonas Andersson wrote: > Perfect, that worked very well. Thanks a lot. > > Another question: > > Using http://ceph.com/howto/deploying-ceph-with-ceph-deploy/ as a guide to > set up my test-cluster I now have a working cluster with 12 osd's in and up. > I've create a client, a 10gb rbd volume, mounted it, written data all good. > > Looking at my ceph.conf it seems it's using all defaults: > [root@ceph02 ~]# cat /etc/ceph/ceph.conf [global] fsid = > f865694c-7a50-46a9-9550-f6b160c00313 > mon_initial_members = ceph02, ceph03, ceph04 mon_host = > 10.130.21.33,10.130.21.34,10.130.21.42 > auth_supported = cephx > osd_journal_size = 1024 > filestore_xattr_use_omap = true > > Is there any way to dump the default running config to the config-file so I > can start tinkering around? What do you mean by dump? You can backup that copy which is the one that ceph-deploy uses and use a new one and push it to your nodes and try that way "ceph --show-config" seems to show all parameters running, but I don't see any mentioning of the monitors/osd's at all? > > Last question: > I had a too low number of pgs, which caused a health warning. Since I type a > lot faster than I think sometimes I adjusted the value to 128 (ceph osd pool > set rbd pg_num 128), which turned out to be too high, but I cant seem to be > able to tune it down again. How do I achieve this? > > Thanks a lot in advance! > > Kind regards > > Jonas > > -Original Message- > From: Alfredo Deza [mailto:alfredo.d...@inktank.com] > Sent: Sunday, December 01, 2013 6:30 PM > To: Jonas Andersson > Cc: ceph-us...@ceph.com > Subject: Re: [ceph-users] ceph-deploy 1.2.2 vs fedora 19 > > On Thu, Nov 28, 2013 at 8:25 AM, Jonas Andersson > wrote: >> Hi all, >> >> >> >> I am seeing some weirdness when trying to deploy Ceph Emperor on >> fedora 19 using ceph-deploy. Problem occurs when trying to install >> ceph-deploy, and seems to point to the version of pushy in your repository: >> >> > > Since ceph-deploy version 1.3 there is no longer a requirement on > pushy. You should update to the latest version (currently at 1.3.3) > >> >> [root@ceph02 ~]# yum install ceph-deploy >> >> Loaded plugins: priorities, protectbase >> >> imc-default >> | 1.1 kB 00:00:00 >> >> imc-shared >> | 1.1 kB 00:00:00 >> >> imc-systemimages >> | 1.1 kB 00:00:00 >> >> imc-systemimages-shared >> | 1.1 kB 00:00:00 >> >> 45 packages excluded due to repository priority protections >> >> 0 packages excluded due to repository protections >> >> Resolving Dependencies >> >> --> Running transaction check >> >> ---> Package ceph-deploy.noarch 0:1.2.2-0 will be installed >> >> --> Processing Dependency: python-pushy >= 0.5.3 for package: >> ceph-deploy-1.2.2-0.noarch >> >> --> Processing Dependency: pushy >= 0.5.3 for package: >> ceph-deploy-1.2.2-0.noarch >> >> --> Processing Dependency: or for package: ceph-deploy-1.2.2-0.noarch >> >> --> Processing Dependency: gdisk for package: >> --> ceph-deploy-1.2.2-0.noarch >> >> --> Running transaction check >> >> ---> Package ceph-deploy.noarch 0:1.2.2-0 will be installed >> >> --> Processing Dependency: python-pushy >= 0.5.3 for package: >> ceph-deploy-1.2.2-0.noarch >> >> --> Processing Dependency: or for package: ceph-deploy-1.2.2-0.noarch >> >> ---> Package gdisk.x86_64 0:0.8.8-1.fc19 will be installed >> >> --> Processing Dependency: libicuuc.so.50()(64bit) for package: >> gdisk-0.8.8-1.fc19.x86_64 >> >> --> Processing Dependency: libicuio.so.50()(64bit) for package: >> gdisk-0.8.8-1.fc19.x86_64 >> >> ---> Package pushy.noarch 0:0.5.3-1 will be installed >> >> --> Running transaction check >> >> ---> Package ceph-deploy.noarch 0:1.2.2-0 will be installed >> >> --> Processing Dependency: python-pushy >= 0.5.3 for package: >> ceph-deploy-1.2.2-0.noarch >> >> --> Processing Dependency: or for package: ceph-deploy-1.2.2-0.noarch >> >> ---> Package libicu.x86_64 0:50.1.2-9.fc19 will be installed >> >> --> Finished Dependency Resolution >> >> Error: Package: ceph-deploy-1.2.2-0.noarch (ceph-extras-noarch) >> >>Requires: python-pushy >= 0.5.3 >> >>Available: pytho
Re: [ceph-users] Journal, SSD and OS
Another option is to run journals on individually presented SSDs, in a 5:1 ratio (spinning-disk:ssd) and have the OS somewhere else. Then the failure domain is smaller. Ideally implement some way to monitor SSD write life SMART data - at least it gives a guide as to device condition compared to its rated life. That can be done with smartmontools, but it would be nice to have it on the InkTank dashboard for example. On 2013-12-05 14:26, Sebastien Han wrote: Hi guys, I won’t do a RAID 1 with SSDs since they both write the same data. Thus, they are more likely to “almost” die at the same time. What I will try to do instead is to use both disk in JBOD mode or (degraded RAID0). Then I will create a tiny root partition for the OS. Then I’ll still have something like /dev/sda2 and /dev/sdb2 and then I can take advantage of the 2 disks independently. The good thing with that is that you can balance your journals across both SSDs. From a performance perspective this is really good. The bad thing as always is that if you loose a SSD you loose all the journals attached to it. Cheers. Sébastien Han Cloud Engineer "Always give 100%. Unless you're giving blood.” Phone: +33 (0)1 49 70 99 72 Mail: sebastien@enovance.com Address : 10, rue de la Victoire - 75009 Paris Web : www.enovance.com - Twitter : @enovance On 05 Dec 2013, at 10:53, Gandalf Corvotempesta wrote: 2013/12/4 Simon Leinen : I think this is a fine configuration - you won't be writing to the root partition too much, outside journals. We also put journals on the same SSDs as root partitions (not that we're very ambitious about performance...). Do you suggest a RAID1 for the OS partitions on SSDs ? Is this safe or a RAID1 will decrease SSD life? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph User Committee Formal Announcement Format
Hi Nathan, Here is a very rough draft of the announcement which is going to be released next monday. It is more a discussion starter than a draft. Feel free to modify at will :-) It includes the names and affiliations of all founding members. There may be more in the days to come and I'll add to it when I receive new applications: http://pad.ceph.com/p/user-committee-announce It basically is a re-iteration of what has been said during the past few weeks. I added two sentences about the scope, in a attempt to say that it is not just about companies but also academics, individuals and non profit ( there are no governmental agencies yet). And that it's not just technical and that the legal environment in which Ceph can prosper is something we should also care about (not just software patents but also the endless amendments to copyright law that may be detrimental to Free Software in general ). Not being a native english speaker it's difficult to get it right ;-) As for the personalized version of the announcement for each founding member, I would love to have one to remember this date. The graphics used http://www.slideshare.net/Inktank_Ceph/erasure-codeceph are under a Free Software license and you're welcome to use them if you want. I can send you high resolution versions. Cheers On 02/12/2013 15:52, Regola, Nathan (Contractor) wrote: > I'm looking forward to working with everyone involved with the Ceph User > Committee > (http://wiki.ceph.com/01Planning/02Blueprints/Firefly/Ceph_User_Committee#D > etailed_Description). I believe that all of the members of the Ceph User > Committee should have received an email from Loic asking them to confirm > their organization's interest in being named a founding member. The formal > announcement is currently being planned for 10 December and we are working > on drafting it. > > Would members prefer a single general announcement or a personalized > announcement? A personalized announcement would probably be something like > an automatically generated PDF file containing a letter (with the member's > name/affiliation) so that members could distribute it. We are open to > suggestions. If you have a preference for a general announcement listing > all of the members or a personalized announcement welcoming the user > (which obviously could include a list of all members), please reply. > > Best Regards, > Nate Regola > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -- Loïc Dachary, Artisan Logiciel Libre signature.asc Description: OpenPGP digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable
On 12/05/2013 09:16 AM, Jan Kalcic wrote: It seems ceph-mon does not exit with success, in fact: ceph-node1 # sudo /usr/bin/ceph-mon -i ceph-node1 --pid-file /var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d 2013-12-05 10:06:27.429602 7fe06baf9780 0 ceph version 0.72.1 (4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 35662 IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable 2013-12-05 10:06:27.436295 7fe06baf9780 -1 failed to create new leveldb store ceph@ceph-node1:/var/run/ceph> sudo /usr/bin/ceph-mon -i ceph-node1 --pid-file /var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable 2013-12-05 10:06:43.961756 7fd71d27c780 0 ceph version 0.72.1 (4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 36216 2013-12-05 10:06:43.964079 7fd71d27c780 -1 failed to create new leveldb store This is oftentime a symptom of another monitor running with the same id, and the monitor you're trying to start is unable to acquire the lock. Make sure /var/lib/ceph/mon/ceph-ceph-node1 exists and is a directory. Make sure you don't have any other monitor with id 'ceph-node1' running. If this error persists, set 'debug mon = 10' on ceph.conf, restart the monitor and send us the resulting log. -Joao Thanks, Jan ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Joao Eduardo Luis Software Engineer | http://inktank.com | http://ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] aborted downloads from Radosgw when multiple clients access same object
We are investigating a curious problem with radosgw: We see intermittent timeouts and http connections breaking when streaming video files through the rados gateway. On server 1 we have Ubuntu 13.10 (saucy) with the stock Apache 2.4 and associated fastcgi (and a mon) On server 2 we also have Ubuntu 13.10 but installed the ceph patched Apache 2.2 with fastcgi (and a mon) We can now do the following: wget a 700 MB file from Server 2 ( wget --no-check-certificate https://staging.server2.example.ch/2f/e4491dbfa00c328828bbbc2c8d128a/test2.mp4 -O /dev/null ) Stream the same file trough Chrome from server 1: -> the wget process is interrupted: 2013-12-05 16:32:18 (364 KB/s) - Connection closed at byte 172487549. Retrying. the apache error log shows this: [Thu Dec 05 16:32:50 2013] [error] [client 2001:6502] FastCGI: comm with server "/var/www/s3gw.fcgi" aborted: idle timeout (30 sec) [Thu Dec 05 16:32:50 2013] [error] [client 2001:xxx:6502] FastCGI: incomplete headers (0 bytes) received from server "/var/www/s3gw.fcgi" [Thu Dec 05 16:55:54 2013] [error] [client 123.123.123.123] FastCGI: comm with server "/var/www/s3gw.fcgi" aborted: idle timeout (30 sec) [Thu Dec 05 16:55:54 2013] [error] [client 123.123.123.123] Handler for fastcgi-script returned invalid result code 1 [Thu Dec 05 16:56:25 2013] [error] [client 123.123.123.123] FastCGI: comm with server "/var/www/s3gw.fcgi" aborted: idle timeout (30 sec) [Thu Dec 05 16:56:25 2013] [error] [client 123.123.123.123] FastCGI: incomplete headers (0 bytes) received from server "/var/www/s3gw.fcgi" radosgw log shows this (from another test, that's why timestamps don't match) 2013-12-05 17:14:02.675911 7f2b9cfeb700 0 ERROR: s->cio->print() returned err=-1 2013-12-05 17:14:02.675942 7f2b9cfeb700 0 ERROR: s->cio->print() returned err=-1 2013-12-05 17:14:02.675946 7f2b9cfeb700 0 ERROR: s->cio->print() returned err=-1 2013-12-05 17:14:02.675948 7f2b9cfeb700 0 ERROR: s->cio->print() returned err=-1 2013-12-05 17:14:02.675973 7f2b9cfeb700 2 req 5:8.176355:s3:GET /2f/e4491dbfa00c328828bbbc2c8d128a/test2.mp4:get_obj:http status=403 2013-12-05 17:14:02.675971 7f2b597fa700 20 rados->get_obj_iterate_cb oid=default.40804.6__shadow__W8D84-M3taNmGJG4UCxDxbmNDJqubhP_5 obj-ofs=17301504 read_ofs=0 len=4194304 2013-12-05 17:14:02.676068 7f2b597fa700 20 rados->aio_operate r=0 bl.length=0 2013-12-05 17:14:02.676299 7f2b9cfeb700 1 == req done req=0x1d60d40 http_status=403 == 2013-12-05 17:14:08.909528 7f2bbcbc0700 2 RGWDataChangesLog::ChangesRenewThread: start 2013-12-05 17:14:30.909803 7f2bbcbc0700 2 RGWDataChangesLog::ChangesRenewThread: start 2013-12-05 17:14:32.321141 7f2bbd3c1700 20 get_obj_aio_completion_cb: io completion ofs=13107200 len=4194304 2013-12-05 17:14:32.321176 7f2bbd3c1700 20 get_obj_aio_completion_cb: io completion ofs=8912896 len=4194304 2013-12-05 17:14:32.321185 7f2bbd3c1700 20 get_obj_aio_completion_cb: io completion ofs=4718592 len=4194304 2013-12-05 17:14:32.321188 7f2bbd3c1700 20 get_obj_aio_completion_cb: io completion ofs=8912896 len=4194304 2013-12-05 17:14:32.321194 7f2bbd3c1700 20 get_obj_aio_completion_cb: io completion ofs=138936320 len=4194304 2013-12-05 17:14:32.321248 7f2b597fa700 20 rados->get_obj_iterate_cb oid=default.40804.6__shadow__W8D84-M3taNmGJG4UCxDxbmNDJqubhP_6 obj-ofs=21495808 read_ofs=0 len=4194304 2013-12-05 17:14:32.321281 7f2b22fad700 20 rados->get_obj_iterate_cb oid=default.40804.6__shadow__W8D84-M3taNmGJG4UCxDxbmNDJqubhP_38 obj-ofs=155713536 read_ofs=0 len=4194304 2013-12-05 17:14:32.321383 7f2b597fa700 20 rados->aio_operate r=0 bl.length=0 2013-12-05 17:14:32.321436 7f2b597fa700 20 rados->get_obj_iterate_cb oid=default.40804.6__shadow__W8D84-M3taNmGJG4UCxDxbmNDJqubhP_7 obj-ofs=25690112 read_ofs=0 len=4194304 2013-12-05 17:14:32.321530 7f2b22fad700 20 rados->aio_operate r=0 bl.length=0 2013-12-05 17:14:32.321672 7f2b597fa700 20 rados->aio_operate r=0 bl.length=0 2013-12-05 17:14:32.321690 7f2b597fa700 20 rados->get_obj_iterate_cb oid=default.40804.6__shadow__W8D84-M3taNmGJG4UCxDxbmNDJqubhP_8 obj-ofs=29884416 read_ofs=0 len=4194304 2013-12-05 17:14:32.321779 7f2b597fa700 20 rados->aio_operate r=0 bl.length=0 2013-12-05 17:14:35.635836 7f2bbd3c1700 20 get_obj_aio_completion_cb: io completion ofs=147324928 len=4194304 2013-12-05 17:14:35.635881 7f2bbd3c1700 20 get_obj_aio_completion_cb: io completion ofs=143130624 len=4194304 2013-12-05 17:14:35.635886 7f2bbd3c1700 20 get_obj_aio_completion_cb: io completion ofs=17301504 len=4194304 2013-12-05 17:14:35.635961 7f2b22fad700 20 rados->get_obj_iterate_cb oid=default.40804.6__shadow__W8D84-M3taNmGJG4UCxDxbmNDJqubhP_39 obj-ofs=159907840 read_ofs=0 len=4194304 2013-12-05 17:14:35.636133 7f2b22fad700 20 rados->aio_operate r=0 bl.length=0 2013-12-05 17:14:35.636157 7f2b22fad700 20 rados->get_obj_iterate_cb oid=default.40804.6__shadow__W8D84-M3taNmGJG4UCxDxbmNDJqubhP_40 obj-ofs=164102144 read_ofs=0 len=4194304 2013-12-05 17:14:
Re: [ceph-users] HEALTH_WARN pool .rgw.buckets has too few pgs
Ah. So that warning compares the objects per pg in that pool vs the objects per pg in the entire system, and if there is too much of a skew, issues a warning. If you look at 'ceph health detail' you will see some of the detail there. The reason you're seeing this is because you have lots and lots of pgs for the rgw pools that typically have almost no objects (.rgw.control, .rgw.root, .rgw.gc, .users, .users.email, possibly others). Generally speaking, it is only the .rgw.buckets pool that needs to get large (that is where most of the data goes). The pg_num value for others can be turned up as needed (see 'ceph df' to see where the objects are). It's not possible to decrease pg_num on a pool right now. For some of these pools, though, it is possible to create a new pool, copy the data, remove the old one, and rename the new one (while the radosgw daemons are stopped). The 'rados' tool has a copy function that can handle this, IIRC. You can also adjust the max skew to make this warning go away and change nothing (mon pg warn max object skew), but the large number of PGs will have a somewhat negative impact on cluster performance... sage On Thu, 5 Dec 2013, Alexis G?NST HORN wrote: > Hello, > > Here it is : > http://pastie.org/private/u5yut673fv6csobuvain9g > > > Thanks a lot for your help > > Best Regards - Cordialement > > Alexis G?NST HORN, > Tel : 0826.206.307 (poste ) > Fax : +33.1.83.62.92.89 > > > > IMPORTANT: The information contained in this message may be privileged > and confidential and protected from disclosure. If the reader of this > message is not the intended recipient, or an employee or agent > responsible for delivering this message to the intended recipient, you > are hereby notified that any dissemination, distribution or copying of > this communication is strictly prohibited. If you have received this > communication in error, please notify us immediately by replying to > the message and deleting it from your computer. > > > > 2013/12/4 Sage Weil : > > Hi, > > > > Can you attach the output from 'ceph osd dump'? > > > > Thanks! > > sage > > > > > > > > "Alexis G?NST HORN" wrote: > >> > >> Hello, > >> > >> I can't understand an error I have since now : > >> > >> HEALTH_WARN pool .rgw.buckets has too few pgs. > >> Do you have any ideas ? > >> > >> Some info : > >> > >> [root@admin ~]# ceph --version > >> ceph version 0.72.1 (4d923861868f6a15dcb33fef7f50f674997322de) > >> > >> [root@admin ~]# ceph osd pool get .rgw.buckets pgp_num > >> pgp_num: 10050 > >> > >> [root@admin ~]# ceph osd pool get .rgw.buckets pg_num > >> pg_num: 10050 > >> > >> [root@admin ~]# ceph -s > >> (...) > >> osdmap e30632: 201 osds: 201 up, 201 in > >> pgmap v4984359: 90666 pgs, 13 pools, 1276 GB data, 340 kobjects > >> 3916 GB used, 727 TB / 731 TB avail > >> 90666 active+clean > >> > >> > >> Thanks a lot, > >> Alexis > >> > >> > >> ceph-users mailing list > >> ceph-users@lists.ceph.com > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > > > > -- > > Sent from Kaiten Mail. Please excuse my brevity. > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] pgs incomplete
ceph pg 4.7d query will tell you which OSDs it wants to talk to in order to make the PG complete (or what other information it needs). sag On Thu, 5 Dec 2013, Rzk wrote: > Hi All, > > I found 6 pgs incomplete while "ceph health detail" after 3 osds down, > but after i manage to start again all 3 osds, only 1 left incomplete pg. > > root@:~# ceph health detail | grep 4.7d > pg 4.7d is stuck inactive for 306404.577611, current state incomplete, last > acting [6,0] > pg 4.7d is stuck unclean for 306404.577947, current state incomplete, last > acting [6,0] > pg 4.7d is incomplete, acting [6,0] > > does anyone know how to fix/repair this "incomplete" state ? > i have tried to add more osd, done almost all suggestion i found in google, > but no luck. > > and also i can't start any instance in the openstack. > > Thanks, > RzK > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] btrfs constant background write activity even at idle
On Thu, 5 Dec 2013, James Harper wrote: > > > > Can you generate an OSD log with 'debug filestore = 20' for an idle period? > > > > Any more tests you would like me to run? I'm going to recreate that osd > as xfs soon. Ah, Ilya tells me that the brfs cleaner is probably chewing on a snapshot cleanup backlog. This is something that other btrfs users see pretty frequently. I don't know enough about the btrfs internals to know how harmful this is to performance, though.. sage ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph-deploy 1.2.2 vs fedora 19
Jonas, You can query the admin sockets of your monitors and osds get a json listing of their running configuration. The command will look something like: # ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show # ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show You can then inject new settings to running daemons with injectargs: # ceph tell osd.* injectargs '--osd_max_backfills 10' Or, your can add those to ceph.conf and restart the daemons. Cheers, Mike Dawson On 12/5/2013 9:54 AM, Jonas Andersson wrote: I mean, I have OSD's and MON's running now, but I see no mention of them in the current config file (/etc/ceph/ceph.conf) so backing that file up would not allow me to see where monitors/objectstores/journals where placed. Is there a nifty command that allows me to push these defaults to something that can be used as a config file that allows me to see how it was setup once I am done with my tests? I want to be able to do performance tests that that I can attach to individual configs which allows me to revert to the best config found for the ultimate performance once I have it tuned and compared the data? -Original Message- From: Alfredo Deza [mailto:alfredo.d...@inktank.com] Sent: Thursday, December 05, 2013 3:34 PM To: Jonas Andersson Cc: ceph-us...@ceph.com Subject: Re: [ceph-users] ceph-deploy 1.2.2 vs fedora 19 On Thu, Dec 5, 2013 at 9:18 AM, Jonas Andersson wrote: Perfect, that worked very well. Thanks a lot. Another question: Using http://ceph.com/howto/deploying-ceph-with-ceph-deploy/ as a guide to set up my test-cluster I now have a working cluster with 12 osd's in and up. I've create a client, a 10gb rbd volume, mounted it, written data all good. Looking at my ceph.conf it seems it's using all defaults: [root@ceph02 ~]# cat /etc/ceph/ceph.conf [global] fsid = f865694c-7a50-46a9-9550-f6b160c00313 mon_initial_members = ceph02, ceph03, ceph04 mon_host = 10.130.21.33,10.130.21.34,10.130.21.42 auth_supported = cephx osd_journal_size = 1024 filestore_xattr_use_omap = true Is there any way to dump the default running config to the config-file so I can start tinkering around? What do you mean by dump? You can backup that copy which is the one that ceph-deploy uses and use a new one and push it to your nodes and try that way "ceph --show-config" seems to show all parameters running, but I don't see any mentioning of the monitors/osd's at all? Last question: I had a too low number of pgs, which caused a health warning. Since I type a lot faster than I think sometimes I adjusted the value to 128 (ceph osd pool set rbd pg_num 128), which turned out to be too high, but I cant seem to be able to tune it down again. How do I achieve this? Thanks a lot in advance! Kind regards Jonas -Original Message- From: Alfredo Deza [mailto:alfredo.d...@inktank.com] Sent: Sunday, December 01, 2013 6:30 PM To: Jonas Andersson Cc: ceph-us...@ceph.com Subject: Re: [ceph-users] ceph-deploy 1.2.2 vs fedora 19 On Thu, Nov 28, 2013 at 8:25 AM, Jonas Andersson wrote: Hi all, I am seeing some weirdness when trying to deploy Ceph Emperor on fedora 19 using ceph-deploy. Problem occurs when trying to install ceph-deploy, and seems to point to the version of pushy in your repository: Since ceph-deploy version 1.3 there is no longer a requirement on pushy. You should update to the latest version (currently at 1.3.3) [root@ceph02 ~]# yum install ceph-deploy Loaded plugins: priorities, protectbase imc-default | 1.1 kB 00:00:00 imc-shared | 1.1 kB 00:00:00 imc-systemimages | 1.1 kB 00:00:00 imc-systemimages-shared | 1.1 kB 00:00:00 45 packages excluded due to repository priority protections 0 packages excluded due to repository protections Resolving Dependencies --> Running transaction check ---> Package ceph-deploy.noarch 0:1.2.2-0 will be installed --> Processing Dependency: python-pushy >= 0.5.3 for package: ceph-deploy-1.2.2-0.noarch --> Processing Dependency: pushy >= 0.5.3 for package: ceph-deploy-1.2.2-0.noarch --> Processing Dependency: or for package: ceph-deploy-1.2.2-0.noarch --> Processing Dependency: gdisk for package: --> ceph-deploy-1.2.2-0.noarch --> Running transaction check ---> Package ceph-deploy.noarch 0:1.2.2-0 will be installed --> Processing Dependency: python-pushy >= 0.5.3 for package: ceph-deploy-1.2.2-0.noarch --> Processing Dependency: or for package: ceph-deploy-1.2.2-0.noarch ---> Package gdisk.x86_64 0:0.8.8-1.fc19 will be installed --> Processing Dependency: libicuuc.so.50()(64bit) for package: gdisk-0.8.8-1.fc19.x86_64 --> Processing Dependency: libicuio.so.50()(64bit) for package: gdisk-0.8.8-1.fc19.x86_64 ---> Package pushy.noarch 0:0.5.3-1 will be installed --> Running transaction check ---> Package ceph-deploy.noarch 0:1.2.2-0 will be installed --> Processing Dependency: python-pushy >= 0.5.3 for package: ceph-deploy-1.2.2-0.noarch --> Processing Dependency: or for package: cep
Re: [ceph-users] Error in starting ceph
Hi,
Thank you for quick reply.
ems is a server from where I ran service ceph start.
These are the steps followed. Please let me know if have anything is
missing or something is wrong.
wget -q -O- '
https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' | sudo
apt-key add -
echo deb http://ceph.com/packages/ceph-extras/debian $(lsb_release -sc)
main | sudo tee /etc/apt/sources.list.d/ceph-extras.list
sudo apt-add-repository 'deb http://ceph.com/debian-emperor/ precise main'
sudo apt-get update && sudo apt-get install ceph
update /etc/ceph/ceph.conf:
[global]
# For version 0.54 and earlier, you may enable
# authentication with the following setting.
# Specifying `cephx` enables authentication;
# and specifying `none` disables authentication.
#auth supported = cephx
# For version 0.55 and beyond, you must explicitly enable
# or disable authentication with "auth" entries in [global].
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
[osd]
osd journal size = 1000
# uncomment the following line if you are mounting with ext4
filestore xattr use omap = true
# For Bobtail (v 0.56) and subsequent versions, you may
# add settings for mkcephfs so that it will create and mount
# the file system for you. Remove the comment `#` character for
# the following settings and replace the values in parenthesis
# with appropriate values, or leave the following settings commented
# out to accept the default values. You must specify the --mkfs
# option with mkcephfs in order for the deployment script to
# utilize the following settings, and you must define the 'devs'
# option for each osd instance; see below.
osd mkfs type = ext4
#osd mkfs options {fs-type} = {mkfs options} # default for xfs is "-f"
osd mount options ext4 = user_xattr,rw,noexec,nodev,noatime,nodiratime
# default mount option is "rw, noatime"
[mon.a]
host = ems
mon addr = :6789
[osd.0]
host = ems
devs = /dev/sdb1
[osd.1]
host = ems
devs = /dev/sdb2
[mds.a]
host = ems
#devs = {path-to-device}
~
Copied the configuration file to /etc/ceph/ceph.conf on client host.
Set the permissions 644 on client machine
On Ceph server host, created directory for each daemon:
mkdir -p /var/lib/ceph/osd/ceph-0
mkdir -p /var/lib/ceph/osd/ceph-1
mkdir -p /var/lib/ceph/mon/ceph-a
mkdir -p /var/lib/ceph/mds/ceph-a
~ Executed the following on the Ceph server host:
cd /etc/ceph
mkcephfs -a -c /etc/ceph/ceph.conf -k ceph.keyring --mkfs
service ceph -a start
Got error:
service ceph -a start
=== mon.a ===
Starting Ceph mon.a on ems...already running
=== mds.a ===
Starting Ceph mds.a on ems...already running
=== osd.0 ===
Mounting ext4 on ems:/var/lib/ceph/osd/ceph-0
Error ENOENT: osd.0 does not exist. create it before updating the crush map
failed: 'timeout 10 /usr/bin/ceph --name=osd.0
--keyring=/var/lib/ceph/osd/ceph-0/keyring
osd crush create-or-move -- 0
0.10root=default
host=ems'
actually this was warning in dumpling version. So started processes
manually.
[root@ip-10-68-107-28 ceph]# ps -eaf | grep ceph
root 16130 1 0 11:19 pts/100:00:01 /usr/bin/ceph-mon -i a
--pid-file /var/run/ceph/mon.a.pid -c /etc/ceph/ceph.conf
root 16232 1 0 11:19 ?00:00:00 /usr/bin/ceph-mds -i a
--pid-file /var/run/ceph/mds.a.pid -c /etc/ceph/ceph.conf
root 16367 1 0 11:19 ?00:00:07 /usr/bin/ceph-osd -i 0
--pid-file /var/run/ceph/osd.0.pid -c /etc/ceph/ceph.conf
root 16531 1 0 11:19 ?00:00:05 /usr/bin/ceph-osd -i 1
--pid-file /var/run/ceph/osd.1.pid -c /etc/ceph/ceph.conf
root 16722 15658 0 11:46 pts/100:00:00 grep ceph
output of mount :
/dev/sdb1 on /var/lib/ceph/osd/ceph-0 type ext4
(rw,noexec,nodev,noatime,nodiratime,user_xattr)
/dev/sdb2 on /var/lib/ceph/osd/ceph-1 type ext4
(rw,noexec,nodev,noatime,nodiratime,user_xattr)
Thanks,
Sahana
On Thu, Dec 5, 2013 at 6:17 PM, Li Wang wrote:
> ems is a remote machine?
> Did you set up the corresponding directories: /var/lib/ceph/osd/ceph-0,
> and called mkcephfs before?
> You can also try starting osd manually by 'ceph-osd -i 0 -c
> /etc/ceph/ceph.conf', then 'pgrep ceph-osd' to see if they are there, then
> 'ceph -s' to check the health.
>
>
> On 2013/12/5 18:26, Sahana wrote:
>
>> Installed ceph-emperor using apt-get in ubuntu 12.04 by following the
>> steps given in installation part of ceph-doc website.
>>
>> http://ceph.com/docs/master/install/get-packages/
>>
>> http://ceph.com/docs/master/install/install-storage-cluster/
>>
>> But get error when this command is run :
>>
>>
>> service ceph -a start
>>
>> service ceph -a start
>> === mon.a ===
>> Starting Ceph
Re: [ceph-users] IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource temporarily unavailable
Hi Joao,
On 12/05/2013 04:29 PM, Joao Eduardo Luis wrote:
On 12/05/2013 09:16 AM, Jan Kalcic wrote:
It seems ceph-mon does not exit with success, in fact:
ceph-node1 # sudo /usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
2013-12-05 10:06:27.429602 7fe06baf9780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 35662
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:27.436295 7fe06baf9780 -1 failed to create new leveldb
store
ceph@ceph-node1:/var/run/ceph> sudo /usr/bin/ceph-mon -i ceph-node1
--pid-file /var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf -d
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
IO error: lock /var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK: Resource
temporarily unavailable
2013-12-05 10:06:43.961756 7fd71d27c780 0 ceph version 0.72.1
(4d923861868f6a15dcb33fef7f50f674997322de), process ceph-mon, pid 36216
2013-12-05 10:06:43.964079 7fd71d27c780 -1 failed to create new leveldb
store
This is oftentime a symptom of another monitor running with the same
id, and the monitor you're trying to start is unable to acquire the lock.
For example the following (3931, 3933) ??
# ps aux | grep ceph
root 3303 0.0 0.7 82184 3564 ?Ss 18:08 0:00 sshd:
ceph [priv]
ceph 3306 0.0 0.4 82312 2164 ?S18:08 0:00 sshd:
ceph@pts/0
ceph 3307 0.0 0.5 14004 2952 pts/0Ss 18:08 0:00 -bash
root 3835 0.0 0.3 11648 1680 ?S18:30 0:00 /bin/sh
/usr/sbin/rcceph -c /etc/ceph/ceph.conf start mon.ceph-node1
root 3930 0.0 0.2 11512 1312 ?S18:30 0:00 bash -c
ulimit -n 32768; /usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf
root 3931 0.0 1.0 76612 5328 ?S18:30 0:00
/usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf
root 3933 0.0 1.0 101688 5096 ?Sl 18:30 0:00
/usr/bin/ceph-mon -i ceph-node1 --pid-file
/var/run/ceph/mon.ceph-node1.pid -c /etc/ceph/ceph.conf
root 3941 0.0 1.7 69468 8796 ?S18:30 0:00 python
/usr/bin/ceph --cluster=ceph --admin-daemon
/var/run/ceph/ceph-mon.ceph-node1.asok mon_status
ceph 3964 0.0 0.1 4940 968 pts/0R+ 18:37 0:00 ps aux
ceph 3965 0.0 0.1 5712 820 pts/0R+ 18:37 0:00 grep ceph
but I am not doing tricky things, just following basic steps in the doc
and running ceph-deploy as indicated.. what's wrong with ceph-deploy then??
Make sure /var/lib/ceph/mon/ceph-ceph-node1 exists and is a directory.
Make sure you don't have any other monitor with id 'ceph-node1'
running. If this error persists, set 'debug mon = 10' on ceph.conf,
restart the monitor and send us the resulting log.
# find /var/lib/ceph/mon/ceph-ceph-node1
/var/lib/ceph/mon/ceph-ceph-node1
/var/lib/ceph/mon/ceph-ceph-node1/sysvinit
/var/lib/ceph/mon/ceph-ceph-node1/store.db
/var/lib/ceph/mon/ceph-ceph-node1/store.db/LOG
/var/lib/ceph/mon/ceph-ceph-node1/store.db/CURRENT
/var/lib/ceph/mon/ceph-ceph-node1/store.db/06.log
/var/lib/ceph/mon/ceph-ceph-node1/store.db/05.ldb
/var/lib/ceph/mon/ceph-ceph-node1/store.db/MANIFEST-04
/var/lib/ceph/mon/ceph-ceph-node1/store.db/LOCK
/var/lib/ceph/mon/ceph-ceph-node1/done
/var/lib/ceph/mon/ceph-ceph-node1/keyring
# cat ceph.conf
[global]
filestore_xattr_use_omap = true
mon_host = 172.17.43.11
fsid = 18203301-f70b-40f1-b93e-fde4731f3bee
mon_initial_members = ceph-node1
auth_supported = cephx
osd_journal_size = 1024
debug mon = 10
...this is actually on ceph-admin but I have verified it is deployed
afterward to ceph-node1 (by running --overwrite-conf)
# ceph-deploy --overwrite-conf mon create ceph-node1
[ceph_deploy.cli][INFO ] Invoked (1.3.3): /usr/bin/ceph-deploy
--overwrite-conf mon create ceph-node1
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-node1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-node1 ...
[ceph-node1][DEBUG ] connected to host: ceph-node1
[ceph-node1][DEBUG ] detect platform information from remote host
[ceph-node1][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: SUSE Linux Enterprise Server 11
x86_64
[ceph-node1][DEBUG ] determining if provided host has same hostname in
remote
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] deploying mon to ceph-node1
[ceph-node1][DEBUG ] get remote short hostname
[ceph-node1][DEBUG ] remote hostname: ceph-node1
[ceph-node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-node1][DEBUG ] create the mon path if it does not exist
[ceph-node1][DEBUG ] checking for done path:
/var/lib/ceph/mon/ceph-ceph-node
[ceph-users] Ceph User Committee List
Hey all, For those who have been following, or are interested in, the Ceph User Committee [0] discussed at CDS [1] there is now a mailing list to discuss all things User Committee. This could include: * Proposed changes to Ceph.com * Event participation and coverage * Community development logistics * Mailing List & IRC moderation [0] http://wiki.ceph.com/01Planning/02Blueprints/Firefly/Ceph_User_Committee [1] http://pad.ceph.com/p/user-committee-announce If you have any questions please let me know. Thanks. Best Regards, Patrick McGarry Director, Community || Inktank http://ceph.com || http://inktank.com @scuttlemonkey || @ceph || @inktank ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph User Committee List
Suppose I should have mentioned, as with the other mailing lists you can find the info to subscribe at: http://lists.ceph.com/listinfo.cgi/ceph-community-ceph.com and mail can be sent to the list at: ceph-commun...@lists.ceph.com Best Regards, Patrick McGarry Director, Community || Inktank http://ceph.com || http://inktank.com @scuttlemonkey || @ceph || @inktank On Thu, Dec 5, 2013 at 3:12 PM, Patrick McGarry wrote: > Hey all, > > For those who have been following, or are interested in, the Ceph User > Committee [0] discussed at CDS [1] there is now a mailing list to > discuss all things User Committee. This could include: > > * Proposed changes to Ceph.com > * Event participation and coverage > * Community development logistics > * Mailing List & IRC moderation > > [0] http://wiki.ceph.com/01Planning/02Blueprints/Firefly/Ceph_User_Committee > > [1] http://pad.ceph.com/p/user-committee-announce > > If you have any questions please let me know. Thanks. > > > Best Regards, > > Patrick McGarry > Director, Community || Inktank > http://ceph.com || http://inktank.com > @scuttlemonkey || @ceph || @inktank ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Mounting Ceph on Linux/Windows
Can someone point me to directions on how to mount a Ceph storage volume on Linux as well as Windows? Thanks in advance for your help. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] centos6.4 + libvirt + qemu + rbd/ceph
I've been working on getting this setup working. I have virtual machines working using rbd based images by editing the domain directly. Is there any way to make the creation process better? We are hoping to be able to use a virsh pool using the rbd driver but it appears that Redhat has not compiled libvirt with rbd support. Thought? Thanks, /Chris C ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Mounting Ceph on Linux/Windows
> > Can someone point me to directions on how to mount a Ceph storage > volume on Linux as well as Windows? > Do you mean cephfs filesystem, or rbd block device? I have ported librbd to windows in a very "alpha" sense - it compiles and I can do things like 'rbd ls' and 'rbd import', but haven't tested it much further than that. I was hoping something like blktap already existed for Windows so I could write a windows application to expose a block device from userspace, but I can't find anything, or at least can't find anything open source. I started writing a driver but haven't had time to do much with it yet. If you wanted to use cephfs then you could probably build on the porting work I've done. I'm not sure what cephfs uses but I assume it sits on top of librados so that work is already done (assuming there is a fuse-like driver for windows - otherwise you'd have to port it all to windows kernel which is more work, as the windows kernel doesn't support C++) James ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Nearly full OSDs with very little (apparent) FS utilization
> How do you mount cephfs, use ceph-fuse or kernel driver? > > Regards > Yan, Zheng I use ceph-fuse. Cheers, MAO ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Mounting Ceph on Linux/Windows
Native block support is coming for Hyper-V next year we hope... would be great to hear from InkTank on anything that can be shared publicly on that front :) On 2013-12-05 22:02, James Harper wrote: Can someone point me to directions on how to mount a Ceph storage volume on Linux as well as Windows? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ephemeral RBD with Havana and Dumpling
Josh, On Tue, Nov 19, 2013 at 4:24 PM, Josh Durgin wrote: >>> I hope I can release or push commits to this branch contains live-migration, >>> incorrect filesystem size fix and ceph-snapshort support in a few days. >> >> Can't wait to see this patch! Are you getting rid of the shared >> storage requirement for live-migration? > > Yes, that's what Haomai's patch will fix for rbd-based ephemeral > volumes (bug https://bugs.launchpad.net/nova/+bug/1250751). We've got a version of a Nova patch that makes live migrations work for non volume-backed instances, and hopefully addresses the concerns raised in code review in https://review.openstack.org/56527, along with a bunch of small bugfixes, e.g. missing max_size parameter in direct_fetch, and a fix for http://tracker.ceph.com/issues/6693. I have submitted it as a pull request to your nova fork on GitHub: https://github.com/jdurgin/nova/pull/1 Our changes depend on the rest of commits on your havana-ephemeral-rbd branch, and the whole patchset is now at 7 commits, which is going to be rather tedious to submit to the OpenStack Gerrit as a series of dependent changes. Do you think we should keep the current commit history in its current form, or would it be easier to squash it down to a more manageable number of patches? Thanks, -- Dmitry Borodaenko ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Mounting Ceph on Linux/Windows
On Thu, 5 Dec 2013, James Harper wrote: > > > > Can someone point me to directions on how to mount a Ceph storage > > volume on Linux as well as Windows? > > > > Do you mean cephfs filesystem, or rbd block device? > > I have ported librbd to windows in a very "alpha" sense - it compiles > and I can do things like 'rbd ls' and 'rbd import', but haven't tested > it much further than that. I would love to see any of this porting work make its way back into the tree. Have you seen the work going on in the wip-port branch? sage ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Mounting Ceph on Linux/Windows
> > On Thu, 5 Dec 2013, James Harper wrote: > > > > > > Can someone point me to directions on how to mount a Ceph storage > > > volume on Linux as well as Windows? > > > > > > > Do you mean cephfs filesystem, or rbd block device? > > > > I have ported librbd to windows in a very "alpha" sense - it compiles > > and I can do things like 'rbd ls' and 'rbd import', but haven't tested > > it much further than that. > > I would love to see any of this porting work make its way back into the > tree. Have you seen the work going on in the wip-port branch? > I wasn't aware it existed. Would that be a good place to start with committing the win32 stuff? Or a new win32 branch based on ports? My stuff is currently against dumpling, but from a brief browse of the commits it looks like we share some of the required changes. For win32 my current scheme is to just throw out all the configure and automake stuff and have a static makefile, so the build procedure is 'cd win32; make'. Do you think that's acceptable? I suppose the configure could be modified to work with a mingw32 cross compile under Linux, but it might be a bit of a stretch to make it work in mingw32 environment under Windows (or maybe it's easy... I've never used mingw32 under Windows). At this time the configure and Makefile stuff would need to be able to build only selective bits as a lot of stuff doesn't currently build under Windows, and some may never build... osd under windows would be a pretty big project for something with very limited appeal (imho) James ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Mounting Ceph on Linux/Windows
On Thu, 5 Dec 2013, James Harper wrote: > > > > On Thu, 5 Dec 2013, James Harper wrote: > > > > > > > > Can someone point me to directions on how to mount a Ceph storage > > > > volume on Linux as well as Windows? > > > > > > > > > > Do you mean cephfs filesystem, or rbd block device? > > > > > > I have ported librbd to windows in a very "alpha" sense - it compiles > > > and I can do things like 'rbd ls' and 'rbd import', but haven't tested > > > it much further than that. > > > > I would love to see any of this porting work make its way back into the > > tree. Have you seen the work going on in the wip-port branch? > > > > I wasn't aware it existed. Would that be a good place to start with > committing the win32 stuff? Or a new win32 branch based on ports? My > stuff is currently against dumpling, but from a brief browse of the > commits it looks like we share some of the required changes. Yeah, I'd try to make sure the efforts are complementary. If stuff in wip-port is general enough, let's make it. > For win32 my current scheme is to just throw out all the configure and > automake stuff and have a static makefile, so the build procedure is 'cd > win32; make'. Do you think that's acceptable? I think that's a fine, certainly as a place to start. (I'm not sure there is going to be a better solution that doesn't rely on mingw32 or cygwin or something.) > I suppose the configure > could be modified to work with a mingw32 cross compile under Linux, but > it might be a bit of a stretch to make it work in mingw32 environment > under Windows (or maybe it's easy... I've never used mingw32 under > Windows). At this time the configure and Makefile stuff would need to be > able to build only selective bits as a lot of stuff doesn't currently > build under Windows, and some may never build... osd under windows would > be a pretty big project for something with very limited appeal (imho) Yep! Anyway, this sounds great! sage ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Mounting Ceph on Linux/Windows
A little info about wip-port. The wip-port branch lags behind master a bit, usually a week or two depending on what I've got going on. There are testers for OSX and FreeBSD, and bringing in windows patches would probably be a nice staging place for them, as I suspect the areas of change will overlap a lot. My bandwidth is limited until Dec 16th, but I'll be able to test on my platforms and pull stuff in. The patch ordering in wip-port is also semi-relevant. The older patches are more likely to have pull requests send up before newer patches, which might be of interest if there are windows things that overlap. Thanks! Noah On Thu, Dec 5, 2013 at 3:55 PM, Sage Weil wrote: > On Thu, 5 Dec 2013, James Harper wrote: >> > >> > On Thu, 5 Dec 2013, James Harper wrote: >> > > > >> > > > Can someone point me to directions on how to mount a Ceph storage >> > > > volume on Linux as well as Windows? >> > > > >> > > >> > > Do you mean cephfs filesystem, or rbd block device? >> > > >> > > I have ported librbd to windows in a very "alpha" sense - it compiles >> > > and I can do things like 'rbd ls' and 'rbd import', but haven't tested >> > > it much further than that. >> > >> > I would love to see any of this porting work make its way back into the >> > tree. Have you seen the work going on in the wip-port branch? >> > >> >> I wasn't aware it existed. Would that be a good place to start with >> committing the win32 stuff? Or a new win32 branch based on ports? My >> stuff is currently against dumpling, but from a brief browse of the >> commits it looks like we share some of the required changes. > > Yeah, I'd try to make sure the efforts are complementary. If stuff in > wip-port is general enough, let's make it. > >> For win32 my current scheme is to just throw out all the configure and >> automake stuff and have a static makefile, so the build procedure is 'cd >> win32; make'. Do you think that's acceptable? > > I think that's a fine, certainly as a place to start. (I'm not sure there > is going to be a better solution that doesn't rely on mingw32 or cygwin or > something.) > >> I suppose the configure >> could be modified to work with a mingw32 cross compile under Linux, but >> it might be a bit of a stretch to make it work in mingw32 environment >> under Windows (or maybe it's easy... I've never used mingw32 under >> Windows). At this time the configure and Makefile stuff would need to be >> able to build only selective bits as a lot of stuff doesn't currently >> build under Windows, and some may never build... osd under windows would >> be a pretty big project for something with very limited appeal (imho) > > Yep! > > Anyway, this sounds great! > sage > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Nearly full OSDs with very little (apparent) FS utilization
On Fri, Dec 6, 2013 at 6:08 AM, Miguel Oliveira wrote: >> How do you mount cephfs, use ceph-fuse or kernel driver? >> >> Regards >> Yan, Zheng > > I use ceph-fuse. > Looks like the issue is not caused by the bug I presume. Could you please run following commands, and send the output to me. rados -p data ls >object.list find /cephmountpoint -printf '%i\t%p\n' >inode.list Regards Yan, Zheng ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
