Re: [ceph-users] One OSD always dieing

[Rottmann, Jonas (centron GmbH)]

Hi,

The Smart Values are looking good as far as I can see, and also if I mark the 
osd down and let the rebuild to other osds happen I'm running into the same 
problem on another osd.

Jonas

Von: ceph-users-boun...@lists.ceph.com 
[mailto:ceph-users-boun...@lists.ceph.com] Im Auftrag von Udo Lembke
Gesendet: Mittwoch, 15. Januar 2014 21:45
An: ceph-users@lists.ceph.com
Betreff: Re: [ceph-users] One OSD always dieing

Hi,
perhaps the disk has an problem?
Have you look with smartctl?

(apt-get install smartmontools; smartctl -A /dev/sdX )

Udo

On 15.01.2014 10:49, Rottmann, Jonas (centron GmbH) wrote:
Hi,

I now did an upgrade to dumpling (ceph version 0.67.5 
(a60ac9194718083a4b6a225fc17cad6096c69bd1)), but the osd still fails at startup 
with a trace.

Heres the trace:

http://paste.ubuntu.com/6755307/

If you need any more infos I will provide them. Can someone please help?

Thanks


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph cluster is unreachable because of authentication failure

[Guang]

I still have bad the luck to figure out what is the problem making 
authentication failure, so in order to get the cluster back, I tried:
  1. stop all daemons (mon & osd)
  2. change the configuration to disable cephx
  3. start mon daemons (3 in total)
  4. start osd daemon one by one

After finishing step 3, the cluster can be reachable ('ceph -s' give results):
-bash-4.1$ sudo ceph -s
  cluster b9cb3ea9-e1de-48b4-9e86-6921e2c537d2
   health HEALTH_WARN 2797 pgs degraded; 107 pgs down; 7503 pgs peering; 917 
pgs recovering; 6079 pgs recovery_wait; 2957 pgs stale; 7771 pgs stuck 
inactive; 2957 pgs stuck stale; 16567 pgs stuck unclean; recovery 
54346804/779462977 degraded (6.972%); 9/259724199 unfound (0.000%); 2 near full 
osd(s); 57/751 in osds are down; 
noout,nobackfill,norecover,noscrub,nodeep-scrub flag(s) set
   monmap e1: 3 mons at 
{osd151=10.194.0.68:6789/0,osd152=10.193.207.130:6789/0,osd153=10.193.207.131:6789/0},
 election epoch 106022, quorum 0,1,2 osd151,osd152,osd153
   osdmap e134893: 781 osds: 694 up, 751 in
pgmap v2388518: 22203 pgs: 26 inactive, 14 active, 79 
stale+active+recovering, 5020 active+clean, 242 stale, 4352 
active+recovery_wait, 616 stale+active+clean, 177 active+recovering+degraded, 
6714 peering, 925 stale+active+recovery_wait, 86 down+peering, 1547 
active+degraded, 32 stale+active+recovering+degraded, 648 stale+peering, 21 
stale+down+peering, 239 stale+active+degraded, 651 
active+recovery_wait+degraded, 30 remapped+peering, 151 
stale+active+recovery_wait+degraded, 4 stale+remapped+peering, 629 
active+recovering; 79656 GB data, 363 TB used, 697 TB / 1061 TB avail; 
54346804/779462977 degraded (6.972%); 9/259724199 unfound (0.000%)
   mdsmap e1: 0/0/1 up
(at this point, all OSDs should be down).

When I tried to start OSD daemon, the starting script got hang, and the process 
hang is:
root  80497  80496  0 08:18 pts/000:00:00 python /usr/bin/ceph 
--name=osd.22 --keyring=/var/lib/ceph/osd/ceph-22/keyring osd crush 
create-or-move -- 22 0.40 root=default host=osd173

When I strace the starting script, I got the following traces (process 75873 is 
the above process), it failed with futex and then do a infinite loop:
   select(0, NULL, NULL, NULL, {0, 16000}) = 0 (Timeout)
Any idea what might trigger this?

=== STRACE (PARTIAL) == 
[pid 75873] futex(0xf707a0, FUTEX_WAIT_PRIVATE, 0, NULL 
[pid 75878] mmap(NULL, 134217728, PROT_NONE, 
MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f5da6529000
[pid 75878] munmap(0x7f5da6529000, 28143616) = 0
[pid 75878] munmap(0x7f5dac00, 38965248) = 0
[pid 75878] mprotect(0x7f5da800, 135168, PROT_READ|PROT_WRITE) = 0
[pid 75878] futex(0xf707a0, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 75873] <... futex resumed> )   = 0
[pid 75873] futex(0xdd3cb0, FUTEX_WAIT_PRIVATE, 0, NULL 
[pid 75878] futex(0xdd3cb0, FUTEX_WAKE_PRIVATE, 1 
[pid 75873] <... futex resumed> )   = -1 EAGAIN (Resource temporarily 
unavailable)
[pid 75878] <... futex resumed> )   = 0
[pid 75873] select(0, NULL, NULL, NULL, {0, 1000} 
[pid 75878] rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [], 8) = 0
[pid 75878] mmap(NULL, 10489856, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f5dadb28000
[pid 75878] mprotect(0x7f5dadb28000, 4096, PROT_NONE) = 0
[ omit some entries…]
[pid 75873] select(0, NULL, NULL, NULL, {0, 16000}) = 0 (Timeout)
[pid 75873] select(0, NULL, NULL, NULL, {0, 32000}) = 0 (Timeout)
[pid 75873] select(0, NULL, NULL, NULL, {0, 5}) = 0 (Timeout)
[pid 75873] select(0, NULL, NULL, NULL, {0, 5}) = 0 (Timeout)
[pid 75873] select(0, NULL, NULL, NULL, {0, 5}) = 0 (Timeout)
[pid 75873] select(0, NULL, NULL, NULL, {0, 5}) = 0 (Timeout)
[pid 75873] select(0, NULL, NULL, NULL, {0, 5}) = 0 (Timeout)


Thanks,
Guang

On Jan 15, 2014, at 5:54 AM, Guang  wrote:

> Thanks Sage.
> 
> -bash-4.1$ sudo ceph --admin-daemon /var/run/ceph/ceph-mon.osd151.asok 
> mon_status
> { "name": "osd151",
>  "rank": 2,
>  "state": "electing",
>  "election_epoch": 85469,
>  "quorum": [],
>  "outside_quorum": [],
>  "extra_probe_peers": [],
>  "sync_provider": [],
>  "monmap": { "epoch": 1,
>  "fsid": "b9cb3ea9-e1de-48b4-9e86-6921e2c537d2",
>  "modified": "0.00",
>  "created": "0.00",
>  "mons": [
>{ "rank": 0,
>  "name": "osd152",
>  "addr": "10.193.207.130:6789\/0"},
>{ "rank": 1,
>  "name": "osd153",
>  "addr": "10.193.207.131:6789\/0"},
>{ "rank": 2,
>  "name": "osd151",
>  "addr": "10.194.0.68:6789\/0"}]}}
> 
> And:
> 
> -bash-4.1$ sudo ceph --admin-daemon /var/run/ceph/ceph-mon.osd151.asok 
> quorum_status
> { "election_epoch": 85480,
>  "quorum": [
>0,
>1,
>2],
>  "quorum_names": [
>"osd151",
>"osd152",
>"osd153"],
>  "quorum_leader_name": "osd152",
>  "monmap": { "epoch": 1,
>  "fsid": "b9cb3ea9-e1de-48b4-9e86-6921e2c537d2",

Re: [ceph-users] Ceph / Dell hardware recommendation

[NEVEU Stephane]

À : ceph-users@lists.ceph.com
Objet : Re: [ceph-users] Ceph / Dell hardware recommendation

On 01/15/2014 08:29 AM, Derek Yarnell wrote:
> On 1/15/14, 9:20 AM, Mark Nelson wrote:
>> I guess I'd probably look at the R520 in an 8 bay configuration with 
>> an
>> E5-2407 and 4 1TB data disks per chassis (along with whatever OS disk 
>> setup you want).  That gives you 4 PCIE slots for the extra network 
>> cards, the option for a hardware raid controller, and some expansion 
>> options if you want to add more disks.  It's not the densest chassis 
>> and I've never tested performance on it, but it's a fairly 
>> straightforward setup with no expanders to worry about and a cheap/fast CPU.
>
> Dell recently will sell now a R720xd with a pure LSI card in it (you 
> will need to ask).  This is important that the PERC controllers do not 
> support JBOD mode and really for Ceph you don't want a hardware raid 
> controller.  They currently give me a hard time about trying to mix 
> and match SSDs though on the 12 bay back-plane which is not a 
> technical problem but a Dell problem.  However you can get two flex 
> bays in it which they allow you to put SSDs in.  So you can get 12 4TB 
> drives and 2 100G eMLC drives in 2U.

That's good to know that they will sell LSI controllers in the R series now!  
JBOD vs RAID has some nuances.  As far as I know, LSI controllers basically 
disable cache when put in JBOD mode (as opposed to something like Areca that 
keeps it enabled), so you give up a lot when you run in JBOD mode.  With SSD 
journals (that have a supercap) this is much less of an issue as you don't have 
to worry about journal writes and data writes competing on the same disk.  With 
journals on disk, it appears that controllers with WB cache can help quite a 
bit.  In this scenario you'd use single disk RAID0 arrays.  This is actually 
what DreamHost does.


Thank you all for comments,

So to sum up a bit, it's a reasonable compromise to buy :
2 x R720 with 2x Intel E5-2660v2, 2.2GHz, 25M Cache, 48Gb RAM, 2 x 146GB, SAS 
6Gbps, 2.5-in, 15K RPM Hard Drive (Hot-plug) Flex Bay for OS and 24 x 1.2TB, 
SAS 6Gbps, 2.5in, 10K RPM Hard Drive for OSDs (journal located on each osd) and 
PERC H710p Integrated RAID Controller, 1GB NV Cache
?
Or is it a better idea to buy 4 servers less powerful instead of 2 ?

>
> Thanks,
> derek
>

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph / Dell hardware recommendation

[Cedric Lemarchand]

Le 16/01/2014 10:16, NEVEU Stephane a écrit :

Thank you all for comments,

So to sum up a bit, it's a reasonable compromise to buy :
2 x R720 with 2x Intel E5-2660v2, 2.2GHz, 25M Cache, 48Gb RAM, 2 x 146GB, SAS 
6Gbps, 2.5-in, 15K RPM Hard Drive (Hot-plug) Flex Bay for OS and 24 x 1.2TB, 
SAS 6Gbps, 2.5in, 10K RPM Hard Drive for OSDs (journal located on each osd) and 
PERC H710p Integrated RAID Controller, 1GB NV Cache
?
Or is it a better idea to buy 4 servers less powerful instead of 2 ?
I think you are facing the well known trade off between 
price/performances/usable storage size.


More servers less powerfull will give you better power computation and 
better iops by usable To, but will be more expensive. An extrapolation 
of that that would be to use a blade for each To => very powerful/very 
expensive.



The choice really depend of the work load you need to handle, witch is 
not an easy thing to estimate.


Cheers

--
Cédric

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v0.75 released

[Ilya Dryomov]

On Wed, Jan 15, 2014 at 5:42 AM, Sage Weil  wrote:
>
> [...]
>
> * rbd: support for 4096 mapped devices, up from ~250 (Ilya Dryomov)

Just a note, v0.75 simply adds some of the infrastructure, the actual
support for this will arrive with kernel 3.14.  The theoretical limit
is 65536 mapped devices, although I admit I haven't tried mapping more
than ~4000 at once.

Thanks,

Ilya
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Come Talk at Ceph Day Frankfurt!

[Patrick McGarry]

Greetings denizens of the Ceph universe!

As you may have noticed, Inktank has announced the next "Ceph Day"
which will be held in Frankfurt, Germany on February 27th [0].  While
Inktank may be the one putting on these events, our hope is that the
real stars will be from the community.  We already have two confirmed
community talks and we'd love to get a few more.

If you are interested in sharing your Ceph experiences with a Ceph Day
audience please let me know!  If you are looking for inspiration, we
currently have a few proposed talks that may make the schedule:

 * Ceph + Openstack (always a favorite)
 * Ceph Performance and Optimization
 * Ceph Integration into Pacemaker Cluster Systems

We also love to hear about specific user experiences.  So if you would
just like to share the details of how you are integrating Ceph into
your system that would also make an acceptable talk.  Shoot me your
ideas and we'll get you on the docket! Thanks.

[0] http://www.eventbrite.com/e/ceph-day-frankfurt-tickets-10173269523


Best Regards,

Patrick McGarry
Director, Community || Inktank
http://ceph.com  ||  http://inktank.com
@scuttlemonkey || @ceph || @inktank
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph cluster is unreachable because of authentication failure

[Sage Weil]

Hi Guang,

On Thu, 16 Jan 2014, Guang wrote:
> I still have bad the luck to figure out what is the problem making 
> authentication failure, so in order to get the cluster back, I tried:
>   1. stop all daemons (mon & osd)
>   2. change the configuration to disable cephx
>   3. start mon daemons (3 in total)
>   4. start osd daemon one by one
>
> After finishing step 3, the cluster can be reachable ('ceph -s' give results):
> -bash-4.1$ sudo ceph -s
>   cluster b9cb3ea9-e1de-48b4-9e86-6921e2c537d2
>health HEALTH_WARN 2797 pgs degraded; 107 pgs down; 7503 pgs peering; 917 
> pgs recovering; 6079 pgs recovery_wait; 2957 pgs stale; 7771 pgs stuck 
> inactive; 2957 pgs stuck stale; 16567 pgs stuck unclean; recovery 
> 54346804/779462977 degraded (6.972%); 9/259724199 unfound (0.000%); 2 near 
> full osd(s); 57/751 in osds are down; 
> noout,nobackfill,norecover,noscrub,nodeep-scrub flag(s) set
>monmap e1: 3 mons at 
> {osd151=10.194.0.68:6789/0,osd152=10.193.207.130:6789/0,osd153=10.193.207.131:6789/0},
>  election epoch 106022, quorum 0,1,2 osd151,osd152,osd153
>osdmap e134893: 781 osds: 694 up, 751 in
> pgmap v2388518: 22203 pgs: 26 inactive, 14 active, 79 
> stale+active+recovering, 5020 active+clean, 242 stale, 4352 
> active+recovery_wait, 616 stale+active+clean, 177 active+recovering+degraded, 
> 6714 peering, 925 stale+active+recovery_wait, 86 down+peering, 1547 
> active+degraded, 32 stale+active+recovering+degraded, 648 stale+peering, 21 
> stale+down+peering, 239 stale+active+degraded, 651 
> active+recovery_wait+degraded, 30 remapped+peering, 151 
> stale+active+recovery_wait+degraded, 4 stale+remapped+peering, 629 
> active+recovering; 79656 GB data, 363 TB used, 697 TB / 1061 TB avail; 
> 54346804/779462977 degraded (6.972%); 9/259724199 unfound (0.000%)
>mdsmap e1: 0/0/1 up
> (at this point, all OSDs should be down).
> 
> When I tried to start OSD daemon, the starting script got hang, and the 
> process hang is:
> root  80497  80496  0 08:18 pts/000:00:00 python /usr/bin/ceph 
> --name=osd.22 --keyring=/var/lib/ceph/osd/ceph-22/keyring osd crush 
> create-or-move -- 22 0.40 root=default host=osd173
> 
> When I strace the starting script, I got the following traces (process 75873 
> is the above process), it failed with futex and then do a infinite loop:
>select(0, NULL, NULL, NULL, {0, 16000}) = 0 (Timeout)
> Any idea what might trigger this?

It is hard to tell from the strace what is going on from this.  Do you see 
that the OSDs are booting in ceph.log (or ceph -w output)?  If not, I 
would look at the osd daemon log for clues.  You may need to turn up 
debugging to see (ceph daemon osd.NNN config set debug_osd 20 to adjust 
the level on the running daemon).

If they are booting, it is mostly a matter of letting it recover and come 
up.  We have seen patterns where configuration or network issues have let 
the system bury itself under a series of osdmap updates.  If you see that 
in the log when you turn up debugging, or see the osds going up and down 
when you try to bring the cluster up, that could be what is going on.  A 
strategy that has worked there is to let all the osds catch up on their 
maps before trying to peer and join the cluster.  To do that, 'ceph osd 
set noup' (which prevents the osds from joining), wait for the ceph-osd 
processes to stop chewing on maps (watch the cpu utilization in top), and 
once they are all ready 'ceph osd unset noup' and let them join and peer 
all at once.

sage

> 
> === STRACE (PARTIAL) == 
> [pid 75873] futex(0xf707a0, FUTEX_WAIT_PRIVATE, 0, NULL 
> [pid 75878] mmap(NULL, 134217728, PROT_NONE, 
> MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f5da6529000
> [pid 75878] munmap(0x7f5da6529000, 28143616) = 0
> [pid 75878] munmap(0x7f5dac00, 38965248) = 0
> [pid 75878] mprotect(0x7f5da800, 135168, PROT_READ|PROT_WRITE) = 0
> [pid 75878] futex(0xf707a0, FUTEX_WAKE_PRIVATE, 1) = 1
> [pid 75873] <... futex resumed> )   = 0
> [pid 75873] futex(0xdd3cb0, FUTEX_WAIT_PRIVATE, 0, NULL 
> [pid 75878] futex(0xdd3cb0, FUTEX_WAKE_PRIVATE, 1 
> [pid 75873] <... futex resumed> )   = -1 EAGAIN (Resource temporarily 
> unavailable)
> [pid 75878] <... futex resumed> )   = 0
> [pid 75873] select(0, NULL, NULL, NULL, {0, 1000} 
> [pid 75878] rt_sigprocmask(SIG_BLOCK, ~[RTMIN RT_1], [], 8) = 0
> [pid 75878] mmap(NULL, 10489856, PROT_READ|PROT_WRITE, 
> MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f5dadb28000
> [pid 75878] mprotect(0x7f5dadb28000, 4096, PROT_NONE) = 0
> [ omit some entries?]
> [pid 75873] select(0, NULL, NULL, NULL, {0, 16000}) = 0 (Timeout)
> [pid 75873] select(0, NULL, NULL, NULL, {0, 32000}) = 0 (Timeout)
> [pid 75873] select(0, NULL, NULL, NULL, {0, 5}) = 0 (Timeout)
> [pid 75873] select(0, NULL, NULL, NULL, {0, 5}) = 0 (Timeout)
> [pid 75873] select(0, NULL, NULL, NULL, {0, 5}) = 0 (Timeout)
> [pid 75873] select(0, NULL, NULL, NULL, {0, 5}) = 0 (Timeout)
> [pid 75

Re: [ceph-users] Ceph / Dell hardware recommendation

[Lincoln Bryant]

For our ~400 TB Ceph deployment, we bought:
(2) R720s w/ dual X5660s and 96 GB of RAM
(1) 10Gb NIC (2 interfaces per card)
(4) MD1200s per machine
...and a boat load of 4TB disks!

In retrospect, I would almost certainly would have gotten more servers. During 
heavy writes we see the load spiking up to ~50 on Emperor and warnings about 
slow OSDs, but we clearly seem to be on the extreme with something like 60 OSDs 
per box :)

Cheers,
Lincoln

On Jan 16, 2014, at 4:09 AM, Cedric Lemarchand wrote:

> 
> Le 16/01/2014 10:16, NEVEU Stephane a écrit :
>> Thank you all for comments,
>> 
>> So to sum up a bit, it's a reasonable compromise to buy :
>> 2 x R720 with 2x Intel E5-2660v2, 2.2GHz, 25M Cache, 48Gb RAM, 2 x 146GB, 
>> SAS 6Gbps, 2.5-in, 15K RPM Hard Drive (Hot-plug) Flex Bay for OS and 24 x 
>> 1.2TB, SAS 6Gbps, 2.5in, 10K RPM Hard Drive for OSDs (journal located on 
>> each osd) and PERC H710p Integrated RAID Controller, 1GB NV Cache
>> ?
>> Or is it a better idea to buy 4 servers less powerful instead of 2 ?
> I think you are facing the well known trade off between 
> price/performances/usable storage size.
> 
> More servers less powerfull will give you better power computation and better 
> iops by usable To, but will be more expensive. An extrapolation of that that 
> would be to use a blade for each To => very powerful/very expensive.
> 
> 
> The choice really depend of the work load you need to handle, witch is not an 
> easy thing to estimate.
> 
> Cheers
> 
> -- 
> Cédric
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Power Cycle Problems

[Dan Ryder (daryder)]

Hello,

On two separate occasions I have lost power to my Ceph cluster. Both times, I 
had trouble bringing the cluster back to good health. I am wondering if I need 
to config something that would solve this problem?
After powering back up the cluster, "ceph health" revealed stale pages, mds 
cluster degraded, 3/3 OSDs down. I tried to issue "sudo /etc/init.d/ceph -a 
start" but I got no output from the command and the health status did not 
change.
I ended up having to re-install the cluster to fix the issue, but as my group 
wants to use Ceph for VM storage in the future, we need to find a solution.

Please note I followed the Storage Cluster Quick Start for installation and did 
not make any Ceph config changes after.

Any help would be appreciated.

Thanks,
Dan

[http://www.cisco.com/web/europe/images/email/signature/logo05.jpg]

Dan Ryder
ENGINEER.SOFTWARE ENGINEERING
CSMTG Performance/Analytics
dary...@cisco.com

Cisco Systems, Inc.
7100-8 Kit Creek Road
PO Box 14987
27709-4987
Research Triangle Park
United States
Cisco.com



[Think before you print.] Think before you print.

This email may contain confidential and privileged material for the sole use of 
the intended recipient. Any review, use, distribution or disclosure by others 
is strictly prohibited. If you are not the intended recipient (or authorized to 
receive for the recipient), please contact the sender by reply email and delete 
all copies of this message.
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html




<><>___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Emperor Upgrade: osds not starting

[Gagandeep Arora]

Hello,

Osds are not starting on any of the nodes after I upgraded ceph-0.67.4 to
emperor 0.72.2. Tried to start osd see the following verbose output. The
same error comes up on all nodes when starting osds.

[root@ceph2 ~]# service ceph -v start osd.20
/usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.20 "user"
=== osd.20 ===
/usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.20 "run dir"
/usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.20 "pid file"
--- ceph2# mkdir -p /var/run/ceph
/usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.20 "log dir"
/usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.20 "auto start"
--- ceph2# [ -e /var/run/ceph/osd.20.pid ] || exit 1   # no pid, presumably
not running
pid=`cat /var/run/ceph/osd.20.pid`
[ -e /proc/$pid ] && grep -q ceph-osd /proc/$pid/cmdline && grep
-qwe -i.20 /proc/$pid/cmdline && exit 0 # running
exit 1  # pid is something else
/usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.20 "copy executable to"
/usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.20 "osd data"
/usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.20 "fs path"
/usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.20 "devs"
/usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.20 "btrfs devs"
/usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.20 "lock file"
/usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.20 "admin socket"
/usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.20 "max open files"
/usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.20 "restart on core dump"
/usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.20 "valgrind"
/usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.20 "osd crush update on
start"
/usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.20 "osd crush location"
/usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.20 "osd crush initial
weight"
/usr/bin/ceph-conf -c /etc/ceph/ceph.conf -n osd.20 "keyring"
--- ceph2# timeout 10 /usr/bin/ceph --name=osd.20
--keyring=/var/lib/ceph/osd/ceph-20/keyring
osd crush create-or-move--20
--- ceph2# df /var/lib/ceph/osd/ceph-20/. | tail -1 | awk '{ d=
$2/1073741824 ; r = sprintf("%.2f", d); print r }'
0.45root=defaulthost=ceph2
Invalid command:  --- doesn't represent a float
osd crush create-or-move   
[...] :  create entry or move existing entry for  
at/to location 
Error EINVAL: invalid command
bash: line 1: 0.45: command not found
failed: 'timeout 10 /usr/bin/ceph   --name=osd.20
--keyring=/var/lib/ceph/osd/ceph-20/keyring
osd crush create-or-move--20
--- ceph2# df /var/lib/ceph/osd/ceph-20/. | tail -1 | awk '{ d=
$2/1073741824 ; r = sprintf("%.2f", d); print r }'
0.45root=defaulthost=ceph2



However, Osds start when I use ceph-osd -c /etc/ceph/ceph.conf -i 
 but not through service ceph or /etc/init.d/ceph. After I stared all the
osds, ceph warning comes up with a message that a  "pool has too few pgs".
I deleted the pool as there wasn't any important data in it. The same
warning now comes up on a different pool.

[root@ceph1 ~]# ceph -s
cluster c0459c67-e2cd-45f7-b580-dec1afc9dea5
 health HEALTH_WARN pool vmware-backups has too few pgs
 monmap e3: 3 mons at {a=
192.168.6.101:6789/0,b=192.168.6.102:6789/0,c=192.168.6.103:6789/0},
election epoch 17684, quorum 0,1,2 a,b,c
 mdsmap e28128: 1/1/1 up {0=a=up:active}, 1 up:standby
 osdmap e7053: 30 osds: 30 up, 30 in
  pgmap v16242514: 6348 pgs, 12 pools, 9867 GB data, 2543 kobjects
19775 GB used, 58826 GB / 78602 GB avail
6343 active+clean
   5 active+clean+scrubbing+deep
  client io 0 B/s rd, 617 kB/s wr, 81 op/s

[root@ceph1 ~]# ceph osd pool delete vmware-backups vmware-backups
--yes-i-really-really-mean-it
pool 'vmware-backups' deleted
[root@ceph1 ~]# ceph -s
cluster c0459c67-e2cd-45f7-b580-dec1afc9dea5
 health HEALTH_WARN pool centaur-backups has too few pgs
 monmap e3: 3 mons at {a=
192.168.6.101:6789/0,b=192.168.6.102:6789/0,c=192.168.6.103:6789/0},
election epoch 17684, quorum 0,1,2 a,b,c
 mdsmap e28128: 1/1/1 up {0=a=up:active}, 1 up:standby
 osdmap e7054: 30 osds: 30 up, 30 in
  pgmap v16243076: 6048 pgs, 12 pools, 4437 GB data, 1181 kobjects
19775 GB used, 58826 GB / 78602 GB avail
6047 active+clean
   1 active+clean+scrubbing+deep
  client io 54836 kB/s rd, 699 op/s


Regards,
Gagan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Come Talk at Ceph Day Frankfurt!

[Loic Dachary]

Hi Patrick,

I would be happy to present the Ceph User Committee first the first time :-)

Cheers

On 16/01/2014 18:30, Patrick McGarry wrote:
> Greetings denizens of the Ceph universe!
> 
> As you may have noticed, Inktank has announced the next "Ceph Day"
> which will be held in Frankfurt, Germany on February 27th [0].  While
> Inktank may be the one putting on these events, our hope is that the
> real stars will be from the community.  We already have two confirmed
> community talks and we'd love to get a few more.
> 
> If you are interested in sharing your Ceph experiences with a Ceph Day
> audience please let me know!  If you are looking for inspiration, we
> currently have a few proposed talks that may make the schedule:
> 
>  * Ceph + Openstack (always a favorite)
>  * Ceph Performance and Optimization
>  * Ceph Integration into Pacemaker Cluster Systems
> 
> We also love to hear about specific user experiences.  So if you would
> just like to share the details of how you are integrating Ceph into
> your system that would also make an acceptable talk.  Shoot me your
> ideas and we'll get you on the docket! Thanks.
> 
> [0] http://www.eventbrite.com/e/ceph-day-frankfurt-tickets-10173269523
> 
> 
> Best Regards,
> 
> Patrick McGarry
> Director, Community || Inktank
> http://ceph.com  ||  http://inktank.com
> @scuttlemonkey || @ceph || @inktank
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

-- 
Loïc Dachary, Artisan Logiciel Libre



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Emperor Upgrade: osds not starting

[Michael]

Hi Gagan,

You have 30 osd with 12 pools and only 6048 PG. Some of your pools must 
have pretty low PG numbers. I think it looks for a 'skew' in numbers and 
issues a warning now as well as if you've pools which have 'too many' 
objects per placement group.


Run: ~$ceph osd dump | grep 'pg_num'
And see the docs: 
http://ceph.com/docs/master/rados/operations/placement-groups/


You can currently increase the number of PG/PGP of a pool but not 
decrease them, so take care if you need to balance them as higher 
numbers increases CPU load.


-Michael



However, Osds start when I use ceph-osd -c /etc/ceph/ceph.conf -i 
  but not through service ceph or /etc/init.d/ceph. After I 
stared all the osds, ceph warning comes up with a message that a 
 "pool has too few pgs". I deleted the pool as there wasn't any 
important data in it. The same warning now comes up on a different pool.


[root@ceph1 ~]# ceph -s
cluster c0459c67-e2cd-45f7-b580-dec1afc9dea5
 health HEALTH_WARN pool vmware-backups has too few pgs
 monmap e3: 3 mons at 
{a=192.168.6.101:6789/0,b=192.168.6.102:6789/0,c=192.168.6.103:6789/0 
}, 
election epoch 17684, quorum 0,1,2 a,b,c

 mdsmap e28128: 1/1/1 up {0=a=up:active}, 1 up:standby
 osdmap e7053: 30 osds: 30 up, 30 in
  pgmap v16242514: 6348 pgs, 12 pools, 9867 GB data, 2543 kobjects
19775 GB used, 58826 GB / 78602 GB avail
6343 active+clean
   5 active+clean+scrubbing+deep
  client io 0 B/s rd, 617 kB/s wr, 81 op/s

[root@ceph1 ~]# ceph osd pool delete vmware-backups vmware-backups 
--yes-i-really-really-mean-it

pool 'vmware-backups' deleted
[root@ceph1 ~]# ceph -s
cluster c0459c67-e2cd-45f7-b580-dec1afc9dea5
 health HEALTH_WARN pool centaur-backups has too few pgs
 monmap e3: 3 mons at 
{a=192.168.6.101:6789/0,b=192.168.6.102:6789/0,c=192.168.6.103:6789/0 
}, 
election epoch 17684, quorum 0,1,2 a,b,c

 mdsmap e28128: 1/1/1 up {0=a=up:active}, 1 up:standby
 osdmap e7054: 30 osds: 30 up, 30 in
  pgmap v16243076: 6048 pgs, 12 pools, 4437 GB data, 1181 kobjects
19775 GB used, 58826 GB / 78602 GB avail
6047 active+clean
   1 active+clean+scrubbing+deep
  client io 54836 kB/s rd, 699 op/s


Regards,
Gagan


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v0.75 released

[Christian Balzer]

On Thu, 16 Jan 2014 15:51:17 +0200 Ilya Dryomov wrote:

> On Wed, Jan 15, 2014 at 5:42 AM, Sage Weil  wrote:
> >
> > [...]
> >
> > * rbd: support for 4096 mapped devices, up from ~250 (Ilya Dryomov)
> 
> Just a note, v0.75 simply adds some of the infrastructure, the actual
> support for this will arrive with kernel 3.14.  The theoretical limit
> is 65536 mapped devices, although I admit I haven't tried mapping more
> than ~4000 at once.
> 
Just for clarification, this is for the client side when using the kernel
module, right?

Not looking at more than about 150 devices per compute node now, but that
might change and there is also the case of failovers...

Regards,

Christian
-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Fusion Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Power Cycle Problems

[Gao, Wei M]

Things to check:

1.   Check the network if the IP address change, if the cable is plugged-in

2.   Check the data directory(/var/lib/ceph/), usually you would mount the
osd data directory to a separate disk.

 

It would be helpful if you can post the logs of ceph here.

 

Best Regards

Wei

 

From: ceph-users-boun...@lists.ceph.com
[mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Dan Ryder (daryder)
Sent: Friday, January 17, 2014 4:07 AM
To: ceph-us...@ceph.com
Subject: [ceph-users] Power Cycle Problems

 

Hello,

 

On two separate occasions I have lost power to my Ceph cluster. Both times,
I had trouble bringing the cluster back to good health. I am wondering if I
need to config something that would solve this problem?

After powering back up the cluster, "ceph health" revealed stale pages, mds
cluster degraded, 3/3 OSDs down. I tried to issue "sudo /etc/init.d/ceph -a
start" but I got no output from the command and the health status did not
change.

I ended up having to re-install the cluster to fix the issue, but as my
group wants to use Ceph for VM storage in the future, we need to find a
solution. 

 

Please note I followed the Storage Cluster Quick Start for installation and
did not make any Ceph config changes after.

 

Any help would be appreciated.

 

Thanks,

Dan

 





Dan Ryder
ENGINEER.SOFTWARE ENGINEERING
CSMTG Performance/Analytics
  dary...@cisco.com

Cisco Systems, Inc.
7100-8 Kit Creek Road
PO Box 14987
27709-4987
Research Triangle Park
United States
  Cisco.com




 


 Think before you print.


This email may contain confidential and privileged material for the sole use
of the intended recipient. Any review, use, distribution or disclosure by
others is strictly prohibited. If you are not the intended recipient (or
authorized to receive for the recipient), please contact the sender by reply
email and delete all copies of this message.

For corporate legal information go to:
 
http://www.cisco.com/web/about/doing_business/legal/cri/index.html

 

 

 

<><>

smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Power Cycle Problems

[Kyle Bader]

> On two separate occasions I have lost power to my Ceph cluster. Both times, I 
> had trouble bringing the cluster back to good health. I am wondering if I 
> need to config something that would solve this problem?

No special configuration should be necessary, I've had the unfortunate
luck of witnessing several power loss events with large Ceph clusters.
In both cases something other than Ceph was the source of frustrations
once power was returned. That said, monitor daemons should be started
first and must form a quorum before the cluster will be usable. It
sounds like you have made it that far if your getting output from
"ceph health" commands. The next step is to get your Ceph OSD daemons
running, which will require the data partitions to be mounted and the
journal device present. In Ubuntu installations this is handled by
udev scripts installed by the Ceph packages (I think this is may be
true for RHEL/CentOS but have not verified). Short of the udev method
you can mount the data partition manually. Once the data partition is
mounted you can start the OSDs manually in the event that init still
doesn't work after mounting, to do so you will need to know the
location of your keyring, ceph.conf and the OSD id. If you are unsure
of what the OSD id is then you can look at the root of the OSD data
partition, after it is mounted, in a file named "whoami". To manually
start:

/usr/bin/ceph-osd -i ${OSD_ID} --pid-file
/var/run/ceph/osd.${OSD_ID}.pid -c /etc/ceph/ceph.conf

After that it's a matter of examining the logs if your still having
issues getting the OSDs to boot.

> After powering back up the cluster, “ceph health” revealed stale pages, mds 
> cluster degraded, 3/3 OSDs down. I tried to issue “sudo /etc/init.d/ceph -a 
> start” but I got no output from the command and the health status did not 
> change.

The placement groups are stale because none of the OSDs have reported
their state recently since they are down.

> I ended up having to re-install the cluster to fix the issue, but as my group 
> wants to use Ceph for VM storage in the future, we need to find a solution.

That's a shame, but at least you will be better prepared if it happens
again, hopefully your luck is not as unfortunate as mine!

-- 

Kyle Bader
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com