Re: [ceph-users] Can't start OSD

[Karan Singh]

Try to make these OSD IN

ceph osd in osd.12 osd.13 osd.14 osd.15

Then restart osd services 


- Karan Singh -

On 08 Aug 2014, at 00:55, O'Reilly, Dan  wrote:

> # idweight  type name   up/down reweight
> -1  7.2 root default
> -2  1.8 host tm1cldosdl01
> 0   0.45osd.0   up  1
> 1   0.45osd.1   up  1
> 2   0.45osd.2   up  1
> 3   0.45osd.3   up  1
> -3  1.8 host tm1cldosdl02
> 4   0.45osd.4   up  1
> 5   0.45osd.5   up  1
> 6   0.45osd.6   up  1
> 7   0.45osd.7   up  1
> -4  1.8 host tm1cldosdl03
> 8   0.45osd.8   up  1
> 9   0.45osd.9   up  1
> 10  0.45osd.10  up  1
> 11  0.45osd.11  up  1
> -5  1.8 host tm1cldosdl04
> 12  0.45osd.12  down0
> 13  0.45osd.13  down0
> 14  0.45osd.14  down0
> 15  0.45osd.15  down0
> [ceph@tm1cldosdl04 ~]$ sudo /etc/init.d/ceph start osd.12
> /etc/init.d/ceph: osd.12 not found (/etc/ceph/ceph.conf defines , 
> /var/lib/ceph defines )
>  
> What am I missing?  Specifically, what would need to be in ceph.conf or 
> /var/lib/ceph?
>  
> Dan O'Reilly
> UNIX Systems Administration
> 
> 9601 S. Meridian Blvd.
> Englewood, CO 80112
> 720-514-6293
>  
>  
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] nf_conntrack overflow crashes OSDs

[Christian Kauhaus]

Hi,

today I'd like to share a severe problem we've found (and fixed) on our Ceph
cluster. We're running 48 OSDs (8 per host). While restarting all OSDs on a
host, the kernel's nf_conntrack table was overflown. This rendered all OSDs on
that machine unusable.

The symptoms were as follows. In the kernel log, we saw lines like:

| Aug  6 15:23:48 cartman06 kernel: [12713575.554784] nf_conntrack: table
full, dropping packet

This is effectively a DoS against the kernel's IP stack.

In the OSD log files, we saw repeated connection attempts like:

| 2014-08-06 15:22:35.348175 7f92f25a8700 10 -- 172.22.4.42:6802/9560 >>
172.22.4.51:0/2025662 pipe(0x7f9208035440 sd=382 :6802 s=2 pgs=26750 cs=1 l=1
c=0x7f92080021c0).fault on lossy channel, failing
| 2014-08-06 15:22:35.348287 7f8fd69e4700 10 -- 172.22.4.42:6802/9560 >>
172.22.4.39:0/3024957 pipe(0x7f9208007b30 sd=149 :6802 s=2 pgs=245725 cs=1 l=1
c=0x7f9208036630).fault on lossy channel, failing
| 2014-08-06 15:22:35.348293 7f8fe24e4700 20 -- 172.22.4.42:6802/9560 >>
172.22.4.38:0/1013265 pipe(0x7f92080476e0 sd=450 :6802 s=4 pgs=32439 cs=1 l=1
c=0x7f9208018e90).writer finishing
| 2014-08-06 15:22:35.348284 7f8fd4fca700  2 -- 172.22.4.42:6802/9560 >>
172.22.4.5:0/3032136 pipe(0x7f92080686b0 sd=305 :6802 s=2 pgs=306100 cs=1 l=1
c=0x7f920805f340).fault 0: Success
| 2014-08-06 15:22:35.348292 7f8fd108b700 20 -- 172.22.4.42:6802/9560 >>
172.22.4.4:0/1000901 pipe(0x7f920802e7d0 sd=401 :6802 s=4 pgs=73173 cs=1 l=1
c=0x7f920802eda0).writer finishing
| 2014-08-06 15:22:35.344719 7f8fd1d98700  2 -- 172.22.4.42:6802/9560 >>
172.22.4.49:0/3026524 pipe(0x7f9208033a80 sd=492 :6802 s=2 pgs=12845 cs=1 l=1
c=0x7f9208033ce0).reader couldn't read tag, Success

and so on, generating 1000s of log lines. The OSDs were spinning with 100%
CPU, trying to re-connect in rapid succession. The repeated connection
attempts stopped nf_conntrack from getting out of its overflown state.

Thus, we saw blocked requests for 15 minutes or so, until the MONs banned the
stuck OSDs from the cluster.

As a short term countermeasure, we stopped all OSDs on the affected hosts and
started them one by one, leaving enough time in between to allow the recovery
settle a bit (10 sec gap between OSDs was enough). During normal operation, we
see only 5000-6000 connections on a host.

As a permanent fix, we have doubled the size of the nf_conntrack table and
reduced some timeouts according to
.
Now a restart of all 8 OSDs on a host works without problems.

Alternatively, we have considered removing nf_conntrack completely. This,
however, is not possible since we use host-based firewalling and nf_conntrack
is wired quite deeply into Linux' firewall code.

Just to share our experience in case someone experiences the same problem.

Regards

Christian

-- 
Dipl.-Inf. Christian Kauhaus <>< · k...@gocept.com · systems administration
gocept gmbh & co. kg · Forsterstraße 29 · 06112 Halle (Saale) · Germany
http://gocept.com · tel +49 345 219401-11
Python, Pyramid, Plone, Zope · consulting, development, hosting, operations
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] nf_conntrack overflow crashes OSDs

[Dan Van Der Ster]

Hi Christian,
This is good advice. Presumably we saw this issue before, since we have the 
following in our cluster’s puppet manifest:

  sysctl { "net.netfilter.nf_conntrack_max": val => "1024000", }
  sysctl { "net.nf_conntrack_max": val => "1024000", }

But I don’t remember when or how we discovered this, and google isn’t helping. 
I suggest that this should be added to ceph.com docs.

Cheers, Dan

-- Dan van der Ster || Data & Storage Services || CERN IT Department --


On 08 Aug 2014, at 10:46, Christian Kauhaus  wrote:

> Hi,
> 
> today I'd like to share a severe problem we've found (and fixed) on our Ceph
> cluster. We're running 48 OSDs (8 per host). While restarting all OSDs on a
> host, the kernel's nf_conntrack table was overflown. This rendered all OSDs on
> that machine unusable.
> 
> The symptoms were as follows. In the kernel log, we saw lines like:
> 
> | Aug  6 15:23:48 cartman06 kernel: [12713575.554784] nf_conntrack: table
> full, dropping packet
> 
> This is effectively a DoS against the kernel's IP stack.
> 
> In the OSD log files, we saw repeated connection attempts like:
> 
> | 2014-08-06 15:22:35.348175 7f92f25a8700 10 -- 172.22.4.42:6802/9560 >>
> 172.22.4.51:0/2025662 pipe(0x7f9208035440 sd=382 :6802 s=2 pgs=26750 cs=1 l=1
> c=0x7f92080021c0).fault on lossy channel, failing
> | 2014-08-06 15:22:35.348287 7f8fd69e4700 10 -- 172.22.4.42:6802/9560 >>
> 172.22.4.39:0/3024957 pipe(0x7f9208007b30 sd=149 :6802 s=2 pgs=245725 cs=1 l=1
> c=0x7f9208036630).fault on lossy channel, failing
> | 2014-08-06 15:22:35.348293 7f8fe24e4700 20 -- 172.22.4.42:6802/9560 >>
> 172.22.4.38:0/1013265 pipe(0x7f92080476e0 sd=450 :6802 s=4 pgs=32439 cs=1 l=1
> c=0x7f9208018e90).writer finishing
> | 2014-08-06 15:22:35.348284 7f8fd4fca700  2 -- 172.22.4.42:6802/9560 >>
> 172.22.4.5:0/3032136 pipe(0x7f92080686b0 sd=305 :6802 s=2 pgs=306100 cs=1 l=1
> c=0x7f920805f340).fault 0: Success
> | 2014-08-06 15:22:35.348292 7f8fd108b700 20 -- 172.22.4.42:6802/9560 >>
> 172.22.4.4:0/1000901 pipe(0x7f920802e7d0 sd=401 :6802 s=4 pgs=73173 cs=1 l=1
> c=0x7f920802eda0).writer finishing
> | 2014-08-06 15:22:35.344719 7f8fd1d98700  2 -- 172.22.4.42:6802/9560 >>
> 172.22.4.49:0/3026524 pipe(0x7f9208033a80 sd=492 :6802 s=2 pgs=12845 cs=1 l=1
> c=0x7f9208033ce0).reader couldn't read tag, Success
> 
> and so on, generating 1000s of log lines. The OSDs were spinning with 100%
> CPU, trying to re-connect in rapid succession. The repeated connection
> attempts stopped nf_conntrack from getting out of its overflown state.
> 
> Thus, we saw blocked requests for 15 minutes or so, until the MONs banned the
> stuck OSDs from the cluster.
> 
> As a short term countermeasure, we stopped all OSDs on the affected hosts and
> started them one by one, leaving enough time in between to allow the recovery
> settle a bit (10 sec gap between OSDs was enough). During normal operation, we
> see only 5000-6000 connections on a host.
> 
> As a permanent fix, we have doubled the size of the nf_conntrack table and
> reduced some timeouts according to
> .
> Now a restart of all 8 OSDs on a host works without problems.
> 
> Alternatively, we have considered removing nf_conntrack completely. This,
> however, is not possible since we use host-based firewalling and nf_conntrack
> is wired quite deeply into Linux' firewall code.
> 
> Just to share our experience in case someone experiences the same problem.
> 
> Regards
> 
> Christian
> 
> -- 
> Dipl.-Inf. Christian Kauhaus <>< · k...@gocept.com · systems administration
> gocept gmbh & co. kg · Forsterstraße 29 · 06112 Halle (Saale) · Germany
> http://gocept.com · tel +49 345 219401-11
> Python, Pyramid, Plone, Zope · consulting, development, hosting, operations
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Show IOps per VM/client to find heavy users...

[Andrija Panic]

Hi,

we just had some new clients, and have suffered very big degradation in
CEPH performance for some reasons (we are using CloudStack).

I'm wondering if there is way to monitor OP/s or similar usage by client
connected, so we can isolate the heavy client ?

Also, what is the general best practice to monitor these kind of changes in
CEPH ? I'm talking about R/W or OP/s change or similar...

Thanks,
-- 

Andrija Panić
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Show IOps per VM/client to find heavy users...

[Wido den Hollander]

On 08/08/2014 01:51 PM, Andrija Panic wrote:

Hi,

we just had some new clients, and have suffered very big degradation in
CEPH performance for some reasons (we are using CloudStack).

I'm wondering if there is way to monitor OP/s or similar usage by client
connected, so we can isolate the heavy client ?



This is not very easy to do with Ceph, but CloudStack keeps track of 
this in the usage database.


With never versions of CloudStack you can also limit the IOps of 
Instances to prevent such situations.



Also, what is the general best practice to monitor these kind of changes
in CEPH ? I'm talking about R/W or OP/s change or similar...

Thanks,
--

Andrija Panić



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




--
Wido den Hollander
42on B.V.
Ceph trainer and consultant

Phone: +31 (0)20 700 9902
Skype: contact42on
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Show IOps per VM/client to find heavy users...

[Andrija Panic]

Thanks Wido, yes I'm aware of CloudStack in that sense, but would prefer
some precise OP/s per ceph Image at least...
Will check CloudStack then...

Thx


On 8 August 2014 13:53, Wido den Hollander  wrote:

> On 08/08/2014 01:51 PM, Andrija Panic wrote:
>
>> Hi,
>>
>> we just had some new clients, and have suffered very big degradation in
>> CEPH performance for some reasons (we are using CloudStack).
>>
>> I'm wondering if there is way to monitor OP/s or similar usage by client
>> connected, so we can isolate the heavy client ?
>>
>>
> This is not very easy to do with Ceph, but CloudStack keeps track of this
> in the usage database.
>
> With never versions of CloudStack you can also limit the IOps of Instances
> to prevent such situations.
>
>  Also, what is the general best practice to monitor these kind of changes
>> in CEPH ? I'm talking about R/W or OP/s change or similar...
>>
>> Thanks,
>> --
>>
>> Andrija Panić
>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>
> --
> Wido den Hollander
> 42on B.V.
> Ceph trainer and consultant
>
> Phone: +31 (0)20 700 9902
> Skype: contact42on
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 

Andrija Panić
--
  http://admintweets.com
--
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] nf_conntrack overflow crashes OSDs

[Robert van Leeuwen]

> today I'd like to share a severe problem we've found (and fixed) on our Ceph
> cluster. We're running 48 OSDs (8 per host). While restarting all OSDs on a
> host, the kernel's nf_conntrack table was overflown. This rendered all OSDs on
> that machine unusable.

It is also possible to specifically not conntrack certain connections.
e.g.
iptables -t raw -A PREROUTING -p tcp --dport 6789 -j CT --notrack

Note that you will have to make the rules in both traffic flows since the 
connections are no longer tracked it does not automatically accepts the return 
packets...

Cheers,
Robert van Leeuwen

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Show IOps per VM/client to find heavy users...

[Wido den Hollander]

On 08/08/2014 02:02 PM, Andrija Panic wrote:

Thanks Wido, yes I'm aware of CloudStack in that sense, but would prefer
some precise OP/s per ceph Image at least...
Will check CloudStack then...



Ceph doesn't really know that since RBD is just a layer on top of RADOS. 
In the end the CloudStack hypervisors are doing I/O towards RADOS 
objects, so giving exact stats of how many IOps you are seeing per image 
is hard to figure out.


The hypervisor knows this best since it sees all the I/O going through.

Wido


Thx


On 8 August 2014 13:53, Wido den Hollander mailto:w...@42on.com>> wrote:

On 08/08/2014 01:51 PM, Andrija Panic wrote:

Hi,

we just had some new clients, and have suffered very big
degradation in
CEPH performance for some reasons (we are using CloudStack).

I'm wondering if there is way to monitor OP/s or similar usage
by client
connected, so we can isolate the heavy client ?


This is not very easy to do with Ceph, but CloudStack keeps track of
this in the usage database.

With never versions of CloudStack you can also limit the IOps of
Instances to prevent such situations.

Also, what is the general best practice to monitor these kind of
changes
in CEPH ? I'm talking about R/W or OP/s change or similar...

Thanks,
--

Andrija Panić



_
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/__listinfo.cgi/ceph-users-ceph.__com




--
Wido den Hollander
42on B.V.
Ceph trainer and consultant

Phone: +31 (0)20 700 9902 
Skype: contact42on
_
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/__listinfo.cgi/ceph-users-ceph.__com





--

Andrija Panić
--
http://admintweets.com
--



--
Wido den Hollander
42on B.V.
Ceph trainer and consultant

Phone: +31 (0)20 700 9902
Skype: contact42on
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Show IOps per VM/client to find heavy users...

[Andrija Panic]

Hm, true...
One final question, I might be a noob...
13923 B/s rd, 4744 kB/s wr, 1172 op/s
what does this op/s represent - is it classic IOps (4k reads/writes) or
something else ? how much is too much :)  - I'm familiar with SATA/SSD IO/s
specs/tests, etc, but not sure what CEPH menas by op/s - could not find
anything with google...

Thanks again Wido.
Andrija


On 8 August 2014 14:07, Wido den Hollander  wrote:

> On 08/08/2014 02:02 PM, Andrija Panic wrote:
>
>> Thanks Wido, yes I'm aware of CloudStack in that sense, but would prefer
>> some precise OP/s per ceph Image at least...
>> Will check CloudStack then...
>>
>>
> Ceph doesn't really know that since RBD is just a layer on top of RADOS.
> In the end the CloudStack hypervisors are doing I/O towards RADOS objects,
> so giving exact stats of how many IOps you are seeing per image is hard to
> figure out.
>
> The hypervisor knows this best since it sees all the I/O going through.
>
> Wido
>
>  Thx
>>
>>
>> On 8 August 2014 13:53, Wido den Hollander > > wrote:
>>
>> On 08/08/2014 01:51 PM, Andrija Panic wrote:
>>
>> Hi,
>>
>> we just had some new clients, and have suffered very big
>> degradation in
>> CEPH performance for some reasons (we are using CloudStack).
>>
>> I'm wondering if there is way to monitor OP/s or similar usage
>> by client
>> connected, so we can isolate the heavy client ?
>>
>>
>> This is not very easy to do with Ceph, but CloudStack keeps track of
>> this in the usage database.
>>
>> With never versions of CloudStack you can also limit the IOps of
>> Instances to prevent such situations.
>>
>> Also, what is the general best practice to monitor these kind of
>> changes
>> in CEPH ? I'm talking about R/W or OP/s change or similar...
>>
>> Thanks,
>> --
>>
>> Andrija Panić
>>
>>
>>
>> _
>> ceph-users mailing list
>> ceph-users@lists.ceph.com 
>> http://lists.ceph.com/__listinfo.cgi/ceph-users-ceph.__com
>>
>> 
>>
>>
>>
>> --
>> Wido den Hollander
>> 42on B.V.
>> Ceph trainer and consultant
>>
>> Phone: +31 (0)20 700 9902 
>> Skype: contact42on
>> _
>> ceph-users mailing list
>> ceph-users@lists.ceph.com 
>> http://lists.ceph.com/__listinfo.cgi/ceph-users-ceph.__com
>>
>> 
>>
>>
>>
>>
>> --
>>
>> Andrija Panić
>> --
>> http://admintweets.com
>> --
>>
>
>
> --
> Wido den Hollander
> 42on B.V.
> Ceph trainer and consultant
>
> Phone: +31 (0)20 700 9902
> Skype: contact42on
>



-- 

Andrija Panić
--
  http://admintweets.com
--
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Show IOps per VM/client to find heavy users...

[Dan Van Der Ster]

Hi,
Here’s what we do to identify our top RBD users.

First, enable log level 10 for the filestore so you can see all the IOs coming 
from the VMs. Then use a script like this (used on a dumpling cluster):

  https://github.com/cernceph/ceph-scripts/blob/master/tools/rbd-io-stats.pl

to summarize the osd logs and identify the top clients.

Then its just a matter of scripting to figure out the ops/sec per volume, but 
for us at least the main use-case has been to identify who is responsible for a 
new peak in overall ops — and daily-granular statistics from the above script 
tends to suffice.

BTW, do you throttle your clients? We found that its absolutely necessary, 
since without a throttle just a few active VMs can eat up the entire iops 
capacity of the cluster.

Cheers, Dan

-- Dan van der Ster || Data & Storage Services || CERN IT Department --


On 08 Aug 2014, at 13:51, Andrija Panic 
mailto:andrija.pa...@gmail.com>> wrote:

Hi,

we just had some new clients, and have suffered very big degradation in CEPH 
performance for some reasons (we are using CloudStack).

I'm wondering if there is way to monitor OP/s or similar usage by client 
connected, so we can isolate the heavy client ?

Also, what is the general best practice to monitor these kind of changes in 
CEPH ? I'm talking about R/W or OP/s change or similar...

Thanks,
--

Andrija Panić

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Can't start OSD

[O'Reilly, Dan]

Nope.  Nothing works.  This is VERY frustrating.

What happened:


-  I rebooted the box, simulating a system failure.

-  When the system came back up, ceph wasn't started, and the osd 
volumes weren't mounted.

-  I did a "service ceph start osd" and the ceph processes don't start

-  I did a "ceph-deploy activate" on the devices,  so they're mounted.  
"service ceph start" still doesn't start anything.

Right now:

# service ceph restart
=== osd.18 ===
=== osd.18 ===
Stopping Ceph osd.18 on tm1cldosdl04...done
=== osd.18 ===
create-or-move updated item name 'osd.18' weight 0.45 at location 
{host=tm1cldosdl04,root=default} to crush map
Starting Ceph osd.18 on tm1cldosdl04...
starting osd.18 at :/0 osd_data /var/lib/ceph/osd/ceph-18 
/var/lib/ceph/osd/ceph-18/journal
=== osd.17 ===
=== osd.17 ===
Stopping Ceph osd.17 on tm1cldosdl04...done
=== osd.17 ===
create-or-move updated item name 'osd.17' weight 0.45 at location 
{host=tm1cldosdl04,root=default} to crush map
Starting Ceph osd.17 on tm1cldosdl04...
starting osd.17 at :/0 osd_data /var/lib/ceph/osd/ceph-17 
/var/lib/ceph/osd/ceph-17/journal
=== osd.19 ===
=== osd.19 ===
Stopping Ceph osd.19 on tm1cldosdl04...done
=== osd.19 ===
create-or-move updated item name 'osd.19' weight 0.45 at location 
{host=tm1cldosdl04,root=default} to crush map
Starting Ceph osd.19 on tm1cldosdl04...
starting osd.19 at :/0 osd_data /var/lib/ceph/osd/ceph-19 
/var/lib/ceph/osd/ceph-19/journal
=== osd.16 ===
=== osd.16 ===
Stopping Ceph osd.16 on tm1cldosdl04...done
=== osd.16 ===
create-or-move updated item name 'osd.16' weight 0.45 at location 
{host=tm1cldosdl04,root=default} to crush map
Starting Ceph osd.16 on tm1cldosdl04...
starting osd.16 at :/0 osd_data /var/lib/ceph/osd/ceph-16 
/var/lib/ceph/osd/ceph-16/journal
[NEW:note: root@tm1cldosdl04 on parent: /root]
# ps -eaf|grep ceph
root  7528  6124  0 07:32 pts/000:00:00 grep ceph
[NEW:note: root@tm1cldosdl04 on parent: /root]
# ceph osd tree
# idweight  type name   up/down reweight
-1  9   root default
-2  1.8 host tm1cldosdl01
0   0.45osd.0   up  1
1   0.45osd.1   up  1
2   0.45osd.2   up  1
3   0.45osd.3   up  1
-3  1.8 host tm1cldosdl02
4   0.45osd.4   up  1
5   0.45osd.5   up  1
6   0.45osd.6   up  1
7   0.45osd.7   up  1
-4  1.8 host tm1cldosdl03
8   0.45osd.8   up  1
9   0.45osd.9   up  1
10  0.45osd.10  up  1
11  0.45osd.11  up  1
-5  3.6 host tm1cldosdl04
12  0.45osd.12  DNE
13  0.45osd.13  DNE
14  0.45osd.14  DNE
15  0.45osd.15  DNE
16  0.45osd.16  down0
17  0.45osd.17  down0
18  0.45osd.18  down0
19  0.45osd.19  down0

I'm missing something here.  I don't know if it's a config issue or what.  But 
the docs haven't helped me.

From: Karan Singh [mailto:karan.si...@csc.fi]
Sent: Friday, August 08, 2014 1:11 AM
To: O'Reilly, Dan
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Can't start OSD

Try to make these OSD IN

ceph osd in osd.12 osd.13 osd.14 osd.15

Then restart osd services


- Karan Singh -

On 08 Aug 2014, at 00:55, O'Reilly, Dan 
mailto:daniel.orei...@dish.com>> wrote:


# idweight  type name   up/down reweight
-1  7.2 root default
-2  1.8 host tm1cldosdl01
0   0.45osd.0   up  1
1   0.45osd.1   up  1
2   0.45osd.2   up  1
3   0.45osd.3   up  1
-3  1.8 host tm1cldosdl02
4   0.45osd.4   up  1
5   0.45osd.5   up  1
6   0.45osd.6   up  1
7   0.45osd.7   up  1
-4  1.8 host tm1cldosdl03
8   0.45osd.8   up  1
9   0.45osd.9   up  1
10  0.45osd.10  up  1
11  0.45osd.11  up  1
-5  1.8 host tm1cldosdl04
12  0.45osd.12  down0
13  0.45osd.13  down0
14  0.45osd.14  down0
15  0.45osd.15  down0
[ceph@tm1cldosdl04 ~]$ sudo /etc/init.d/ceph start osd.12
/etc/init.d/ceph: osd.12 not found (/etc/ceph/ceph.conf defines , /var/lib/ceph 
defines )

What am I missing?  Specifically, what would need to be in ceph.conf or 
/var/lib/ceph?

Dan O'R

Re: [ceph-users] Show IOps per VM/client to find heavy users...

[Andrija Panic]

Hi Dan,

thank you very much for the script, will check it out...no thortling so
far, but I guess it will have to be done...

This seems to read only gziped logs? so since read only I guess it is safe
to run it on proudction cluster now... ?
The script will also check for mulitply OSDs as far as I can understadn,
not just osd.0 given in script comment ?

Thanks a lot.
Andrija




On 8 August 2014 15:44, Dan Van Der Ster  wrote:

>  Hi,
> Here’s what we do to identify our top RBD users.
>
>  First, enable log level 10 for the filestore so you can see all the IOs
> coming from the VMs. Then use a script like this (used on a dumpling
> cluster):
>
>
> https://github.com/cernceph/ceph-scripts/blob/master/tools/rbd-io-stats.pl
>
>  to summarize the osd logs and identify the top clients.
>
>  Then its just a matter of scripting to figure out the ops/sec per
> volume, but for us at least the main use-case has been to identify who is
> responsible for a new peak in overall ops — and daily-granular statistics
> from the above script tends to suffice.
>
>  BTW, do you throttle your clients? We found that its absolutely
> necessary, since without a throttle just a few active VMs can eat up the
> entire iops capacity of the cluster.
>
>  Cheers, Dan
>
> -- Dan van der Ster || Data & Storage Services || CERN IT Department --
>
>
>  On 08 Aug 2014, at 13:51, Andrija Panic  wrote:
>
>  Hi,
>
>  we just had some new clients, and have suffered very big degradation in
> CEPH performance for some reasons (we are using CloudStack).
>
>  I'm wondering if there is way to monitor OP/s or similar usage by client
> connected, so we can isolate the heavy client ?
>
>  Also, what is the general best practice to monitor these kind of changes
> in CEPH ? I'm talking about R/W or OP/s change or similar...
>
>  Thanks,
> --
>
> Andrija Panić
>
>   ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>


-- 

Andrija Panić
--
  http://admintweets.com
--
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Show IOps per VM/client to find heavy users...

[Dan Van Der Ster]

Hi,

On 08 Aug 2014, at 15:55, Andrija Panic 
mailto:andrija.pa...@gmail.com>> wrote:

Hi Dan,

thank you very much for the script, will check it out...no thortling so far, 
but I guess it will have to be done...

This seems to read only gziped logs?

Well it’s pretty simple, and it zcat’s each input file. So yes, only gz files 
in the current script. But you can change that pretty trivially ;)

so since read only I guess it is safe to run it on proudction cluster now… ?

I personally don’t do anything new on a Friday just before leaving ;)

But its just grepping the log files, so start with one, then two, then...

The script will also check for mulitply OSDs as far as I can understadn, not 
just osd.0 given in script comment ?


Yup, what I do is gather all of the OSD logs for a single day in a single 
directory (in CephFS ;), then run that script on all of the OSDs. It takes 
awhile, but it will give you the overall daily totals for the whole cluster.

If you are only trying to find the top users, then it is sufficient to check a 
subset of OSDs, since by their nature the client IOs are spread across most/all 
OSDs.

Cheers, Dan

Thanks a lot.
Andrija




On 8 August 2014 15:44, Dan Van Der Ster 
mailto:daniel.vanders...@cern.ch>> wrote:
Hi,
Here’s what we do to identify our top RBD users.

First, enable log level 10 for the filestore so you can see all the IOs coming 
from the VMs. Then use a script like this (used on a dumpling cluster):

  https://github.com/cernceph/ceph-scripts/blob/master/tools/rbd-io-stats.pl

to summarize the osd logs and identify the top clients.

Then its just a matter of scripting to figure out the ops/sec per volume, but 
for us at least the main use-case has been to identify who is responsible for a 
new peak in overall ops — and daily-granular statistics from the above script 
tends to suffice.

BTW, do you throttle your clients? We found that its absolutely necessary, 
since without a throttle just a few active VMs can eat up the entire iops 
capacity of the cluster.

Cheers, Dan

-- Dan van der Ster || Data & Storage Services || CERN IT Department --


On 08 Aug 2014, at 13:51, Andrija Panic 
mailto:andrija.pa...@gmail.com>> wrote:

Hi,

we just had some new clients, and have suffered very big degradation in CEPH 
performance for some reasons (we are using CloudStack).

I'm wondering if there is way to monitor OP/s or similar usage by client 
connected, so we can isolate the heavy client ?

Also, what is the general best practice to monitor these kind of changes in 
CEPH ? I'm talking about R/W or OP/s change or similar...

Thanks,
--

Andrija Panić

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




--

Andrija Panić
--
  http://admintweets.com
--

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Show IOps per VM/client to find heavy users...

[Andrija Panic]

Thanks again, and btw, beside being Friday I'm also on vacation - so double
the joy of troubleshooting performance problmes :)))

Thx :)


On 8 August 2014 16:01, Dan Van Der Ster  wrote:

>  Hi,
>
>  On 08 Aug 2014, at 15:55, Andrija Panic  wrote:
>
>  Hi Dan,
>
>  thank you very much for the script, will check it out...no thortling so
> far, but I guess it will have to be done...
>
>  This seems to read only gziped logs?
>
>
>  Well it’s pretty simple, and it zcat’s each input file. So yes, only gz
> files in the current script. But you can change that pretty trivially ;)
>
>  so since read only I guess it is safe to run it on proudction cluster
> now… ?
>
>
>  I personally don’t do anything new on a Friday just before leaving ;)
>
>  But its just grepping the log files, so start with one, then two, then...
>
>   The script will also check for mulitply OSDs as far as I can
> understadn, not just osd.0 given in script comment ?
>
>
>  Yup, what I do is gather all of the OSD logs for a single day in a
> single directory (in CephFS ;), then run that script on all of the OSDs. It
> takes awhile, but it will give you the overall daily totals for the whole
> cluster.
>
>  If you are only trying to find the top users, then it is sufficient to
> check a subset of OSDs, since by their nature the client IOs are spread
> across most/all OSDs.
>
>  Cheers, Dan
>
>  Thanks a lot.
> Andrija
>
>
>
>
> On 8 August 2014 15:44, Dan Van Der Ster 
> wrote:
>
>> Hi,
>> Here’s what we do to identify our top RBD users.
>>
>>  First, enable log level 10 for the filestore so you can see all the IOs
>> coming from the VMs. Then use a script like this (used on a dumpling
>> cluster):
>>
>>
>> https://github.com/cernceph/ceph-scripts/blob/master/tools/rbd-io-stats.pl
>>
>>  to summarize the osd logs and identify the top clients.
>>
>>  Then its just a matter of scripting to figure out the ops/sec per
>> volume, but for us at least the main use-case has been to identify who is
>> responsible for a new peak in overall ops — and daily-granular statistics
>> from the above script tends to suffice.
>>
>>  BTW, do you throttle your clients? We found that its absolutely
>> necessary, since without a throttle just a few active VMs can eat up the
>> entire iops capacity of the cluster.
>>
>>  Cheers, Dan
>>
>> -- Dan van der Ster || Data & Storage Services || CERN IT Department --
>>
>>
>>   On 08 Aug 2014, at 13:51, Andrija Panic 
>> wrote:
>>
>>Hi,
>>
>>  we just had some new clients, and have suffered very big degradation in
>> CEPH performance for some reasons (we are using CloudStack).
>>
>>  I'm wondering if there is way to monitor OP/s or similar usage by
>> client connected, so we can isolate the heavy client ?
>>
>>  Also, what is the general best practice to monitor these kind of
>> changes in CEPH ? I'm talking about R/W or OP/s change or similar...
>>
>>  Thanks,
>> --
>>
>> Andrija Panić
>>
>>___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>>
>
>
>  --
>
> Andrija Panić
> --
>   http://admintweets.com
> --
>
>
>


-- 

Andrija Panić
--
  http://admintweets.com
--
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Show IOps per VM/client to find heavy users...

[Wido den Hollander]

On 08/08/2014 03:44 PM, Dan Van Der Ster wrote:

Hi,
Here’s what we do to identify our top RBD users.

First, enable log level 10 for the filestore so you can see all the IOs
coming from the VMs. Then use a script like this (used on a dumpling
cluster):

https://github.com/cernceph/ceph-scripts/blob/master/tools/rbd-io-stats.pl

to summarize the osd logs and identify the top clients.

Then its just a matter of scripting to figure out the ops/sec per
volume, but for us at least the main use-case has been to identify who
is responsible for a new peak in overall ops — and daily-granular
statistics from the above script tends to suffice.

BTW, do you throttle your clients? We found that its absolutely
necessary, since without a throttle just a few active VMs can eat up the
entire iops capacity of the cluster.


+1

I'd strongly advise to set I/O limits for Instances. I've had multiple 
occasions where a runaway script inside a VM was hammering on the 
underlying storage killing all I/O.


Not only with Ceph, but over the many years I've worked with storage. 
I/O == expensive


CloudStack supports I/O limiting, so I recommend you set a limit. Set it 
to 750 write IOps for example. That way one Instance can't kill the 
whole cluster, but it still has enough I/O to run. (usually).


Wido



Cheers, Dan

-- Dan van der Ster || Data & Storage Services || CERN IT Department --


On 08 Aug 2014, at 13:51, Andrija Panic mailto:andrija.pa...@gmail.com>> wrote:


Hi,

we just had some new clients, and have suffered very big degradation
in CEPH performance for some reasons (we are using CloudStack).

I'm wondering if there is way to monitor OP/s or similar usage by
client connected, so we can isolate the heavy client ?

Also, what is the general best practice to monitor these kind of
changes in CEPH ? I'm talking about R/W or OP/s change or similar...

Thanks,
--

Andrija Panić

___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




--
Wido den Hollander
42on B.V.
Ceph trainer and consultant

Phone: +31 (0)20 700 9902
Skype: contact42on
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] nf_conntrack overflow crashes OSDs

[Christian Kauhaus]

Am 08.08.2014 um 14:05 schrieb Robert van Leeuwen:
> It is also possible to specifically not conntrack certain connections.
> e.g.
> iptables -t raw -A PREROUTING -p tcp --dport 6789 -j CT --notrack

Thanks Robert. This is really an interesting approach. We will test it.

Regards

Christian

-- 
Dipl.-Inf. Christian Kauhaus <>< · k...@gocept.com · systems administration
gocept gmbh & co. kg · Forsterstraße 29 · 06112 Halle (Saale) · Germany
http://gocept.com · tel +49 345 219401-11
Python, Pyramid, Plone, Zope · consulting, development, hosting, operations
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Can't start OSD

[German Anders]

How about the logs? Is something there?

ls /var/log/ceph/


German Anders



















--- Original message ---
Asunto: Re: [ceph-users] Can't start OSD
De: "O'Reilly, Dan" 
Para: Karan Singh 
Cc: ceph-users@lists.ceph.com 
Fecha: Friday, 08/08/2014 10:53



Nope.  Nothing works.  This is VERY frustrating.

What happened:

-  I rebooted the box, simulating a system failure.
-  When the system came back up, ceph wasn’t started, and 
the osd volumes weren’t mounted.
-  I did a “service ceph start osd” and the ceph processes 
don’t start
-  I did a “ceph-deploy activate” on the devices,  so 
they’re mounted.  “service ceph start” still doesn’t start 
anything.


Right now:

# service ceph restart
=== osd.18 ===
=== osd.18 ===
Stopping Ceph osd.18 on tm1cldosdl04...done
=== osd.18 ===
create-or-move updated item name 'osd.18' weight 0.45 at location 
{host=tm1cldosdl04,root=default} to crush map

Starting Ceph osd.18 on tm1cldosdl04...
starting osd.18 at :/0 osd_data /var/lib/ceph/osd/ceph-18 
/var/lib/ceph/osd/ceph-18/journal

=== osd.17 ===
=== osd.17 ===
Stopping Ceph osd.17 on tm1cldosdl04...done
=== osd.17 ===
create-or-move updated item name 'osd.17' weight 0.45 at location 
{host=tm1cldosdl04,root=default} to crush map

Starting Ceph osd.17 on tm1cldosdl04...
starting osd.17 at :/0 osd_data /var/lib/ceph/osd/ceph-17 
/var/lib/ceph/osd/ceph-17/journal

=== osd.19 ===
=== osd.19 ===
Stopping Ceph osd.19 on tm1cldosdl04...done
=== osd.19 ===
create-or-move updated item name 'osd.19' weight 0.45 at location 
{host=tm1cldosdl04,root=default} to crush map

Starting Ceph osd.19 on tm1cldosdl04...
starting osd.19 at :/0 osd_data /var/lib/ceph/osd/ceph-19 
/var/lib/ceph/osd/ceph-19/journal

=== osd.16 ===
=== osd.16 ===
Stopping Ceph osd.16 on tm1cldosdl04...done
=== osd.16 ===
create-or-move updated item name 'osd.16' weight 0.45 at location 
{host=tm1cldosdl04,root=default} to crush map

Starting Ceph osd.16 on tm1cldosdl04...
starting osd.16 at :/0 osd_data /var/lib/ceph/osd/ceph-16 
/var/lib/ceph/osd/ceph-16/journal

[NEW:note: root@tm1cldosdl04 on parent: /root]
# ps -eaf|grep ceph
root  7528  6124  0 07:32 pts/000:00:00 grep ceph
[NEW:note: root@tm1cldosdl04 on parent: /root]
# ceph osd tree
# idweight  type name   up/down reweight
-1  9   root default
-2  1.8 host tm1cldosdl01
0   0.45osd.0   up  1
1   0.45osd.1   up  1
2   0.45osd.2   up  1
3   0.45osd.3   up  1
-3  1.8 host tm1cldosdl02
4   0.45osd.4   up  1
5   0.45osd.5   up  1
6   0.45osd.6   up  1
7   0.45osd.7   up  1
-4  1.8 host tm1cldosdl03
8   0.45osd.8   up  1
9   0.45osd.9   up  1
10  0.45osd.10  up  1
11  0.45osd.11  up  1
-5  3.6 host tm1cldosdl04
12  0.45osd.12  DNE
13  0.45osd.13  DNE
14  0.45osd.14  DNE
15  0.45osd.15  DNE
16  0.45osd.16  down0
17  0.45osd.17  down0
18  0.45osd.18  down0
19  0.45osd.19  down0

I’m missing something here.  I don’t know if it’s a config issue 
or what.  But the docs haven’t helped me.




From: Karan Singh [mailto:karan.si...@csc.fi]
Sent: Friday, August 08, 2014 1:11 AM
To: O'Reilly, Dan
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Can't start OSD


Try to make these OSD IN



ceph osd in osd.12 osd.13 osd.14 osd.15



Then restart osd services






- Karan Singh -



On 08 Aug 2014, at 00:55, O'Reilly, Dan  
wrote:






# idweight  type name   up/down reweight

-1  7.2 root default

-2  1.8 host tm1cldosdl01

0   0.45osd.0   up  1

1   0.45osd.1   up  1

2   0.45osd.2   up  1

3   0.45osd.3   up  1

-3  1.8 host tm1cldosdl02

4   0.45osd.4   up  1

5   0.45osd.5   up  1

6   0.45osd.6   up  1

7   0.45osd.7   up  1

-4  1.8 host tm1cldosdl03

8   0.45osd.8   up  1

9   0.45osd.9   up  1

10  0.45osd.10  up  1

11  0.45osd.11  up  1

-5  1.8 host tm1cldosdl04

12  0.45osd.12  down0

13  0.45osd.13  down0

14  0.45osd.14  down0

15  0.45   

Re: [ceph-users] Show IOps per VM/client to find heavy users...

[Andrija Panic]

Will do so definitively, thanks Wido and Dan...
Cheers guys


On 8 August 2014 16:13, Wido den Hollander  wrote:

> On 08/08/2014 03:44 PM, Dan Van Der Ster wrote:
>
>> Hi,
>> Here’s what we do to identify our top RBD users.
>>
>> First, enable log level 10 for the filestore so you can see all the IOs
>> coming from the VMs. Then use a script like this (used on a dumpling
>> cluster):
>>
>> https://github.com/cernceph/ceph-scripts/blob/master/
>> tools/rbd-io-stats.pl
>>
>> to summarize the osd logs and identify the top clients.
>>
>> Then its just a matter of scripting to figure out the ops/sec per
>> volume, but for us at least the main use-case has been to identify who
>> is responsible for a new peak in overall ops — and daily-granular
>> statistics from the above script tends to suffice.
>>
>> BTW, do you throttle your clients? We found that its absolutely
>> necessary, since without a throttle just a few active VMs can eat up the
>> entire iops capacity of the cluster.
>>
>
> +1
>
> I'd strongly advise to set I/O limits for Instances. I've had multiple
> occasions where a runaway script inside a VM was hammering on the
> underlying storage killing all I/O.
>
> Not only with Ceph, but over the many years I've worked with storage. I/O
> == expensive
>
> CloudStack supports I/O limiting, so I recommend you set a limit. Set it
> to 750 write IOps for example. That way one Instance can't kill the whole
> cluster, but it still has enough I/O to run. (usually).
>
> Wido
>
>
>> Cheers, Dan
>>
>> -- Dan van der Ster || Data & Storage Services || CERN IT Department --
>>
>>
>> On 08 Aug 2014, at 13:51, Andrija Panic > > wrote:
>>
>>  Hi,
>>>
>>> we just had some new clients, and have suffered very big degradation
>>> in CEPH performance for some reasons (we are using CloudStack).
>>>
>>> I'm wondering if there is way to monitor OP/s or similar usage by
>>> client connected, so we can isolate the heavy client ?
>>>
>>> Also, what is the general best practice to monitor these kind of
>>> changes in CEPH ? I'm talking about R/W or OP/s change or similar...
>>>
>>> Thanks,
>>> --
>>>
>>> Andrija Panić
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com 
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>
> --
> Wido den Hollander
> 42on B.V.
> Ceph trainer and consultant
>
> Phone: +31 (0)20 700 9902
> Skype: contact42on
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 

Andrija Panić
--
  http://admintweets.com
--
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Can't start OSD

[O'Reilly, Dan]

I’m afraid I don’t know exactly how to interpret this, but after a reboot:

2014-08-08 08:48:44.616005 7f0c3b1447a0  0 ceph version 0.80.1 
(a38fe1169b6d2ac98b427334c12d7cf81f809b74), process ceph-osd, pid 2978
2014-08-08 08:48:44.635680 7f0c3b1447a0  0 filestore(/var/lib/ceph/osd/ceph-10) 
mount detected xfs (libxfs)
2014-08-08 08:48:44.635730 7f0c3b1447a0  1 filestore(/var/lib/ceph/osd/ceph-10) 
 disabling 'filestore replica fadvise' due to known issues with 
fadvise(DONTNEED) on xfs
2014-08-08 08:48:44.681911 7f0c3b1447a0  0 
genericfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_features: FIEMAP 
ioctl is supported and appears to work
2014-08-08 08:48:44.681959 7f0c3b1447a0  0 
genericfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_features: FIEMAP 
ioctl is disabled via 'filestore fiemap' config option
2014-08-08 08:48:44.748483 7f0c3b1447a0  0 
genericfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_features: 
syscall(SYS_syncfs, fd) fully supported
2014-08-08 08:48:44.748605 7f0c3b1447a0  0 
xfsfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_feature: extsize is 
supported
2014-08-08 08:48:44.889826 7f0c3b1447a0  0 filestore(/var/lib/ceph/osd/ceph-10) 
mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2014-08-08 08:48:45.064198 7f0c3b1447a0 -1 filestore(/var/lib/ceph/osd/ceph-10) 
mount failed to open journal /var/lib/ceph/osd/ceph-10/journal: (2) No such 
file or directory
2014-08-08 08:48:45.074220 7f0c3b1447a0 -1  ** ERROR: error converting store 
/var/lib/ceph/osd/ceph-10: (2) No such file or directory
2014-08-08 08:49:19.957725 7f2c40c1a7a0  0 ceph version 0.80.1 
(a38fe1169b6d2ac98b427334c12d7cf81f809b74), process ceph-osd, pid 4707
2014-08-08 08:49:19.973896 7f2c40c1a7a0  0 filestore(/var/lib/ceph/osd/ceph-10) 
mount detected xfs (libxfs)
2014-08-08 08:49:19.973931 7f2c40c1a7a0  1 filestore(/var/lib/ceph/osd/ceph-10) 
 disabling 'filestore replica fadvise' due to known issues with 
fadvise(DONTNEED) on xfs
2014-08-08 08:49:20.016413 7f2c40c1a7a0  0 
genericfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_features: FIEMAP 
ioctl is supported and appears to work
2014-08-08 08:49:20.016444 7f2c40c1a7a0  0 
genericfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_features: FIEMAP 
ioctl is disabled via 'filestore fiemap' config option
2014-08-08 08:49:20.083052 7f2c40c1a7a0  0 
genericfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_features: 
syscall(SYS_syncfs, fd) fully supported
2014-08-08 08:49:20.083179 7f2c40c1a7a0  0 
xfsfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_feature: extsize is 
supported
2014-08-08 08:49:20.134213 7f2c40c1a7a0  0 filestore(/var/lib/ceph/osd/ceph-10) 
mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2014-08-08 08:49:20.136710 7f2c40c1a7a0 -1 filestore(/var/lib/ceph/osd/ceph-10) 
mount failed to open journal /var/lib/ceph/osd/ceph-10/journal: (2) No such 
file or directory
2014-08-08 08:49:20.146797 7f2c40c1a7a0 -1  ** ERROR: error converting store 
/var/lib/ceph/osd/ceph-10: (2) No such file or directory

From: German Anders [mailto:gand...@despegar.com]
Sent: Friday, August 08, 2014 8:23 AM
To: O'Reilly, Dan
Cc: Karan Singh; ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Can't start OSD

How about the logs? Is something there?

ls /var/log/ceph/

German Anders















--- Original message ---
Asunto: Re: [ceph-users] Can't start OSD
De: "O'Reilly, Dan" mailto:daniel.orei...@dish.com>>
Para: Karan Singh mailto:karan.si...@csc.fi>>
Cc: ceph-users@lists.ceph.com 
mailto:ceph-users@lists.ceph.com>>
Fecha: Friday, 08/08/2014 10:53


Nope.  Nothing works.  This is VERY frustrating.

What happened:

-  I rebooted the box, simulating a system failure.
-  When the system came back up, ceph wasn’t started, and the osd 
volumes weren’t mounted.
-  I did a “service ceph start osd” and the ceph processes don’t start
-  I did a “ceph-deploy activate” on the devices,  so they’re mounted.  
“service ceph start” still doesn’t start anything.

Right now:

# service ceph restart
=== osd.18 ===
=== osd.18 ===
Stopping Ceph osd.18 on tm1cldosdl04...done
=== osd.18 ===
create-or-move updated item name 'osd.18' weight 0.45 at location 
{host=tm1cldosdl04,root=default} to crush map
Starting Ceph osd.18 on tm1cldosdl04...
starting osd.18 at :/0 osd_data /var/lib/ceph/osd/ceph-18 
/var/lib/ceph/osd/ceph-18/journal
=== osd.17 ===
=== osd.17 ===
Stopping Ceph osd.17 on tm1cldosdl04...done
=== osd.17 ===
create-or-move updated item name 'osd.17' weight 0.45 at location 
{host=tm1cldosdl04,root=default} to crush map
Starting Ceph osd.17 on tm1cldosdl04...
starting osd.17 at :/0 osd_data /var/lib/ceph/osd/ceph-17 
/var/lib/ceph/osd/ceph-17/journal
=== osd.19 ===
=== osd.19 ===
Stopping Ceph osd.19 on tm1cldosdl04...done
=== osd.19 ===
create-or-move updated item name 'osd.19' weight 0.45 at location 
{host=tm1cldosdl04,root=default} to crush map
Starting Ceph osd

Re: [ceph-users] Ceph runs great then falters

[Chris Kitzmiller]

On Aug 4, 2014, at 10:53 PM, Christian Balzer wrote:
> On Mon, 4 Aug 2014 15:11:39 -0400 Chris Kitzmiller wrote:
>> On Aug 2, 2014, at 12:03 AM, Christian Balzer wrote:
>>> On Fri, 1 Aug 2014 14:23:28 -0400 Chris Kitzmiller wrote:
 I have 3 nodes each running a MON and 30 OSDs. 
 ...
 When I test my cluster
 with either rados bench or with fio via a 10GbE client using RBD I get
 great initial speeds >900MBps and I max out my 10GbE links for a
 while. Then, something goes wrong the performance falters and the
 cluster stops responding all together. I'll see a monitor call for a
 new election and then my OSDs mark each other down, they complain
 that they've been wrongly marked down, I get slow request warnings of
 30 and >60 seconds. This eventually resolves itself and the cluster
 recovers but it then recurs again right away. Sometimes, via fio, I'll get
 an I/O error and it will bail.

This appears to still be happening. :(

From your advice, Christian, I monitored my cluster with atop and found that I 
did have one HDD which was pegged at 100% while the rest of the cluster was at 
0% utilization. I replaced that disk and set my cluster back up again. Wrote 
~20T of data into a 3x pool and that went very smoothly. My speeds did decrease 
from ~600MBps down to ~230MBps over the course of that write but I was still 
getting steady responsive writes.

Today, I'm seeing the problem recur. The trouble is that I don't have any drive 
that is at 100% like I used to. In fact, I have no drives which aren't at 0% 
utilization during these incidents. `ceph osd perf` doesn't seem to have any 
useful information in it. dump_historic_ops has what looks like interesting 
information but I'm lost when it comes to interpreting it's output (e.g. 
http://pastebin.com/raw.php?i=4KHFuyGi ). So right now I have two main 
questions:

1) How do I figure out what is going on? What explains the periods of no 
activity seen here http://pastebin.com/raw.php?i=Mv2y3Tka if not a slow OSD 
drive like before?

2) Why does fio exit with IO errors like these?

fio: io_u error on file /mnt/image1/temp.58.fio: Input/output error
 write offset=79754690560, buflen=4194304
fio: io_u error on file /mnt/image1/temp.69.fio: Input/output error
 write offset=67515711488, buflen=4194304
fio: io_u error on file /mnt/image1/temp.71.fio: Input/output error
 write offset=38646317056, buflen=4194304
fio: io_u error on file /mnt/image1/temp.68.fio: Input/output error
 write offset=103263764480, buflen=4194304
fio: pid=10972, err=5/file:io_u.c:1373, func=io_u error, error=Input/output 
error

4m-randwrite: (groupid=0, jobs=1): err= 5 (file:io_u.c:1373, func=io_u error, 
error=Input/output error): pid=10972: Fri Aug  8 11:01:48 2014

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Apache on Trusty

[Craig Lewis]

Is anybody running Ubuntu Trusty, but using Ceph's apache 2.2 and fastcgi
packages?

I'm a bit of a Ubuntu noob.  I can't figure out the correct
/etc/apt/preferences.d/ configs to prioritize  Ceph's version of the
packages.  I keep getting Ubuntu's apache 2.4 packages.

Can somebody that has this working tell me what configs I need to change?



Alternately, should I just ditch apache entirely, and migrate to Firefly
and Civetweb?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] PGs stuck creating

[Brian Rak]

ceph version 0.80.4 (7c241cfaa6c8c068bc9da8578ca00b9f4fc7567f)

I recently managed to cause some problems for one of our clusters, we 
had 1/3 of the OSDs fail and lose all the data.


I removed all the failed OSDs from the crush map, and did 'ceph osd 
rm'.  Once it finished recovering, I was left with a whole bunch of 
'stale+active+clean' PGs.  These had been hosted entirely on the OSDs 
that failed.


So, there will be some data loss here.  Luckily the majority of the data 
is easily replaceable.  I couldn't do a whole lot with these PGs, so I 
ended up forcing ceph to recreate them, with:


ceph health detail | grep pg | awk '{ print $2 }'  | xargs -n1 ceph pg 
force_create_pg


This fixed most of them, though I'm now left with one that's hanging on 
'creating'.  Any suggestions for what I can do?  There isn't any data to 
lose in this pg, so I would be okay removing it, but I don't see any way 
to do that.  How can I force the OSD to create it again?


cluster e312b58c-0391-43d0-98e6-25a41bea6a70
 health HEALTH_WARN 1 pgs stuck inactive; 1 pgs stuck unclean
 monmap e3: 3 mons at {snip}, election epoch 50, quorum 0,1,2 {snip}
 osdmap e3922: 11 osds: 11 up, 11 in
  pgmap v1261502: 4722 pgs, 14 pools, 4344 GB data, 3314 kobjects
8668 GB used, 11803 GB / 20472 GB avail
   1 creating
4721 active+clean
  client io 449 kB/s rd, 0 B/s wr, 643 op/s

# ceph pg dump | grep creating
dumped all in format plain
3.15c   0   0   0   0   0   0   0 
creating2014-08-08 16:18:38.781245  0'0 0:0 [4,2]   
4   [2,4]   2   0'0 0.000'0 0.00


# ceph pg 3.15c query
Error ENOENT: i don't have pgid 3.15c

# ceph pg 3.15c mark_unfound_lost revert
Error ENOENT: i don't have pgid 3.15c

If I try to force a scrub:

2014-08-08 16:41:38.016388 7f33270cd700  0 osd.2 3926 do_command r=0
2014-08-08 16:41:39.775253 7f33270cd700  0 osd.2 3926 do_command r=0
2014-08-08 16:41:42.491501 7f33270cd700  0 osd.2 3926 do_command r=0
2014-08-08 16:41:42.497906 7f33270cd700  0 osd.2 3926 do_command r=-2 i 
don't have pgid 3.15c
2014-08-08 16:41:42.497911 7f33270cd700  0 log [INF] : i don't have pgid 
3.15c


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] PGs stuck creating

[Brian Rak]

Ahh figured it out.  I hadn't removed the dead OSDs from the crush map, 
which was apparently confusing ceph.


I just did 'ceph osd crush rm XXX' for all of them, restarted all the 
online OSDs, and the pg got created!


On 8/8/2014 4:51 PM, Brian Rak wrote:

ceph version 0.80.4 (7c241cfaa6c8c068bc9da8578ca00b9f4fc7567f)

I recently managed to cause some problems for one of our clusters, we 
had 1/3 of the OSDs fail and lose all the data.


I removed all the failed OSDs from the crush map, and did 'ceph osd 
rm'.  Once it finished recovering, I was left with a whole bunch of 
'stale+active+clean' PGs.  These had been hosted entirely on the OSDs 
that failed.


So, there will be some data loss here.  Luckily the majority of the 
data is easily replaceable.  I couldn't do a whole lot with these PGs, 
so I ended up forcing ceph to recreate them, with:


ceph health detail | grep pg | awk '{ print $2 }'  | xargs -n1 ceph pg 
force_create_pg


This fixed most of them, though I'm now left with one that's hanging 
on 'creating'.  Any suggestions for what I can do?  There isn't any 
data to lose in this pg, so I would be okay removing it, but I don't 
see any way to do that.  How can I force the OSD to create it again?


cluster e312b58c-0391-43d0-98e6-25a41bea6a70
 health HEALTH_WARN 1 pgs stuck inactive; 1 pgs stuck unclean
 monmap e3: 3 mons at {snip}, election epoch 50, quorum 0,1,2 {snip}
 osdmap e3922: 11 osds: 11 up, 11 in
  pgmap v1261502: 4722 pgs, 14 pools, 4344 GB data, 3314 kobjects
8668 GB used, 11803 GB / 20472 GB avail
   1 creating
4721 active+clean
  client io 449 kB/s rd, 0 B/s wr, 643 op/s

# ceph pg dump | grep creating
dumped all in format plain
3.15c   0   0   0   0   0   0   0 
creating2014-08-08 16:18:38.781245  0'0 0:0 [4,2]   
4   [2,4]   2   0'0 0.000'0 0.00


# ceph pg 3.15c query
Error ENOENT: i don't have pgid 3.15c

# ceph pg 3.15c mark_unfound_lost revert
Error ENOENT: i don't have pgid 3.15c

If I try to force a scrub:

2014-08-08 16:41:38.016388 7f33270cd700  0 osd.2 3926 do_command r=0
2014-08-08 16:41:39.775253 7f33270cd700  0 osd.2 3926 do_command r=0
2014-08-08 16:41:42.491501 7f33270cd700  0 osd.2 3926 do_command r=0
2014-08-08 16:41:42.497906 7f33270cd700  0 osd.2 3926 do_command r=-2 
i don't have pgid 3.15c
2014-08-08 16:41:42.497911 7f33270cd700  0 log [INF] : i don't have 
pgid 3.15c


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [Ceph-community] working ceph.conf file?

[Andrew Woodward]

Dan,

It is not necessary to specify the OSD data in ceph.conf anymore. Ceph has
two auto-start functions besides this method.

udev rules:
ceph uses a udev rule to scan and attempt to mount (and activate)
partitions with specific GUID set for the partition typecode

> sgdisk --typecode=<>:<> /dev/<>


the exact GUID's to use can be found
https://github.com/ceph/ceph/blob/master/udev/95-ceph-osd.rules. These are
set automaticly using ceph-disk (or ceph-deploy) if it creates the
partition from an empty disk, in the case that it does not, you have to set
them by hand, all be it should probably do this, or at least tell you
should do this.

ceph init script:
the ceph init script will scan /var/lib/ceph/osd (or the otherwise
configured location) for -  (default cluster name is
ceph) folders and attempt to start the osd service for each of them if they
look correct

lastly, and possibly the most annoying way is that you can configure each
OSD and path in ceph.conf, I don't have any good examples as the two prior
are more flexible / require less config.




On Fri, Aug 8, 2014 at 8:53 AM, O'Reilly, Dan 
wrote:

> Does anybody have a good sample ceph.conf file I can use for reference?
> I’m having a problem where OSD’s won’t come back up after a sysem reboot.
>
>
>
> Dan O'Reilly
>
> UNIX Systems Administration
>
> [image: cid:638154011@09122011-048B]
>
> 9601 S. Meridian Blvd.
>
> Englewood, CO 80112
>
> 720-514-6293
>
>
>
>
>
> ___
> Ceph-community mailing list
> ceph-commun...@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-community-ceph.com
>
>


-- 
Andrew
Mirantis
Ceph community
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [Ceph-community] OSD won't restart after system boot

[Andrew Woodward]

Dan,

I replied to your other thread, Please note that ceph-users is a better
list for this type of subject matter. Also please pop on IRC #ceph on
oftc.net as you are more likely going to be able to hash it out with folks
there if you are so desperate for help.


On Fri, Aug 8, 2014 at 8:55 AM, O'Reilly, Dan 
wrote:

> When I reboot an OSD host, the OSD’s won’t restart.  I’m a newbie on this
> and desperately need help
>
>
>
> Dan O'Reilly
>
> UNIX Systems Administration
>
> [image: cid:638154011@09122011-048B]
>
> 9601 S. Meridian Blvd.
>
> Englewood, CO 80112
>
> 720-514-6293
>
>
>
>
>
> ___
> Ceph-community mailing list
> ceph-commun...@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-community-ceph.com
>
>


-- 
Andrew
Mirantis
Ceph community
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Introductions

[Zach Hill]

Hi all,

I'm Zach Hill, the storage lead at Eucalyptus .
We're working on adding Ceph RBD support for our scale-out block storage
(EBS API). Things are going well, and we've been happy with Ceph thus far.
We are a RHEL/CentOS shop mostly, so any other tips there would be greatly
appreciated.

Our basic architecture is that we have a storage control node that issues
control-plan operations: create image, delete, snapshot, etc. This
controller uses librbd directly via JNA bindings. VMs access the Ceph RBD
Images as block devices exposed via the Qemu/KVM RBD driver on our "Node
Controller" hosts. It's similar to OpenStack Cinder in many ways.

One of the questions we often get is:
Can I run OSDs on my servers that also host VMs?

Generally, we recommend strongly against such a deployment in order to
ensure performance and failure isolation between the compute and storage
sides of the system. But, I'm curious if anyone is doing this in practice
and if they've found reasonable ways to make it work in production.

Thanks for any info in advance, and we're happy to be joining this
community in a more active way.

-Zach
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Can't start OSD

[Matt Harlum]

Hi,

Can you run ls -lah /var/lib/ceph/osd/ceph-10/journal

It’s saying it can’t find the journal

Regards,
Matt 

On 9 Aug 2014, at 12:51 am, O'Reilly, Dan  wrote:

> I’m afraid I don’t know exactly how to interpret this, but after a reboot:
>  
> 2014-08-08 08:48:44.616005 7f0c3b1447a0  0 ceph version 0.80.1 
> (a38fe1169b6d2ac98b427334c12d7cf81f809b74), process ceph-osd, pid 2978
> 2014-08-08 08:48:44.635680 7f0c3b1447a0  0 
> filestore(/var/lib/ceph/osd/ceph-10) mount detected xfs (libxfs)
> 2014-08-08 08:48:44.635730 7f0c3b1447a0  1 
> filestore(/var/lib/ceph/osd/ceph-10)  disabling 'filestore replica fadvise' 
> due to known issues with fadvise(DONTNEED) on xfs
> 2014-08-08 08:48:44.681911 7f0c3b1447a0  0 
> genericfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_features: FIEMAP 
> ioctl is supported and appears to work
> 2014-08-08 08:48:44.681959 7f0c3b1447a0  0 
> genericfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_features: FIEMAP 
> ioctl is disabled via 'filestore fiemap' config option
> 2014-08-08 08:48:44.748483 7f0c3b1447a0  0 
> genericfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_features: 
> syscall(SYS_syncfs, fd) fully supported
> 2014-08-08 08:48:44.748605 7f0c3b1447a0  0 
> xfsfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_feature: extsize is 
> supported
> 2014-08-08 08:48:44.889826 7f0c3b1447a0  0 
> filestore(/var/lib/ceph/osd/ceph-10) mount: enabling WRITEAHEAD journal mode: 
> checkpoint is not enabled
> 2014-08-08 08:48:45.064198 7f0c3b1447a0 -1 
> filestore(/var/lib/ceph/osd/ceph-10) mount failed to open journal 
> /var/lib/ceph/osd/ceph-10/journal: (2) No such file or directory
> 2014-08-08 08:48:45.074220 7f0c3b1447a0 -1  ** ERROR: error converting store 
> /var/lib/ceph/osd/ceph-10: (2) No such file or directory
> 2014-08-08 08:49:19.957725 7f2c40c1a7a0  0 ceph version 0.80.1 
> (a38fe1169b6d2ac98b427334c12d7cf81f809b74), process ceph-osd, pid 4707
> 2014-08-08 08:49:19.973896 7f2c40c1a7a0  0 
> filestore(/var/lib/ceph/osd/ceph-10) mount detected xfs (libxfs)
> 2014-08-08 08:49:19.973931 7f2c40c1a7a0  1 
> filestore(/var/lib/ceph/osd/ceph-10)  disabling 'filestore replica fadvise' 
> due to known issues with fadvise(DONTNEED) on xfs
> 2014-08-08 08:49:20.016413 7f2c40c1a7a0  0 
> genericfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_features: FIEMAP 
> ioctl is supported and appears to work
> 2014-08-08 08:49:20.016444 7f2c40c1a7a0  0 
> genericfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_features: FIEMAP 
> ioctl is disabled via 'filestore fiemap' config option
> 2014-08-08 08:49:20.083052 7f2c40c1a7a0  0 
> genericfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_features: 
> syscall(SYS_syncfs, fd) fully supported
> 2014-08-08 08:49:20.083179 7f2c40c1a7a0  0 
> xfsfilestorebackend(/var/lib/ceph/osd/ceph-10) detect_feature: extsize is 
> supported
> 2014-08-08 08:49:20.134213 7f2c40c1a7a0  0 
> filestore(/var/lib/ceph/osd/ceph-10) mount: enabling WRITEAHEAD journal mode: 
> checkpoint is not enabled
> 2014-08-08 08:49:20.136710 7f2c40c1a7a0 -1 
> filestore(/var/lib/ceph/osd/ceph-10) mount failed to open journal 
> /var/lib/ceph/osd/ceph-10/journal: (2) No such file or directory
> 2014-08-08 08:49:20.146797 7f2c40c1a7a0 -1  ** ERROR: error converting store 
> /var/lib/ceph/osd/ceph-10: (2) No such file or directory
>  
> From: German Anders [mailto:gand...@despegar.com] 
> Sent: Friday, August 08, 2014 8:23 AM
> To: O'Reilly, Dan
> Cc: Karan Singh; ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Can't start OSD
>  
> How about the logs? Is something there?
> 
> ls /var/log/ceph/
>  
> German Anders
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>  
> --- Original message --- 
> Asunto: Re: [ceph-users] Can't start OSD 
> De: "O'Reilly, Dan"  
> Para: Karan Singh  
> Cc: ceph-users@lists.ceph.com  
> Fecha: Friday, 08/08/2014 10:53
> 
> 
> Nope.  Nothing works.  This is VERY frustrating.
>  
> What happened:
>  
> -  I rebooted the box, simulating a system failure.
> -  When the system came back up, ceph wasn’t started, and the osd 
> volumes weren’t mounted.
> -  I did a “service ceph start osd” and the ceph processes don’t start
> -  I did a “ceph-deploy activate” on the devices,  so they’re 
> mounted.  “service ceph start” still doesn’t start anything.
>  
> Right now:
>  
> # service ceph restart
> === osd.18 ===
> === osd.18 ===
> Stopping Ceph osd.18 on tm1cldosdl04...done
> === osd.18 ===
> create-or-move updated item name 'osd.18' weight 0.45 at location 
> {host=tm1cldosdl04,root=default} to crush map
> Starting Ceph osd.18 on tm1cldosdl04...
> starting osd.18 at :/0 osd_data /var/lib/ceph/osd/ceph-18 
> /var/lib/ceph/osd/ceph-18/journal
> === osd.17 ===
> === osd.17 ===
> Stopping Ceph osd.17 on tm1cldosdl04...done
> === osd.17 ===
> create-or-move updated item name 'osd.17' weight 0.45 at location 
> {host=tm1cldosdl04,root=default} to crush map
> Starting Ceph osd.17 on tm1cldosdl04...
> starting o

Re: [ceph-users] [Ceph-community] working ceph.conf file?

[Matt Harlum]

One thing to add, I had a similar issue with manually created OSD’s not coming 
back up with a reboot, they were being mounted but not started
To resolve this I had to create a file on each OSD called sysvinit

Regards,
Matt

On 9 Aug 2014, at 7:57 am, Andrew Woodward  wrote:

> Dan,
> 
> It is not necessary to specify the OSD data in ceph.conf anymore. Ceph has 
> two auto-start functions besides this method.
> 
> udev rules:
> ceph uses a udev rule to scan and attempt to mount (and activate) partitions 
> with specific GUID set for the partition typecode
> sgdisk --typecode=<>:<> /dev/<>
> 
> the exact GUID's to use can be found 
> https://github.com/ceph/ceph/blob/master/udev/95-ceph-osd.rules. These are 
> set automaticly using ceph-disk (or ceph-deploy) if it creates the partition 
> from an empty disk, in the case that it does not, you have to set them by 
> hand, all be it should probably do this, or at least tell you should do this.
> 
> ceph init script:
> the ceph init script will scan /var/lib/ceph/osd (or the otherwise configured 
> location) for -  (default cluster name is ceph) folders and 
> attempt to start the osd service for each of them if they look correct
> 
> lastly, and possibly the most annoying way is that you can configure each OSD 
> and path in ceph.conf, I don't have any good examples as the two prior are 
> more flexible / require less config.
> 
> 
> 
> 
> On Fri, Aug 8, 2014 at 8:53 AM, O'Reilly, Dan  wrote:
> Does anybody have a good sample ceph.conf file I can use for reference?  I’m 
> having a problem where OSD’s won’t come back up after a sysem reboot.
> 
>  
> 
> Dan O'Reilly
> 
> UNIX Systems Administration
> 
> 
> 
> 9601 S. Meridian Blvd.
> 
> Englewood, CO 80112
> 
> 720-514-6293
> 
>  
> 
>  
> 
> 
> ___
> Ceph-community mailing list
> ceph-commun...@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-community-ceph.com
> 
> 
> 
> 
> -- 
> Andrew
> Mirantis
> Ceph community
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Introductions

[debian Only]

As i konw , it is not recommend to run Ceph OSD (RBD server) same as the VM
host like KVM.
in another hand,  more service in same host, it is hard to maintenance, and
not good performance for each service.


2014-08-09 7:33 GMT+07:00 Zach Hill :

> Hi all,
>
> I'm Zach Hill, the storage lead at Eucalyptus .
> We're working on adding Ceph RBD support for our scale-out block storage
> (EBS API). Things are going well, and we've been happy with Ceph thus far.
> We are a RHEL/CentOS shop mostly, so any other tips there would be greatly
> appreciated.
>
> Our basic architecture is that we have a storage control node that issues
> control-plan operations: create image, delete, snapshot, etc. This
> controller uses librbd directly via JNA bindings. VMs access the Ceph RBD
> Images as block devices exposed via the Qemu/KVM RBD driver on our "Node
> Controller" hosts. It's similar to OpenStack Cinder in many ways.
>
> One of the questions we often get is:
> Can I run OSDs on my servers that also host VMs?
>
> Generally, we recommend strongly against such a deployment in order to
> ensure performance and failure isolation between the compute and storage
> sides of the system. But, I'm curious if anyone is doing this in practice
> and if they've found reasonable ways to make it work in production.
>
> Thanks for any info in advance, and we're happy to be joining this
> community in a more active way.
>
> -Zach
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] CRUSH map advice

[John Morris]

Our experimental Ceph cluster is performing terribly (with the operator 
to blame!), and while it's down to address some issues, I'm curious to 
hear advice about the following ideas.


The cluster:
- two disk nodes (6 * CPU, 16GB RAM each)
- 8 OSDs (4 each)
- 3 monitors
- 10Gb front + back networks
- 2TB Enterprise SATA drives
- HP RAID controller w/battery-backed cache
- one SSD journal drive for each two OSDs

First, I'd like to play with taking one machine down, but with the other 
node continuing to serve the cluster.  To maintain redundancy in this 
scenario, I'm thinking of setting the pool size to 4 and the min_size to 
2, with the idea that a proper CRUSH map should always keep two copies 
on each disk node.  Again, *this is for experimentation* and probably 
raises red flags for production, but I'm just asking if it's *possible*: 
 Could one node go down and the other node continue to serve r/w data? 
 Any anecdotes of performance differences between size=4 and size=3 in 
other clusters?


Second, does it make any sense to divide the CRUSH map into an extra 
level for the SSD disks, which each hold journals for two OSDs?  This 
might increase redundancy in case of a journal disk failure, but ISTR 
something about too few OSDs in a bucket causing problems with the CRUSH 
algorithm.


Thanks-

John
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com