Re: [ceph-users] HA Filesystem mode (MON, OSD, MDS) with Ceph and HAof MDS daemon.

2017-06-27 Thread Daniel Carrasco
Hello, I just write to say that after more than a week the server still working without problem and the OSD are not marked as down erroneously. On my tests the webpage stop working for less than a minute when i stop an OSD, so the failover is working fine. Greetings and thanks for all your help!!

Re: [ceph-users] HA Filesystem mode (MON, OSD, MDS) with Ceph and HAof MDS daemon.

2017-06-15 Thread Daniel Carrasco
Hello, thanks for the info. I'll give a try tomorrow. On one of my test I got the messages that yo say (wrongfully marked), but i've lowered other options and now is fine. For now the OSD are not reporting down messages even with an high load test, but I'll see the logs tomorrow to confirm. The m

Re: [ceph-users] HA Filesystem mode (MON, OSD, MDS) with Ceph and HAof MDS daemon.

2017-06-15 Thread David Turner
osd_heartbeat_grace is a setting for how many seconds since the last time an osd received a successful response from another osd before telling the mons that it's down. This is one you may want to lower from its default value of 20 seconds. mon_osd_min_down_reporters is a setting for how many osd

Re: [ceph-users] HA Filesystem mode (MON, OSD, MDS) with Ceph and HAof MDS daemon.

2017-06-15 Thread Daniel Carrasco
I forgot to say that after upgrade the machine RAM to 4Gb, the OSD daemons has started to use only a 5% (about 200MB). Is like magic, and now I've about 3.2Gb of free RAM. Greetings!! 2017-06-15 15:08 GMT+02:00 Daniel Carrasco : > Finally, the problem was W3Total Cache, that seems to be unable t

Re: [ceph-users] HA Filesystem mode (MON, OSD, MDS) with Ceph and HAof MDS daemon.

2017-06-15 Thread Daniel Carrasco
Finally, the problem was W3Total Cache, that seems to be unable to manage HA and when the master redis host is down, it stop working without try the slave. I've added some options to make it faster to detect a down OSD and the page is online again in about 40s. [global] fsid = Hidden mon_initial_

Re: [ceph-users] HA Filesystem mode (MON, OSD, MDS) with Ceph and HAof MDS daemon.

2017-06-14 Thread David Turner
I've used the kernel client and the ceph-fuse driver for mapping the cephfs volume. I didn't notice any network hiccups while failing over, but I was reading large files during my tests (and live) and some caching may have hidden hidden network hiccups for my use case. Going back to the memory po

Re: [ceph-users] HA Filesystem mode (MON, OSD, MDS) with Ceph and HAof MDS daemon.

2017-06-14 Thread Daniel Carrasco
Is strange because on my test cluster (three nodes) with two nodes with OSD, and all with MON and MDS, I've configured the size to 2 and min_size to 1, I've restarted all nodes one by one and the client loose the connection for about 5 seconds until connect to other MDS. Are you using ceph client

Re: [ceph-users] HA Filesystem mode (MON, OSD, MDS) with Ceph and HAof MDS daemon.

2017-06-14 Thread David Turner
I have 3 ceph nodes, size 3, min_size 2, and I can restart them all 1 at a time to do ceph and kernel upgrades. The VM's running out of ceph, the clients accessing MDS, etc all keep working fine without any problem during these restarts. What is your full ceph configuration? There must be someth

Re: [ceph-users] HA Filesystem mode (MON, OSD, MDS) with Ceph and HAof MDS daemon.

2017-06-14 Thread Daniel Carrasco
El 14 jun. 2017 10:08 p. m., "David Turner" escribió: Not just the min_size of your cephfs data pool, but also your cephfs_metadata pool. Both were at 1. I don't know why because I don't remember to have changed the min_size and the cluster has 3 odd from beginning (I did it on another cluster

Re: [ceph-users] HA Filesystem mode (MON, OSD, MDS) with Ceph and HAof MDS daemon.

2017-06-14 Thread David Turner
Not just the min_size of your cephfs data pool, but also your cephfs_metadata pool. On Wed, Jun 14, 2017 at 4:07 PM David Turner wrote: > Ceph recommends 1GB of RAM for ever 1TB of OSD space. Your 2GB nodes are > definitely on the low end. 50GB OSDs... I don't know what that will > require, bu

Re: [ceph-users] HA Filesystem mode (MON, OSD, MDS) with Ceph and HAof MDS daemon.

2017-06-14 Thread David Turner
Ceph recommends 1GB of RAM for ever 1TB of OSD space. Your 2GB nodes are definitely on the low end. 50GB OSDs... I don't know what that will require, but where you're running the mon and mds on the same node, I'd still say that 2GB is low. The Ceph OSD daemon using 1GB of RAM is not surprising,

Re: [ceph-users] HA Filesystem mode (MON, OSD, MDS) with Ceph and HAof MDS daemon.

2017-06-14 Thread Daniel Carrasco
Finally I've created three nodes, I've increased the size of pools to 3 and I've created 3 MDS (active, standby, standby). Today the server has decided to fail and I've noticed that failover is not working... The ceph -s command shows like everything was OK but the clients weren't able to connect

Re: [ceph-users] HA Filesystem mode (MON, OSD, MDS) with Ceph and HAof MDS daemon.

2017-06-12 Thread Mazzystr
Since your app is an Apache / php app is it possible for you to reconfigure the app to use S3 module rather than a posix open file()? Then with Ceph drop CephFS and configure Civetweb S3 gateway? You can have "active-active" endpoints with round robin dns or F5 or something. You would also have

Re: [ceph-users] HA Filesystem mode (MON, OSD, MDS) with Ceph and HAof MDS daemon.

2017-06-12 Thread Daniel Carrasco
2017-06-12 16:10 GMT+02:00 David Turner : > I have an incredibly light-weight cephfs configuration. I set up an MDS > on each mon (3 total), and have 9TB of data in cephfs. This data only has > 1 client that reads a few files at a time. I haven't noticed any downtime > when it fails over to a s

Re: [ceph-users] HA Filesystem mode (MON, OSD, MDS) with Ceph and HAof MDS daemon.

2017-06-12 Thread David Turner
I have an incredibly light-weight cephfs configuration. I set up an MDS on each mon (3 total), and have 9TB of data in cephfs. This data only has 1 client that reads a few files at a time. I haven't noticed any downtime when it fails over to a standby MDS. So it definitely depends on your workl

Re: [ceph-users] HA Filesystem mode (MON, OSD, MDS) with Ceph and HAof MDS daemon.

2017-06-12 Thread John Petrini
We use the following in our ceph.conf for MDS failover. We're running one active and one standby. Last time it failed over there was about 2 minutes of downtime before the mounts started responding again but it did recover gracefully. [mds] max_mds = 1 mds_standby_for_rank = 0 mds_standby_replay =

Re: [ceph-users] HA Filesystem mode (MON, OSD, MDS) with Ceph and HAof MDS daemon.

2017-06-12 Thread Daniel Carrasco
2017-06-12 10:49 GMT+02:00 Burkhard Linke < burkhard.li...@computational.bio.uni-giessen.de>: > Hi, > > > On 06/12/2017 10:31 AM, Daniel Carrasco wrote: > >> Hello, >> >> I'm very new on Ceph, so maybe this question is a noob question. >> >> We have an architecture that have some web servers (ngin

Re: [ceph-users] HA Filesystem mode (MON, OSD, MDS) with Ceph and HAof MDS daemon.

2017-06-12 Thread Burkhard Linke
Hi, On 06/12/2017 10:31 AM, Daniel Carrasco wrote: Hello, I'm very new on Ceph, so maybe this question is a noob question. We have an architecture that have some web servers (nginx, php...) with a common File Server through NFS. Of course that is a SPOF, so we want to create a multi FS to a