Hi, Thanks for the replies. Likely will not choose this method but wanted to make sure that it was a good technical reason rather than just a "best practice". I did not quite think of "conntracker" at the time so this is a good one to consider.
Thanks Pieter On 12 November 2014 14:30, Haomai Wang <haomaiw...@gmail.com> wrote: > Actually, our production cluster(up to ten) all are that ceph-osd ran > on compute-node(KVM). > > The primary action is that you need to constrain the cpu and memory. > For example, you can alloc a ceph cpu-set and memory group, let > ceph-osd run with it within limited cores and memory. > > The another risk is the network. Because compute-node and ceph-osd > shared the same kernel network stack, it exists some risks that VM may > ran out of network resources such as conntracker in netfilter > framework. > > On Wed, Nov 12, 2014 at 10:23 PM, Mark Nelson <mark.nel...@inktank.com> > wrote: > > Technically there's no reason it shouldn't work, but it does complicate > > things. Probably the biggest worry would be that if something bad > happens > > on the compute side (say it goes nuts with network or memory transfers) > it > > could slow things down enough that OSDs start failing heartbeat checks > > causing ceph to go into recovery and maybe cause a vicious cycle of > > nastiness. > > > > You can mitigate some of this with cgroups and try to dedicate specific > > sockets and memory banks to Ceph/Compute, but we haven't done a lot of > > testing yet afaik. > > > > Mark > > > > > > On 11/12/2014 07:45 AM, Pieter Koorts wrote: > >> > >> Hi, > >> > >> A while back on a blog I saw mentioned that Ceph should not be run on > >> compute nodes and in the general sense should be on dedicated hardware. > >> Does this really still apply? > >> > >> An example, if you have nodes comprised of > >> > >> 16+ cores > >> 256GB+ RAM > >> Dual 10GBE Network > >> 2+8 OSD (SSD log + HDD store) > >> > >> I understand that Ceph can use a lot of IO and CPU in some cases but if > >> the nodes are powerful enough does it not make it an option to run > >> compute and storage on the same hardware to either increase density of > >> compute or save money on additional hardware? > >> > >> What are the reasons for not running Ceph on the Compute nodes. > >> > >> Thanks > >> > >> Pieter > >> > >> > >> _______________________________________________ > >> ceph-users mailing list > >> ceph-users@lists.ceph.com > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > > > > _______________________________________________ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- > Best Regards, > > Wheat > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
