Re: [ceph-users] nf_conntrack overflow crashes OSDs

2014-08-08 Thread Christian Kauhaus
Am 08.08.2014 um 14:05 schrieb Robert van Leeuwen: > It is also possible to specifically not conntrack certain connections. > e.g. > iptables -t raw -A PREROUTING -p tcp --dport 6789 -j CT --notrack Thanks Robert. This is really an interesting approach. We will test it. Regards Christian -- Di

Re: [ceph-users] nf_conntrack overflow crashes OSDs

2014-08-08 Thread Robert van Leeuwen
> today I'd like to share a severe problem we've found (and fixed) on our Ceph > cluster. We're running 48 OSDs (8 per host). While restarting all OSDs on a > host, the kernel's nf_conntrack table was overflown. This rendered all OSDs on > that machine unusable. It is also possible to specifically

Re: [ceph-users] nf_conntrack overflow crashes OSDs

2014-08-08 Thread Dan Van Der Ster
Hi Christian, This is good advice. Presumably we saw this issue before, since we have the following in our cluster’s puppet manifest: sysctl { "net.netfilter.nf_conntrack_max": val => "1024000", } sysctl { "net.nf_conntrack_max": val => "1024000", } But I don’t remember when or how we discov

[ceph-users] nf_conntrack overflow crashes OSDs

2014-08-08 Thread Christian Kauhaus
Hi, today I'd like to share a severe problem we've found (and fixed) on our Ceph cluster. We're running 48 OSDs (8 per host). While restarting all OSDs on a host, the kernel's nf_conntrack table was overflown. This rendered all OSDs on that machine unusable. The symptoms were as follows. In the k