-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi guys,
we running in a described problem on our storage nodes.
The object auditor process throws errors because the system has no
available ports.
Mar 9 13:05:37 swift2 object-replicator: Error syncing with node:
{'replication_port': 6000, 'zone': 1, 'weight': 100.0, 'ip':
'10.0.0.22', 'region': 1, 'port': 6000, 'replication_ip': '10.0.0.22',
'meta': u'', 'device': 'sda5', 'id': 0}: #012Traceback (most recent call
last):#012 File
"/usr/lib/python2.7/dist-packages/swift/obj/replicator.py", line 282, in
update#012 '', headers=self.headers).getresponse()#012 File
"/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py", line
157, in http_connect#012 ipaddr, port, method, path, headers,
query_string, ssl)#012 File
"/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py", line
189, in http_connect_raw#012 conn.endheaders()#012 File
"/usr/lib/python2.7/httplib.py", line 954, in endheaders#012
self._send_output(message_body)#012 File
"/usr/lib/python2.7/httplib.py", line 814, in _send_output#012
self.send(msg)#012 File "/usr/lib/python2.7/httplib.py", line 776, in
send#012 self.connect()#012 File
"/usr/lib/python2.7/dist-packages/swift/common/bufferedhttp.py", line
108, in connect#012 return HTTPConnection.connect(self)#012 File
"/usr/lib/python2.7/httplib.py", line 757, in connect#012
self.timeout, self.source_address)#012 File
"/usr/lib/python2.7/dist-packages/eventlet/green/socket.py", line 59, in
create_connection#012 raise error, msg#012error: [Errno 99] EADDRNOTAVAIL
:~# netstat --inet | grep TIME_WAIT | wc -l
63038
This value of used ports is on all nodes nearly the same and fluctuates
extremely. So i tuned the kernel and network stack of the Linux kernel
but without success.
# disable TIME_WAIT.. wait..
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_tw_reuse=1
# disable syn cookies
net.ipv4.tcp_syncookies = 0
# double amount of allowed conntrack
net.ipv4.netfilter.ip_conntrack_max = 262144
net.ipv4.ip_local_port_range = 18000 65535
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 1
net.netfilter.nf_conntrack_tcp_timeout_established=600
net.netfilter.nf_conntrack_tcp_timeout_time_wait=30
net.ipv4.tcp_fin_timeout=15
net.ipv4.tcp_keepalive_intvl=30
net.ipv4.tcp_keepalive_probes=5
The object-server conf-file:
[object-replicator]
recon_enable = yes
concurrency = 2
run_pause = 60
reclaim_age = 259200
interval = 60
[object-updater]
concurrency = 4
recon_enable = yes
recon_cache_path = /var/cache/swift
slowdown = 0.1
[object-auditor]
bytes_per_second = 3000000
files_per_second = 10
concurrency = 4
recon_enable = yes
recon_cache_path = /var/cache/swift
Have anyone a hint for me ?
Greetings
Heiko
- --
anynines.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQEcBAEBAgAGBQJU/ZtBAAoJELxFogM4ixOFS88IANZZBzzFnFymWXRyuAQGjVpz
X7Os9Y9Jn41EOph4HHS9eablTc14YX4YiB/JvKj1KKJAUOkVoPfB5oC154hQ5Goa
i3f1qSWg3qEqv/lo5EvtX++B92Ut/68OSUblie1XGkivs6ZIfzeByzJqDgwdS2kV
UEMzyEw9K4oNFkyURts8vH4NX4FgqKIoaPaQh6qOe27YKEdWw9NJn3NbRzWncwVJ
R181jaerubZo8gYOVO9zYLHoPFLSxVft7zC6M0fHK6SqDUosA8zjperlvWChx2ZD
UnL3LAEs1BCSxnJw876AvH9nxwFwkZwioQeVW5inTtqxmvZRn0RnsCY/qzv51Oc=
=2fqn
-----END PGP SIGNATURE-----
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
