Hi,
my Swift cluster is working for three weeks now as a mail storage and the
performance is getting slower and slower.
This is my hardware configuration:
- 1 hardware-loadbalancer
- 2 proxy-server DELL R420 16 GB RAM 2 Quadcore CPU
- 5 proxy-server DELL R720xd 16 GB RAM 2 Quadcore CPU with 3x 2TB HDD SATA 6GB/s
The cluster contains:
279.000 containers
8.170.000 objects = 984 GByte
I have about 3.000.000 GETs per day and 1.500.000 PUTs.
The load is raising up to 12 now:
top - 11:06:39 up 114 days, 23:10, 1 user, load average: 10.88, 11.26, 11.22
Tasks: 175 total, 3 running, 172 sleeping, 0 stopped, 0 zombie
%Cpu(s): 15.6 us, 9.8 sy, 0.0 ni, 69.3 id, 4.5 wa, 0.0 hi, 0.8 si, 0.0 st
KiB Mem: 16386408 total, 14829360 used, 1557048 free, 30796 buffers
KiB Swap: 17575932 total, 132604 used, 17443328 free, 5572360 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25880 swift 20 0 81400 15m 3352 D 34.5 0.1 2838:36 /usr/bin/python
/usr/bin/swift-container-replicator /etc/swift/container-server.conf
24282 swift 20 0 81084 14m 3244 S 19.2 0.1 3835:23 /usr/bin/python
/usr/bin/swift-container-auditor /etc/swift/container-server.conf
29816 swift 20 0 1523m 27m 2032 D 9.6 0.2 1132:58 /usr/bin/python
/usr/bin/swift-object-server /etc/swift/object-server.conf
29812 swift 20 0 1523m 27m 2032 R 7.0 0.2 1169:50 /usr/bin/python
/usr/bin/swift-object-server /etc/swift/object-server.conf
29815 swift 20 0 1523m 27m 2032 D 6.6 0.2 1132:45 /usr/bin/python
/usr/bin/swift-object-server /etc/swift/object-server.conf
29809 swift 20 0 1523m 26m 2032 D 5.6 0.2 1163:53 /usr/bin/python
/usr/bin/swift-object-server /etc/swift/object-server.conf
30684 swift 20 0 84496 16m 2468 D 5.6 0.1 1959:57 /usr/bin/python
/usr/bin/swift-container-server /etc/swift/container-server.conf
13853 snmp 20 0 51844 3356 1800 S 5.3 0.0 32:14.90 /usr/sbin/snmpd
-Lsd -Lf /dev/null -u snmp -g snmp -I -smux -p /var/run/snmpd.pid
29810 swift 20 0 1523m 27m 2032 D 5.3 0.2 1132:09 /usr/bin/python
/usr/bin/swift-object-server /etc/swift/object-server.conf
29813 swift 20 0 1525m 27m 2032 D 5.3 0.2 1177:23 /usr/bin/python
/usr/bin/swift-object-server /etc/swift/object-server.conf
29811 swift 20 0 1524m 28m 2032 D 5.0 0.2 1169:37 /usr/bin/python
/usr/bin/swift-object-server /etc/swift/object-server.conf
30677 swift 20 0 84932 17m 2468 D 5.0 0.1 1992:49 /usr/bin/python
/usr/bin/swift-container-server /etc/swift/container-server.conf
29814 swift 20 0 1524m 27m 2032 D 4.6 0.2 1181:11 /usr/bin/python
/usr/bin/swift-object-server /etc/swift/object-server.conf
30681 swift 20 0 84980 17m 2468 S 4.3 0.1 1999:12 /usr/bin/python
/usr/bin/swift-container-server /etc/swift/container-server.conf
30678 swift 20 0 84684 17m 2468 D 4.0 0.1 1973:14 /usr/bin/python
/usr/bin/swift-container-server /etc/swift/container-server.conf
30682 swift 20 0 84224 16m 2468 S 4.0 0.1 1960:55 /usr/bin/python
/usr/bin/swift-container-server /etc/swift/container-server.conf
30685 swift 20 0 84356 16m 2468 D 3.6 0.1 1979:40 /usr/bin/python
/usr/bin/swift-container-server /etc/swift/container-server.conf
7086 root 20 0 0 0 0 S 3.3 0.0 0:00.78 [kworker/6:1]
30683 swift 20 0 84160 16m 2468 S 3.3 0.1 1981:38 /usr/bin/python
/usr/bin/swift-container-server /etc/swift/container-server.conf
30679 swift 20 0 83852 16m 2468 S 3.0 0.1 1965:53 /usr/bin/python
/usr/bin/swift-container-server /etc/swift/container-server.conf
5935 root 20 0 0 0 0 S 2.7 0.0 0:16.38 [kworker/2:1]
6832 root 20 0 0 0 0 S 2.7 0.0 0:02.78 [kworker/4:0]
7153 swift 20 0 80300 9.8m 2040 D 2.7 0.1 0:00.08 /usr/bin/python
/usr/bin/swift-container-updater /etc/swift/container-server.conf
1840 root 20 0 0 0 0 S 2.3 0.0 0:59.87 [kworker/0:1]
9630 swift 20 0 75796 6272 1708 D 2.0 0.0 1889:16 /usr/bin/python
/usr/bin/swift-object-auditor /etc/swift/object-server.conf
9656 swift 20 0 87424 17m 2304 S 2.0 0.1 748:29.97 /usr/bin/python
/usr/bin/swift-account-server /etc/swift/account-server.conf
9659 swift 20 0 86908 17m 2304 S 2.0 0.1 803:09.64 /usr/bin/python
/usr/bin/swift-account-server /etc/swift/account-server.conf
9653 swift 20 0 85816 16m 2304 S 1.7 0.1 723:34.42 /usr/bin/python
/usr/bin/swift-account-server /etc/swift/account-server.conf
9654 swift 20 0 87500 17m 2304 R 1.7 0.1 769:23.97 /usr/bin/python
/usr/bin/swift-account-server /etc/swift/account-server.conf
9655 swift 20 0 85956 16m 2408 S 1.7 0.1 783:28.18 /usr/bin/python
/usr/bin/swift-account-server /etc/swift/account-server.conf
9658 swift 20 0 87808 18m 2304 S 1.7 0.1 767:31.82 /usr/bin/python
/usr/bin/swift-account-server /etc/swift/account-server.conf
32616 root 20 0 0 0 0 S 1.7 0.0 0:18.13 [kworker/3:0]
1826 syslog 20 0 243m 5124 856 S 1.3 0.0 329:10.12 rsyslogd -c5
6944 root 20 0 0 0 0 S 1.3 0.0 0:01.08 [kworker/7:2]
9652 swift 20 0 86896 17m 2412 S 1.3 0.1 663:32.21 /usr/bin/python
/usr/bin/swift-account-server /etc/swift/account-server.conf
9657 swift 20 0 86016 16m 2408 S 1.3 0.1 706:22.08 /usr/bin/python
/usr/bin/swift-account-server /etc/swift/account-server.conf
51 root 20 0 0 0 0 S 1.0 0.0 134:26.37 [kswapd0]
4548 root 20 0 0 0 0 S 1.0 0.0 0:11.69 [kworker/1:2]
5362 root 20 0 0 0 0 S 1.0 0.0 0:10.95 [kworker/5:3]
3 root 20 0 0 0 0 S 0.7 0.0 604:49.60 [ksoftirqd/0]
52 root 20 0 0 0 0 S 0.7 0.0 68:32.58 [kswapd1]
7412 swift 20 0 80300 9704 2780 S 0.7 0.1 566:00.53 /usr/bin/python
/usr/bin/swift-container-updater /etc/swift/container-server.conf
9580 swift 20 0 75632 7140 2728 D 0.7 0.0 2218:06 /usr/bin/python
/usr/bin/swift-object-auditor /etc/swift/object-server.conf
22178 root 20 0 7404 992 700 D 0.7 0.0 20:28.93
/usr/bin/updatedb.mlocate
Is this a memory problem or a harddisk problem?
And second question: On the server the mlocate database is rising. Can I
disable the updatedb.mlocate process and empty the database?
Thanks
Klaus
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
