+ Check the proxys ram ... if there starving, try restarting them and check if your performance recover
On Mon, Aug 5, 2013 at 7:10 PM, Morten Møller Riis <m...@gigahost.dk> wrote: > I second this. > > We have millions of objects on several accounts. Eg. one account has about > 7 million images. These are split into containers according to their > filename: > > /images_000/something.jpg > ... > /images_fff/something_else.jpg > > Also, I recommend looking at tuning the configs. Some examples: > > > object-server.conf: > > [object-replicator] > concurrency = 4 > run_pause = 60 > > [object-updater] > concurrency = 4 > > [object-auditor] > files_per_second = 5 > bytes_per_second = 2500000 > concurrency = 25 > > > container-server.conf: > > [container-replicator] > concurrency = 8 > run_pause = 60 > > [container-updater] > concurrency = 4 > > [container-sync] > interval = 300 > container_time = 60 > > > account-server.conf: > > [account-replicator] > concurrency = 4 > run_pause = 45 > > [account-reaper] > concurrency = 25 > > > > Mvh / Best regards > Morten Møller Riis > Gigahost ApS > m...@gigahost.dk > > > > > On Aug 6, 2013, at 12:15 AM, John Dickinson <m...@not.mn> wrote: > > There are a few things you can look at. > > First, the slowdowns you are seeing may be because of the increased > numbers of files on disk. This causes the background processes to do more > work, which can cause some issues for the server processes. > > Second (and perhaps more likely), how is your data arranged? Are you > putting every object into the same container? Swift is optimized for access > across the entire namespace, so spreading your data across many containers > can help improve write performance. Since you are running with account and > proxy servers on the same spinning media as the object servers, as you get > your cluster more full, write performance will degrade significantly with > many objects in a single container. With your setup, you should probably > have a max of about a million objects in a single container. If you > separate your account and container servers to use SSDs (and perhaps > separate machines), you could do 100x this number effectively. Note that > this issue doesn't affect read performance. Object eead performance should > stay high no matter the cardinality of objects in a single container. > > > --John > > > > On Aug 5, 2013, at 4:09 AM, Jonathan Lu <jojokur...@gmail.com> wrote: > > Hi, > I suffer the same issue that our swift system gets slower and slower. I > tried almost all the guide in the document but it does little help. > > Maybe swift is not suitable to the circumstance of putting and getting > many little files. Maybe is it more suitable to the large object such as > the image of OS? I am not quite sure. > > Best Regards > Jonathan Lu > > On 2013/8/5 18:35, Hua ZZ Zhang wrote: > > Here's the deployment guide which can give you options for tuning > container replicator and auditor, such as concurrency. > http://docs.openstack.org/developer/swift/deployment_guide.html > hope this can help you. > > Best Regards, > > > Edward Zhang(张华) > Advisory Software Engineer > Software Standards & Open Source Software > Emerging Technology Institute(ETI) > IBM China Software Development Lab > e-mail: zhu...@cn.ibm.com > <Mail Attachment.gif> > <Mail Attachment.gif> > <Mail Attachment.gif> > > <Mail Attachment.gif> > <Mail Attachment.gif>Klaus Schürmann ---08/05/2013 06:13:43 PM---Klaus > Schürmann <klaus.schuerm...@mediabeam.com> > > Klaus Schürmann <klaus.schuerm...@mediabeam.com> > 08/05/2013 06:14 PM > > <Mail Attachment.gif> > To > <Mail Attachment.gif> > Hua ZZ Zhang/China/IBM@IBMCN, > <Mail Attachment.gif> > cc > <Mail Attachment.gif> > "openstack@lists.openstack.org" <openstack@lists.openstack.org> > <Mail Attachment.gif> > Subject > <Mail Attachment.gif> > AW: [Openstack] [SWIFT] PUTs and GETs getting slower > <Mail Attachment.gif> <Mail Attachment.gif> > Here is my container-server.conf: > > [DEFAULT] > log_name = container-server > log_facility = LOG_LOCAL6 > log_level = INFO > log_address = /dev/log > bind_ip = 10.12.1.1 > workers = 8 > log_statsd_host = 10.11.2.1 > log_statsd_port = 8125 > log_statsd_default_sample_rate = 0.25 > log_statsd_metric_prefix = Storage01 > > [pipeline:main] > pipeline = recon container-server > > [filter:recon] > use = egg:swift#recon > recon_cache_path = /var/cache/swift > > [app:container-server] > use = egg:swift#container > > [container-replicator] > concurrency = 2 > > [container-updater] > concurrency = 1 > > [container-auditor] > > [container-sync] > > > -------- > For the second question, Swift doesn't use updatedb.mlocate in my mind. > it should be safe to disable it. > Based on the TOP output, it seems that the CPU workload of container > replicator and auditor are too high. > Perhaps you need to tune the config file to decrease their workload. paste > your config file of > container servers. > > Best Regards, > Edward Zhang(张华) > Advisory Software Engineer > Software Standards & Open Source Software > Emerging Technology Institute(ETI) > IBM China Software Development Lab > e-mail: zhu...@cn.ibm.com > <Mail Attachment.gif> > <Mail Attachment.gif> > <Mail Attachment.gif> > <Mail Attachment.gif> <Mail Attachment.gif> > <Mail Attachment.gif>Klaus Schürmann ---08/05/2013 05:43:08 PM---Klaus > Schürmann <klaus.schuerm...@mediabeam.com> > Klaus Schürmann <klaus.schuerm...@mediabeam.com> > 08/05/2013 05:26 PM > > <Mail Attachment.gif> > To > <Mail Attachment.gif> > "openstack@lists.openstack.org" <openstack@lists.openstack.org>, > <Mail Attachment.gif> > cc > <Mail Attachment.gif> > <Mail Attachment.gif> > Subject > <Mail Attachment.gif> > [Openstack] [SWIFT] PUTs and GETs getting slower > > > <Mail Attachment.gif> <Mail Attachment.gif> > Hi, > > my Swift cluster is working for three weeks now as a mail storage and the > performance is getting slower and slower. > > This is my hardware configuration: > > - 1 hardware-loadbalancer > - 2 proxy-server DELL R420 16 GB RAM 2 Quadcore CPU > - 5 proxy-server DELL R720xd 16 GB RAM 2 Quadcore CPU with 3x 2TB HDD SATA > 6GB/s > > The cluster contains: > > 279.000 containers > 8.170.000 objects = 984 GByte > > I have about 3.000.000 GETs per day and 1.500.000 PUTs. > > The load is raising up to 12 now: > > top - 11:06:39 up 114 days, 23:10, 1 user, load average: 10.88, 11.26, > 11.22 > Tasks: 175 total, 3 running, 172 sleeping, 0 stopped, 0 zombie > %Cpu(s): 15.6 us, 9.8 sy, 0.0 ni, 69.3 id, 4.5 wa, 0.0 hi, 0.8 si, > 0.0 st > KiB Mem: 16386408 total, 14829360 used, 1557048 free, 30796 buffers > KiB Swap: 17575932 total, 132604 used, 17443328 free, 5572360 cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 25880 swift 20 0 81400 15m 3352 D 34.5 0.1 2838:36 > /usr/bin/python /usr/bin/swift-container-replicator > /etc/swift/container-server.conf > 24282 swift 20 0 81084 14m 3244 S 19.2 0.1 3835:23 > /usr/bin/python /usr/bin/swift-container-auditor > /etc/swift/container-server.conf > 29816 swift 20 0 1523m 27m 2032 D 9.6 0.2 1132:58 > /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf > 29812 swift 20 0 1523m 27m 2032 R 7.0 0.2 1169:50 > /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf > 29815 swift 20 0 1523m 27m 2032 D 6.6 0.2 1132:45 > /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf > 29809 swift 20 0 1523m 26m 2032 D 5.6 0.2 1163:53 > /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf > 30684 swift 20 0 84496 16m 2468 D 5.6 0.1 1959:57 > /usr/bin/python /usr/bin/swift-container-server > /etc/swift/container-server.conf > 13853 snmp 20 0 51844 3356 1800 S 5.3 0.0 32:14.90 > /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp -I -smux -p > /var/run/snmpd.pid > 29810 swift 20 0 1523m 27m 2032 D 5.3 0.2 1132:09 > /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf > 29813 swift 20 0 1525m 27m 2032 D 5.3 0.2 1177:23 > /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf > 29811 swift 20 0 1524m 28m 2032 D 5.0 0.2 1169:37 > /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf > 30677 swift 20 0 84932 17m 2468 D 5.0 0.1 1992:49 > /usr/bin/python /usr/bin/swift-container-server > /etc/swift/container-server.conf > 29814 swift 20 0 1524m 27m 2032 D 4.6 0.2 1181:11 > /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf > 30681 swift 20 0 84980 17m 2468 S 4.3 0.1 1999:12 > /usr/bin/python /usr/bin/swift-container-server > /etc/swift/container-server.conf > 30678 swift 20 0 84684 17m 2468 D 4.0 0.1 1973:14 > /usr/bin/python /usr/bin/swift-container-server > /etc/swift/container-server.conf > 30682 swift 20 0 84224 16m 2468 S 4.0 0.1 1960:55 > /usr/bin/python /usr/bin/swift-container-server > /etc/swift/container-server.conf > 30685 swift 20 0 84356 16m 2468 D 3.6 0.1 1979:40 > /usr/bin/python /usr/bin/swift-container-server > /etc/swift/container-server.conf > 7086 root 20 0 0 0 0 S 3.3 0.0 0:00.78 [kworker/6:1] > 30683 swift 20 0 84160 16m 2468 S 3.3 0.1 1981:38 > /usr/bin/python /usr/bin/swift-container-server > /etc/swift/container-server.conf > 30679 swift 20 0 83852 16m 2468 S 3.0 0.1 1965:53 > /usr/bin/python /usr/bin/swift-container-server > /etc/swift/container-server.conf > 5935 root 20 0 0 0 0 S 2.7 0.0 0:16.38 [kworker/2:1] > 6832 root 20 0 0 0 0 S 2.7 0.0 0:02.78 [kworker/4:0] > 7153 swift 20 0 80300 9.8m 2040 D 2.7 0.1 0:00.08 > /usr/bin/python /usr/bin/swift-container-updater > /etc/swift/container-server.conf > 1840 root 20 0 0 0 0 S 2.3 0.0 0:59.87 [kworker/0:1] > 9630 swift 20 0 75796 6272 1708 D 2.0 0.0 1889:16 > /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf > 9656 swift 20 0 87424 17m 2304 S 2.0 0.1 748:29.97 > /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf > 9659 swift 20 0 86908 17m 2304 S 2.0 0.1 803:09.64 > /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf > 9653 swift 20 0 85816 16m 2304 S 1.7 0.1 723:34.42 > /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf > 9654 swift 20 0 87500 17m 2304 R 1.7 0.1 769:23.97 > /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf > 9655 swift 20 0 85956 16m 2408 S 1.7 0.1 783:28.18 > /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf > 9658 swift 20 0 87808 18m 2304 S 1.7 0.1 767:31.82 > /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf > 32616 root 20 0 0 0 0 S 1.7 0.0 0:18.13 [kworker/3:0] > 1826 syslog 20 0 243m 5124 856 S 1.3 0.0 329:10.12 rsyslogd -c5 > 6944 root 20 0 0 0 0 S 1.3 0.0 0:01.08 [kworker/7:2] > 9652 swift 20 0 86896 17m 2412 S 1.3 0.1 663:32.21 > /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf > 9657 swift 20 0 86016 16m 2408 S 1.3 0.1 706:22.08 > /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf > 51 root 20 0 0 0 0 S 1.0 0.0 134:26.37 [kswapd0] > 4548 root 20 0 0 0 0 S 1.0 0.0 0:11.69 [kworker/1:2] > 5362 root 20 0 0 0 0 S 1.0 0.0 0:10.95 [kworker/5:3] > 3 root 20 0 0 0 0 S 0.7 0.0 604:49.60 [ksoftirqd/0] > 52 root 20 0 0 0 0 S 0.7 0.0 68:32.58 [kswapd1] > 7412 swift 20 0 80300 9704 2780 S 0.7 0.1 566:00.53 > /usr/bin/python /usr/bin/swift-container-updater > /etc/swift/container-server.conf > 9580 swift 20 0 75632 7140 2728 D 0.7 0.0 2218:06 > /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf > 22178 root 20 0 7404 992 700 D 0.7 0.0 20:28.93 > /usr/bin/updatedb.mlocate > > Is this a memory problem or a harddisk problem? > > And second question: On the server the mlocate database is rising. Can I > disable the updatedb.mlocate process and empty the database? > > Thanks > Klaus > > > > _______________________________________________ > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openstack@lists.openstack.org > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > > > _______________________________________________ > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > Post to : > openstack@lists.openstack.org > > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > > _______________________________________________ > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openstack@lists.openstack.org > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > > _______________________________________________ > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openstack@lists.openstack.org > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > > > _______________________________________________ > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openstack@lists.openstack.org > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > >
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
