I second this. We have millions of objects on several accounts. Eg. one account has about 7 million images. These are split into containers according to their filename:
/images_000/something.jpg ... /images_fff/something_else.jpg Also, I recommend looking at tuning the configs. Some examples: object-server.conf: [object-replicator] concurrency = 4 run_pause = 60 [object-updater] concurrency = 4 [object-auditor] files_per_second = 5 bytes_per_second = 2500000 concurrency = 25 container-server.conf: [container-replicator] concurrency = 8 run_pause = 60 [container-updater] concurrency = 4 [container-sync] interval = 300 container_time = 60 account-server.conf: [account-replicator] concurrency = 4 run_pause = 45 [account-reaper] concurrency = 25 Mvh / Best regards Morten Møller Riis Gigahost ApS m...@gigahost.dk On Aug 6, 2013, at 12:15 AM, John Dickinson <m...@not.mn> wrote: > There are a few things you can look at. > > First, the slowdowns you are seeing may be because of the increased numbers > of files on disk. This causes the background processes to do more work, which > can cause some issues for the server processes. > > Second (and perhaps more likely), how is your data arranged? Are you putting > every object into the same container? Swift is optimized for access across > the entire namespace, so spreading your data across many containers can help > improve write performance. Since you are running with account and proxy > servers on the same spinning media as the object servers, as you get your > cluster more full, write performance will degrade significantly with many > objects in a single container. With your setup, you should probably have a > max of about a million objects in a single container. If you separate your > account and container servers to use SSDs (and perhaps separate machines), > you could do 100x this number effectively. Note that this issue doesn't > affect read performance. Object eead performance should stay high no matter > the cardinality of objects in a single container. > > > --John > > > > On Aug 5, 2013, at 4:09 AM, Jonathan Lu <jojokur...@gmail.com> wrote: > >> Hi, >> I suffer the same issue that our swift system gets slower and slower. I >> tried almost all the guide in the document but it does little help. >> >> Maybe swift is not suitable to the circumstance of putting and getting >> many little files. Maybe is it more suitable to the large object such as the >> image of OS? I am not quite sure. >> >> Best Regards >> Jonathan Lu >> >> On 2013/8/5 18:35, Hua ZZ Zhang wrote: >>> Here's the deployment guide which can give you options for tuning container >>> replicator and auditor, such as concurrency. >>> http://docs.openstack.org/developer/swift/deployment_guide.html >>> hope this can help you. >>> >>> Best Regards, >>> >>> >>> Edward Zhang(张华) >>> Advisory Software Engineer >>> Software Standards & Open Source Software >>> Emerging Technology Institute(ETI) >>> IBM China Software Development Lab >>> e-mail: zhu...@cn.ibm.com >>> <Mail Attachment.gif> >>> <Mail Attachment.gif> >>> <Mail Attachment.gif> >>> >>> <Mail Attachment.gif> >>> <Mail Attachment.gif>Klaus Schürmann ---08/05/2013 06:13:43 PM---Klaus >>> Schürmann <klaus.schuerm...@mediabeam.com> >>> >>> Klaus Schürmann <klaus.schuerm...@mediabeam.com> >>> 08/05/2013 06:14 PM >>> >>> <Mail Attachment.gif> >>> To >>> <Mail Attachment.gif> >>> Hua ZZ Zhang/China/IBM@IBMCN, >>> <Mail Attachment.gif> >>> cc >>> <Mail Attachment.gif> >>> "openstack@lists.openstack.org" <openstack@lists.openstack.org> >>> <Mail Attachment.gif> >>> Subject >>> <Mail Attachment.gif> >>> AW: [Openstack] [SWIFT] PUTs and GETs getting slower >>> <Mail Attachment.gif> <Mail Attachment.gif> >>> Here is my container-server.conf: >>> >>> [DEFAULT] >>> log_name = container-server >>> log_facility = LOG_LOCAL6 >>> log_level = INFO >>> log_address = /dev/log >>> bind_ip = 10.12.1.1 >>> workers = 8 >>> log_statsd_host = 10.11.2.1 >>> log_statsd_port = 8125 >>> log_statsd_default_sample_rate = 0.25 >>> log_statsd_metric_prefix = Storage01 >>> >>> [pipeline:main] >>> pipeline = recon container-server >>> >>> [filter:recon] >>> use = egg:swift#recon >>> recon_cache_path = /var/cache/swift >>> >>> [app:container-server] >>> use = egg:swift#container >>> >>> [container-replicator] >>> concurrency = 2 >>> >>> [container-updater] >>> concurrency = 1 >>> >>> [container-auditor] >>> >>> [container-sync] >>> >>> >>> -------- >>> For the second question, Swift doesn't use updatedb.mlocate in my mind. it >>> should be safe to disable it. >>> Based on the TOP output, it seems that the CPU workload of container >>> replicator and auditor are too high. >>> Perhaps you need to tune the config file to decrease their workload. paste >>> your config file of >>> container servers. >>> >>> Best Regards, >>> Edward Zhang(张华) >>> Advisory Software Engineer >>> Software Standards & Open Source Software >>> Emerging Technology Institute(ETI) >>> IBM China Software Development Lab >>> e-mail: zhu...@cn.ibm.com >>> <Mail Attachment.gif> >>> <Mail Attachment.gif> >>> <Mail Attachment.gif> >>> <Mail Attachment.gif> <Mail Attachment.gif> >>> <Mail Attachment.gif>Klaus Schürmann ---08/05/2013 05:43:08 PM---Klaus >>> Schürmann <klaus.schuerm...@mediabeam.com> >>> Klaus Schürmann <klaus.schuerm...@mediabeam.com> >>> 08/05/2013 05:26 PM >>> >>> <Mail Attachment.gif> >>> To >>> <Mail Attachment.gif> >>> "openstack@lists.openstack.org" <openstack@lists.openstack.org>, >>> <Mail Attachment.gif> >>> cc >>> <Mail Attachment.gif> >>> <Mail Attachment.gif> >>> Subject >>> <Mail Attachment.gif> >>> [Openstack] [SWIFT] PUTs and GETs getting slower >>> >>> >>> <Mail Attachment.gif> <Mail Attachment.gif> >>> Hi, >>> >>> my Swift cluster is working for three weeks now as a mail storage and the >>> performance is getting slower and slower. >>> >>> This is my hardware configuration: >>> >>> - 1 hardware-loadbalancer >>> - 2 proxy-server DELL R420 16 GB RAM 2 Quadcore CPU >>> - 5 proxy-server DELL R720xd 16 GB RAM 2 Quadcore CPU with 3x 2TB HDD SATA >>> 6GB/s >>> >>> The cluster contains: >>> >>> 279.000 containers >>> 8.170.000 objects = 984 GByte >>> >>> I have about 3.000.000 GETs per day and 1.500.000 PUTs. >>> >>> The load is raising up to 12 now: >>> >>> top - 11:06:39 up 114 days, 23:10, 1 user, load average: 10.88, 11.26, >>> 11.22 >>> Tasks: 175 total, 3 running, 172 sleeping, 0 stopped, 0 zombie >>> %Cpu(s): 15.6 us, 9.8 sy, 0.0 ni, 69.3 id, 4.5 wa, 0.0 hi, 0.8 si, >>> 0.0 st >>> KiB Mem: 16386408 total, 14829360 used, 1557048 free, 30796 buffers >>> KiB Swap: 17575932 total, 132604 used, 17443328 free, 5572360 cached >>> >>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >>> 25880 swift 20 0 81400 15m 3352 D 34.5 0.1 2838:36 >>> /usr/bin/python /usr/bin/swift-container-replicator >>> /etc/swift/container-server.conf >>> 24282 swift 20 0 81084 14m 3244 S 19.2 0.1 3835:23 >>> /usr/bin/python /usr/bin/swift-container-auditor >>> /etc/swift/container-server.conf >>> 29816 swift 20 0 1523m 27m 2032 D 9.6 0.2 1132:58 >>> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf >>> 29812 swift 20 0 1523m 27m 2032 R 7.0 0.2 1169:50 >>> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf >>> 29815 swift 20 0 1523m 27m 2032 D 6.6 0.2 1132:45 >>> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf >>> 29809 swift 20 0 1523m 26m 2032 D 5.6 0.2 1163:53 >>> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf >>> 30684 swift 20 0 84496 16m 2468 D 5.6 0.1 1959:57 >>> /usr/bin/python /usr/bin/swift-container-server >>> /etc/swift/container-server.conf >>> 13853 snmp 20 0 51844 3356 1800 S 5.3 0.0 32:14.90 >>> /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp -I -smux -p >>> /var/run/snmpd.pid >>> 29810 swift 20 0 1523m 27m 2032 D 5.3 0.2 1132:09 >>> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf >>> 29813 swift 20 0 1525m 27m 2032 D 5.3 0.2 1177:23 >>> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf >>> 29811 swift 20 0 1524m 28m 2032 D 5.0 0.2 1169:37 >>> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf >>> 30677 swift 20 0 84932 17m 2468 D 5.0 0.1 1992:49 >>> /usr/bin/python /usr/bin/swift-container-server >>> /etc/swift/container-server.conf >>> 29814 swift 20 0 1524m 27m 2032 D 4.6 0.2 1181:11 >>> /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf >>> 30681 swift 20 0 84980 17m 2468 S 4.3 0.1 1999:12 >>> /usr/bin/python /usr/bin/swift-container-server >>> /etc/swift/container-server.conf >>> 30678 swift 20 0 84684 17m 2468 D 4.0 0.1 1973:14 >>> /usr/bin/python /usr/bin/swift-container-server >>> /etc/swift/container-server.conf >>> 30682 swift 20 0 84224 16m 2468 S 4.0 0.1 1960:55 >>> /usr/bin/python /usr/bin/swift-container-server >>> /etc/swift/container-server.conf >>> 30685 swift 20 0 84356 16m 2468 D 3.6 0.1 1979:40 >>> /usr/bin/python /usr/bin/swift-container-server >>> /etc/swift/container-server.conf >>> 7086 root 20 0 0 0 0 S 3.3 0.0 0:00.78 [kworker/6:1] >>> 30683 swift 20 0 84160 16m 2468 S 3.3 0.1 1981:38 >>> /usr/bin/python /usr/bin/swift-container-server >>> /etc/swift/container-server.conf >>> 30679 swift 20 0 83852 16m 2468 S 3.0 0.1 1965:53 >>> /usr/bin/python /usr/bin/swift-container-server >>> /etc/swift/container-server.conf >>> 5935 root 20 0 0 0 0 S 2.7 0.0 0:16.38 [kworker/2:1] >>> 6832 root 20 0 0 0 0 S 2.7 0.0 0:02.78 [kworker/4:0] >>> 7153 swift 20 0 80300 9.8m 2040 D 2.7 0.1 0:00.08 >>> /usr/bin/python /usr/bin/swift-container-updater >>> /etc/swift/container-server.conf >>> 1840 root 20 0 0 0 0 S 2.3 0.0 0:59.87 [kworker/0:1] >>> 9630 swift 20 0 75796 6272 1708 D 2.0 0.0 1889:16 >>> /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf >>> 9656 swift 20 0 87424 17m 2304 S 2.0 0.1 748:29.97 >>> /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf >>> 9659 swift 20 0 86908 17m 2304 S 2.0 0.1 803:09.64 >>> /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf >>> 9653 swift 20 0 85816 16m 2304 S 1.7 0.1 723:34.42 >>> /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf >>> 9654 swift 20 0 87500 17m 2304 R 1.7 0.1 769:23.97 >>> /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf >>> 9655 swift 20 0 85956 16m 2408 S 1.7 0.1 783:28.18 >>> /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf >>> 9658 swift 20 0 87808 18m 2304 S 1.7 0.1 767:31.82 >>> /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf >>> 32616 root 20 0 0 0 0 S 1.7 0.0 0:18.13 [kworker/3:0] >>> 1826 syslog 20 0 243m 5124 856 S 1.3 0.0 329:10.12 rsyslogd -c5 >>> 6944 root 20 0 0 0 0 S 1.3 0.0 0:01.08 [kworker/7:2] >>> 9652 swift 20 0 86896 17m 2412 S 1.3 0.1 663:32.21 >>> /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf >>> 9657 swift 20 0 86016 16m 2408 S 1.3 0.1 706:22.08 >>> /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf >>> 51 root 20 0 0 0 0 S 1.0 0.0 134:26.37 [kswapd0] >>> 4548 root 20 0 0 0 0 S 1.0 0.0 0:11.69 [kworker/1:2] >>> 5362 root 20 0 0 0 0 S 1.0 0.0 0:10.95 [kworker/5:3] >>> 3 root 20 0 0 0 0 S 0.7 0.0 604:49.60 [ksoftirqd/0] >>> 52 root 20 0 0 0 0 S 0.7 0.0 68:32.58 [kswapd1] >>> 7412 swift 20 0 80300 9704 2780 S 0.7 0.1 566:00.53 >>> /usr/bin/python /usr/bin/swift-container-updater >>> /etc/swift/container-server.conf >>> 9580 swift 20 0 75632 7140 2728 D 0.7 0.0 2218:06 >>> /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf >>> 22178 root 20 0 7404 992 700 D 0.7 0.0 20:28.93 >>> /usr/bin/updatedb.mlocate >>> >>> Is this a memory problem or a harddisk problem? >>> >>> And second question: On the server the mlocate database is rising. Can I >>> disable the updatedb.mlocate process and empty the database? >>> >>> Thanks >>> Klaus >>> >>> >>> >>> _______________________________________________ >>> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>> Post to : openstack@lists.openstack.org >>> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>> >>> >>> >>> _______________________________________________ >>> Mailing list: >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>> >>> Post to : >>> openstack@lists.openstack.org >>> >>> Unsubscribe : >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> >> _______________________________________________ >> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> Post to : openstack@lists.openstack.org >> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > _______________________________________________ > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openstack@lists.openstack.org > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
