That's right, the maximum upload size is 5GB. As a short term solution I'm going to implement a check for the file size prior to upload to prevent these useless uploads.
And yes, the "WARNING: MD5 Sums don't match!" message is misleading. That's already fixed in s3cmd 0.9.9 though. Michal Fr. Robert Bower wrote: > According to Amazon > http://docs.amazonwebservices.com/AmazonS3/2006-03-01/gsg/ > > "Each object can hold up to 5 GB of data." > > You may have got lucky with the 5. 2 file and it finally failed on the 5.5 > > Thanks, > > Fr. Robert Bower > Batushka's Web Hosting & Design > frrob...@batushkas.com > 574.850.7982 > http://www.batushkas.com > http://www.batushkas.net > > > > Alex Efros wrote: >> Hi! >> >> I've large file (~5.5 GB - /home dir backup), and tried several times to >> upload it to S3 using s3cmd 0.9.8.4. Each time it took several hours and >> finally failed. I've just tried to upgrade to 0.9.9, it also failed but >> with another error message (see at bottom of this message). >> >> All other ~1600 files uploaded without troubles using 'sync' command, >> while this file failed both using 'sync' and 'put' commands. >> Also I should note this file is largest from all ~1600 files - next >> large file size is ~5.2GB and it uploaded normally. >> >> # time md5sum home.full.20090302_031507.tar.gpg >> d3722332b583f2d2e5f53413ee14fe57 home.full.20090302_031507.tar.gpg >> >> real 2m10.812s >> user 0m25.490s >> sys 0m8.570s >> >> # ls -l home.full.20090302_031507.tar.gpg >> -rw-r--r-- 1 backup backup 5586801747 2009-03-02 03:57 >> home.full.20090302_031507.tar.gpg >> >> # time s3cmd put home.full.20090302_031507.tar.gpg s3://backup.ourcompany >> WARNING: MD5 Sums don't match! >> WARNING: MD5 Sums don't match! >> WARNING: MD5 Sums don't match! >> WARNING: MD5 Sums don't match! >> ERROR: Upload of 'home.full.20090302_031507.tar.gpg' failed too many times. >> Skipping that file. >> >> real 303m38.849s >> user 7m56.840s >> sys 1m8.640s >> >> # time /etc/cron.daily/backup_to_s3 >> Compiling list of local files... >> Retrieving list of remote files... >> Found 1663 local files, 1632 remote files >> Verifying checksums... >> Summary: 8 local files to upload, 0 remote files to delete >> WARNING: MD5 Sums don't match! >> WARNING: MD5 Sums don't match! >> WARNING: MD5 Sums don't match! >> WARNING: MD5 Sums don't match! >> ERROR: home.full.20090302_031507.tar.gpg: upload failed too many times. >> Skipping that file. >> File 'mysql.incr.20090304_030008.tar.gpg' stored as >> s3://backup.ourcompany/mysql.incr.20090304_030008.tar.gpg (80084541 bytes in >> 171.0 seconds, 457.26 kB/s) [2 of 8] >> File 'home.incr.20090304_030010.tar.gpg' stored as >> s3://backup.ourcompany/home.incr.20090304_030010.tar.gpg (14791469 bytes in >> 1.9 seconds, 7.47 MB/s) [3 of 8] >> File 'mysql.incr.20090304_030010.tar.gpg' stored as >> s3://backup.ourcompany/mysql.incr.20090304_030010.tar.gpg (1554 bytes in 0.0 >> seconds, 9.43 MB/s) [4 of 8] >> File 'svn.incr.20090304_030010.tar.gpg' stored as >> s3://backup.ourcompany/svn.incr.20090304_030010.tar.gpg (3553 bytes in 0.0 >> seconds, 22.17 MB/s) [5 of 8] >> File 'config.incr.20090304_071720.tar.gpg' stored as >> s3://backup.ourcompany/config.incr.20090304_071720.tar.gpg (32459 bytes in >> 0.3 seconds, 125.75 kB/s) [6 of 8] >> File 'home.incr.20090304_071720.tar.gpg' stored as >> s3://backup.ourcompany/home.incr.20090304_071720.tar.gpg (2705201 bytes in >> 1.0 seconds, 2.60 MB/s) [7 of 8] >> File 'mysql.incr.20090304_071720.tar.gpg' stored as >> s3://backup.ourcompany/mysql.incr.20090304_071720.tar.gpg (979 bytes in 0.0 >> seconds, 6.06 MB/s) [8 of 8] >> Done. Uploaded 97619756 bytes in 34733.5 seconds, 2.74 kB/s >> >> real 599m43.248s >> user 10m35.850s >> sys 2m7.890s >> >> >> 0.9.9 fail in different way: >> >> # time s3cmd put home.full.20090302_031507.tar.gpg s3://backup.ourcompany >> home.full.20090302_031507.tar.gpg -> s3://backup.ourcompany/ [1 of 1] >> 5586801747 of 5586801747 100% in 11194s 487.39 kB/s done >> ERROR: S3 error: 400 (MalformedXML): The XML you provided was not >> well-formed or did not validate against our published schema >> >> real 186m34.334s >> user 2m27.220s >> sys 0m28.430s >> >> >> > > ------------------------------------------------------------------------------ > Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA > -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise > -Strategies to boost innovation and cut costs with open source participation > -Receive a $600 discount off the registration fee with the source code: SFAD > http://p.sf.net/sfu/XcvMzF8H > _______________________________________________ > S3tools-general mailing list > S3tools-general@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/s3tools-general > ------------------------------------------------------------------------------ Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA -OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise -Strategies to boost innovation and cut costs with open source participation -Receive a $600 discount off the registration fee with the source code: SFAD http://p.sf.net/sfu/XcvMzF8H _______________________________________________ S3tools-general mailing list S3tools-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/s3tools-general
