This 'bash' script calculates the
Amazon S3 Etag value for large files
and/or files that were uploaded using the
parallel transfer approach.
The first parameter is the part or
chunk size to use. Files <=
that size have a simple MD5 hash
calculated. Larger files have
the MD5 hash of the MD5 hashes
of all the parts calculated.
Sent this out yesterday but for some
reason the Sourceforge mail archive
does not show that messages.
Removed the shebang /bin/sh on
the first line in case that was
causing a filter problem.
#
# Calculate Amazon S3 object Etag for given part size.
#
# For files less than or equal to the part size,
# return the MD5 hash for the part.
#
# For files larger than one part, the MD5 hash of each
# part is calculated and appended in binary form
# to a hash string. Then MD5 hash of the resulting hash
# string is returned suffixed with the hyphen character
# followed by the part count.
#
set -e
SEGSZU=${1?}
shift
FILES=${@}
if test $(expr "${SEGSZU?}" : '[0-9]\+$') -eq 0 \
-o -z "${FILES?}"
then
echo "usage: $0 part_size_mb file1 file2 file3 . . ."
exit 1
fi
UNITSZ=1048576
SEGSZY=$(expr ${SEGSZU:?} \* ${UNITSZ:?})
EXITCODE=0
for FILE in ${FILES:?}; do
if test ! -f ${FILE:?}; then
echo 1>&2 "\"${FILE:?}\" not found or not a regular file"
EXITCODE=1
else
FSIZE=$(stat -c '%s' ${FILE:?})
UNITCT=$(expr \( ${FSIZE:?} + ${UNITSZ:?} - 1 \) / ${UNITSZ:?})
SEGCNT=$(expr \( ${FSIZE:?} + ${SEGSZY:?} - 1 \) / ${SEGSZY:?})
if test ${SEGCNT:?} -le 1; then
md5sum --binary ${FILE:?} \
| awk '{printf "%-35s %s\n", $1, substr($2,2)}'
else
SKIP=0
while test ${SKIP:?} -lt ${UNITCT:?}; do
dd bs=${UNITSZ:?} count=${SEGSZU:?} skip=${SKIP:?} \
if=${FILE:?} 2>/dev/null \
| md5sum --binary \
| awk '{print $1}' \
| xxd -r -p
SKIP=$(expr ${SKIP:?} + ${SEGSZU:?})
done \
| md5sum --binary \
| awk -v SEGCNT=${SEGCNT:?} -v FILE=${FILE:?} \
'{printf "%-35s %s\n", $1 "-" SEGCNT, FILE}'
fi
fi
done
exit ${EXITCODE:?}
------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead.
Download for free and get started troubleshooting in minutes.
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
S3tools-general mailing list
S3tools-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/s3tools-general
