At 03:10 PM 8/3/2013 -0400, starli...@binnacle.cx wrote:
>Sometime in the next week I'll post a proper
>'az3md5' script that performs the S3 hash
>calculation for a specified segment size.
As promised.
#!/bin/sh
#
# Calculate Amazon S3 object Etag for given part size.
#
# For files less than or equal to the part size,
# return the MD5 hash for the part.
#
# For files larger than one part, the MD5 hash of each
# part is calculated and appended in binary form
# to a hash string. Then MD5 hash of the resulting hash
# string is returned suffixed with the hyphen character
# followed by the part count.
#
set -e
SEGSZU=${1?}
shift
FILES=${@}
if test $(expr "${SEGSZU?}" : '[0-9]\+$') -eq 0 \
-o -z "${FILES?}"
then
echo "usage: $0 part_size_mb file1 file2 file3 . . ."
exit 1
fi
UNITSZ=1048576
SEGSZY=$(expr ${SEGSZU:?} \* ${UNITSZ:?})
EXITCODE=0
for FILE in ${FILES:?}; do
if test ! -f ${FILE:?}; then
echo 1>&2 "\"${FILE:?}\" not found or not a regular file"
EXITCODE=1
else
FSIZE=$(stat -c '%s' ${FILE:?})
UNITCT=$(expr \( ${FSIZE:?} + ${UNITSZ:?} - 1 \) / ${UNITSZ:?})
SEGCNT=$(expr \( ${FSIZE:?} + ${SEGSZY:?} - 1 \) / ${SEGSZY:?})
if test ${SEGCNT:?} -le 1; then
md5sum --binary ${FILE:?} \
| awk '{printf "%-35s %s\n", $1, substr($2,2)}'
else
SKIP=0
while test ${SKIP:?} -lt ${UNITCT:?}; do
dd bs=${UNITSZ:?} count=${SEGSZU:?} skip=${SKIP:?} \
if=${FILE:?} 2>/dev/null \
| md5sum --binary \
| awk '{print $1}' \
| xxd -r -p
SKIP=$(expr ${SKIP:?} + ${SEGSZU:?})
done \
| md5sum --binary \
| awk -v SEGCNT=${SEGCNT:?} -v FILE=${FILE:?} \
'{printf "%-35s %s\n", $1 "-" SEGCNT, FILE}'
fi
fi
done
exit ${EXITCODE:?}
------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite!
It's a free troubleshooting tool designed for production.
Get down to code-level detail for bottlenecks, with <2% overhead.
Download for free and get started troubleshooting in minutes.
http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk
_______________________________________________
S3tools-general mailing list
S3tools-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/s3tools-general
