costin 01/01/31 20:38:14
Added: src/build/nightly README build_once.sh build_tomcat.sh
cvs_get.sh functions_build.sh nightly.sh
run_watchdog.sh start_tomcat.sh stop_tomcat.sh
weekly.sh
Log:
Added the shell scripts I use for the nightly build. All is an ugly
hack, but it works ( it started with a simple shell to just build
tomcat ).
Revision Changes Path
1.1 jakarta-tomcat/src/build/nightly/README
Index: README
===================================================================
- I tested the scripts on linux and solaris - only standard bourne shell
is used
- this is based on my early attempt based on ant - I'll try again later,
now .sh does the job and it's enough
To run the tests:
1. make sure .antrc and .antproperties are clean
2. You need the following directory structure ( or edit the .sh files to
match your - the scripts are not very configurable, it's just a tool to
build ):
$HOME: ( the home directory of the user who have the crontab job ):
$HOME/bin/nightly/* -> the scripts
$HOME/java/jsse -> jsse jars ( jcert.jar jnet.jar jsse.jar)
$HOME/java/xml -> jaxp jars ( jaxp.jar parser.jar )
$HOME/java/java1.1 -> link to a working 1.1 jdk installation
$HOME/java/java1.2 -> link to a working 1.2+ jdk
$HOME/opt/ant-1.2 -> link to ant 1.2
$HOME/opt/ant-1.1 -> link to ant 1.1 ( needed by watchdog )
$HOME/opt/ws/log -> empty dir
$HOME/opt/ws/zip -> empty dir
$HOME/opt/ws/build -> empty dir
You must do "cvs login" first, so the passowrd is stored in .cvspass.
- If behind a firewall, you can use "runsocks nightly.sh" ( or
"socksify nightly.sh" ).
You must run "weekly.sh" to get the initial CVS repository and build
servletapi and watchdog for the first time.
Then nightly.sh will only update ( cvs update ) to minimize time.
What's inside ?
===============
weekly.sh -> get jakarta-tomcat, watchdog, tools, build everything for the
first time ( it's a fresh checkout - old dirs are removed )
nightly.sh -> main driver, call various build and test functions and
scripts with various JDKs and CLASSPATHS
start_tomcat.sh -> a tricky one - start tomcat and wait till it's ready
( by grep-ping the log file for HttpInterceptor starting )
stop_tomcat.sh -> ATTENTION: it'll first try to be nice, but then it'll
kill all java processes ( pkill, killall ) - use a separate
account for building ( and not root ) (XXX do a grep first )
run_watchdog.sh -> start, run watchdog 3 times, run sanity, grep the
results for error counts
cvs_get.sh -> update or clean checkoug, for tomcat or all
build_tomcat.sh -> just build and zip the result ( after adding jaxp )
functions_build.sh -> various shell functions ( I use to source it and
use them in command line to repeat individual steps )
NOTE: don't expect nice code, it's just a hack to build and test tomcat.
It may be useful - if it's not, ignore this directory.
Costin
1.1 jakarta-tomcat/src/build/nightly/build_once.sh
Index: build_once.sh
===================================================================
#!/bin/sh
. $HOME/bin/nightly/functions_build.sh
# override ANT_HOME
ANT_HOME=$HOME/opt/ant-1.1
export ANT_HOME
######## MAIN
echo Building watchdog
echo ---------------------------------------------------
ant_build jakarta-servletapi servletapi servletapi-build.log
ant_build jakarta-watchdog watchdog watchdog-build.log
tail -6 $LOGDIR/watchdog-build.log
echo
1.1 jakarta-tomcat/src/build/nightly/build_tomcat.sh
Index: build_tomcat.sh
===================================================================
#!/bin/sh
. $HOME/bin/nightly/functions_build.sh
if [ "$1" != "" ] ; then
SUFIX=$1
fi
######## MAIN
ant_build jakarta-tomcat tomcat tomcat-3.3-build-$SUFIX.log "main watchdog-web-based
dist"
fix_tomcat
zip_dist tomcat tomcat-3.3-$SUFIX
1.1 jakarta-tomcat/src/build/nightly/cvs_get.sh
Index: cvs_get.sh
===================================================================
#!/bin/sh
. $HOME/bin/nightly/functions_build.sh
GET=""
if [ "$1" = "-f" ]; then
GET="-f"
shift
cvs_get jakarta-tomcat
else
cvs_update jakarta-tomcat
fi
zip_src jakarta-tomcat tomcat-3.3-src.zip
if [ "$1" = "all" ]; then
if [ "$GET" = "-f" ]; then
cvs_get jakarta-watchdog
cvs_get jakarta-tools
cvs_get jakarta-servletapi
else
cvs_update jakarta-watchdog
cvs_update jakarta-tools
cvs_update jakarta-servletapi
fi
zip_src jakarta-watchdog watchdog-src.zip
zip_src jakarta-servletapi servletapi-src.zip
fi
zip_src jakarta-tomcat tomcat-3.3-src.zip
1.1 jakarta-tomcat/src/build/nightly/functions_build.sh
Index: functions_build.sh
===================================================================
#!/bin/sh
# Config: ZIPDIR CVSROOT ANT_HOME JAVA_HOME
# PATH
# JDK11_HOME JDK12_HOME
if [ -f $HOME/.nightlyrc ] ; then
. $HOME/.nightlyrc
fi
# Defaults
if [ "$WS" = "" ] ; then
WS=$HOME/ws
fi
if [ "$JAVA_HOME" = "" ] ; then
JAVA_HOME=$HOME/java/java1.2
fi
ZIP=zip
ZIPDIR=$WS/zip
LOGDIR=$WS/log
CVSROOT=:pserver:[EMAIL PROTECTED]:/home/cvspublic
PATH=$HOME/bin:$HOME/bin/nightly:/usr/local/bin:$PATH
JSSE=$HOME/java/jsse
EXTENTSIONS=$JSSE/jsse.jar:$JSSE/jnet.jar:$JSSE/jcert.jar:.
ANT_HOME=$HOME/opt/ant-1.2
export JAVA_HOME
export ZIP
export ZIPDIR
export LOGDIR
export CVSROOT
export PATH
export JSSE
export EXTENSIONS
export ANT_HOME
## Make sure all dirs exists
mkdir -p $ZIPDIR
mkdir -p $LOGDIR
help_nightly() {
# nothing yet
echo
}
check() {
# Java version
# JSSE
# Ant 1, 2
echo
}
# functions
cvs_get() {
MOD=$1
cd $WS
rm -rf $MOD
echo cvs -d $CVSROOT co $MOD
cvs co $MOD >$LOGDIR/cvs-get-$MOD.log 2>&1
wc $LOGDIR/cvs-get-$MOD.log
tail -5 $LOGDIR/cvs-get-$MOD.log
echo
}
cvs_update() {
MOD=$1
cd $WS/$MOD
echo cvs -d -P $CVSROOT update
cvs update >$LOGDIR/cvs-update-$MOD.log 2>&1
wc $LOGDIR/cvs-update-$MOD.log
tail -5 $LOGDIR/cvs-update-$MOD.log
echo
}
ant_build() {
MOD=$1
DIST=$2
LOG=$LOGDIR/$3
TARGET=$4
# ANT_HOME
# JAVA_HOME
if [ "$TARGET" = "" ] ;
then TARGET=dist ; fi
echo ---------- $DIST BUILD `date` ----------
echo JAVA_HOME=$JAVA_HOME
echo LOG=$LOG
echo WS=$WS/$MOD
echo CLASSPATH=$CLASSPATH
echo
echo ---------- $DIST BUILD `date` ---------- >> $LOG 2>&1
echo JAVA_HOME=$JAVA_HOME >> $LOG 2>&1
echo LOG=$LOG >> $LOG 2>&1
echo WS=$WS/$MOD >> $LOG 2>&1
echo CLASSPATH=$CLASSPATH >> $LOG 2>&1
echo >> $LOG 2>&1
$JAVA_HOME/bin/java -version >> $LOG 2>&1
cd $WS/$MOD
echo rm -rf $WS/build/$DIST >> $LOG 2>&1
echo rm -rf $WS/dist/$DIST >> $LOG 2>&1
rm -rf $WS/build/$DIST >> $LOG 2>&1
rm -rf $WS/dist/$DIST >> $LOG 2>&1
JAVACMD=$JAVA_HOME/bin/java
export JAVACMD
echo Building with $ANT_HOME/bin/ant $TARGET 2>&1 >> $LOG
$ANT_HOME/bin/ant $TARGET 2>&1 >> $LOG
grep "BUILD SUCCESSFUL" $LOG
if [ "$?" != "0" ]; then
echo BUILD FAILED.
echo ---------- HEAD:
head -10 $LOG
echo ---------- TAIL:
tail -20 $LOG
echo ----------
echo
fi
echo ---------- DONE $DIST BUILD `date` ----------
}
ant_build_xml() {
MOD=$1
DIST=$2
LOG=$LOGDIR/$3
# ANT_HOME
# JAVA_HOME
echo ---------- $DIST BUILD `date` ----------
echo JAVA_HOME=$JAVA_HOME
echo LOG=$LOG
echo WS=$WS/$MOD
echo CLASSPATH=$CLASSPATH
echo
echo ---------- $DIST BUILD `date` ---------- >> $LOG 2>&1
echo JAVA_HOME=$JAVA_HOME >> $LOG 2>&1
echo LOG=$LOG >> $LOG 2>&1
echo WS=$WS/$MOD >> $LOG 2>&1
echo CLASSPATH=$CLASSPATH >> $LOG 2>&1
echo >> $LOG 2>&1
$JAVA_HOME/bin/java -version >> $LOG 2>&1
cd $WS/$MOD
echo rm -rf $WS/$MOD/build >> $LOG 2>&1
rm -rf $WS/$MOD/build >> $LOG 2>&1
JAVACMD=$JAVA_HOME/bin/java
export JAVACMD
$ANT_HOME/bin/ant dist 2>&1 >> $LOG
grep "BUILD SUCCESSFUL" $LOG
if [ "$?" != "0" ]; then
echo BUILD FAILED.
echo ---------- HEAD:
head -10 $LOG
echo ---------- TAIL:
tail -20 $LOG
echo ----------
echo
fi
echo ---------- DONE $DIST BUILD `date` ----------
}
zip_src() {
MOD=$1
ZIPNAME=$2
cd $WS
echo
echo zip -r $ZIPNAME $MOD
zip -r $MOD >/dev/null
mv zip.zip $ZIPDIR/$ZIPNAME
echo
}
zip_dist() {
DIST=$1
ARCH=$2
echo
echo Creating $DIST distribution in $ZIPDIR:
echo $ARCH.tar.gz $ARCH.zip
cd $WS/dist
rm -f $ZIPDIR/$ARCH.tar >/dev/null 2>&1
rm -f $ZIPDIR/$ARCH.tar.gz >/dev/null 2>&1
tar cvf $ZIPDIR/$ARCH.tar $DIST >/dev/null 2>&1
gzip $ZIPDIR/$ARCH.tar
rm $ZIPDIR/$ARCH.zip >/dev/null 2>&1
zip -r $DIST >/dev/null
mv zip.zip $ZIPDIR/$ARCH.zip
echo
}
fix_tomcat() {
cp $ANT_HOME/lib/parser.jar $WS/dist/tomcat/lib
cp $ANT_HOME/lib/jaxp.jar $WS/dist/tomcat/lib
}
count_errors() {
BASELOG=$1
grep -v "^OK" $BASELOG.log | \
grep -v "The args attribute" | grep -v "In compareWeek" \
> $BASELOG-errors.log
ERRORS=`grep FAIL $BASELOG-errors.log | grep -v "FAIL\*\*\*" |wc -l`
OK=`grep "^OK" $BASELOG.log |wc -l`
echo
echo LOG=$BASELOG.log
echo OK: $OK ERRORS: $ERRORS
echo
echo Watchdog tail:
echo --------------------
tail -10 $BASELOG.log
echo --------------------
}
1.1 jakarta-tomcat/src/build/nightly/nightly.sh
Index: nightly.sh
===================================================================
#!/bin/sh
. $HOME/bin/nightly/functions_build.sh
PATH=$HOME/bin/nightly:$PATH
# Special builds - no ext
CLASSPATH=.
JAVA_HOME=$HOME/java/java1.2
TOMCAT_HOME=/home/costin/ws/dist/tomcat
export TOMCAT_HOME
export CLASSPATH
export JAVA_HOME
cvs_get.sh
echo rm -f $LOGDIR/*
rm -f $LOGDIR/*
echo BUILD `date` >$LOGDIR/nightly.log
build_tomcat.sh noext >>$LOGDIR/nightly.log 2>&1
run_watchdog.sh noext >>$LOGDIR/nightly.log 2>&1
echo BUILD `date` >>$LOGDIR/nightly.log
# Special tomcat build - 1.1
JAVA_HOME=$HOME/java/java1.1
LD_LIBRARY_PATH=$JAVA_HOME/lib:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH
CLASSPATH=.
echo BUILD `date` >>$LOGDIR/nightly.log
build_tomcat.sh jdk11 >>$LOGDIR/nightly.log 2>&1
#THREADS_FLAG=green
#export THREADS_FLAG
# run_watchdog.sh jdk11 >>$LOGDIR/nightly.log 2>&1
rm -rf $WS/dist/tomcat-1.1
echo mv $WS/dist/tomcat $WS/dist/tomcat-1.1
mv $WS/dist/tomcat $WS/dist/tomcat-1.1
echo done
#echo BUILD `date` >>$LOGDIR/nightly.log
#THREADS_FLAG=native
#unset LD_LIBARY_PATH
# Full build
CLASSPATH=$EXTENSIONS
JAVA_HOME=$HOME/java/java1.2
echo $EXT
echo BUILD `date` >>$LOGDIR/nightly.log
build_tomcat.sh full >>$LOGDIR/nightly.log 2>&1
run_watchdog.sh full >>$LOGDIR/nightly.log 2>&1
echo security test
run_watchdog.sh security >>$LOGDIR/nightly.log 2>&1
echo security done
echo BUILD `date` >>$LOGDIR/nightly.log
# mail -s "Tomcat3.3 nightly build " \
# [EMAIL PROTECTED] \
# < $LOGDIR/nightly.log
1.1 jakarta-tomcat/src/build/nightly/run_watchdog.sh
Index: run_watchdog.sh
===================================================================
#!/bin/sh
. $HOME/bin/nightly/functions_build.sh
######## MAIN
ROOT=$WS/dist
if [ "$1" = "-b" ] ; then
ROOT=$WS/build
BUILD=true
echo Using build $1
shift
fi
if [ "$1" = "-s" ] ; then
NOSTART=true;
echo No start
shift
fi
EXT=$1
CLASSPATH=$ANT_HOME/lib/parser.jar:$CLASSPATH
CLASSPATH=$ANT_HOME/lib/jaxp.jar:$CLASSPATH
CLASSPATH=$ANT_HOME/lib/ant.jar:$CLASSPATH
export CLASSPATH
cd $ROOT
cp $ANT_HOME/lib/ant.jar $ROOT/watchdog/lib
#rm -rf tomcat/webapps/jsp-tests >/dev/null 2>&1
#rm -rf tomcat/webapps/servlet-tests >/dev/null 2>&1
if [ "$BUILD" != "" ]; then
ln -s $ROOT/watchdog/webapps/jsp-tests tomcat/webapps >/dev/null 2>&1
ln -s $ROOT/watchdog/webapps/servlet-tests tomcat/webapps >/dev/null 2>&1
else
ln -s $ROOT/watchdog/webapps/jsp-tests.war tomcat/webapps >/dev/null 2>&1
ln -s $ROOT/watchdog/webapps/servlet-tests.war tomcat/webapps >/dev/null 2>&1
fi
echo ---------- Start tomcat `date` ----------
echo WS=$ROOT EXT=$EXT
echo Start `date` >$LOGDIR/watchdog-$EXT-1.log 2>&1
# Port 9080
mv $ROOT/tomcat/conf/server.xml $ROOT/tomcat/conf/server.xml.orig
sed s/8080/9080/ <$ROOT/tomcat/conf/server.xml.orig >$ROOT/tomcat/conf/server.xml
if [ "$NOSTART" = "" ]; then
if [ "$BUILD" = "" ]; then
start_tomcat.sh $EXT
if [ "$?" = "1" ]; then
exit 1
fi
else
start_tomcat.sh -b $EXT
if [ "$?" = "1" ]; then
exit 1
fi
fi
fi
cd $ROOT/watchdog
WATCHDOG_HOME=`pwd`
export WATCHDOG_HOME
# We know watchdog can fail with JDK1.1.
echo
echo ---------- Running watchdog `date` ----------
echo JAVA_VERSION `$JAVA_HOME/bin/java -version`
rm -rf $TOMCAT_HOME/work/DEFAULT/jsp-tests/jsp/tagext
time bin/watchdog.sh all localhost 9080 >>$LOGDIR/watchdog-$EXT-1.log 2>&1
echo Watchdog1 done `date`
count_errors $LOGDIR/watchdog-$EXT-1
$JAVA_HOME/bin/java -version 2>&1 | grep 1.1 >/dev/null 2>&1
if [ "$?" = "0" ] ; then
echo Detected java1.1 - run again with 1.2
JAVA_HOME=$HOME/java/java1.2
echo
echo Running watchdog 1.2 `date`
time bin/watchdog.sh all localhost 9080 >$LOGDIR/watchdog-$EXT-1.2.log 2>&1
echo Watchdog1.2 done `date`
count_errors $LOGDIR/watchdog-$EXT-1.2
fi
echo
echo Running watchdog 2 `date`
rm -rf $TOMCAT_HOME/work/DEFAULT/jsp-tests/jsp/tagext
time bin/watchdog.sh all localhost 9080 >$LOGDIR/watchdog-$EXT-2.log 2>&1
echo Watchdog2 done `date`
count_errors $LOGDIR/watchdog-$EXT-2
echo
echo Running watchdog 3 `date`
rm -rf $TOMCAT_HOME/work/DEFAULT/jsp-tests/jsp/tagext
time bin/watchdog.sh all localhost 9080 >$LOGDIR/watchdog-$EXT-3.log 2>&1
echo Watchdog3 done `date`
count_errors $LOGDIR/watchdog-$EXT-3
X=$PATH
PATH=$HOME/opt/ant-1.2/bin:$PATH
export PATH
time sh $TOMCAT_HOME/webapps/admin/WEB-INF/scripts/run-test.sh -Dport=9080
>$LOGDIR/sanity-$EXT.log 2>&1
count_errors $LOGDIR/sanity-$EXT
PATH=$X
export PATH
if [ "$NOSTART" = "" ]; then
stop_tomcat.sh >>$LOGDIR/watchdog-$EXT-1.log 2>&1
fi
1.1 jakarta-tomcat/src/build/nightly/start_tomcat.sh
Index: start_tomcat.sh
===================================================================
#!/bin/sh
. $HOME/bin/nightly/functions_build.sh
######## MAIN
ROOT=$WS/dist
if [ "$1" = "-b" ] ; then
ROOT=$WS/build
shift
fi
EXT=$1
CLASSPATH=$ANT_HOME/lib/parser.jar:$CLASSPATH
CLASSPATH=$ANT_HOME/lib/jaxp.jar:$CLASSPATH
CLASSPATH=$ANT_HOME/lib/ant.jar:$CLASSPATH
export CLASSPATH
cd $ROOT
echo Start tomcat
echo TOMCAT_HOME=`pwd`
echo LOG=$LOGDIR/tomcat-run$EXT.log
$JAVA_HOME/bin/java -version
# Make sure no tomcat is running
cd $ROOT/tomcat
echo Make sure tomcat is stopped
bin/tomcat.sh stop >/dev/null 2>&1
cd $ROOT/tomcat
rm -f $LOGDIR/tomcat-run$EXT.log >/dev/null 2>&1
$JAVA_HOME/bin/java -version > $LOGDIR/tomcat-run$EXT.log 2>&1 &
if [ "$EXT" = "security" ] ; then
bin/tomcat.sh run -security >> $LOGDIR/tomcat-run$EXT.log 2>&1 &
else
bin/tomcat.sh run >> $LOGDIR/tomcat-run$EXT.log 2>&1 &
fi
i=0
while [ "$STARTED" != "0" ] ; do
sleep 1
# read
# echo $REPLY |
grep "Http10Interceptor" $LOGDIR/tomcat-run$EXT.log \
>/dev/null 2>&1
STARTED=$?
i=`expr $i + 1`
if [ "$i" = "120" ] ; then
echo "Can't start after 120 seconds "
tail -20 $LOGDIR/tomcat-run$EXT.log
exit 1
fi
# echo "Wait - not ready $i $STARTED"
# ls -l $LOGDIR/tomcat-run$EXT.log
done
sleep 2
echo Started ok
1.1 jakarta-tomcat/src/build/nightly/stop_tomcat.sh
Index: stop_tomcat.sh
===================================================================
#!/bin/sh
. $HOME/bin/nightly/functions_build.sh
######## MAIN
ROOT=$WS/dist
if [ "$1" = "-b" ] ; then
ROOT=$WS/build
fi
CLASSPATH=$ANT_HOME/lib/parser.jar:$CLASSPATH
CLASSPATH=$ANT_HOME/lib/jaxp.jar:$CLASSPATH
CLASSPATH=$ANT_HOME/lib/ant.jar:$CLASSPATH
export CLASSPATH
cd $ROOT
# Make sure no tomcat is running
cd $ROOT/tomcat
bin/tomcat.sh stop
pkill java
killall -9 java
1.1 jakarta-tomcat/src/build/nightly/weekly.sh
Index: weekly.sh
===================================================================
#!/bin/sh
. $HOME/bin/nightly/functions_build.sh
PATH=$HOME/bin/nightly:$PATH
echo "BUILD `date`" >$LOGDIR/weekly.log
CLASSPATH=$EXTENSIONS
JAVA_HOME=$HOME/java/java1.2
export CLASSPATH
export JAVA_HOME
cvs_get.sh -f all >>$LOGDIR/weekly.log 2>&1
build_once.sh >>$LOGDIR/weekly.log 2>&1
build_tomcat.sh full >>$LOGDIR/weekly.log 2>&1
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, email: [EMAIL PROTECTED]
