Phil (and anyone else interested), The script is now written and I have spent most of the day testing it. It has currently been running for about three hours now and seems to be running correctly. I'm attaching the script and a set of instructions on how to configure and run it. The script creates and maintains a "last known good" copy of the weewx.sdb database at whatever interval the user specifies. It also attempts to be as robust as possible, notifying the user (via email) of any exception conditions rather than simply "failing silently". If you (or anyone else) runs it, I'd be interested in getting feedback as to any issues discovered.
Dave -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
Instructions.pdf
Description: Adobe PDF document
#! /usr/bin/env python3 ################################################################################ # # # sdbCheck.py # # # # The primary purpose of this script is to create and maintain a "last known # # good" copy of the weeWX database. To do this, it makes a copy of the weeWX # # database in the script starting directory (waiting, if necessary, until the # # database is not being updated). The integrity of the copy is checked using # # sqlite3 and, if it is good, it replaces the previous copy in the ./Backup # # directory. If a problem is found an email is sent to the user. A small log # # file is also created indicating the results of the run. # # # # The script is intended to be run periodically by the operating system using # # whatever support the OS provides (cron, Task Scheduler, etc.). It is also # # intended to run as "safely" as possible without user intervention, so the # # first thing the script does is to verify that the previous run happened when # # it was supposed to and that it completed correctly. If not, an email is # # sent to the user. # # # # There are a number of script options that can be configured by the user, # # such as email address, period of weeWX updates, etc. These are listed below # # and must be set prior to operation. To assist in verifying that they are # # set correctly, the script can be manually started on the command line with # # anything as a parameter. When the script is started, it checks the length # # of sys.argv and, if it is more than 1, it will perform a configuration check # # and then exit after printing the results on the screen and sending a test # # email. # # # # For support issues, please contact [email protected]. Feel free to modify # # and / or redistribute this script. If you add a useful feature, please let # # me know. # # # ################################################################################ ################################################################################ # Import the modules that will be used throughout this script. # ################################################################################ import sys import os import os.path import time from datetime import datetime import shutil from shutil import copyfile ################################################################################ # The following options can be used to tailor the script to a specific # # environment. # ################################################################################ WEEWX_DIR = "" # location of the weewx.sdb file WEEWX_UPDATE = -1 # number of seconds between weewx.sdb updates SCRIPT_UPDATE = -1 # number of seconds between runs of this script INSTALL_DIR = "" # full path to where this script is installed EMAIL_ADDR = "" # login address on email host EMAIL_PASSWD = "" # email host login password EMAIL_HOST = "" # email host EMAIL_PORT = -1 # port number used by host smtp EMAIL_TARGET = "" # address to send email to (may be same as EMAIL_ADDR) EMAIL_TIMEOUT = 10 # configCheck() timeout in seconds - increase if needed SLOP = 30 # number of seconds to use as a "slop factor" when checking timeouts ################################################################################ # Python requires functions to be defined before they are used, so this next # # section contains all of them prior to the script entry point. This first # # one takes as input a string which will be written to the log file. # ################################################################################ def logIt(msg): lfd = open("sdbCheck.log", "w") lfd.write(msg) lfd.close() ################################################################################ # This function will send an email to the user. It takes as input the body of # # the message to be sent. The function will log into the user email host, # # create and send the message and then log back out again. # ################################################################################ def sendEmail(body): import smtplib # import required email modules from email.mime.multipart import MIMEMultipart from email.mime.text import MIMEText s = smtplib.SMTP(host = EMAIL_HOST, port = EMAIL_PORT) s.starttls() s.login(EMAIL_ADDR, EMAIL_PASSWD) msg = MIMEMultipart() msg['From'] = EMAIL_ADDR msg['To'] = EMAIL_TARGET msg['Subject'] = 'sdbCheck.py Notification' msg.attach(MIMEText(body, 'plain')) s.send_message(msg) s.quit() ################################################################################ # This function is called if there is anything at all on the command line # # following the script name. It will go through each of the configuration # # options and verify that the option works. The script must be started in the # # installation directory. # ################################################################################ def configCheck(): import smtplib # import required email modules from email.mime.multipart import MIMEMultipart from email.mime.text import MIMEText ############################################################################## # First step - verify the INSTALL_DIR is set and that we started in that # # directory. # ############################################################################## print("Checking INSTALL_DIR: ", end="") if INSTALL_DIR == "": print("ERROR - directory not set\n") sys.exit(1) if INSTALL_DIR != os.getcwd(): print("ERROR - INSTALL_DIR doesn't match") print(" INSTALL_DIR: " + INSTALL_DIR) print(" Current Directory: " + os.getcwd() + ")\n") sys.exit(1) print("OK") ############################################################################## # Next, make sure that the WEEWX_DIR directory has been set and that a copy # # of weewx.sdb can be found in that directory. # ############################################################################## print("Checking WEEWX_DIR: ", end="") if WEEWX_DIR == "": print("ERROR - directory not set\n") sys.exit(1) if not os.path.isfile(WEEWX_DIR + "/weewx.sdb"): print("ERROR - " + WEEWX_DIR + "/weewx.sdb not found\n") sys.exit(1) print("OK") ############################################################################## # There is no easy way to verify the values for the UPDATE settings but at # # a check can be made to see that a value was entered. # ############################################################################## print("Checking WEEWX_UPDATE: ", end="") if WEEWX_UPDATE == -1: print("ERROR - value not set\n") sys.exit(1) else: print("OK") print("Checking SCRIPT_UPDATE: ", end="") if SCRIPT_UPDATE == -1: print("ERROR - value not set\n") sys.exit(1) else: print("OK") ############################################################################## # Next, check to make sure values are set for the email parameters. # ############################################################################## print("Checking EMAIL_ADDR: ", end="") if EMAIL_ADDR == "": print("ERROR - value not set\n") sys.exit(1) print("OK") print("Checking EMAIL_PASSWD: ", end="") if EMAIL_PASSWD == "": print("ERROR - value not set\n") sys.exit(1) print("OK") print("Checking EMAIL_HOST: ", end="") if EMAIL_HOST == "": print("ERROR - value not set\n") sys.exit(1) print("OK") print("Checking EMAIL_PORT: ", end="") if EMAIL_PORT == -1: print("ERROR - value not set\n") sys.exit(1) print("OK") print("Checking EMAIL_TARGET: ", end="") if EMAIL_TARGET == "": print("ERROR - value not set\n") sys.exit(1) print("OK") ############################################################################## # Now make sure that sqlite3 is installed on the system and is in the PATH # # and executable. # ############################################################################## print("Checking sqlite3: ", end="") if not shutil.which("sqlite3"): print("ERROR - sqlite3 executable not found, check installation") sys.exit(1) print("OK") ############################################################################## # Last step - try to send a test email, error checking as we go. The check # # uses a timeout value of 10 seconds. If this step is failing and you feel # # the settings are correct, try increasing this value. # ############################################################################## print("Opening SMTP host: ", end="") try: s = smtplib.SMTP(host=EMAIL_HOST, port=EMAIL_PORT, timeout=EMAIL_TIMEOUT) except: print("ERROR - connect timeout after " + str(EMAIL_TIMEOUT) + " seconds\n") sys.exit(1) print("OK") s.starttls() print("Logging into email account: ", end="") try: s.login(EMAIL_ADDR, EMAIL_PASSWD) except: print("ERROR - unable to login with " + EMAIL_ADDR + ", " + EMAIL_PASSWD) sys.exit(1) print("OK") msg = MIMEMultipart() msg['From'] = EMAIL_ADDR msg['To'] = EMAIL_TARGET msg['Subject'] = 'Email test from sdbCheck.py' msg.attach(MIMEText('All configuration checks completed successfully', 'plain')) s.send_message(msg) print("Sending test email - check your inbox") ################################################################################ # # # SCRIPT ENTRY POINT # # # # Everything preceding this point has been constant and function definitions. # # This begins the actual script body. The first thing that is done is to see # # if any command line parameter was specified. If so, the program calls # # configCheck() and then exits. Otherwise, the script continues with a number # # of checks that try to detect any existing problems prior to updating the # # known good copy of weewx.sdb. Since the script is expected to run without # # having to be constantly monitored, it is important to avoid a situation # # where it "fails silently", leaving the user thinking that everything is # # running fine because no errors have been reported. Thus, the checks start # # out as simply as possible and build up to a point where it should be safe to # # try updating the backup copy of weewx.sdb. If any of the checks fail, the # # user is sent an email indicating the problem. The checks consist of: # # # # 1) Making sure the INSTALL_DIR still exists and can be switched to. # # 2) Making sure the Backup directory exists and creating it if not. # # 3) Checking that weeWX has updated weewx.sdb within the WEEWX_UPDATE time. # # 4) Making sure a sdbCheck.log file exists and creating and empty one if # # not. This will happen the first time the script is run. # # 5) Checking that log file timestamp to make sure this script ran when it # # was supposed to. # # 6) Reading the log file and verifying that the previous run completed # # without errors. # # # ################################################################################ if len(sys.argv) > 1: # see if configCheck requested configCheck() print("configCheck() passed") sys.exit(0) if not os.path.isdir(INSTALL_DIR): # first make sure the directory still exists msg = "ERROR - the INSTALL_DIR directory can no longer be found!" sendEmail(msg) sys.exit(1) os.chdir(INSTALL_DIR) # make sure the program is running in the correct place if not os.path.isdir("./Backup"): try: os.mkdir("./Backup") except: msg = "ERROR - the ./Backup directory does not exist and was unable " msg += "to be created. This must be resolved to run the script." sendEmail(msg) sys.exit(1) if not os.path.isfile(WEEWX_DIR + "/weewx.sdb"): msg = "ERROR - the " + WEEWX_DIR + "/weewx.sdb file cannot be located!" sendEmail(msg) sys.exit(1) curTime = int(time.mktime(time.localtime())) sMtime = int(os.path.getmtime(WEEWX_DIR + "/weewx.sdb")) chkTime = sMtime + WEEWX_UPDATE + SLOP if chkTime < curTime: # check last database update mtimeStr = datetime.fromtimestamp(sMtime).strftime("%Y/%m/%d %H:%M") msg = "ERROR - weeWX has not updated database since " + mtimeStr sendEmail(msg) sys.exit(1) if os.path.isfile("sdbCheck.log"): # perform only if the log file exists lMtime = int(os.path.getmtime("sdbCheck.log")) chkTime = lMtime + SCRIPT_UPDATE + SLOP if curTime > chkTime: # check last time sdbCheck.py was run mtimeStr = datetime.fromtimestamp(lMtime).strftime("%Y/%m/%d %H:%M") msg = "Warning - sdbCheck.py has not been run since " + mtimeStr msg += ".\nA new run is being attempted." sendEmail(msg) with open("sdbCheck.log", "r") as lfd: # check log file for completion success data = lfd.read() lfd.close() if "successfully" not in data: # make sure the last run succeeded os.rename("sdbCheck.log", "sdbCheck.err") msg = "Error - previous run of sdbCheck.py did not complete successfully.\n" msg += "Previous sdbCheck.log has been saved as sdbCheck.err. Check it " msg += "for possible error information. If the file is empty, then the " msg += "previous run aborted prior to completion. Try running it manually " msg += "to see what's wrong. A new run is being attempted." sendEmail(msg) else: logIt("") # create an empty file for next time in case we abort early ################################################################################ # This section is the "heart" of the script. The script predicts the next # # weewx.sdb update time based on the the sMtime value and WEEWX_UPDATE. If # # the current time is 10 seconds or less of this time, it will wait until # # after the update occurs. The file is then copied to the local (INSTALL_DIR) # # directory and checked with sqlite3. If the integrity check passes, the file # # is moved into the ./Backup directory. Otherwise an email is sent indicating # # that the test failed. The very last step is to create a new sdbCheck.log # # file showing the results of the run. # ################################################################################ waitTime = (sMtime + WEEWX_UPDATE) - curTime if waitTime < 10: waitTime += 15 # wait until 15 seconds after completion before doing copy time.sleep(waitTime) copyfile(WEEWX_DIR + "/weewx.sdb", INSTALL_DIR + "/weewx.sdb") # make local copy cmd = "sqlite3 weewx.sdb 'pragma integrity_check'" rtn = os.system(cmd) if rtn: cmd += " 2> sqlite3.err" os.system(cmd) if os.path.isfile(INSTALL_DIR + "/Backup/weewx.sdb"): bkupMtime = os.path.getmtime(INSTALL_DIR + "/Backup/weewx.sdb") bkupStr = datetime.fromtimestamp(bkupMtime).strftime("%Y/%m%d %H:%M:%S") msg = "ERROR - The 'pragma integrity_check' of weewx.sdb failed and the " msg += "sqlite3 error message has been saved in the file sqlite3.err. The" msg += " last known good backup from " + bkupStr + " is in the directory " msg += INSTALL_DIR + "/Backup." sendEmail(msg) else: msg = "ERROR - The 'pragma integrity_check' of weewx.sdb failed and the " msg += "sqlite3 error message has been saved in the file sqlite3.err. " msg += "Unfortunately, there is no previous backup of the database." sendEmail(msg) lfd = open("sdbCheck.log", "w") lfd.write("The sqlite3 integrity check failed. Check sqlite3.err for ") lfd.write("further information.\n") lfd.close() else: shutil.move("weewx.sdb", "./Backup/weewx.sdb") lfd = open("sdbCheck.log", "w") lfd.write("sdbCheck.py completed successfully\n") sys.exit(0)
