Wow. I just mislead you. My fault. What you need is a copy of V4.0.0, but with a patch that allows a secure authorization connection to be used, but with insecure data. Try the attached.
My apologies! -tk On Sat, May 30, 2020 at 5:29 AM Tom Keffer <[email protected]> wrote: > Hello, > > 1. You don't want the latest. You are still on V4.0.0, so you want to > revert to that version of ftpupload.py. You can find a copy here: > https://github.com/weewx/weewx/blob/v4.0.0/bin/weeutil/ftpupload.py > 2. Once you get that sorted, try again with secure_data set to false. > 3. The case of true/false does not matter. > > -tk > > On Sat, May 30, 2020 at 3:26 AM Richard Cropper < > [email protected]> wrote: > >> Sorry >> >> This looks like the same error as with your first replacement: >> >> May 30 11:20:54 raspberrypi weewx[21515] ERROR weewx.reportengine: Caught >> unrecoverable exception in generator 'weewx.reportengine.FtpGenerator' >> >> May 30 11:20:54 raspberrypi weewx[21515] ERROR weewx.reportengine: >> **** __init__() got an unexpected keyword argument 'max_tries' >> >> May 30 11:20:54 raspberrypi weewx[21515] ERROR weewx.reportengine: >> **** Traceback (most recent call last): >> >> May 30 11:20:54 raspberrypi weewx[21515] ERROR weewx.reportengine: >> **** File "/usr/share/weewx/weewx/reportengine.py", line 197, in run >> >> May 30 11:20:54 raspberrypi weewx[21515] ERROR weewx.reportengine: >> **** obj.start() >> >> May 30 11:20:54 raspberrypi weewx[21515] ERROR weewx.reportengine: >> **** File "/usr/share/weewx/weewx/reportengine.py", line 280, in start >> >> May 30 11:20:54 raspberrypi weewx[21515] ERROR weewx.reportengine: >> **** self.run() >> >> May 30 11:20:54 raspberrypi weewx[21515] ERROR weewx.reportengine: >> **** File "/usr/share/weewx/weewx/reportengine.py", line 320, in run >> >> May 30 11:20:54 raspberrypi weewx[21515] ERROR weewx.reportengine: >> **** secure_data=to_bool(self.skin_dict.get('secure_data', True))) >> >> May 30 11:20:54 raspberrypi weewx[21515] ERROR weewx.reportengine: >> **** TypeError: __init__() got an unexpected keyword argument 'max_tries' >> >> May 30 11:20:54 raspberrypi weewx[21515] ERROR weewx.reportengine: >> **** Generator terminated >> >> >> I've double checked to make sure I've used your latest. >> >> >> I've tried with secure_data set to both true and false. >> >> >> Incidentally, are capital letters critical in the true/false settings? >> >> >> Regards >> >> >> Richard >> >> >> >> On Saturday, May 30, 2020 at 1:33:20 AM UTC+1, Tom Keffer wrote: >>> >>> Well then, let's try setting the PROT setting, even though the data will >>> not be secure. Try this version. >>> >>> On Fri, May 29, 2020 at 6:15 AM Richard Cropper <[email protected]> >>> wrote: >>> >>>> I'm on v 4.0.0 - it can't have been more than a week since I installed >>>> it >>>> >>>> Yes I did try setting secure_data to false. With the previous version >>>> of ftpupload.py I then get >>>> >>>> May 29 14:10:56 raspberrypi weewx[4780] ERROR weeutil.ftpupload: >>>> Attempt #1. Failed uploading /public_html/weewx/weekwind.png to >>>> polaris.servers.prgn.misp.co.uk. Reason: 521 Data connection cannot be >>>> opened with this PROT setting. >>>> >>>> May 29 14:10:57 raspberrypi weewx[4780] ERROR weeutil.ftpupload: >>>> Attempt #2. Failed uploading /public_html/weewx/weekwind.png to >>>> polaris.servers.prgn.misp.co.uk. Reason: 521 Data connection cannot be >>>> opened with this PROT setting. >>>> >>>> May 29 14:10:57 raspberrypi weewx[4780] ERROR weeutil.ftpupload: >>>> Attempt #3. Failed uploading /public_html/weewx/weekwind.png to >>>> polaris.servers.prgn.misp.co.uk. Reason: 521 Data connection cannot be >>>> opened with this PROT setting. >>>> >>>> May 29 14:10:57 raspberrypi weewx[4780] ERROR weeutil.ftpupload: Failed >>>> to upload file /public_html/weewx/weekwind.png >>>> >>>> >>>> Regards >>>> >>>> Richard >>>> On Friday, May 29, 2020 at 12:40:44 PM UTC+1, Tom Keffer wrote: >>>>> >>>>> The file I posted was to fix a problem introduced in V4.1, which I had >>>>> assumed you were using. Apparently not. Disregard the file. Sorry for any >>>>> confusion. >>>>> >>>>> Did you try setting secure_data to false? >>>>> >>>>> -tk >>>>> >>>>> On Fri, May 29, 2020 at 4:22 AM Richard Cropper <[email protected]> >>>>> wrote: >>>>> >>>>>> Incidentally, I've been nowhere near reportengine.py >>>>>> >>>>>> On Friday, May 29, 2020 at 12:19:38 PM UTC+1, Richard Cropper wrote: >>>>>>> >>>>>>> Hi Tom >>>>>>> >>>>>>> This what I get with the new ftpupload.py >>>>>>> >>>>>>> May 29 12:01:40 raspberrypi weewx[2431] INFO weewx.reportengine: >>>>>>> Copied 5 files to /var/www/html/weewx >>>>>>> >>>>>>> May 29 12:01:40 raspberrypi weewx[2431] ERROR weewx.reportengine: >>>>>>> Caught unrecoverable exception in generator >>>>>>> 'weewx.reportengine.FtpGenerator' >>>>>>> >>>>>>> May 29 12:01:40 raspberrypi weewx[2431] ERROR weewx.reportengine: >>>>>>> **** __init__() got an unexpected keyword argument 'max_tries' >>>>>>> >>>>>>> May 29 12:01:40 raspberrypi weewx[2431] ERROR weewx.reportengine: >>>>>>> **** Traceback (most recent call last): >>>>>>> >>>>>>> May 29 12:01:40 raspberrypi weewx[2431] ERROR weewx.reportengine: >>>>>>> **** File "/usr/share/weewx/weewx/reportengine.py", line 197, >>>>>>> in >>>>>>> run >>>>>>> >>>>>>> May 29 12:01:40 raspberrypi weewx[2431] ERROR weewx.reportengine: >>>>>>> **** obj.start() >>>>>>> >>>>>>> May 29 12:01:40 raspberrypi weewx[2431] ERROR weewx.reportengine: >>>>>>> **** File "/usr/share/weewx/weewx/reportengine.py", line 280, >>>>>>> in >>>>>>> start >>>>>>> >>>>>>> May 29 12:01:40 raspberrypi weewx[2431] ERROR weewx.reportengine: >>>>>>> **** self.run() >>>>>>> >>>>>>> May 29 12:01:40 raspberrypi weewx[2431] ERROR weewx.reportengine: >>>>>>> **** File "/usr/share/weewx/weewx/reportengine.py", line 320, >>>>>>> in >>>>>>> run >>>>>>> >>>>>>> May 29 12:01:40 raspberrypi weewx[2431] ERROR weewx.reportengine: >>>>>>> **** secure_data=to_bool(self.skin_dict.get('secure_data', >>>>>>> True))) >>>>>>> >>>>>>> May 29 12:01:40 raspberrypi weewx[2431] ERROR weewx.reportengine: >>>>>>> **** TypeError: __init__() got an unexpected keyword argument >>>>>>> 'max_tries' >>>>>>> >>>>>>> May 29 12:01:40 raspberrypi weewx[2431] ERROR weewx.reportengine: >>>>>>> **** Generator terminated >>>>>>> >>>>>>> I get the same message whether secure_data is set to True or False. >>>>>>> >>>>>>> Regards >>>>>>> >>>>>>> Richard >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thursday, May 28, 2020 at 9:28:49 PM UTC+1, Tom Keffer wrote: >>>>>>>> >>>>>>>> I just noticed that in refactoring the ftp uploader, the function >>>>>>>> _make_remote_dir() never gets called! Try this version of >>>>>>>> weeutil/ftpupload.py. >>>>>>>> >>>>>>>> -tk >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, May 28, 2020 at 12:50 PM Tom Keffer <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hard to say, but I think you're getting hit by issue #284 >>>>>>>>> <https://github.com/weewx/weewx/issues/284>. Try adding an option >>>>>>>>> secure_data and setting it to false: >>>>>>>>> >>>>>>>>> [[FTP]] >>>>>>>>> ... >>>>>>>>> secure_ftp = True >>>>>>>>> secure_data = False >>>>>>>>> >>>>>>>>> -tk >>>>>>>>> >>>>>>>>> On Thu, May 28, 2020 at 9:36 AM Richard Cropper < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Thanks Tom >>>>>>>>>> >>>>>>>>>> I had in fact tried those steps but did so again, without success. >>>>>>>>>> >>>>>>>>>> Suspecting that the directory might be the cause even though its >>>>>>>>>> permissions looked good, I deleted the weewx directory and then >>>>>>>>>> created it >>>>>>>>>> again. >>>>>>>>>> >>>>>>>>>> This had made a difference but not solved the problem. >>>>>>>>>> >>>>>>>>>> Here's an extract from the log: >>>>>>>>>> >>>>>>>>>> May 28 17:24:35 raspberrypi weewx[723] ERROR weeutil.ftpupload: >>>>>>>>>> Attempt #1. Failed uploading /public_html/weewx/daytempfeel.png to >>>>>>>>>> polaris.servers.prgn.misp.co.uk. Reason: [Errno 32] Broken pipe >>>>>>>>>> >>>>>>>>>> May 28 17:24:35 raspberrypi weewx[723] ERROR weeutil.ftpupload: >>>>>>>>>> Attempt #2. Failed uploading /public_html/weewx/daytempfeel.png to >>>>>>>>>> polaris.servers.prgn.misp.co.uk. Reason: [Errno 32] Broken pipe >>>>>>>>>> >>>>>>>>>> May 28 17:24:35 raspberrypi weewx[723] ERROR weeutil.ftpupload: >>>>>>>>>> Attempt #3. Failed uploading /public_html/weewx/daytempfeel.png to >>>>>>>>>> polaris.servers.prgn.misp.co.uk. Reason: [Errno 32] Broken pipe >>>>>>>>>> >>>>>>>>>> May 28 17:24:35 raspberrypi weewx[723] ERROR weeutil.ftpupload: >>>>>>>>>> Failed to upload file /public_html/weewx/daytempfeel.png >>>>>>>>>> >>>>>>>>>> I have tried all possible file paths again, as you suggest but I >>>>>>>>>> always get this broken pipe message. If I extend the file path any >>>>>>>>>> higher, >>>>>>>>>> I get the 'cannot open this file' message. >>>>>>>>>> >>>>>>>>>> I have been able to upload the index file from >>>>>>>>>> /var/www/html/weewx to the weewx directory on my server using curl. >>>>>>>>>> >>>>>>>>>> Richard >>>>>>>>>> >>>>>>>>>> On Wednesday, May 27, 2020 at 7:56:28 PM UTC+1, Tom Keffer wrote: >>>>>>>>>>> >>>>>>>>>>> Two things to try: >>>>>>>>>>> >>>>>>>>>>> 1. Make sure the FTP username has write permissions on the >>>>>>>>>>> server. >>>>>>>>>>> >>>>>>>>>>> 2. Try different variations on the FTP path. For example, >>>>>>>>>>> >>>>>>>>>>> path = public_html/weewx/ # What you have >>>>>>>>>>> path = public_html/weewx >>>>>>>>>>> path = /public_html/weewx/ >>>>>>>>>>> path = /public_html/weewx >>>>>>>>>>> path = weewx >>>>>>>>>>> path = /weewx >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, May 27, 2020 at 10:02 AM Richard Cropper < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Thanks Tom >>>>>>>>>>>> >>>>>>>>>>>> Yes all the files are in /var/www/html/weewx >>>>>>>>>>>> >>>>>>>>>>>> If I understand you correctly, this has more to do with the >>>>>>>>>>>> file path of the remote host server. >>>>>>>>>>>> >>>>>>>>>>>> I am at a loss to understand what is going on at that end. >>>>>>>>>>>> >>>>>>>>>>>> I can at least connect but I either get a message as you have >>>>>>>>>>>> seen or a 'broken pipe' message if I play around with the file >>>>>>>>>>>> path. >>>>>>>>>>>> >>>>>>>>>>>> I guess this is more to do with the remote server than weewx >>>>>>>>>>>> but thanks for your advice. >>>>>>>>>>>> >>>>>>>>>>>> Regards >>>>>>>>>>>> >>>>>>>>>>>> Richard >>>>>>>>>>>> On Wednesday, May 27, 2020 at 3:17:35 PM UTC+1, Tom Keffer >>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hello, Richard >>>>>>>>>>>>> >>>>>>>>>>>>> Unfortunately, the error message is misleading (and should be >>>>>>>>>>>>> changed). When it says >>>>>>>>>>>>> >>>>>>>>>>>>> Failed uploading public_html/weewx/monthtempfeel.png to >>>>>>>>>>>>> ftp.tq7weather.uk. Reason: 553 Can't open that file: No such >>>>>>>>>>>>> file or directory >>>>>>>>>>>>> >>>>>>>>>>>>> what it means is it could not upload to the *target* >>>>>>>>>>>>> public_html/weewx/monthtempfeel.png*.* The unfortunate part >>>>>>>>>>>>> is that the error message doesn't give the source. >>>>>>>>>>>>> >>>>>>>>>>>>> Take a look in /var/www/html/weewx and make sure the files you >>>>>>>>>>>>> expect to be in there are actually in there. >>>>>>>>>>>>> >>>>>>>>>>>>> -tk >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, May 27, 2020 at 5:50 AM Richard Cropper < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi >>>>>>>>>>>>>> Having secured a connection, I still don't seem to be able >>>>>>>>>>>>>> to upload files >>>>>>>>>>>>>> >>>>>>>>>>>>>> I get the following error message on the log >>>>>>>>>>>>>> >>>>>>>>>>>>>> May 27 13:30:36 raspberrypi weewx[899] ERROR >>>>>>>>>>>>>> weeutil.ftpupload: Attempt #1. Failed uploading >>>>>>>>>>>>>> public_html/weewx/monthtempfeel.png to ftp.tq7weather.uk. >>>>>>>>>>>>>> Reason: 553 Can't open that file: No such file or directory >>>>>>>>>>>>>> >>>>>>>>>>>>>> May 27 13:30:36 raspberrypi weewx[899] ERROR >>>>>>>>>>>>>> weeutil.ftpupload: Attempt #2. Failed uploading >>>>>>>>>>>>>> public_html/weewx/monthtempfeel.png to ftp.tq7weather.uk. >>>>>>>>>>>>>> Reason: 553 Can't open that file: No such file or directory >>>>>>>>>>>>>> >>>>>>>>>>>>>> May 27 13:30:37 raspberrypi weewx[899] ERROR >>>>>>>>>>>>>> weeutil.ftpupload: Attempt #3. Failed uploading >>>>>>>>>>>>>> public_html/weewx/monthtempfeel.png to ftp.tq7weather.uk. >>>>>>>>>>>>>> Reason: 553 Can't open that file: No such file or directory >>>>>>>>>>>>>> >>>>>>>>>>>>>> May 27 13:30:37 raspberrypi weewx[899] ERROR >>>>>>>>>>>>>> weeutil.ftpupload: Failed to upload file >>>>>>>>>>>>>> public_html/weewx/monthtempfeel.png >>>>>>>>>>>>>> >>>>>>>>>>>>>> May 27 13:30:37 raspberrypi weewx[899] ERROR >>>>>>>>>>>>>> weeutil.ftpupload: Attempt #1. Failed uploading >>>>>>>>>>>>>> public_html/weewx/yearwind.png to ftp.tq7weather.uk. Reason: >>>>>>>>>>>>>> 553 Can't open that file: No such file or directory >>>>>>>>>>>>>> >>>>>>>>>>>>>> May 27 13:30:38 raspberrypi weewx[899] ERROR >>>>>>>>>>>>>> weeutil.ftpupload: Attempt #2. Failed uploading >>>>>>>>>>>>>> public_html/weewx/yearwind.png to ftp.tq7weather.uk. Reason: >>>>>>>>>>>>>> 553 Can't open that file: No such file or directory >>>>>>>>>>>>>> >>>>>>>>>>>>>> May 27 13:30:38 raspberrypi weewx[899] ERROR >>>>>>>>>>>>>> weeutil.ftpupload: Attempt #3. Failed uploading >>>>>>>>>>>>>> public_html/weewx/yearwind.png to ftp.tq7weather.uk. Reason: >>>>>>>>>>>>>> 553 Can't open that file: No such file or directory >>>>>>>>>>>>>> >>>>>>>>>>>>>> May 27 13:30:38 raspberrypi weewx[899] ERROR >>>>>>>>>>>>>> weeutil.ftpupload: Failed to upload file >>>>>>>>>>>>>> public_html/weewx/yearwind.png >>>>>>>>>>>>>> >>>>>>>>>>>>>> May 27 13:30:39 raspberrypi weewx[899] ERROR >>>>>>>>>>>>>> weeutil.ftpupload: Attempt #1. Failed uploading >>>>>>>>>>>>>> public_html/weewx/statistics.html to ftp.tq7weather.uk. >>>>>>>>>>>>>> Reason: 553 Can't open that file: No such file or directory >>>>>>>>>>>>>> >>>>>>>>>>>>>> May 27 13:30:39 raspberrypi weewx[899] ERROR >>>>>>>>>>>>>> weeutil.ftpupload: Attempt #2. Failed uploading >>>>>>>>>>>>>> public_html/weewx/statistics.html to ftp.tq7weather.uk. >>>>>>>>>>>>>> Reason: 553 Can't open that file: No such file or directory >>>>>>>>>>>>>> >>>>>>>>>>>>>> May 27 13:30:40 raspberrypi weewx[899] ERROR >>>>>>>>>>>>>> weeutil.ftpupload: Attempt #3. Failed uploading >>>>>>>>>>>>>> public_html/weewx/statistics.html to ftp.tq7weather.uk. >>>>>>>>>>>>>> Reason: 553 Can't open that file: No such file or directory >>>>>>>>>>>>>> >>>>>>>>>>>>>> May 27 13:30:40 raspberrypi weewx[899] ERROR >>>>>>>>>>>>>> weeutil.ftpupload: Failed to upload file >>>>>>>>>>>>>> public_html/weewx/statistics.html >>>>>>>>>>>>>> >>>>>>>>>>>>>> May 27 13:30:41 raspberrypi weewx[899] ERROR >>>>>>>>>>>>>> weeutil.ftpupload: Attempt #1. Failed uploading >>>>>>>>>>>>>> public_html/weewx/weekbarometer.png to ftp.tq7weather.uk. >>>>>>>>>>>>>> Reason: 553 Can't open that file: No such file or directory >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Here's the relevant bit of my weewx.conf: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [[FTP]] >>>>>>>>>>>>>> >>>>>>>>>>>>>> # FTP'ing the results to a webserver is treated as >>>>>>>>>>>>>> just another report, >>>>>>>>>>>>>> >>>>>>>>>>>>>> # albeit one with an unusual report generator! >>>>>>>>>>>>>> >>>>>>>>>>>>>> skin = Ftp >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> # If you wish to use FTP, set "enable" to "true", then >>>>>>>>>>>>>> >>>>>>>>>>>>>> # fill out the next four lines. >>>>>>>>>>>>>> >>>>>>>>>>>>>> # Use quotes around passwords to guard against >>>>>>>>>>>>>> parsing errors. >>>>>>>>>>>>>> >>>>>>>>>>>>>> enable = true >>>>>>>>>>>>>> >>>>>>>>>>>>>> user =[user name] >>>>>>>>>>>>>> >>>>>>>>>>>>>> password = [password] >>>>>>>>>>>>>> >>>>>>>>>>>>>> server = [server name] # The ftp server name, >>>>>>>>>>>>>> e.g, www.myserver.org >>>>>>>>>>>>>> >>>>>>>>>>>>>> path = public_html/weewx/ # The destination >>>>>>>>>>>>>> directory, e.g., /weather >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> # Set to True for an FTP over TLS (FTPS) connection. >>>>>>>>>>>>>> Not all servers >>>>>>>>>>>>>> >>>>>>>>>>>>>> # support this. >>>>>>>>>>>>>> >>>>>>>>>>>>>> secure_ftp = True >>>>>>>>>>>>>> >>>>>>>>>>>>>> secure_data = True >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> # To upload files from something other than what >>>>>>>>>>>>>> HTML_ROOT is set >>>>>>>>>>>>>> >>>>>>>>>>>>>> # to above, specify a different HTML_ROOT here. >>>>>>>>>>>>>> >>>>>>>>>>>>>> HTML_ROOT = /var/www/html/weewx/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> # Most FTP servers use port 21 >>>>>>>>>>>>>> >>>>>>>>>>>>>> port = 21 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> # Set to 1 to use passive mode, zero for active mode >>>>>>>>>>>>>> >>>>>>>>>>>>>> passive = 1 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> The connection is FTP over TLS. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> My weather data is from an Aercus Weather Sleuth using >>>>>>>>>>>>>> interceptor on a Raspberry pi. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> It's as if FTP is looking for the wrong path on my pi but >>>>>>>>>>>>>> maybe I misunderstand the error message. That's why I tried >>>>>>>>>>>>>> removing the >>>>>>>>>>>>>> hash before HTML_ROOT but I got the same result with the hash. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Otherwise very pleased with weewx which I can view on my pi >>>>>>>>>>>>>> using nginx >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Any help gratefully received. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Richard >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>>>> Google Groups "weewx-user" group. >>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>>>> https://groups.google.com/d/msgid/weewx-user/35d77ba3-e9a1-4919-9b2e-bb0680490b8a%40googlegroups.com >>>>>>>>>>>>>> <https://groups.google.com/d/msgid/weewx-user/35d77ba3-e9a1-4919-9b2e-bb0680490b8a%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>>>> . >>>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>> Google Groups "weewx-user" group. >>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>>> https://groups.google.com/d/msgid/weewx-user/65f68c3f-31e6-4e6f-95aa-0268a0a82f78%40googlegroups.com >>>>>>>>>>>> <https://groups.google.com/d/msgid/weewx-user/65f68c3f-31e6-4e6f-95aa-0268a0a82f78%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>>>> . >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "weewx-user" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to [email protected]. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/weewx-user/cc51fa47-91a9-4a06-838b-18a3ed7b3c2f%40googlegroups.com >>>>>>>>>> <https://groups.google.com/d/msgid/weewx-user/cc51fa47-91a9-4a06-838b-18a3ed7b3c2f%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>> . >>>>>>>>>> >>>>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "weewx-user" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/weewx-user/c53bbd3e-00ee-4291-872d-6de13557e08c%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/weewx-user/c53bbd3e-00ee-4291-872d-6de13557e08c%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "weewx-user" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/weewx-user/67c65342-a58e-440b-a92a-8ed9dd6b3395%40googlegroups.com >>>> <https://groups.google.com/d/msgid/weewx-user/67c65342-a58e-440b-a92a-8ed9dd6b3395%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "weewx-user" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/weewx-user/6dd53e7f-7f1a-434c-8d44-66b2ff6619e4%40googlegroups.com >> <https://groups.google.com/d/msgid/weewx-user/6dd53e7f-7f1a-434c-8d44-66b2ff6619e4%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEDYgP_zepNSQ9SZCYtwmzT8u-h0Xs7N5gugx6UCH%3DZyOQ%40mail.gmail.com.
# # Copyright (c) 2009-2015 Tom Keffer <[email protected]> # # See the file LICENSE.txt for your full rights. # """For uploading files to a remove server via FTP""" from __future__ import absolute_import from __future__ import print_function from __future__ import with_statement import ftplib import logging import os import sys import time from six.moves import cPickle log = logging.getLogger(__name__) class FtpUpload(object): """Uploads a directory and all its descendants to a remote server. Keeps track of when a file was last uploaded, so it is uploaded only if its modification time is newer.""" def __init__(self, server, user, password, local_root, remote_root, port=21, name="FTP", passive=True, max_tries=3, secure=False, debug=0, secure_data=True): """Initialize an instance of FtpUpload. After initializing, call method run() to perform the upload. server: The remote server to which the files are to be uploaded. user, password : The user name and password that are to be used. name: A unique name to be given for this FTP session. This allows more than one session to be uploading from the same local directory. [Optional. Default is 'FTP'.] passive: True to use passive mode; False to use active mode. [Optional. Default is True (passive mode)] max_tries: How many times to try creating a directory or uploading a file before giving up [Optional. Default is 3] secure: Set to True to attempt an FTP over TLS (FTPS) session. debug: Set to 1 for extra debug information, 0 otherwise. secure_data: If a secure session is requested (option secure=True), should we attempt a secure data connection as well? This option is useful due to a bug in the Python FTP client library. See Issue #284. [Optional. Default is True] """ self.server = server self.user = user self.password = password self.local_root = os.path.normpath(local_root) self.remote_root = os.path.normpath(remote_root) self.port = port self.name = name self.passive = passive self.max_tries = max_tries self.secure = secure self.debug = debug self.secure_data = secure_data def run(self): """Perform the actual upload. returns: the number of files uploaded.""" if self.secure: try: FTPClass = ftplib.FTP_TLS except AttributeError: FTPClass = ftplib.FTP log.debug("Your version of Python does not support FTPS. Using insecure connection.") self.secure = False else: FTPClass = ftplib.FTP # Get the timestamp and members of the last upload: (timestamp, fileset) = self.getLastUpload() n_uploaded = 0 # Try to connect to the ftp server up to max_tries times: try: if self.secure: log.debug("Attempting secure connection to %s" % self.server) else: log.debug("Attempting connection to %s" % self.server) for count in range(self.max_tries): try: ftp_server = FTPClass() ftp_server.connect(self.server, self.port) if self.debug: ftp_server.set_debuglevel(self.debug) ftp_server.login(self.user, self.password) ftp_server.set_pasv(self.passive) if self.secure: ftp_server.prot_p() log.debug("Secure data connection to %s" % self.server) else: log.debug("Connected to %s" % self.server) break except ftplib.all_errors as e: log.error("Unable to connect or log into server : %s" % e) else: # This is executed only if the loop terminates naturally (without a break statement), # meaning the ftp connection failed max_tries times. Abandon ftp upload log.error("Attempted %d times to connect to server %s. Giving up." % (self.max_tries, self.server)) return n_uploaded # Walk the local directory structure for (dirpath, unused_dirnames, filenames) in os.walk(self.local_root): # Strip out the common local root directory. What is left # will be the relative directory both locally and remotely. local_rel_dir_path = dirpath.replace(self.local_root, '.') if self._skipThisDir(local_rel_dir_path): continue # This is the absolute path to the remote directory: remote_dir_path = os.path.normpath(os.path.join(self.remote_root, local_rel_dir_path)) # Make the remote directory if necessary: self._make_remote_dir(ftp_server, remote_dir_path) # Now iterate over all members of the local directory: for filename in filenames: full_local_path = os.path.join(dirpath, filename) # See if this file can be skipped: if self._skipThisFile(timestamp, fileset, full_local_path): continue full_remote_path = os.path.join(remote_dir_path, filename) STOR_cmd = "STOR %s" % full_remote_path # Retry up to max_tries times: for count in range(self.max_tries): # If we have to retry, we should probably reopen the file as well. # Hence, the open is in the inner loop: with open(full_local_path, 'rb') as fd: try: ftp_server.storbinary(STOR_cmd, fd) except ftplib.all_errors as e: # Unsuccessful. Log it and go around again. log.error("Attempt #%d. Failed uploading %s to %s. Reason: %s" % (count + 1, full_remote_path, self.server, e)) ftp_server.set_pasv(self.passive) else: # Success. Log it, break out of the loop n_uploaded += 1 fileset.add(full_local_path) log.debug("Uploaded file %s" % full_remote_path) break else: # This is executed only if the loop terminates naturally (without a break statement), # meaning the upload failed max_tries times. Log it, move on to the next file. log.error("Failed to upload file %s" % full_remote_path) finally: try: ftp_server.quit() except: pass timestamp = time.time() self.saveLastUpload(timestamp, fileset) return n_uploaded def getLastUpload(self): """Reads the time and members of the last upload from the local root""" timeStampFile = os.path.join(self.local_root, "#%s.last" % self.name) # If the file does not exist, an IOError exception will be raised. # If the file exists, but is truncated, an EOFError will be raised. # Either way, be prepared to catch it. try: with open(timeStampFile, "rb") as f: timestamp = cPickle.load(f) fileset = cPickle.load(f) except (IOError, EOFError, cPickle.PickleError, AttributeError): timestamp = 0 fileset = set() # Either the file does not exist, or it is garbled. # Either way, it's safe to remove it. try: os.remove(timeStampFile) except OSError: pass return (timestamp, fileset) def saveLastUpload(self, timestamp, fileset): """Saves the time and members of the last upload in the local root.""" timeStampFile = os.path.join(self.local_root, "#%s.last" % self.name) with open(timeStampFile, "wb") as f: cPickle.dump(timestamp, f) cPickle.dump(fileset, f) def _make_remote_dir(self, ftp_server, remote_dir_path): """Make a remote directory if necessary.""" # Try to make the remote directory up max_tries times, then give up. for unused_count in range(self.max_tries): try: ftp_server.mkd(remote_dir_path) except ftplib.all_errors as e: # Got an exception. It might be because the remote directory already exists: if sys.exc_info()[0] is ftplib.error_perm: msg = str(e).strip() # If a directory already exists, some servers respond with a '550' ("Requested action not taken") code, # others with a '521' ("Access denied" or "Pathname already exists") code. if msg.startswith('550') or msg.startswith('521'): # Directory already exists return log.error("Got error while attempting to make remote directory %s" % remote_dir_path) log.error(" **** Error: %s" % e) else: log.debug("Made directory %s" % remote_dir_path) return else: log.error("Unable to create remote directory %s" % remote_dir_path) raise IOError("Unable to create remote directory %s" % remote_dir_path) def _skipThisDir(self, local_dir): return os.path.basename(local_dir) in ('.svn', 'CVS') def _skipThisFile(self, timestamp, fileset, full_local_path): filename = os.path.basename(full_local_path) if filename[-1] == '~' or filename[0] == '#': return True if full_local_path not in fileset: return False if os.stat(full_local_path).st_mtime > timestamp: return False # Filename is in the set, and is up to date. return True if __name__ == '__main__': import socket import configobj import weewx import weeutil.logger weewx.debug = 1 weeutil.logger.setup('ftpupload', {}) if len(sys.argv) < 2: print("""Usage: ftpupload.py path-to-configuration-file [path-to-be-ftp'd]""") sys.exit(weewx.CMD_ERROR) try: config_dict = configobj.ConfigObj(sys.argv[1], file_error=True, encoding='utf-8') except IOError: print("Unable to open configuration file %s" % sys.argv[1]) raise if len(sys.argv) == 2: try: ftp_dir = os.path.join(config_dict['WEEWX_ROOT'], config_dict['StdReport']['HTML_ROOT']) except KeyError: print("No HTML_ROOT in configuration dictionary.") sys.exit(1) else: ftp_dir = sys.argv[2] socket.setdefaulttimeout(10) ftp_upload = FtpUpload(config_dict['StdReport']['FTP']['server'], config_dict['StdReport']['FTP']['user'], config_dict['StdReport']['FTP']['password'], ftp_dir, config_dict['StdReport']['FTP']['path'], 'FTP', config_dict['StdReport']['FTP'].as_bool('passive'), config_dict['StdReport']['FTP'].as_int('max_tries')) ftp_upload.run()
