Hey Gary, its looking perfect. I just separated the logs for each year and right now I am testing the 6th year in a row, doing a quick dry-run. The script is finding some duplicates, but it looks like every complete year from 01/01 to 12/31 should be imported into the database.
One more question from my side: Would you prefer a MySQL database with all this mass of data? Thank you very much! Peter Peter Süß schrieb am Montag, 1. Februar 2021 um 06:42:53 UTC+1: > Thank you Gary. > This might be the error from my side. I just copied the whole bunch of > logs into my import directory. > I will make subdir for every year, then I am going to test again. > Maybe going backward from 2020 to 2005 might be a good idea to find out my > problems. > > Version of my WeeWX is 4.4.0, running on a fresh Raspian image, installed > on Friday. > > Let me test tonight and I will report you back. > > Peter > > gjr80 schrieb am Montag, 1. Februar 2021 um 04:38:54 UTC+1: > >> Peter, >> >> I ran wee_import under WeeWX v4.4.0/python3 using the import config file >> and log files posted and the import processed without error: >> >> gary@buster31:~$ /home/weewx/bin/wee_import >> --import-config=/var/tmp/wd-import-test.conf --from=2020-12-01T00:00 >> --to=2020-12-31T23:59 --verbose >> Using WeeWX configuration file /home/weewx/weewx.conf >> >> Starting wee_import... >> Weather Display monthly log files in the '/tmp/wdisplay' directory will >> be imported >> The following options will be used: >> config=/home/weewx/weewx.conf, >> import-config=/var/tmp/wd-import-test.conf >> >> from=2020-12-01T00:00, to=2020-12-31T23:59 >> dry-run=None, calc_missing=True, ignore_invalid_data=True >> monthly logs are in METRIC units >> tranche=250, interval=1 >> UV=True, radiation=True ignore extreme temperature and humidity=True >> Using database binding 'wx_binding', which is bound to database >> 'weewx.sdb' >> Destination table 'archive' unit system is '0x01' (US). >> Missing derived observations will be calculated. >> Observations timestamped after 2020-12-01 00:00:00 AEST (1606744800) and >> up to and >> including 2020-12-31 23:59:00 AEST (1609423140) will be imported. >> >> Starting import ... >> Records covering multiple periods have been identified for import. >> Obtaining raw import data for period 1 ... >> Raw import data read successfully for period 1. >> Mapping raw import data for period 1 ... >> Mapped 44639 records. >> Raw import data mapped successfully for period 1. >> Saving mapped data to archive for period 1 ... >> 44639 records identified for import. >> Proceeding will save all imported records in the WeeWX archive. >> Are you sure you want to proceed (y/n)? y >> Unique records processed: 44639; Last timestamp: 2020-12-31 23:59:00 AEST >> (1609423140) >> Mapped data saved to archive successfully for period 1. >> Calculating missing derived observations ... >> Processing record: 44639; Last record: 2021-01-01 00:00:00 AEST >> (1609423200) >> Recalculating daily summaries... >> Records processed: 44000; time: 2020-12-31 13:20:00 AEST (1609384800) >> Finished recalculating daily summaries >> Finished calculating missing derived observations >> Finished import >> 44639 records were processed and 44639 unique records imported in 249.23 >> seconds. >> Those records with a timestamp already in the archive will not have been >> imported. Confirm successful import in the WeeWX log file. >> >> So I am confident there is nothing wrong with the import config file and >> log files posted. Granted it was a setup.py install but should not >> really make a difference to wee_import. >> >> You said you had 15 years of WD data, when you did the import were there >> other older log files in /tmp/wdisplay? The reason I ask is that whilst >> you can limit the time frame over which wee_import imports data, all >> that does is apply a filter to the final processing/saving of records. >> wee_import will still go through and read all log files it finds >> starting at the oldest. So if you had 15 years of WD data in a directory >> and tried to import the most recent month all 15 years of log files will be >> read and processed in a limited manner. The initial processing of the raw >> import data involves assembling the individual date and time components >> from each row in the log. That appears to be where your import failed. When >> I wrote the WD component of wee_import I found a number of data >> inconsistencies in the WD log files I was using so I guess it is possible >> you may have similar inconsistencies. Have a look through the oldest WD log >> file set in /tmp/wdisplay and see if you can spot an obvious >> error/inconsistency in the date and time data. If you can't spot anything >> could you post the respective log files here. Alternatively, if you let me >> know what version of WeeWX you are running I can provide you with an >> instrumented version of wee_import to log details of the offending >> record/log. >> >> Gary >> On Monday, 1 February 2021 at 00:59:35 UTC+10 peter.su...@gmail.com >> wrote: >> >>> Hi Gary, >>> >>> thanks for your quick answer. >>> Please check the attached file. All I edited was the path to data and >>> filenames. I limited it to only one type of file, with my last change. >>> >>> No, problem when it takes time. >>> Thanks for checking out >>> >>> Peter >>> gjr80 schrieb am Sonntag, 31. Januar 2021 um 13:58:21 UTC+1: >>> >>>> Hi, >>>> >>>> Nothing leaps out at me having a quick look at the output and log files >>>> you posted. I'll endeavour to have a look at this tomorrow (bit late >>>> here). >>>> Could I ask you to post the import config file you are using, >>>> /etc/weewx/wd-import-test.conf. There should be nothing sensitive in >>>> the file. >>>> >>>> Gary >>>> >>>> On Sunday, 31 January 2021 at 22:28:32 UTC+10 peter.su...@gmail.com >>>> wrote: >>>> >>>>> Hello, >>>>> after using my Vantage pro 2 with Weather Display for about 15 years, >>>>> I think now its time for a change. :) >>>>> I am making the first tests for transferring data, before switch the >>>>> live system. >>>>> >>>>> My problem, I cannot import the log data from wdisplay. >>>>> I'm getting this error message: >>>>> >>>>> pi@WEEWX-DEMO:/etc/weewx $ wee_import >>>>> --import-config=wd-import-test.conf --from=2020-12-01T00:00 >>>>> --to=2020-12-31T23:59 --verbose >>>>> Using WeeWX configuration file /etc/weewx/weewx.conf >>>>> Starting wee_import... >>>>> Weather Display monthly log files in the '/tmp/wdisplay' directory >>>>> will be imported >>>>> The following options will be used: >>>>> config=/etc/weewx/weewx.conf, >>>>> import-config=/etc/weewx/wd-import-test.conf >>>>> from=2020-12-01T00:00, to=2020-12-31T23:59 >>>>> dry-run=None, calc_missing=True, ignore_invalid_data=True >>>>> monthly logs are in METRIC units >>>>> tranche=250, interval=1 >>>>> UV=True, radiation=True ignore extreme temperature and >>>>> humidity=True >>>>> Using database binding 'wx_binding', which is bound to database >>>>> 'weewx.sdb' >>>>> Destination table 'archive' unit system is '0x01' (US). >>>>> Missing derived observations will be calculated. >>>>> Observations timestamped after 2020-12-01 00:00:00 CET (1606777200) >>>>> and up to and >>>>> including 2020-12-31 23:59:00 CET (1609455540) will be imported. >>>>> Starting import ... >>>>> Records covering multiple periods have been identified for import. >>>>> Period 1 ... >>>>> Obtaining raw import data for period 1 ... >>>>> Traceback (most recent call last): >>>>> File "/usr/share/weewx/wee_import", line 903, in <module> >>>>> main() >>>>> File "/usr/share/weewx/wee_import", line 833, in main >>>>> source_obj.run() >>>>> File "/usr/share/weewx/weeimport/weeimport.py", line 369, in run >>>>> _raw_data = self.getRawData(period) >>>>> File "/usr/share/weewx/weeimport/wdimport.py", line 598, in >>>>> getRawData >>>>> _ts = "%s" % int(time.mktime(_tt)) >>>>> OverflowError: mktime argument out of range >>>>> >>>>> >>>>> Any ideas? File content is looking ok. >>>>> It is happening with all time tests across this 15 years. >>>>> Here a month sample data. >>>>> >>>>> Thanks for any ideas... >>>>> Peter >>>>> >>>>> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/c5f91ae0-d5c1-4a99-8004-32e515a3908fn%40googlegroups.com.
