Hi Gary,
thanks for your quick answer.
Please check the attached file. All I edited was the path to data and
filenames. I limited it to only one type of file, with my last change.
No, problem when it takes time.
Thanks for checking out
Peter
gjr80 schrieb am Sonntag, 31. Januar 2021 um 13:58:21 UTC+1:
> Hi,
>
> Nothing leaps out at me having a quick look at the output and log files
> you posted. I'll endeavour to have a look at this tomorrow (bit late here).
> Could I ask you to post the import config file you are using,
> /etc/weewx/wd-import-test.conf. There should be nothing sensitive in the
> file.
>
> Gary
>
> On Sunday, 31 January 2021 at 22:28:32 UTC+10 peter.su...@gmail.com wrote:
>
>> Hello,
>> after using my Vantage pro 2 with Weather Display for about 15 years, I
>> think now its time for a change. :)
>> I am making the first tests for transferring data, before switch the live
>> system.
>>
>> My problem, I cannot import the log data from wdisplay.
>> I'm getting this error message:
>>
>> pi@WEEWX-DEMO:/etc/weewx $ wee_import --import-config=wd-import-test.conf
>> --from=2020-12-01T00:00 --to=2020-12-31T23:59 --verbose
>> Using WeeWX configuration file /etc/weewx/weewx.conf
>> Starting wee_import...
>> Weather Display monthly log files in the '/tmp/wdisplay' directory will
>> be imported
>> The following options will be used:
>> config=/etc/weewx/weewx.conf,
>> import-config=/etc/weewx/wd-import-test.conf
>> from=2020-12-01T00:00, to=2020-12-31T23:59
>> dry-run=None, calc_missing=True, ignore_invalid_data=True
>> monthly logs are in METRIC units
>> tranche=250, interval=1
>> UV=True, radiation=True ignore extreme temperature and humidity=True
>> Using database binding 'wx_binding', which is bound to database
>> 'weewx.sdb'
>> Destination table 'archive' unit system is '0x01' (US).
>> Missing derived observations will be calculated.
>> Observations timestamped after 2020-12-01 00:00:00 CET (1606777200) and
>> up to and
>> including 2020-12-31 23:59:00 CET (1609455540) will be imported.
>> Starting import ...
>> Records covering multiple periods have been identified for import.
>> Period 1 ...
>> Obtaining raw import data for period 1 ...
>> Traceback (most recent call last):
>> File "/usr/share/weewx/wee_import", line 903, in <module>
>> main()
>> File "/usr/share/weewx/wee_import", line 833, in main
>> source_obj.run()
>> File "/usr/share/weewx/weeimport/weeimport.py", line 369, in run
>> _raw_data = self.getRawData(period)
>> File "/usr/share/weewx/weeimport/wdimport.py", line 598, in getRawData
>> _ts = "%s" % int(time.mktime(_tt))
>> OverflowError: mktime argument out of range
>>
>>
>> Any ideas? File content is looking ok.
>> It is happening with all time tests across this 15 years.
>> Here a month sample data.
>>
>> Thanks for any ideas...
>> Peter
>>
>>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/fc65d014-fec6-4b8b-b1cf-fc6b150ebec9n%40googlegroups.com.
# EXAMPLE CONFIGURATION FILE FOR IMPORTING FROM WEATHER DISPLAY (WD)
#
# Copyright (c) 2009-2019 Tom Keffer <tkef...@gmail.com> and Gary Roderick.
# See the file LICENSE.txt for your rights.
##############################################################################
# Specify our source. Available options are:
# CSV - import obs from a single CSV format file
# WU - import obs from a Weather Underground PWS history
# Cumulus - import obs from a one or more Cumulus monthly log files
# WD - import obs from a one or more WD monthly log files
# Format is:
# source = (CSV | WU | Cumulus | WD)
source = WD
##############################################################################
[WD]
# Parameters used when importing WD monthly log files
#
# Directory containing WD monthly log files to be imported. Format is:
# directory = full path without trailing /
directory = /tmp/wdisplay
# When importing WD monthly log file data the following WeeWX database
# fields will be populated directly by the imported data:
# barometer
# dateTime
# dewpoint
# extraHumid1 (if MMYYYvantageextrasensorslog.csv is available and field
hum1 has data)
# extraHumid2 (if MMYYYvantageextrasensorslog.csv is available and field
hum2 has data)
# extraTemp1 (if MMYYYvantageextrasensorslog.csv is available and field
temp1 has data)
# extraTemp2 (if MMYYYvantageextrasensorslog.csv is available and field
temp2 has data)
# extraTemp3 (if MMYYYvantageextrasensorslog.csv is available and field
temp3 has data)
# heatindex
# outHumidity
# outTemp
# radiation (if MMYYYYvantagelog.txt or MMYYYYvantagelogcsv.csv is
available)
# rain
# soilTemp1 (if MMYYYYvantagelog.txt or MMYYYYvantagelogcsv.csv is
available)
# soilMoist1 (if MMYYYYvantagelog.txt or MMYYYYvantagelogcsv.csv is
available)
# UV (if MMYYYYvantagelog.txt or MMYYYYvantagelogcsv.csv is
available)
# windDir
# windGust
# windSpeed
#
# The following WeeWX database fields will be populated from other
# settings/config files:
# interval
# usUnits
#
# The following WeeWX database fields will be populated with values derived
# from the imported data provided the --calc-missing command line option is
# used during import:
# altimeter
# ET
# pressure
# rainRate
# windchill
#
# The following WeeWX fields will be populated with values direct from the
# imported data. These fields will only be saved to the WeeWX database if
# the WeeWX schema has been modified to accept them.
# extraHumid3 (if MMYYYvantageextrasensorslog.csv is available and field
hum3 has data)
# extraHumid4 (if MMYYYvantageextrasensorslog.csv is available and field
hum4 has data)
# extraHumid5 (if MMYYYvantageextrasensorslog.csv is available and field
hum5 has data)
# extraHumid6 (if MMYYYvantageextrasensorslog.csv is available and field
hum6 has data)
# extraHumid7 (if MMYYYvantageextrasensorslog.csv is available and field
hum7 has data)
# extraTemp4 (if MMYYYvantageextrasensorslog.csv is available and field
temp4 has data)
# extraTemp5 (if MMYYYvantageextrasensorslog.csv is available and field
temp5 has data)
# extraTemp6 (if MMYYYvantageextrasensorslog.csv is available and field
temp6 has data)
# extraTemp7 (if MMYYYvantageextrasensorslog.csv is available and field
temp7 has data)
#
# The following WeeWX fields will be populated with derived values from the
# imported data provided the --calc-missing command line option is used
# during import. These fields will only be saved to the WeeWX database if
# the WeeWX schema has been modified to accept them. Note that the pyephem
# module is required in order to calculate maxSolarRad - refer WeeWX Users
# Guide.
# appTemp
# cloudbase
# humidex
# maxSolarRad
# windrun
# WD uses multiple log files some of which are in space delimited text
# format, some are in csv format and some in both. wee_import can process
# the following WD log files (actual log file names have 5 or 6 digits
# prepended representing a 1 or 2 digit month and a 4 digit year, these
# digits have been omitted for clarity):
# - lg.txt (same content as lgcsv.csv)
# - lgcsv.csv (same content as lg.txt)
# - vantageextrasensorslog.csv
# - vantagelog.txt (same content as vantagelogcsv.csv)
# - vantagelogcsv.csv (same content as vantagelog.txt)
# Specify log files to be imported. Format is a comma separated list
# including at least one of the supported log files. Do not include
# prepended month and year digits. Default is lg.txt, vantagelog.txt
# and vantageextrasensorslog.csv.
logs_to_process = lg.txt
# How will the interval field be determined for the imported records.
# Available options are:
# derive - Derive the interval field from the timestamp of successive
# records. This setting is best used when the imported records
# are equally spaced in time and there are no missing records.
# conf - Use the interval setting from weewx.conf. This setting is
# best used if the records to be imported have been produced by
# WeeWX using the same archive interval as set in weewx.conf on
# this machine.
# x - Use a fixed interval of x minutes for every record. This
# setting is best used if the records to be imported are
# equally based in time but there are some missing records.
# This setting is recommended for WU and WD imports.
# To import WD monthyl log data it is recommended that the interval setting
# be set to 1 minute.
# Format is:
# interval = (derive | conf | x)
interval = 1
# Should the [StdQC] max/min limits in weewx.conf be applied to the
# imported data. This may be useful if the source has extreme values that
# are clearly incorrect for some observations. Available options are:
# True - weewx.conf [StdQC] max/min limits are applied.
# False - weewx.conf [StdQC] max/min limits are not applied.
# Format is:
# qc = (True | False)
qc = True
# Should any missing derived observations be calculated from the imported
# data if possible. Available options are:
# True - Any missing derived observations are calculated.
# False - Any missing derived observations are not calculated.
# Format is:
# calc_missing = (True | False)
calc_missing = True
# Specify the character used as the field delimiter in .txt monthly log
# files. Normally set to the space character. The character must be
# enclosed in quotes. Must not be the sam# as the decimal setting below.
# Format is:
# txt_delimiter = ' '
txt_delimiter = ' '
# Specify the character used as the field delimiter in .csv monthly log
# files. Normally set to a comma. The character must be enclosed in
# quotes. Must not be the same as the decimal setting below. Format is:
# csv_delimiter = ','
csv_delimiter = ','
# Specify the character used as the decimal point. WD monthly log files
# normally use a full stop character as the decimal point. The character
# must be enclosed in quotes. Must not be the same as the xt_delimiter
# or csv_delimiter settings. Format is:
# decimal = '.'
decimal = '.'
# Specify whether missing log files are to be ignored or abort the import.
# WD log files are complete in themselves and a missing log file will have
# no effect on any other records (eg rain as a delta).
# Format is:
# ignore_missing_log = (True | False)
# Default is True
ignore_missing_log = True
# Specify how imported data fields that contain invalid data (eg a numeric
# field containing non-numeric data) are handled. Available options are:
# True - The invalid data is ignored, the WeeWX target field is set to
# None and the import continues.
# False - The import is halted.
# Format is:
# ignore_invalid_data = (True | False)
# Default is True.
ignore_invalid_data = True
# Imported records are written to archive in transactions of tranche
# records at a time. Increase for faster throughput, decrease to reduce
# memory requirements. Format is:
# tranche = x
# where x is an integer
tranche = 250
# Specify whether a UV sensor was used to produce any UV observations.
# Available options are:
# True - UV sensor was used and UV data will be imported.
# False - UV sensor was not used and any UV data will not be imported.
# UV fields will be set to None/NULL.
# For a WD monthly log file import UV_sensor should be set to False if a UV
# sensor was NOT present when the import data was created. Otherwise it may
# be set to True or omitted. Format is:
# UV_sensor = (True | False)
# The default is True.
UV_sensor = True
# Specify whether a solar radiation sensor was used to produce any solar
# radiation observations. Available options are:
# True - Solar radiation sensor was used and solar radiation data will
# be imported.
# False - Solar radiation sensor was not used and any solar radiation
# data will not be imported. radiation fields will be set to
# None/NULL.
# For a WD monthly log file import solar_sensor should be set to False if a
# solar radiation sensor was NOT present when the import data was created.
# Otherwise it may be set to True or omitted. Format is:
# solar_sensor = (True | False)
# The default is True.
solar_sensor = True
# Specify whether to ignore temperature and humidity reading of 255.0.
# WD logs can include values of 255.0 or 255. These values are usually
# associated with an absent or disconnected senor. In WeeWX the lack of a
# sensor/sensor data results in the value None (or null in SQL) being
# recorded. If ignore_extreme_temp_hum is set to True temperature and
# humidity values of 255 are ignored. Format is:
# ignore_extreme_temp_hum = (True | False)
# The default is True
ignore_extreme_temp_hum = True
# For correct import of the monthly logs wee_import needs to know what
# units are used in the imported data. The units used in the WD log files
# are set to either Metric or US in WD via the 'Log File' setting under
# 'Units' on the 'Units/Wind Chill' tab of the WD universal setup. If
# Metric or US units have been used in the log files then it is usually
# sufficient to set the units config option in the [[Units]] stanza to
# Metric or US. For example:
# [[Units]]
# units = metric
# However, in cases where Metric or US may not have been used (eg a custom
# log file) then the units config option should not be used but rather
# units should be specified for temperature, pressure, rainfall and speed
# groups using the temperature, pressure, rain and speed config options.
# In each case the config option should be set to a WeeWX unit option. For
# example:
# [[Units]]
# temperature = degree_C # options are 'degree_F' or 'degree_C'
# pressure = hPa # options are 'inHg', 'mbar' or 'hPa'
# rain = mm # options are 'inch' or 'mm'
# speed = km_per_hour # options are 'mile_per_hour',
# # 'km_per_hour', 'knot' or
# # 'meter_per_second'
# Note that either the units config option should be set or the individual
# unit group config options but not both.
[[Units]]
units = metric
# Map WD log fields to WeeWX archive fields. Format is:
#
# weewx_archive_field_name = wd_field_name
#
# where:
# weewx_archive_field_name - An observation name in the WeeWX database
# schema.
# wd_field_name - The name of a WD field from the CSV file.
# For example,
# outTemp = temperature
# would map the WD log field temperature to the archive field outTemp.
#
# Note that due to some issues with the field names used in some WD logs
# weeimport uses the following field names when reading WD log data:
# temperature
# humidity
# dewpoint
# barometer
# windspeed
# gustspeed
# direction
# rainlastmin
# heatindex
# radiation in lieu of 'Solar radation' and ''Solar ,radiat.'
# uv in lieu of 'UV'
# soilmoist in lieu of 'soil moist'
# soiltemp in lieu of 'soil temp'
# temp1
# temp2
# temp3
# temp4
# temp5
# temp6
# temp7
# hum1
# hum2
# hum3
# hum4
# hum5
# hum6
# hum7
#
# WeeWX archive fields that do not exist in the logs to be imported may be
# omitted (it is usually safest to omit fields for which you know there is
# no data to import as otherwise you may end up with a value of 0 (or
# something else) in the WeeWX archive when the appropriate value is
# None/null). Any omitted fields that are derived (eg dewpoint) may be
# calculated during import using the equivalent of the WeeWX StdWXCalculate
# service through setting the calc-missing config option above. Note also
# that whilst there may be a mapping for a field and the WD log may contain
# valid data the only fields that will be saved to the WeeWX archive are
# those fields in the in-use schema. Refer to the 'Archive types' appendix
# and the 'Customizing the database' section of the Customization Guide.
[[FieldMap]]
outTemp = temperature
outHumidity = humidity
dewpoint = dewpoint
barometer = barometer
windSpeed = windspeed
gustSpeed = gustspeed
windDir = direction
rain = rainlastmin
heatindex = heatindex
radiation = radiation
UV = uv
soilMoist1 = soilmoist
soilTemp1 = soiltemp
extraTemp1 = temp1
extraTemp2 = temp2
extraTemp3 = temp3
extraTemp4 = temp4
extraTemp5 = temp5
extraTemp6 = temp6
extraTemp7 = temp7
extraHumid1 = hum1
extraHumid2 = hum2
extraHumid3 = hum3
extraHumid4 = hum4
extraHumid5 = hum5
extraHumid6 = hum6
extraHumid7 = hum7
