Using my old import-csv.conf file (attached), I get the following error:
sudo weectl import --import-config=/home/piToad/Documents/csv-import.conf
--dry-runUsing configuration file /etc/weewx/weewx.conf
This is a dry run. Nothing will actually be done.
Starting weectl import...
**** Unknown units 'unix-epoch' specified for field 'dateTime' in
/home/piToad/Documents/csv-import.conf.
**** Nothing done, exiting.
On Sunday, April 21, 2024 at 12:27:22 PM UTC-4 tarob...@gmail.com wrote:
> Hello,
>
> I am trying to import some missing data via CSV file. I've done this
> numerous times pre weewx 5.0 and weectl (using wee_import). I've setup a
> new import config file using the new layout for [[FieldMap]]. I've attached
> the import config I used. However, during the --dry-run I am getting the
> below error. I've attached the config file.
>
> Column Headings:
> ['Timestamp','d5n1Temp', 'd5n1humid', 'd5n1Pres', 'rain5n1',
> 'windSpeed5n1',
> 'windAvg', 'windDir', 'UV', 'luminosity', 'd5n1Humid', 'rain',
> 'windSpeed', 'inTemp', 'inHumid', 'LiTemp', 'LiHumid', 'LiStrikes',
> 'LiDistance', 'LiPres', 'GTemp', 'GHumid', 'radiation']
>
> It's a source_field error but can't figure out which source_field is
> giving the error. I've looked them over but could be missing one.
>
> sudo weectl import
> --import-config=/home/piToad/Documents/csv-import-weectl.conf --dry-run
> Using configuration file /etc/weewx/weewx.conf
> This is a dry run. Nothing will actually be done.
> Starting weectl import...
> Traceback (most recent call last):
> File "/usr/share/weewx/weectl.py", line 74, in <module>
> main()
> File "/usr/share/weewx/weectl.py", line 66, in main
> namespace.func(namespace)
> File "/usr/share/weewx/weectllib/__init__.py", line 121, in dispatch
> namespace.action_func(config_dict, namespace)
> File "/usr/share/weewx/weectllib/import_cmd.py", line 82, in import_func
> weectllib.import_actions.obs_import(config_dict,
> File "/usr/share/weewx/weectllib/import_actions.py", line 54, in
> obs_import
> source_obj =
> weeimport.weeimport.Source.source_factory(config_dict['config_path'],
>
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/usr/share/weewx/weeimport/weeimport.py", line 316, in
> source_factory
> return get_object(module_class)(config_path,
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/usr/share/weewx/weeimport/csvimport.py", line 110, in __init__
> self.map = self.parse_map(_map,
> ^^^^^^^^^^^^^^^^^^^^
> File "/usr/share/weewx/weeimport/weeimport.py", line 551, in parse_map
> _mapped_source_fields = [config['source_field'] for field, config in
> _map.items()]
>
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/usr/share/weewx/weeimport/weeimport.py", line 551, in <listcomp>
> _mapped_source_fields = [config['source_field'] for field, config in
> _map.items()]
> ~~~~~~^^^^^^^^^^^^^^^^
> File "/usr/lib/python3/dist-packages/configobj/__init__.py", line 554,
> in __getitem__
> val = dict.__getitem__(self, key)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^
> KeyError: 'source_field'
>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/3ae8edb0-9438-423e-9342-1b6c95735f0an%40googlegroups.com.
# EXAMPLE CONFIGURATION FILE FOR IMPORTING FROM CSV FILES
#
# Copyright (c) 2009-2019 Tom Keffer <tkef...@gmail.com> and Gary Roderick.
# See the file LICENSE.txt for your rights.
##############################################################################
# Specify the source. Available options are:
# CSV - import obs from a single CSV format file
# WU - import obs from a Weather Underground PWS history
# Cumulus - import obs from a one or more Cumulus monthly log files
# WD - import obs from a one or more WD monthly log files
# Format is:
# source = (CSV | WU | Cumulus)
source = CSV
##############################################################################
[CSV]
# Parameters used when importing from a CSV file
# Path and name of our CSV source file. Format is:
# file = full path and filename
file = /var/tmp/dataCorrection.csv
# The character used to separate fields. Format is:
# delimiter = <single character>
# Default is , (comma).
delimiter = ','
# If there is no mapped interval field how will the interval field be
# determined for the imported records. Available options are:
# derive - Derive the interval field from the timestamp of successive
# records. This setting is best used when the imported records
# are equally spaced in time and there are no missing records.
# conf - Use the interval setting from weewx.conf. This setting is
# best used if the records to be imported have been produced by
# WeeWX using the same archive interval as set in weewx.conf on
# this machine.
# x - Use a fixed interval of x minutes for every record. This
# setting is best used if the records to be imported are
# equally based in time but there are some missing records.
#
# Note: If there is a mapped interval field then this setting will be
# ignored.
# Format is:
# interval = (derive | conf | x)
interval = 5
# Should the [StdQC] max/min limits in weewx.conf be applied to the
# imported data. This may be useful if the source has extreme values that
# are clearly incorrect for some observations. Available options are:
# True - weewx.conf [StdQC] max/min limits are applied.
# False - weewx.conf [StdQC] max/min limits are not applied.
# Format is:
# qc = (True | False)
qc = True
# Should any missing derived observations be calculated from the imported
# data if possible. Available options are:
# True - Any missing derived observations are calculated.
# False - Any missing derived observations are not calculated.
# Format is:
# calc_missing = (True | False)
calc_missing = True
# Specify how imported data fields that contain invalid data (eg a numeric
# field containing non-numeric data) are handled. Available options are:
# True - The invalid data is ignored, the WeeWX target field is set to
# None and the import continues.
# False - The import is halted.
# Format is:
# ignore_invalid_data = (True | False)
# Default is True.
ignore_invalid_data = True
# Imported records are written to archive in transactions of tranche
# records at a time. Increase for faster throughput, decrease to reduce
# memory requirements. Format is:
# tranche = x
# where x is an integer
tranche = 250
# Specify whether a UV sensor was used to produce any UV observations.
# Available options are:
# True - UV sensor was used and UV data will be imported.
# False - UV sensor was not used and any UV data will not be imported.
# UV fields will be set to None/NULL.
# For a CSV import UV_sensor should be set to False if a UV sensor was
# NOT present when the import data was created. Otherwise it may be set to
# True or omitted. Format is:
# UV_sensor = (True | False)
UV_sensor = True
# Specify whether a solar radiation sensor was used to produce any solar
# radiation observations. Available options are:
# True - Solar radiation sensor was used and solar radiation data will
# be imported.
# False - Solar radiation sensor was not used and any solar radiation
# data will not be imported. radiation fields will be set to
# None/NULL.
# For a CSV import solar_sensor should be set to False if a solar radiation
# sensor was NOT present when the import data was created. Otherwise it may
# be set to True or omitted. Format is:
# solar_sensor = (True | False)
solar_sensor = True
# Date-time format of CSV field from which the WeeWX archive record
# dateTime field is to be extracted. wee_import first attempts to interpret
# date/time info in this format, if this fails it then attempts to
# interpret it as a timestamp and if this fails it then raises an error.
# Uses Python strptime() format codes.
# raw_datetime_format = Python strptime() format string
raw_datetime_format = %Y-%m-%d %H:%M:%S
# Does the imported rain field represent the total rainfall since the last
# record or a cumulative value. Available options are:
# discrete - rain field represents total rainfall since last record
# cumulative - rain field represents a cumulative rainfall reset at
# midnight
# rain = (discrete | cumulative)
rain = cumulative
# Lower and upper bounds for imported wind direction. It is possible,
# particularly for a calculated direction, to have a value (eg -45) outside
# of the WeeWX limits (0 to 360 inclusive). Format is:
#
# wind_direction = lower,upper
#
# where :
# lower is the lower limit of acceptable wind direction in degrees
# (may be negative)
# upper is the upper limit of acceptable wind direction in degrees
#
# Imported values from lower to upper will be normalised to the range 0 to
# 360. Values outside of the parameter range will be stored as None.
# Default is -360,360.
wind_direction = -360,360
# Map CSV record fields to WeeWX archive fields. Format is:
#
# weewx_archive_field_name = csv_field_name, weewx_unit_name
#
# where:
# weewx_archive_field_name - An observation name in the WeeWX database
# schema.
# csv_field_name - The name of a field from the CSV file.
# weewx_unit_name - The name of the units, as defined in WeeWX,
# used by csv_field_name. wee_import will do
# the necessary conversions to the unit system
# used by the WeeWX archive.
# For example,
# outTemp = Temp, degree_C
# would map the CSV field Temp, in degrees C, to the archive field outTemp.
#
# A mapping for WeeWX field dateTime is mandatory and the WeeWX unit name
# for the dateTime mapping must be unix_epoch. For example,
# dateTime = csv_date_and_time, unix_epoch
# would map the CSV field csv_date_and_time to the WeeWX dateTime field with
# the csv_date_and_time field being interpreted first using the format
# specified at the raw_datetime_format config option and if that fails as a
# unix epoch timestamp.
#
# Field mapping to the WeeWX usUnits archive field is currently not
# supported. If a usUnits field exists in the CSV data it should not be
# mapped, rather WeeWX unit names should included against each field to be
# imported in the field map.
#
# WeeWX archive fields that do not exist in the CSV data may be omitted.
# Any omitted fields that are derived (eg dewpoint) may be calculated
# during import using the equivalent of the WeeWX StdWXCalculate service
# through setting the calc-missing parameter above.
[[FieldMap]]
dateTime = Timestamp, unix-epoch
interval =
barometer = d5n1Pres, inHg
pressure =
altimeter =
inTemp = inTemp, degree_F
outTemp = d5n1Temp, degree_F
inHumidity = inHumid, percent
outHumidity = d5n1humid, percent
windSpeed = windAvg, mile_per_hour
windDir = windDir, degree_compass
windGust = windSpeed, mile_per_hour
windGustDir =
rainRate =
rain = rain, inch
dewpoint =
windchill =
heatindex =
ET =
radiation = radiation, watt_per_meter_squared
UV = UV, uv_index
luminosity = group_illuminance, lux
extraTemp1 = extraTemp1, degree_F
extraHumid1 = extraHumid1, percent
extraTemp2 = LiTemp, degree_F
extraHumid2 = LiHumid, percent
lightning_distance = LiDistance, mile
strikes_total =
