>
> If you want to use weectl import to import data into existing archive
> records, each record in your import source data (in your case your CSV
> data) needs to include the existing data for all fields as well as the
> updated rainfall data. You then need to use the (as yet undocumented)
> --update command line option when running weectl import.
So rather than just the records I'm *missing*, I need to go from the first
missing record to the last missing record, inclusive of all in-between
records? I can easily do that, just not 100% on whether it's required.
Alternatively, you could avoid using the --update command line option by
> first deleting the records to be imported from the WeeWX database and then
> importing your merged/updated CSV data.
If you'll save me the research, how do I delete specific records/periods,
either a single period record, or a range? I just recently noticed that the
soilMoist1 observation, on my weewx testing install, recorded about 84
hours of very wrong values, that my production weewx install didn't (both
getting the data from the same mqtt topic). I know how to drop entire
observation columns via weectl database drop-columns NAME, but in this case
it's just range of periods within the column that need purging. Or is it as
simple as, zeroing out the values in a range via sqlite> update archive set
soilMoist1=0.0 where dateTime > {startepochtime} and dateTime <=
{endepochtime]; ?
On Fri, Mar 14, 2025 at 1:17 AM gjr80 <gjroder...@gmail.com> wrote:
> To answer your CSV import config file questions first:
>
> interval. The description for the interval config option in the example
> CSV import config file reads:
>
> # If there is no mapped interval field how will the interval field be
> # determined for the imported records. Available options are:
> # derive - Derive the interval field from the timestamp of successive
> # records. This setting is best used when there are no
> missing
> # records from period being imported. Missing records will
> cause
> # the interval field to be incorrectly calculated for some
> # records.
> # conf - Use the interval setting from weewx.conf. This setting is
> best
> # used if the records to be imported have been produced by
> WeeWX
> # or some other means with the same archive interval as set
> in
> # weewx.conf on this machine.
> # x - Use a fixed interval of 'x' minutes for every record
> where 'x'
> # is a number. This setting is best used if the records to
> be
> # imported are equally spaced in time but there are some
> missing
> # records.
> #
> # Note: If there is a mapped interval field this setting will be
> ignored.
> # Format is:
> # interval = (derive | conf | x)
> # Default is derive.
>
> If you use interval = derive the interval field in a given imported
> record will be calculated as the difference in timestamps of the current
> record being imported and the previous record that was imported. Given your
> irregular spacing in time of records to be imported interval = derive
> will not give you the desired result (1 minute). interval = conf will set
> the interval field for all imported records to the archive_interval
> setting from [StdArchive] in weewx.conf. If archive_interval is set to
> one minute you will get the desired result. If you set interval = 1 the
> interval field for all imported records will be set to one minute, again
> the desired result. Sounds like you should be using interval = 1 or maybe
> interval
> = conf.
>
> is_cumulative. is_cumulative doesn't expect data in a particular format,
> rather the is_cumulative setting (True or False) specifies the format the
> data should be in (cumulative or per-archive period respectively). Again
> reading the description in the example CSV import config file:
>
> # is_cumulative - Config option specifying whether the CSV
> # field being mapped is cumulative,
> # e.g: dayrain. Optional, default value is
> # False.
>
> WeeWX records the likes of rain and lightning strike count etc as
> per-archive period values rather than as cumulative values; however, many
> stations provides such data as cumulative values. The is_cumulative
> setting is used to tell weectl import that the field being imported is a
> cumulative value (is_cumulative = True) in which case weectl import
> calculates the value for the field concerned as the difference between
> successive imported values. If the field being imported is already a
> per-archive period value setting is_cumulative = False will cause the
> value to be imported without change. It sounds as though in your case the
> original rainfall data was cumulative but the data you intend to import has
> been converted to per-period values. In that case you should be using
> is_cumulative
> = False. (An aside: in your case with imported records needing to be
> slotted in between existing archive records you have to import per-period
> rainfall data; if you were to import using cumulative values the resulting
> per-archive period value calculated by weectl import would be incorrect
> due to the interspersed existing archive records).
>
> You should also keep in mind the manner in which weectl import (nee
> weeimport) operates. It was originally designed to import large blocks of
> archive record data from other sources; eg other weather station software
> or (shudder) WeatherUnderground etc. Essentially archive record like data
> was read from a source, the data was massaged into a format compatible with
> a WeeWX archive record and the re-formatted data written to WeeWX archive.
> Initially weeimport only allowed new archive records to be added;
> existing records could not be modified and any attempt to do so resulted in
> the imported record being discarded. As of WeeWX v5.1.0 existing records
> can be updated, but the update performed is the overwriting of an existing
> archive record with the imported archive record. The existing archive
> record is not merged with the imported record and the merged record written
> to the WeeWX database. So what? If you want to use weectl import to
> import data into existing archive records, each record in your import
> source data (in your case your CSV data) needs to include the existing data
> for all fields as well as the updated rainfall data. You then need to use
> the (as yet undocumented) --update command line option when running weectl
> import. Alternatively, you could avoid using the --update command line
> option by first deleting the records to be imported from the WeeWX database
> and then importing your merged/updated CSV data.
>
> Gary
> On Friday, 14 March 2025 at 02:58:14 UTC+10 Andrew McGinnis wrote:
>
>> I had a rain event on Monday that saw 0.66" fall on my station. All of
>> the consoles that receive direct from the sensor (433mhz) recorded the same
>> 0.66".
>>
>> My weewx install uses MQTTSubscribe for all of my observations, via
>> rtl_433's mqtt output. It recorded 0.48" in total.
>>
>> My sensor is a cumulative output, so the mqtt topic would read something
>> like this
>> rtl_433/pi4b8/devices/Acurite-5n1/A/3030/rain_mm 510.28601
>> rtl_433/pi4b8/devices/Acurite-5n1/A/3030/rain_mm 511.5
>> rtl_433/pi4b8/devices/Acurite-5n1/A/3030/rain_mm 516.2
>>
>> The rain observation is set this way:
>> [[topics]]
>> [[[rtl_433/pi4b8/devices/Acurite-5n1/A/3030/rain_mm]]]
>> name = rain
>> contains_total = true
>>
>> I dumped out the timeframe of the rain event from my weewx.sdb, and
>> looked at where the missing data might've been:
>> echo "select datetime(dateTime, 'unixepoch', 'localtime'), rain from
>> archive where datetime(dateTime,'unixepoch','localtime') > "2025-03-10
>> 09:00:00" limit 120;" | sqlite3 weewx.sdb
>> ...
>> 2025-03-10 09:49:00|
>> 2025-03-10 09:50:00|0.0199988188976378
>> 2025-03-10 09:51:00|
>> 2025-03-10 09:52:00|0.0199996062992129
>> 2025-03-10 09:53:00|0.0499996062992106
>> 2025-03-10 09:54:00|0.0599996062992136
>> 2025-03-10 09:55:00|0.0300007874015751
>> 2025-03-10 09:56:00|0.0499984251968492
>> 2025-03-10 09:57:00|0.0400007874015758
>> 2025-03-10 09:58:00|
>> 2025-03-10 09:59:00|0.0100000000000008
>> 2025-03-10 10:00:00|0.0
>> 2025-03-10 10:01:00|0.0
>> 2025-03-10 10:02:00|0.00999999999999852
>> 2025-03-10 10:03:00|
>> 2025-03-10 10:04:00|0.0
>> ...
>>
>> I've pieced together the 15 missing data points to get that day's total
>> rainfall to balance with the other consoles.
>>
>> So with a csv file that looks like this:
>> Time,Rain
>> 3/10/2025 09:14:00,0.00999881889763705
>> 3/10/2025 09:19:00,0.0100000000000008
>> 3/10/2025 09:24:00,0.0100000000000008
>> 3/10/2025 09:31:00,0.0
>> 3/10/2025 09:38:00,0.0100000000000008
>> 3/10/2025 09:42:00,0.0199984251968514
>> 3/10/2025 09:49:00,0.04
>> 3/10/2025 09:51:00,0.0200000000000016
>> 3/10/2025 10:03:00,0.0
>> 3/10/2025 10:06:00,0.0200000000000015
>> 3/10/2025 10:09:00,0.0
>> 3/10/2025 10:12:00,0.0
>> 3/10/2025 10:17:00,0.0
>> 3/10/2025 10:18:00,0.00999999999999852
>> 3/10/2025 10:19:00,0.0
>>
>> I'm not confident that my csv.conf import file is correctly configured.
>> Here's what I've got:
>> source = CSV
>>
>> [CSV]
>> file = /var/lib/weewx/rain.csv
>> delimiter = ','
>> decimal = '.'
>> interval = conf
>> qc = True
>> calc_missing = True
>> ignore_invalid_data = True
>> tranche = 250
>> raw_datetime_format = %m/%d/%Y %H:%M:%S
>> wind_direction = -360,360
>>
>> # is_cumulative - Config option specifying whether the
>> CSV
>> # field being mapped is cumulative,
>> # e.g: dayrain. Optional, default value
>> is
>> # False.
>> [[FieldMap]]
>> [[[dateTime]]]
>> source_field = Time
>> unit = unix_epoch
>> [[[rain]]]
>> source_field = Rain
>> unit = mm
>> is_cumulative = False
>>
>> I'm unsure of the interval = conf setting. My weewx.conf uses a
>> 60-second archive_interval, and the missing data points are what would've
>> been recorded in those missing periods... Or should it be interval = x
>> (in my case, 1 minute)?
>>
>> For [[FieldMap]] > [[[rain]]] > is_cumulative, I'm unsure what it should
>> be, and subsequently what my csv data should be. My data points are the
>> values of what fell in those minute intervals, the current.[running total]
>> at each of those periods. is is_cumulative expecting data that is a
>> running total (510.2,511.5,516.2), or just the delta (1.3,0.7,etc)?
>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "weewx-user" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/weewx-user/1KX3-8lbU34/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> weewx-user+unsubscr...@googlegroups.com.
> To view this discussion visit
> https://groups.google.com/d/msgid/weewx-user/23be3120-1e80-4e80-93c4-4249428864ecn%40googlegroups.com
> <https://groups.google.com/d/msgid/weewx-user/23be3120-1e80-4e80-93c4-4249428864ecn%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion visit
https://groups.google.com/d/msgid/weewx-user/CAH3iRs1GPVORbFDZ5qs0zg2uj0PEHBL2v-g-C8w0L6Qy5i_bEw%40mail.gmail.com.
