Sorry for the delay, I wanted to run your data and import config through wee_import before replying.
I ran your data and import config through wee_import and the data was imported successfully and dewpoint, windchill and heatindex calculated correctly. altimeter and pressure were not calculated; the reason is a little complicated. Provided one of the pressures (altimeter, barometer or pressure) is present WeeWX can calculate the other two; however, when only altimeter or barometer is present pressure must be calculated first and pressure requires temperature from 12 hours earlier. In my case I did not have temperature from 12 hours earlier so pressure was not calculated. When calculating altimeter (or barometer) pressure must exist (hence why pressure is calculated first) so in my case altimeter could not be calculated either. In a typical import where temperature 12 hours ago is missing from the first 12 hours of import data you often see 12 hours of two missing pressures before all three are calculated once the temperature 12 hours ago kicks in. I digress. I guess the next thing is to find your log entries. You are looking for whatever log file WeeWX logs to. That may be /var/log/syslog or perhaps /var/log/messages unless you have explicitly changed it. If using a RPi it will likely be /var/log/syslog. The wiki page where are my logs? <https://github.com/weewx/weewx/wiki/faq-where-are-my-logs> may help. If WeeWX is running the log should be very easily spotted as the most recent log file (whichever name it is) will contain clearly labelled WeeWX log entries. The wee_import entries will be clearly labelled as such, you should be looking for something like this: Aug 9 21:50:03 bullseye4 wee_import[1156] INFO __main__: Starting wee_import... Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.csvimport: A CSV import from source file '/var/tmp/test_data.csv' has been requested. Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.csvimport: Using database binding 'wx_binding', which is bound to database 'weewx.sdb' Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.csvimport: Destination table 'archive' unit system is '0x01' (US). Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.csvimport: Missing derived observations will be calculated. Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.csvimport: All WeeWX UV fields will be set to None. Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.csvimport: All WeeWX radiation fields will be set to None. Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: Obtaining raw import data for period 1 ... Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: The following imported field-to-WeeWX field map will be used: Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: source field 'dateTime' in units 'unix_epoch' --> WeeWX field 'dateTime' Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: source field 'barometer' in units 'hPa' --> WeeWX field 'barometer' Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: source field 'insideTemp' in units 'degree_C' --> WeeWX field 'inTemp' Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: source field 'outsideTemp' in units 'degree_C' --> WeeWX field 'outTemp' Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: source field 'insideHum' in units 'percent' --> WeeWX field 'inHumidity' Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: source field 'outsideHum' in units 'percent' --> WeeWX field 'outHumidity' Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: source field 'windSpeed' in units 'km_per_hour' --> WeeWX field 'windSpeed' Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: source field 'windDirection' in units 'degree_compass' --> WeeWX field 'windDir' Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: source field 'hiWindSpeed' in units 'km_per_hour' --> WeeWX field 'windGust' Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: source field 'hiWindDirection' in units 'degree_compass' --> WeeWX field 'windGustDir' Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: source field 'hiRainRate' in units 'mm_per_hour' --> WeeWX field 'rainRate' Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: source field 'rain' in units 'mm' --> WeeWX field 'rain' Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: Raw import data read successfully for period 1. Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: Mapping raw import data for period 1 ... Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: Mapped 5 records. Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: Raw import data mapped successfully for period 1. Aug 9 21:50:03 bullseye4 wee_import[1156] INFO weeimport.weeimport: Saving mapped data to archive for period 1 ... Aug 9 21:50:05 bullseye4 wee_import[1156] ERROR weewx.manager: Unable to add record 2021-08-01 00:01:00 AEST (1627740060) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime Aug 9 21:50:05 bullseye4 wee_import[1156] ERROR weewx.manager: Unable to add record 2021-08-01 00:02:00 AEST (1627740120) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime Aug 9 21:50:05 bullseye4 wee_import[1156] ERROR weewx.manager: Unable to add record 2021-08-01 00:03:00 AEST (1627740180) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime Aug 9 21:50:05 bullseye4 wee_import[1156] ERROR weewx.manager: Unable to add record 2021-08-01 00:04:00 AEST (1627740240) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime Aug 9 21:50:05 bullseye4 wee_import[1156] ERROR weewx.manager: Unable to add record 2021-08-01 00:05:00 AEST (1627740300) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weeimport.weeimport: Mapped data saved to archive successfully for period 1. Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weeimport.weeimport: Calculating missing derived observations ... Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.engine: StdConvert target unit is 0x1 Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.wxservices: StdWXCalculate will use data binding wx_binding Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.engine: Archive will use data binding wx_binding Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.engine: Record generation will be attempted in 'hardware' Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.engine: Using archive interval of 60 seconds (specified by hardware) Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.restx: StationRegistry: Registration not requested. Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.restx: Wunderground: Posting not enabled. Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.restx: PWSweather: Posting not enabled. Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.restx: CWOP: Posting not enabled. Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.restx: WOW: Posting not enabled. Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.restx: AWEKAS: Posting not enabled. Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.engine: 'pyephem' detected, extended almanac data is available Aug 9 21:50:05 bullseye4 wee_import[1156] INFO weewx.wxservices: StdWXCalculate will use data binding wx_binding Aug 9 21:50:06 bullseye4 wee_import[1156] INFO weewx.manager: Starting backfill of daily summaries Aug 9 21:50:06 bullseye4 wee_import[1156] INFO weewx.manager: Processed 5 records to backfill 1 day summaries in 0.01 seconds Aug 9 21:50:06 bullseye4 wee_import[1156] INFO weecfg.database: Processed 1 day consisting of 5 records. 1 day consisting of 5 records were updated in 0 .04 seconds. Aug 9 21:50:06 bullseye4 wee_import[1156] INFO weeimport.weeimport: Finished calculating missing derived observations Aug 9 21:50:06 bullseye4 wee_import[1156] INFO weeimport.weeimport: Finished import Aug 9 21:50:06 bullseye4 wee_import[1156] INFO weeimport.weeimport: 5 records were processed and 5 unique records imported in 2.45 seconds. (Note in this case the error lines are due to there already being records with the imported data timestamps in the archive) Please post all of the import logs, don't leave anything out. Gary On Monday, 8 August 2022 at 02:50:21 UTC+10 [email protected] wrote: > Thanks for your answer. > The two conditions you listed are satisfied: the necessary other values > are present in my csv, and no records were already existing for such period > in the database. > > Answering your questions: > > - Those are my data (a sample): > CSV data (more fields than used in the import process are present) > > dateTime,outsideTemp,hiOutsideTemp,lowOutsideTemp,outsideHum,windSpeed,windDirection,hiWindSpeed,hiWindDirection,barometer,rain,hiRainRate,insideTemp,insideHum,numWindSamples,archiveInterval > > > 2021-08-01T00:01,27.61,27.61,27.61,60.0,1.61,SW,3.22,SW,1006.30,0.00,0.00,30.33,49.0,22,1 > > > 2021-08-01T00:02,27.56,27.61,27.56,60.0,1.61,W,1.61,SW,1006.16,0.00,0.00,30.33,49.0,24,1 > > > 2021-08-01T00:03,27.50,27.56,27.50,60.0,1.61,W,1.61,W,1006.27,0.00,0.00,30.33,49.0,23,1 > > > 2021-08-01T00:04,27.50,27.50,27.50,60.0,0.00,SW,3.22,SW,1006.33,0.00,0.00,30.33,49.0,22,1 > > > 2021-08-01T00:05,27.50,27.50,27.50,60.0,3.22,WSW,4.83,WSW,1006.37,0.00,0.00,30.33,49.0,24,1 > > Resulting db records (extracted querying the `archive`. I here removed > all null values for reading convenience) > > dateTime,usUnits,interval,appTemp,barometer,cloudbase,humidex,inDewpoint,inHumidity,inTemp,outHumidity,outTemp,rain,rainRate,windDir,windGust,windGustDir,windrun,windSpeed > > > 1627768860,16,1,30.5895137231874,1006.3,1167.77458102164,34.4779454220626,18.4017421866142,49,30.33,60,27.61,0,0,225,3.22,225,0.0268333333333333,1.61 > > > 1627768920,16,1,30.5182757269146,1006.16,1167.3911940789,34.3910793180734,18.4017421866142,49,30.33,60,27.56,0,0,270,1.61,225,0.0268333333333333,1.61 > > > 1627768980,16,1,30.4328612401526,1006.27,1166.93122058363,34.286968264113,18.4017421866142,49,30.33,60,27.5,0,0,270,1.61,270,0.0268333333333333,1.61 > > > 1627769040,16,1,30.7459167959586,1006.33,1166.93122058363,34.286968264113,18.4017421866142,49,30.33,60,27.5,0,0,,3.22,225,0,0 > > > 1627769100,16,1,30.1198056843466,1006.37,1166.93122058363,34.286968264113,18.4017421866142,49,30.33,60,27.5,0,0,247.5,4.83,247.5,0.0536666666666667,3.22 > > - No data were already present for that period > - I checked resulting records by querying the database directly > - The query I ran (for the sample): SELECT * FROM archive WHERE dateTime > >= 1627768860 and dateTime < 1627769160 ORDER BY dateTime > - I imported data few weeks ago, so I cannot exactly say what the output > of the import command was, but - as I can remember - nothing remarkable was > printed out. I noticed just now that some fields are missing. Can I still > check those logs? how to? > > Thanks, > Andrea > Il giorno domenica 7 agosto 2022 alle 02:09:20 UTC+2 gjr80 ha scritto: > >> Your import config file is fine and provided a couple of conditions were >> met the missing fields should have been calculated. The two conditions are >> (1) any pre-requisites for calculating the fields concerned must exist, for >> example for heatindex, windchill and dewpoint you must have a non-None >> outTemp value. If a pre-requisite is missing the calculated value will >> be None. (2) Even if the calculated values were calculated there must be no >> record already existing in the archive with the same timestamp as the >> record being imported. If such a record already exists in the archive the >> imported record is discarded in it's entirety (not just the calculated >> fields). So if these conditions were met your imported data including >> calculated fields should have been imported and saved to archive, if they >> were not imported then something went wrong. >> >> A few questions to try to narrow down the cause. What does your data look >> like? Can you post some of the records here or if you don't want to can you >> reply privately to this post with your raw import data. Did records already >> exist in your archive for the period covered by the data you were >> importing? How did you check the imported data? Did you query the database >> archive table directly or rely on WeeWX generated output? If the former >> what command did you use and what results were presented? Did the other >> non-calculated fields import correctly? What was presented on-screen when >> you ran wee_import? Were there any errors or warnings? wee-import will >> have written some entries to the log at the time of the import, they should >> be clearly evident with a 'wee_import' label, what do these entries say? >> >> Gary >> >> On Sunday, 7 August 2022 at 00:23:31 UTC+10 [email protected] wrote: >> >>> I imported ~1 year of data from csv, using this `import.conf` >>> configuration: >>> source = CSV >>> [CSV] >>> file = my-data.csv >>> interval = 1 >>> qc = True >>> calc_missing = True >>> ignore_invalid_data = True >>> tranche = 250 >>> UV_sensor = False >>> solar_sensor = False >>> raw_datetime_format = %Y-%m-%dT%H:%M >>> rain = discrete >>> wind_direction = 0,360 >>> [[FieldMap]] >>> dateTime = dateTime, unix_epoch >>> usUnits = >>> interval = >>> barometer = barometer, hPa >>> pressure = >>> altimeter = >>> inTemp = insideTemp, degree_C >>> outTemp = outsideTemp, degree_C >>> inHumidity = insideHum, percent >>> outHumidity = outsideHum, percent >>> windSpeed = windSpeed, km_per_hour >>> windDir = windDirection, degree_compass >>> windGust = hiWindSpeed, km_per_hour >>> windGustDir = hiWindDirection, degree_compass >>> rainRate = hiRainRate, mm_per_hour >>> rain = rain, mm >>> dewpoint = >>> windchill = >>> heatindex = >>> ET = >>> radiation = >>> UV = >>> >>> What I expected was that the following fields would have been >>> calculated, starting from the other available fields: >>> altimeter >>> dewpoint >>> heatindex >>> pressure >>> windchill >>> But in the `archive` table, those columns are empty. >>> >>> Did I do something wrong, or is the behavior correct? So I cannot have >>> such fields calculated...? >>> >> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/496c3bcd-dafa-42e1-908e-df37c68ebfcdn%40googlegroups.com.
