Thank you very much for your patience Gary, Indeed I did not provide you with all the data from the CSV file in order to simplify. However, the fields were in the correct order with the correct wording.
Regarding the --dry-run command, I actually remove it for integration. This is why all my fields are saved in the database with the timestamp (except for rain). Actually, to make it simple for you, I have a weather station that doesn't have a USB output. The data is sent to AWEKAS. So in python, for 5 minutes I retrieve this data regularly, make an average for certain fields (temperatures, hygrometry etc.) and for others take the maximum (rain rate) during these 5 minutes. For rain, the station gives me the total since 00:00 which I calculate into rain over the 5 minute time slot. After 5 minutes, I generate a CSV which will be integrated into the weewx database with Wee_import. Weewx recalculates all the data, generates the html page and the csv file is deleted. We start again every 5 minutes. The other day after the changes you told me about there was a very light rain. I had this rain over two 5-minute time periods with a correct rain rate in the CSV file. However in the weewx database nothing... This morning, I did another test manually and put a value of 2mm in the rain field and 4mm/h in the rainRate. I manually tried the wee_import command (exactly the same one that is in the crontab) and I had a correct import this time! So the problem really comes from me and not from weewx... Do you think that a rainRate at 0 with a rain value in the rain field could cause a problem during the integration and set both fields to 0? I will first try to artificially simulate a rain in python in the CSV file and see if this value now integrates correctly or not automatically with the crontab. Secondly, I will monitor the actual conditions (rain is normally forecast this weekend...) and keep you informed. *Thank you Gary for helping me out and giving me the solution! ;-)* *Rémy LAVABRE* Le mar. 29 août 2023 à 22:42, gjr80 <gjroder...@gmail.com> a écrit : > I just setup a WeeWX instance with your import config file in your first > post but using rain = discrete and a csv data file using the two records > in your last post. I ran wee_import using these files and --dry-run and > --verbose and received somewhat different output to what you posted, in > my case there were warnings of missing fields in the csv data file: > > $ sudo /home/weewx/bin/wee_import --import-config=/var/tmp/csv.conf > --verbose > /home/weewx/bin/wee_import:719: DeprecationWarning: The distutils package > is deprecated and slated for removal in Python 3.12. Use setuptools or > check PEP 632 for potential alternatives > from distutils.version import StrictVersion > Using WeeWX configuration file /home/weewx/weewx.conf > Starting wee_import... > A CSV import from source file '/var/tmp/remy.csv' has been requested. > The following options will be used: > config=/home/weewx/weewx.conf, import-config=/var/tmp/csv.conf > source=/var/tmp/remy.csv, from=None, to=None > dry-run=None, calc_missing=True, ignore_invalid_data=True > tranche=250, interval=conf, date/time_string_format=%Y-%m-%d %H:%M:%S > delimiter=',', rain=discrete, wind_direction=[-360.0, 360.0] > UV=True, radiation=True > Using database binding 'wx_binding', which is bound to database 'weewx.sdb' > Destination table 'archive' unit system is '0x01' (US). > Missing derived observations will be calculated. > Starting import ... > Obtaining raw import data for period 1 ... > The following imported field-to-WeeWX field map will be used: > source field 'dateTime' in units 'unix_epoch' --> WeeWX field > 'dateTime' > source field 'barometer' in units 'hPa' --> WeeWX field 'barometer' > source field 'inTemp' in units 'degree_C' --> WeeWX field 'inTemp' > source field 'outTemp' in units 'degree_C' --> WeeWX field 'outTemp' > source field 'inHumidity' in units 'percent' --> WeeWX field > 'inHumidity' > source field 'outHumidity' in units 'percent' --> WeeWX field > 'outHumidity' > source field 'windSpeed' in units 'km_per_hour' --> WeeWX field > 'windSpeed' > source field 'windDir' in units 'degree_compass' --> WeeWX field > 'windDir' > source field 'windGust' in units 'km_per_hour' --> WeeWX field > 'windGust' > source field 'rainRate' in units 'mm_per_hour' --> WeeWX field > 'rainRate' > source field 'rain' in units 'mm' --> WeeWX field 'rain' > source field 'dewpoint' in units 'degree_C' --> WeeWX field 'dewpoint' > source field 'windchill' in units 'degree_C' --> WeeWX field > 'windchill' > source field 'radiation' in units 'watt_per_meter_squared' --> WeeWX > field 'radiation' > source field 'UV' in units 'uv_index' --> WeeWX field 'UV' > source field 'extraTemp1' in units 'degree_C' --> WeeWX field > 'extraTemp1' > Raw import data read successfully for period 1. > Mapping raw import data for period 1 ... > Warning: Import field 'inTemp' is mapped to WeeWX field 'inTemp' but the > import field 'inTemp' could not be found in one or more records. > WeeWX field 'inTemp' will be set to 'None' in these records. > Warning: Import field 'inHumidity' is mapped to WeeWX field 'inHumidity' > but the > import field 'inHumidity' could not be found in one or more > records. > WeeWX field 'inHumidity' will be set to 'None' in these records. > Warning: Import field 'radiation' is mapped to WeeWX field 'radiation' but > the > import field 'radiation' could not be found in one or more > records. > WeeWX field 'radiation' will be set to 'None' in these records. > Warning: Import field 'UV' is mapped to WeeWX field 'UV' but the > import field 'UV' could not be found in one or more records. > WeeWX field 'UV' will be set to 'None' in these records. > Warning: Import field 'extraTemp1' is mapped to WeeWX field 'extraTemp1' > but the > import field 'extraTemp1' could not be found in one or more > records. > WeeWX field 'extraTemp1' will be set to 'None' in these records. > Mapped 2 records. > Raw import data mapped successfully for period 1. > Saving mapped data to archive for period 1 ... > 2 records identified for import. > Proceeding will save all imported records in the WeeWX archive. > Are you sure you want to proceed (y/n)? y > Unique records processed: 2; Last timestamp: 2023-08-28 18:35:00 AEST > (1693211700) > Mapped data saved to archive successfully ""for period 1. > Calculating missing derived observations ... > Processing record: 2; Last record: 2023-08-29 00:00:00 AEST (1693231200) > Recalculating daily summaries... > > Finished recalculating daily summaries > Finished calculating missing derived observations > Finished import > 2 records were processed and 2 unique records imported in 2.87 seconds. > Those records with a timestamp already in the archive will not have been > imported. Confirm successful import in the WeeWX log file. > > These warnings are inconsequential for your import, but they make me > wonder if you have posted the correct output for the command/config/data > specified? Importantly, in my test the data was imported as expected: > > $ sqlite3 /home/weewx/archive/weewx.sdb > SQLite version 3.37.2 2022-01-06 13:25:41 > Enter ".help" for usage hints. > sqlite> SELECT dateTime,datetime(dateTime, 'unixepoch', > 'localtime'),outTemp,rain,rainRate FROM archive ORDER BY dateTime DESC > LIMIT 6; > 1693340400|2023-08-30 06:20:00|32.1130389185491|0.0|0.0 > 1693340100|2023-08-30 06:15:00|32.3124634933352|0.0|0.0 > 1693339800|2023-08-30 06:10:00|32.5203063775446|0.0|0.0 > 1693339500|2023-08-30 06:05:00|32.6532104624026|0.0|0.0 > 1693211700|2023-08-28 18:35:00|60.8|0.0078740157480315|0.2362204722 > 1693211400|2023-08-28 18:30:00|63.68|0.0118110236220472|0.14173228332 > sqlite> > > (note the fields are in the native units (US customary) for my database ie > rain and rainRate values are in inches and inches/hour) > > Also, the command and response specified in your posts include the use of > the --dry-run command line option; if --dry-run is used data will never > be saved to database - that is the point of --dry-run. Have you run the > same command without --dry-run? If not I suggest you try that and if it > does not import correctly post the exact command line used and the exact > output received. If you have run wee_import without --dry-run please post > the corresponding wee_import output. > > One other thing, how are you determining that the data was not imported? > The most reliable way is to directly query the database, if you are relying > upon WeeWX generated output this can be unreliable/untimely due to caching, > delays in generation etc. > > Gary > > On Tuesday, 29 August 2023 at 22:34:05 UTC+10 remy.l...@gmail.com wrote: > >> Hi Gary, >> I followed your advice: >> >> 1/ I now have in the rain field of my csv file the amount of rain that >> fell over the previous 5 minutes. >> 2/ I changed the config file so that rain=discrete is now the config value >> >> Unfortunately, the integration of the csv file in the weewx database with >> this new configuration did not solve the problem, the rain and rainRate >> fields systematically remain at zero while the integration of all the other >> fields works wonderfully! >> Do you think there is something wrong or is it a weewx bug? >> >> >> *My new configuration file :* >> * rain = discrete* >> wind_direction = 0,360 >> >> [[FieldMap]] >> dateTime = dateTime, unix_epoch >> barometer = barometer, hPa >> inTemp = inTemp, degree_C >> outTemp = outTemp, degree_C >> inHumidity = inHumidity, percent >> outHumidity = outHumidity, percent >> windSpeed = windSpeed, km_per_hour >> windDir = windDir, degree_compass >> windGust = windGust, km_per_hour >> >> * rainRate = rainRate, mm_per_hour rain = rain, >> mm* >> dewpoint = dewpoint, degree_C >> windchill = windchill, degree_C >> radiation = radiation, watt_per_meter_squared >> UV = UV, uv_index >> extraTemp1 = extraTemp1, degree_C >> >> *Example of CSV file with rain & rainRate *: >> >> dateTime,outTemp,outHumidity,dewpoint,appTemp,barometer,windchill,windSpeed,windDir,windGust >> *,rain,rainRate* >> 2023-08-28 18:30:00,17.6,66.0,11.3,13.8,1009.5,17.6,3.8,0,6.1,*0.3,3.6* >> 2023-08-28 18:35:00,16.0,80.0,12.6,13.7,1009.4,16.0,3.8,1,7.1,*0.2,6.0* >> >> >> *Result of the test :* >> sudo wee_import --config=/etc/weewx/weewx.conf >> --import-config=/home/pi/Documents/Bresser7in1/csv-Bresser.conf --dry-run >> --verbose >> Using WeeWX configuration file /etc/weewx/weewx.conf >> Starting wee_import... >> A CSV import from source file '/home/pi/Documents/Bresser7in1/data5.csv' >> has been requested. >> The following options will be used: >> config=/etc/weewx/weewx.conf, >> import-config=/home/pi/Documents/Bresser7in1/csv-Bresser.conf >> source=/home/pi/Documents/Bresser7in1/data5.csv, from=None, to=None >> dry-run=True, calc_missing=True, ignore_invalid_data=True >> tranche=250, interval=conf, date/time_string_format=%Y-%m-%d %H:%M:%S >> delimiter=',', *rain=discrete,* wind_direction=[0.0, 360.0] >> >> UV=True, radiation=True >> Using database binding 'wx_binding', which is bound to database >> 'weewx.sdb' >> Destination table 'archive' unit system is '0x01' (US). >> Missing derived observations will be calculated. >> This is a dry run, imported data will not be saved to archive. >> Starting dry run import ... >> Obtaining raw import data for period 1 ... >> The following imported field-to-WeeWX field map will be used: >> source field 'dateTime' in units 'unix_epoch' --> WeeWX field >> 'dateTime' >> source field 'barometer' in units 'hPa' --> WeeWX field 'barometer' >> source field 'inTemp' in units 'degree_C' --> WeeWX field 'inTemp' >> source field 'outTemp' in units 'degree_C' --> WeeWX field 'outTemp' >> source field 'inHumidity' in units 'percent' --> WeeWX field >> 'inHumidity' >> source field 'outHumidity' in units 'percent' --> WeeWX field >> 'outHumidity' >> source field 'windSpeed' in units 'km_per_hour' --> WeeWX field >> 'windSpeed' >> source field 'windDir' in units 'degree_compass' --> WeeWX field >> 'windDir' >> source field 'windGust' in units 'km_per_hour' --> WeeWX field >> 'windGust' >> >> >> * source field 'rainRate' in units 'mm_per_hour' --> WeeWX field >> 'rainRate' source field 'rain' in units 'mm' --> WeeWX field 'rain'* >> source field 'dewpoint' in units 'degree_C' --> WeeWX field 'dewpoint' >> source field 'windchill' in units 'degree_C' --> WeeWX field >> 'windchill' >> source field 'radiation' in units 'watt_per_meter_squared' --> WeeWX >> field 'radiation' >> source field 'UV' in units 'uv_index' --> WeeWX field 'UV' >> source field 'extraTemp1' in units 'degree_C' --> WeeWX field >> 'extraTemp1' >> source field 'extraTemp2' in units 'degree_C' --> WeeWX field >> 'extraTemp2' >> source field 'extraTemp3' in units 'degree_C' --> WeeWX field >> 'extraTemp3' >> source field 'extraTemp4' in units 'degree_C' --> WeeWX field >> 'extraTemp4' >> source field 'extraTemp6' in units 'degree_C' --> WeeWX field >> 'extraTemp6' >> source field 'extraHumid1' in units 'percent' --> WeeWX field >> 'extraHumid1' >> source field 'extraHumid2' in units 'percent' --> WeeWX field >> 'extraHumid2' >> source field 'extraHumid3' in units 'percent' --> WeeWX field >> 'extraHumid3' >> source field 'extraHumid4' in units 'percent' --> WeeWX field >> 'extraHumid4' >> *Raw import data read successfully for period 1.* >> Mapping raw import data for period 1 ... >> Mapped 1 records. >> *Raw import data mapped successfully for period 1.* >> 1 records identified for import. >> Unique records processed: 1; Last timestamp: 2023-08-29 14:30:00 CEST >> (1693312200) >> >> Finished dry run import >> 1 records were processed and 1 unique records would have been imported. >> >> *Rémy LAVABRE* >> >> >> Le dim. 27 août 2023 à 05:02, gjr80 <gjrod...@gmail.com> a écrit : >> >>> If you are importing a single record and using rain = cumulative the >>> imported rain and rainRate values will always be zero, this is expected >>> behaviour. The reason being that WeeWX records rainfall data on a per >>> archive interval basis; in other words each archive record contains the >>> rainfall that was recorded in that archive period, for example in the last >>> five minutes if using a five minute archive interval. When the import >>> source rainfall data is cumulative or a running total wee_import >>> calculates the rain field value as the difference in this cumulative or >>> running total value between successive records. The If there is only one >>> record then this difference cannot be calculated and hence rain is >>> recorded as zero. Likewise for rainRate; if there is no rain data then >>> there can be no rainRate. >>> >>> The use of the rain config option in a CSV import config file is >>> covered here <http://weewx.com/docs.html/latest/utilities.htm#csv_rain> >>> in the Utilities Guide <http://weewx.com/docs.html/latest/utilities.htm> >>> . >>> >>> If you must import a single record at a time you must convert your >>> source rain data to be a per-record value and import using rain = >>> discrete. wee_import was not designed with the regular import of single >>> records in mind, rather it was designed for the bulk import of >>> days/months/years of data. >>> >>> Gary >>> On Sunday, 27 August 2023 at 04:34:16 UTC+10 remy.l...@gmail.com wrote: >>> >>>> >>>> >>>> >>>> *Good morning,I am using the wee_import.py utility to integrate AWEKAS >>>> data into my weewx database.command executed: *sudo wee_import >>>> --config=/etc/weewx/weewx.conf >>>> --import-config=/home/pi/Documents/Bresser7in1/csv-Bresser.conf --dry-run >>>> --verbose >>>> >>>> *My CSVFile :* >>>> >>>> >>>> dateTime,outTemp,outHumidity,dewpoint,appTemp,barometer,windchill,windSpeed,windDir,windGust, >>>> *rain,rainRate*,UV,radiation,inTemp,inHumidity,extraTemp1 >>>> 2023-08-26 16:35:00,23.6,61.0,15.7,18.4,1010.6,23.6,3.2,214,3.9, >>>> *6.9,0,*1.1,101.0,26.1,53.0,24.9,22.0 >>>> >>>> >>>> >>>> *My wee_import configuration file:* >>>> >>>> source = CSV >>>> >>>> >>>> ############################################################################## >>>> >>>> [CSV] >>>> file = /home/pi/Documents/Bresser7in1/data5.csv >>>> delimiter = ',' >>>> decimal = '.' >>>> interval = conf >>>> qc = True >>>> calc_missing = True >>>> ignore_invalid_data = True >>>> tranche = 250 >>>> UV_sensor = True >>>> solar_sensor = True >>>> raw_datetime_format = %Y-%m-%d %H:%M:%S >>>> >>>> rain = cumulative >>>> wind_direction = 0,360 >>>> >>>> >>>> [[FieldMap]] >>>> dateTime = dateTime, unix_epoch >>>> barometer = barometer, hPa >>>> inTemp = inTemp, degree_C >>>> outTemp = outTemp, degree_C >>>> inHumidity = inHumidity, percent >>>> outHumidity = outHumidity, percent >>>> windSpeed = windSpeed, km_per_hour >>>> windDir = windDir, degree_compass >>>> windGust = windGust, km_per_hour >>>> >>>> * rainRate = rainRate, mm_per_hour rain = rain, mm* >>>> dewpoint = dewpoint, degree_C >>>> windchill = windchill, degree_C >>>> radiation = radiation, watt_per_meter_squared >>>> UV = UV, uv_index >>>> extraTemp1 = extraTemp1, degree_C >>>> extraTemp2 = extraTemp2, degree_C >>>> extraTemp3 = extraTemp3, degree_C >>>> extraTemp4 = extraTemp4, degree_C >>>> extraTemp6 = extraTemp6, degree_C >>>> extraHumid1 = extraHumid1, percent >>>> extraHumid2 = extraHumid2, percent >>>> extraHumid3 = extraHumid3, percent >>>> extraHumid4 = extraHumid4, percent >>>> >>>> >>>> >>>> *Response from WeeWX :* >>>> >>>> Using WeeWX configuration file /etc/weewx/weewx.conf >>>> Starting wee_import... >>>> A CSV import from source file >>>> '/home/pi/Documents/Bresser7in1/data5.csv' has been requested. >>>> The following options will be used: >>>> config=/etc/weewx/weewx.conf, >>>> import-config=/home/pi/Documents/Bresser7in1/csv-Bresser.conf >>>> source=/home/pi/Documents/Bresser7in1/data5.csv, from=None, to=None >>>> dry-run=True, calc_missing=True, ignore_invalid_data=True >>>> tranche=250, interval=conf, date/time_string_format=%Y-%m-%d >>>> %H:%M:%S >>>> delimiter=',', rain=cumulative, wind_direction=[0.0, 360.0] >>>> UV=True, radiation=True >>>> Using database binding 'wx_binding', which is bound to database >>>> 'weewx.sdb' >>>> Destination table 'archive' unit system is '0x01' (US). >>>> Missing derived observations will be calculated. >>>> This is a dry run, imported data will not be saved to archive. >>>> Starting dry run import ... >>>> Obtaining raw import data for period 1 ... >>>> The following imported field-to-WeeWX field map will be used: >>>> source field 'dateTime' in units 'unix_epoch' --> WeeWX field >>>> 'dateTime' >>>> source field 'barometer' in units 'hPa' --> WeeWX field 'barometer' >>>> source field 'inTemp' in units 'degree_C' --> WeeWX field 'inTemp' >>>> source field 'outTemp' in units 'degree_C' --> WeeWX field >>>> 'outTemp' >>>> source field 'inHumidity' in units 'percent' --> WeeWX field >>>> 'inHumidity' >>>> source field 'outHumidity' in units 'percent' --> WeeWX field >>>> 'outHumidity' >>>> source field 'windSpeed' in units 'km_per_hour' --> WeeWX field >>>> 'windSpeed' >>>> source field 'windDir' in units 'degree_compass' --> WeeWX field >>>> 'windDir' >>>> source field 'windGust' in units 'km_per_hour' --> WeeWX field >>>> 'windGust' >>>> >>>> * source field 'rainRate' in units 'mm_per_hour' --> WeeWX field >>>> 'rainRate' source field 'rain' in units 'mm' --> WeeWX field 'rain'* >>>> source field 'dewpoint' in units 'degree_C' --> WeeWX field >>>> 'dewpoint' >>>> source field 'windchill' in units 'degree_C' --> WeeWX field >>>> 'windchill' >>>> source field 'radiation' in units 'watt_per_meter_squared' --> >>>> WeeWX field 'radiation' >>>> source field 'UV' in units 'uv_index' --> WeeWX field 'UV' >>>> source field 'extraTemp1' in units 'degree_C' --> WeeWX field >>>> 'extraTemp1' >>>> source field 'extraTemp2' in units 'degree_C' --> WeeWX field >>>> 'extraTemp2' >>>> source field 'extraTemp3' in units 'degree_C' --> WeeWX field >>>> 'extraTemp3' >>>> source field 'extraTemp4' in units 'degree_C' --> WeeWX field >>>> 'extraTemp4' >>>> source field 'extraTemp6' in units 'degree_C' --> WeeWX field >>>> 'extraTemp6' >>>> source field 'extraHumid1' in units 'percent' --> WeeWX field >>>> 'extraHumid1' >>>> source field 'extraHumid2' in units 'percent' --> WeeWX field >>>> 'extraHumid2' >>>> source field 'extraHumid3' in units 'percent' --> WeeWX field >>>> 'extraHumid3' >>>> source field 'extraHumid4' in units 'percent' --> WeeWX field >>>> 'extraHumid4' >>>> Raw import data read successfully for period 1. >>>> Mapping raw import data for period 1 ... >>>> Mapped 1 records. >>>> Raw import data mapped successfully for period 1. >>>> 1 records identified for import. >>>> Unique records processed: 1; Last timestamp: 2023-08-26 16:55:00 CEST >>>> (1693061700) >>>> Finished dry run import >>>> 1 records were processed and 1 unique records would have been imported. >>>> >>>> >>>> >>>> >>>> *Everything seems to be correct.However, when I run the command without >>>> the --dry run option, ALL the fields are correctly entered in the weewx >>>> database except SYSTEMATIC "rain" and "rainRate" which always remain at 0!I >>>> don't understand why these two fields are never updated in weewx database. >>>> Here for example rain is at 6.9 (since 00H00) and in the database rain >>>> remains at 0 (option cumulative actived). Same thing for rainRate! And >>>> these are the only two fields that pose a problem!Is it a wee_import.py >>>> bug? Does anyone have a solution... Thanks a lot!* >>>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "weewx-user" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/weewx-user/nWTJnx2UNlU/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> weewx-user+...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/weewx-user/0a5739b7-436c-4ccf-b170-576969ae6357n%40googlegroups.com >>> <https://groups.google.com/d/msgid/weewx-user/0a5739b7-436c-4ccf-b170-576969ae6357n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to weewx-user+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-user/80122a9d-9540-41a1-b6b4-77f50fb4448dn%40googlegroups.com > <https://groups.google.com/d/msgid/weewx-user/80122a9d-9540-41a1-b6b4-77f50fb4448dn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CADhm9YfVoDhor%2Btu-3CY%3D8fnZ7Btyi3o-zDaerWWc9PqUj%3DPAQ%40mail.gmail.com.
