Docs are at https://www.weewx.com/docs/5.0/utilities/weectl-database/ but what I did for a pip installation using Blaine's db was:
source ~/weewx-venv/bin/activate weectl database add-column appTemp weectl calc-missing (answer y when prompted above) If you have a packaged install of weewx you can skip the activate step above. Note - the calc-missing takes quite a while and will peg your cpu while working, so if you have a lot of data so you might need to use the --from and --to options or even the --tranche option to split up the processing a bit into pieces. I got lucky with this db on pi3plus with a bit of patience waiting for calc-missing to complete.... (weewx-venv) pi@pi3plus:~/weewx-data $ weectl database calc-missing -y Using configuration file /home/pi/weewx-data/weewx.conf Missing derived observations will be calculated for all records. Calculating missing derived observations... Processing record: 959536; Last record: 2024-01-19 00:00:00 PST (1705651200) Recalculating daily summaries... Records processed: 959000; time: 2024-01-16 15:55:00 PST (1705449300) Finished recalculating daily summaries Missing derived observations calculated in 2986.21 seconds And after restarting weewx things look good !!!! 2024-01-18T19:45:15.438555-08:00 pi3plus weewxd[1797]: INFO weewx.manager: Added record 2024-01-18 19:45:00 PST (1705635900) to database 'weewx.sdb' 2024-01-18T19:45:15.458841-08:00 pi3plus weewxd[1797]: INFO weewx.manager: Added record 2024-01-18 19:45:00 PST (1705635900) to daily summary in 'weewx.sdb' 2024-01-18T19:45:19.659002-08:00 pi3plus weewxd[1797]: INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 3.92 seconds 2024-01-18T19:45:20.884377-08:00 pi3plus weewxd[1797]: INFO weewx.imagegenerator: Generated 15 images for report SeasonsReport in 1.14 seconds 2024-01-18T19:45:20.893901-08:00 pi3plus weewxd[1797]: INFO weewx.reportengine: Copied 5 files to /home/pi/weewx-data/public_html 2024-01-18T19:45:20.967437-08:00 pi3plus weewxd[1797]: INFO user.belchertown: version 1.3.1 2024-01-18T19:45:37.910258-08:00 pi3plus weewxd[1797]: INFO weewx.cheetahgenerator: Generated 12 files for report Belchertown in 16.96 seconds 2024-01-18T19:45:37.947603-08:00 pi3plus weewxd[1797]: INFO weewx.reportengine: Copied 40 files to /home/pi/weewx-data/public_html/belchertown 2024-01-18T19:50:15.410986-08:00 pi3plus weewxd[1797]: INFO weewx.manager: Added record 2024-01-18 19:50:00 PST (1705636200) to database 'weewx.sdb' 2024-01-18T19:50:15.438866-08:00 pi3plus weewxd[1797]: INFO weewx.manager: Added record 2024-01-18 19:50:00 PST (1705636200) to daily summary in 'weewx.sdb' 2024-01-18T19:50:18.223483-08:00 pi3plus weewxd[1797]: INFO weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 2.65 seconds 2024-01-18T19:50:19.311313-08:00 pi3plus weewxd[1797]: INFO weewx.imagegenerator: Generated 15 images for report SeasonsReport in 1.08 seconds 2024-01-18T19:50:19.313932-08:00 pi3plus weewxd[1797]: INFO weewx.reportengine: Copied 0 files to /home/pi/weewx-data/public_html 2024-01-18T19:50:31.266819-08:00 pi3plus weewxd[1797]: INFO weewx.cheetahgenerator: Generated 12 files for report Belchertown in 11.89 seconds 2024-01-18T19:50:31.277235-08:00 pi3plus weewxd[1797]: INFO weewx.reportengine: Copied 3 files to /home/pi/weewx-data/public_html/belchertown So the commands above 'did' fix this particular db. (thanks Tom) On Thursday, January 18, 2024 at 6:21:01 PM UTC-8 Richard Whitcombe wrote: > Can you explain in laymans terms how to do that? > Do you mean add a new DB field using SQL or similar and leave it blank or > something else? > > Or outline trouble shooting steps to isolate the cause in simple terms (ie > familiar with linux and the basics of weewx but not the core detailed > functionality). > > On Friday 19 January 2024 at 08:12:14 UTC+9 Tom Keffer wrote: > >> Using Blaine's database, I was able to isolate the performance problems. >> >> It's in the template records/index.html.tmpl >> >> Specifically, apparent temperature (appTemp). It does not appear in the >> database, so a simple tag such as $alltime.appTemp.max requires searching >> the engine database, calculating apparent temperature for each record, then >> finding the max value. >> >> In V4, this would not have even been attempted. >> >> The solution is to add appTemp to the database. >> >> On Thu, Jan 18, 2024 at 1:33 PM vince <vince...@gmail.com> wrote: >> >>> Tom - the NOAA stuff builds from this db ok for me v5 pip on pi3+. >>> The Seasons skin NOAA file for 2024-01 'is' being updated every archive >>> period. >>> The historical files dating back to 2014 generated fine. >>> >>> Unrelated but interesting..... >>> >>> - this db shows the Belchertown taking too long issue >>> - running htop I see it peg two cpus and StdReport aborts due to >>> time vs. the Simulator 300 sec on a pi3+ >>> - Belchertown 'always' takes longer than 300 secs and weewx always >>> aborts due to StdReport not completing in 300 secs >>> - yet the same installation with 'my' db works fine with no issues, >>> so it 'has' to be content of the db being used doesn't it ? >>> - I have been unable to get Belchertown to succeed even at 600 secs >>> archive period. >>> >>> Question - why would the db be 'locked' if a skin is reading it in order >>> to process its contents ?????? >>> >>> last restart/failure... >>> >>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO __main__: Starting up weewx >>> version 5.0.0 >>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Clock error is >>> -0.21 seconds (positive is fast) >>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Using binding >>> 'wx_binding' to database 'weewx.sdb' >>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.manager: Starting >>> backfill of daily summaries >>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.manager: Daily >>> summaries up to date >>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Starting main >>> packet loop. >>> Jan 18 13:20:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record >>> 2024-01-18 13:20:00 PST (1705612800) to database 'weewx.sdb' >>> Jan 18 13:20:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record >>> 2024-01-18 13:20:00 PST (1705612800) to daily summary in 'weewx.sdb' >>> Jan 18 13:20:19 pi3plus weewxd[1756]: INFO weewx.cheetahgenerator: >>> Generated 8 files for report SeasonsReport in 3.96 seconds >>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO weewx.imagegenerator: >>> Generated 30 images for report SeasonsReport in 4.80 seconds >>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO weewx.reportengine: Copied 5 >>> files to /home/pi/weewx-data/public_html >>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO user.belchertown: version >>> 1.3.1 >>> Jan 18 13:30:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record >>> 2024-01-18 13:30:00 PST (1705613400) to database 'weewx.sdb' >>> Jan 18 13:30:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record >>> 2024-01-18 13:30:00 PST (1705613400) to daily summary in 'weewx.sdb' >>> Jan 18 13:30:20 pi3plus weewxd[1756]: INFO weewx.engine: Main loop >>> exiting. Shutting engine down. >>> Jan 18 13:30:20 pi3plus weewxd[1756]: INFO weewx.engine: Shutting down >>> StdReport thread >>> Jan 18 13:30:40 pi3plus weewxd[1756]: ERROR weewx.engine: Unable to shut >>> down StdReport thread >>> Jan 18 13:30:40 pi3plus weewxd[1756]: CRITICAL __main__: Database >>> OperationalError exception: database is locked >>> Jan 18 13:30:40 pi3plus weewxd[1756]: CRITICAL __main__: **** >>> Waiting 2 minutes then retrying... >>> Jan 18 13:30:42 pi3plus weewxd[1756]: INFO weewx.cheetahgenerator: >>> Generated 12 files for report Belchertown in 617.64 seconds >>> Jan 18 13:30:42 pi3plus weewxd[1756]: INFO weewx.reportengine: Copied 40 >>> files to /home/pi/weewx-data/public_html/belchertown >>> >>> >>> weewx related open files: >>> >>> pi@pi3plus:~/weewx-data $ lsof|grep weew >>> python3 1633 pi mem REG 179,2 >>> 217360 272588 >>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf >>> python3 1633 pi mem REG 179,2 >>> 224592 272587 /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf >>> python3 1633 pi mem REG 179,2 >>> 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ >>> imaging.cpython-311-arm-linux-gnueabihf.so >>> python3 1633 pi mem REG 179,2 >>> 2528660 271317 /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_ >>> libastro.cpython-311-arm-linux-gnueabihf.so >>> python3 1633 pi mem REG 179,2 >>> 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ >>> imagingmath.cpython-311-arm-linux-gnueabihf.so >>> python3 1633 pi mem REG 179,2 >>> 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ >>> imagingft.cpython-311-arm-linux-gnueabihf.so >>> python3 1633 pi mem REG 179,2 >>> 45064 271812 /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_ >>> namemapper.cpython-311-arm-linux-gnueabihf.so >>> >>> When its cpus are pegged trying to process Belchertown: >>> >>> python3 1633 pi cwd DIR 179,2 >>> 4096 272808 /home/pi/weewx-data/skins/Belchertown >>> python3 1633 pi mem REG 179,2 >>> 217360 272588 >>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf >>> python3 1633 pi mem REG 179,2 >>> 224592 272587 /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf >>> python3 1633 pi mem REG 179,2 >>> 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ >>> imaging.cpython-311-arm-linux-gnueabihf.so >>> python3 1633 pi mem REG 179,2 >>> 2528660 271317 /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_ >>> libastro.cpython-311-arm-linux-gnueabihf.so >>> python3 1633 pi mem REG 179,2 >>> 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ >>> imagingmath.cpython-311-arm-linux-gnueabihf.so >>> python3 1633 pi mem REG 179,2 >>> 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ >>> imagingft.cpython-311-arm-linux-gnueabihf.so >>> python3 1633 pi mem REG 179,2 >>> 45064 271812 /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_ >>> namemapper.cpython-311-arm-linux-gnueabihf.so >>> python3 1633 pi 4uw REG 179,2 >>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb >>> python3 1633 pi 5uw REG 179,2 >>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb >>> python3 1633 pi 6u REG 179,2 >>> 87200 275903 /home/pi/weewx-data/archive/weewx.sdb-journal >>> python3 1633 1853 python3 pi cwd DIR 179,2 >>> 4096 272808 /home/pi/weewx-data/skins/Belchertown >>> python3 1633 1853 python3 pi mem REG 179,2 >>> 217360 272588 >>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf >>> python3 1633 1853 python3 pi mem REG 179,2 >>> 224592 272587 /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf >>> python3 1633 1853 python3 pi mem REG 179,2 >>> 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ >>> imaging.cpython-311-arm-linux-gnueabihf.so >>> python3 1633 1853 python3 pi mem REG 179,2 >>> 2528660 271317 /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_ >>> libastro.cpython-311-arm-linux-gnueabihf.so >>> python3 1633 1853 python3 pi mem REG 179,2 >>> 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ >>> imagingmath.cpython-311-arm-linux-gnueabihf.so >>> python3 1633 1853 python3 pi mem REG 179,2 >>> 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ >>> imagingft.cpython-311-arm-linux-gnueabihf.so >>> python3 1633 1853 python3 pi mem REG 179,2 >>> 45064 271812 /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_ >>> namemapper.cpython-311-arm-linux-gnueabihf.so >>> python3 1633 1853 python3 pi 4uw REG 179,2 >>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb >>> python3 1633 1853 python3 pi 5uw REG 179,2 >>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb >>> python3 1633 1853 python3 pi 6u REG 179,2 >>> 87200 275903 /home/pi/weewx-data/archive/weewx.sdb-journal >>> >>> >>> On Thursday, January 18, 2024 at 12:18:18 PM UTC-8 Blaine wrote: >>> >>>> Thank you for the reply! Yes, I have tried this many times. It >>>> continues to regenerate them with the missing data. >>>> >>>> On Wednesday, January 17, 2024 at 6:29:41 PM UTC-8 Tom Keffer wrote: >>>> >>>> Have you tried completely deleting all NOAA reports and letting weewx >>>> regenerate them? >>>> >>>> On Wed, Jan 17, 2024 at 9:32 AM Blaine <bgri...@gmail.com> wrote: >>>> >>>> Way back in 2020 my NOAA reports stopped populating their daily values. >>>> The average values for the month continue to be populated at the bottom of >>>> the table. Since 2020 every year or so I have spent hours banging my head >>>> against the wall following any and all instructions on the wiki, github, >>>> google and this user group to attempt to repair this issue. None of the >>>> published instructions seem to help. I have never been successful. >>>> >>>> If this is a problem that can be solved I would be more than happy to >>>> pay someone to help fix this issue knowing full well there are no >>>> guarantees recovery is possible. This data is important to me and having >>>> essentially lost years worth with no chance of recovery is a real bummer. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "weewx-user" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to weewx-user+...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/weewx-user/6be270a2-961f-4952-85b0-6074332671een%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/weewx-user/6be270a2-961f-4952-85b0-6074332671een%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "weewx-user" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to weewx-user+...@googlegroups.com. >>> >> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/weewx-user/2bed5055-542d-4ef2-8e32-312e306af59cn%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/weewx-user/2bed5055-542d-4ef2-8e32-312e306af59cn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/c35e4cd9-8a75-4f05-8d0e-d60f301d97a9n%40googlegroups.com.
