Using Blaine's database, I was able to isolate the performance problems. It's in the template records/index.html.tmpl
Specifically, apparent temperature (appTemp). It does not appear in the database, so a simple tag such as $alltime.appTemp.max requires searching the engine database, calculating apparent temperature for each record, then finding the max value. In V4, this would not have even been attempted. The solution is to add appTemp to the database. On Thu, Jan 18, 2024 at 1:33 PM vince <vinceska...@gmail.com> wrote: > Tom - the NOAA stuff builds from this db ok for me v5 pip on pi3+. > The Seasons skin NOAA file for 2024-01 'is' being updated every archive > period. > The historical files dating back to 2014 generated fine. > > Unrelated but interesting..... > > - this db shows the Belchertown taking too long issue > - running htop I see it peg two cpus and StdReport aborts due to time > vs. the Simulator 300 sec on a pi3+ > - Belchertown 'always' takes longer than 300 secs and weewx always > aborts due to StdReport not completing in 300 secs > - yet the same installation with 'my' db works fine with no issues, so > it 'has' to be content of the db being used doesn't it ? > - I have been unable to get Belchertown to succeed even at 600 secs > archive period. > > Question - why would the db be 'locked' if a skin is reading it in order > to process its contents ?????? > > last restart/failure... > > Jan 18 13:15:10 pi3plus weewxd[1756]: INFO __main__: Starting up weewx > version 5.0.0 > Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Clock error is > -0.21 seconds (positive is fast) > Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Using binding > 'wx_binding' to database 'weewx.sdb' > Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.manager: Starting > backfill of daily summaries > Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.manager: Daily summaries > up to date > Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Starting main > packet loop. > Jan 18 13:20:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record > 2024-01-18 13:20:00 PST (1705612800) to database 'weewx.sdb' > Jan 18 13:20:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record > 2024-01-18 13:20:00 PST (1705612800) to daily summary in 'weewx.sdb' > Jan 18 13:20:19 pi3plus weewxd[1756]: INFO weewx.cheetahgenerator: > Generated 8 files for report SeasonsReport in 3.96 seconds > Jan 18 13:20:24 pi3plus weewxd[1756]: INFO weewx.imagegenerator: Generated > 30 images for report SeasonsReport in 4.80 seconds > Jan 18 13:20:24 pi3plus weewxd[1756]: INFO weewx.reportengine: Copied 5 > files to /home/pi/weewx-data/public_html > Jan 18 13:20:24 pi3plus weewxd[1756]: INFO user.belchertown: version 1.3.1 > Jan 18 13:30:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record > 2024-01-18 13:30:00 PST (1705613400) to database 'weewx.sdb' > Jan 18 13:30:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record > 2024-01-18 13:30:00 PST (1705613400) to daily summary in 'weewx.sdb' > Jan 18 13:30:20 pi3plus weewxd[1756]: INFO weewx.engine: Main loop > exiting. Shutting engine down. > Jan 18 13:30:20 pi3plus weewxd[1756]: INFO weewx.engine: Shutting down > StdReport thread > Jan 18 13:30:40 pi3plus weewxd[1756]: ERROR weewx.engine: Unable to shut > down StdReport thread > Jan 18 13:30:40 pi3plus weewxd[1756]: CRITICAL __main__: Database > OperationalError exception: database is locked > Jan 18 13:30:40 pi3plus weewxd[1756]: CRITICAL __main__: **** Waiting > 2 minutes then retrying... > Jan 18 13:30:42 pi3plus weewxd[1756]: INFO weewx.cheetahgenerator: > Generated 12 files for report Belchertown in 617.64 seconds > Jan 18 13:30:42 pi3plus weewxd[1756]: INFO weewx.reportengine: Copied 40 > files to /home/pi/weewx-data/public_html/belchertown > > > weewx related open files: > > pi@pi3plus:~/weewx-data $ lsof|grep weew > python3 1633 pi mem REG 179,2 > 217360 272588 > /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf > python3 1633 pi mem REG 179,2 > 224592 272587 /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf > python3 1633 pi mem REG 179,2 > 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ > imaging.cpython-311-arm-linux-gnueabihf.so > python3 1633 pi mem REG 179,2 > 2528660 271317 /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_ > libastro.cpython-311-arm-linux-gnueabihf.so > python3 1633 pi mem REG 179,2 > 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ > imagingmath.cpython-311-arm-linux-gnueabihf.so > python3 1633 pi mem REG 179,2 > 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ > imagingft.cpython-311-arm-linux-gnueabihf.so > python3 1633 pi mem REG 179,2 > 45064 271812 /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_ > namemapper.cpython-311-arm-linux-gnueabihf.so > > When its cpus are pegged trying to process Belchertown: > > python3 1633 pi cwd DIR 179,2 > 4096 272808 /home/pi/weewx-data/skins/Belchertown > python3 1633 pi mem REG 179,2 > 217360 272588 > /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf > python3 1633 pi mem REG 179,2 > 224592 272587 /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf > python3 1633 pi mem REG 179,2 > 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ > imaging.cpython-311-arm-linux-gnueabihf.so > python3 1633 pi mem REG 179,2 > 2528660 271317 /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_ > libastro.cpython-311-arm-linux-gnueabihf.so > python3 1633 pi mem REG 179,2 > 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ > imagingmath.cpython-311-arm-linux-gnueabihf.so > python3 1633 pi mem REG 179,2 > 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ > imagingft.cpython-311-arm-linux-gnueabihf.so > python3 1633 pi mem REG 179,2 > 45064 271812 /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_ > namemapper.cpython-311-arm-linux-gnueabihf.so > python3 1633 pi 4uw REG 179,2 > 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb > python3 1633 pi 5uw REG 179,2 > 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb > python3 1633 pi 6u REG 179,2 > 87200 275903 /home/pi/weewx-data/archive/weewx.sdb-journal > python3 1633 1853 python3 pi cwd DIR 179,2 > 4096 272808 /home/pi/weewx-data/skins/Belchertown > python3 1633 1853 python3 pi mem REG 179,2 > 217360 272588 > /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf > python3 1633 1853 python3 pi mem REG 179,2 > 224592 272587 /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf > python3 1633 1853 python3 pi mem REG 179,2 > 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ > imaging.cpython-311-arm-linux-gnueabihf.so > python3 1633 1853 python3 pi mem REG 179,2 > 2528660 271317 /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_ > libastro.cpython-311-arm-linux-gnueabihf.so > python3 1633 1853 python3 pi mem REG 179,2 > 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ > imagingmath.cpython-311-arm-linux-gnueabihf.so > python3 1633 1853 python3 pi mem REG 179,2 > 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_ > imagingft.cpython-311-arm-linux-gnueabihf.so > python3 1633 1853 python3 pi mem REG 179,2 > 45064 271812 /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_ > namemapper.cpython-311-arm-linux-gnueabihf.so > python3 1633 1853 python3 pi 4uw REG 179,2 > 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb > python3 1633 1853 python3 pi 5uw REG 179,2 > 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb > python3 1633 1853 python3 pi 6u REG 179,2 > 87200 275903 /home/pi/weewx-data/archive/weewx.sdb-journal > > > On Thursday, January 18, 2024 at 12:18:18 PM UTC-8 Blaine wrote: > >> Thank you for the reply! Yes, I have tried this many times. It continues >> to regenerate them with the missing data. >> >> On Wednesday, January 17, 2024 at 6:29:41 PM UTC-8 Tom Keffer wrote: >> >> Have you tried completely deleting all NOAA reports and letting weewx >> regenerate them? >> >> On Wed, Jan 17, 2024 at 9:32 AM Blaine <bgri...@gmail.com> wrote: >> >> Way back in 2020 my NOAA reports stopped populating their daily values. >> The average values for the month continue to be populated at the bottom of >> the table. Since 2020 every year or so I have spent hours banging my head >> against the wall following any and all instructions on the wiki, github, >> google and this user group to attempt to repair this issue. None of the >> published instructions seem to help. I have never been successful. >> >> If this is a problem that can be solved I would be more than happy to pay >> someone to help fix this issue knowing full well there are no guarantees >> recovery is possible. This data is important to me and having essentially >> lost years worth with no chance of recovery is a real bummer. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "weewx-user" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to weewx-user+...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/weewx-user/6be270a2-961f-4952-85b0-6074332671een%40googlegroups.com >> <https://groups.google.com/d/msgid/weewx-user/6be270a2-961f-4952-85b0-6074332671een%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> -- > You received this message because you are subscribed to the Google Groups > "weewx-user" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to weewx-user+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/weewx-user/2bed5055-542d-4ef2-8e32-312e306af59cn%40googlegroups.com > <https://groups.google.com/d/msgid/weewx-user/2bed5055-542d-4ef2-8e32-312e306af59cn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEDCMUPVw2GFaCgxg_NcGXPcmagL8KLwQPhuuH-%2BdDzSsQ%40mail.gmail.com.
