Re: the Belchertown issue many seem to be facing, I notice that Blaine's db
uses the original old wview schema, not the newer bigger 'wview_extended'
schema that came out a few years ago. I suspect this is the pattern we've
missed. Folks who have the old schema do not have any of the elements for
many of the Extended Observations in Belchertown's skin.conf file.
Blaine's db schema misses these items in Belchertown's skin.conf extended
observations list....
appTemp = Apparent Temperature
cloudbase = Cloud Base
visibility = Visibility
windrun = Wind Run
cloud_cover = Cloud Cover
aqi = AQI
The wview_extended schema is missing these, so I'd expect users with data
in those somehow might have the same slowness issue until they add db
elements and calc-missing to generate summary tables for them.....
visibility = Visibility
cloud_cover = Cloud Cover
Re: Michael's question about what to do about the NOAA files, I'd suggest
pre-calculating them offline and dropping them into place in his NOAA
subdirectory with the appropriate owner/group/mode so they don't have to be
recalculated. I'd omit the current month+year which should regenerate
quickly on weewx startup. That would be a good thing to try. We've
verified the db is ok, so it 'should' work on Blaine's system too
(hopefully). The test then would be to simply look at the date+time last
modified for the current month and year NOAA file. It should change every
archive period as StdReport runs.
(reminder - if you have Belchertown 'and' Seasons, you need to drop the
pre-calculated NOAA files into both places to preseed both skin output
directories)
On Thursday, January 18, 2024 at 8:36:20 PM UTC-8 michael.k...@gmx.at wrote:
> Or, to speed things up, do the "calc-missing" on another machine with a
> more potent CPU.
>
> Back to Blaine's problem: if the NOAA files are generated on other
> installations with his database, there is obviously another problem with
> his installation.I don't have an idea what to look for, if deleted NOAA
> data will be regenerated in the same way, as before.
>
> vince schrieb am Freitag, 19. Januar 2024 um 04:55:13 UTC+1:
>
>> Docs are at https://www.weewx.com/docs/5.0/utilities/weectl-database/
>> but what I did for a pip installation using Blaine's db was:
>>
>> source ~/weewx-venv/bin/activate
>> weectl database add-column appTemp
>> weectl calc-missing
>> (answer y when prompted above)
>>
>> If you have a packaged install of weewx you can skip the activate step
>> above.
>>
>> Note - the calc-missing takes quite a while and will peg your cpu while
>> working, so if you have a lot of data so you might need to use the --from
>> and --to options or even the --tranche option to split up the processing a
>> bit into pieces. I got lucky with this db on pi3plus with a bit of
>> patience waiting for calc-missing to complete....
>>
>> (weewx-venv) pi@pi3plus:~/weewx-data $ weectl database calc-missing -y
>> Using configuration file /home/pi/weewx-data/weewx.conf
>> Missing derived observations will be calculated for all records.
>> Calculating missing derived observations...
>> Processing record: 959536; Last record: 2024-01-19 00:00:00 PST
>> (1705651200)
>> Recalculating daily summaries...
>> Records processed: 959000; time: 2024-01-16 15:55:00 PST (1705449300)
>> Finished recalculating daily summaries
>> Missing derived observations calculated in 2986.21 seconds
>>
>> And after restarting weewx things look good !!!!
>>
>> 2024-01-18T19:45:15.438555-08:00 pi3plus weewxd[1797]: INFO
>> weewx.manager: Added record 2024-01-18 19:45:00 PST (1705635900) to
>> database 'weewx.sdb'
>> 2024-01-18T19:45:15.458841-08:00 pi3plus weewxd[1797]: INFO
>> weewx.manager: Added record 2024-01-18 19:45:00 PST (1705635900) to daily
>> summary in 'weewx.sdb'
>> 2024-01-18T19:45:19.659002-08:00 pi3plus weewxd[1797]: INFO
>> weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 3.92
>> seconds
>> 2024-01-18T19:45:20.884377-08:00 pi3plus weewxd[1797]: INFO
>> weewx.imagegenerator: Generated 15 images for report SeasonsReport in 1.14
>> seconds
>> 2024-01-18T19:45:20.893901-08:00 pi3plus weewxd[1797]: INFO
>> weewx.reportengine: Copied 5 files to /home/pi/weewx-data/public_html
>> 2024-01-18T19:45:20.967437-08:00 pi3plus weewxd[1797]: INFO
>> user.belchertown: version 1.3.1
>> 2024-01-18T19:45:37.910258-08:00 pi3plus weewxd[1797]: INFO
>> weewx.cheetahgenerator: Generated 12 files for report Belchertown in 16.96
>> seconds
>> 2024-01-18T19:45:37.947603-08:00 pi3plus weewxd[1797]: INFO
>> weewx.reportengine: Copied 40 files to
>> /home/pi/weewx-data/public_html/belchertown
>>
>> 2024-01-18T19:50:15.410986-08:00 pi3plus weewxd[1797]: INFO
>> weewx.manager: Added record 2024-01-18 19:50:00 PST (1705636200) to
>> database 'weewx.sdb'
>> 2024-01-18T19:50:15.438866-08:00 pi3plus weewxd[1797]: INFO
>> weewx.manager: Added record 2024-01-18 19:50:00 PST (1705636200) to daily
>> summary in 'weewx.sdb'
>> 2024-01-18T19:50:18.223483-08:00 pi3plus weewxd[1797]: INFO
>> weewx.cheetahgenerator: Generated 8 files for report SeasonsReport in 2.65
>> seconds
>> 2024-01-18T19:50:19.311313-08:00 pi3plus weewxd[1797]: INFO
>> weewx.imagegenerator: Generated 15 images for report SeasonsReport in 1.08
>> seconds
>> 2024-01-18T19:50:19.313932-08:00 pi3plus weewxd[1797]: INFO
>> weewx.reportengine: Copied 0 files to /home/pi/weewx-data/public_html
>> 2024-01-18T19:50:31.266819-08:00 pi3plus weewxd[1797]: INFO
>> weewx.cheetahgenerator: Generated 12 files for report Belchertown in 11.89
>> seconds
>> 2024-01-18T19:50:31.277235-08:00 pi3plus weewxd[1797]: INFO
>> weewx.reportengine: Copied 3 files to
>> /home/pi/weewx-data/public_html/belchertown
>>
>> So the commands above 'did' fix this particular db.
>> (thanks Tom)
>>
>> On Thursday, January 18, 2024 at 6:21:01 PM UTC-8 Richard Whitcombe wrote:
>>
>>> Can you explain in laymans terms how to do that?
>>> Do you mean add a new DB field using SQL or similar and leave it blank
>>> or something else?
>>>
>>> Or outline trouble shooting steps to isolate the cause in simple terms
>>> (ie familiar with linux and the basics of weewx but not the core detailed
>>> functionality).
>>>
>>> On Friday 19 January 2024 at 08:12:14 UTC+9 Tom Keffer wrote:
>>>
>>>> Using Blaine's database, I was able to isolate the performance problems.
>>>>
>>>> It's in the template records/index.html.tmpl
>>>>
>>>> Specifically, apparent temperature (appTemp). It does not appear in the
>>>> database, so a simple tag such as $alltime.appTemp.max requires searching
>>>> the engine database, calculating apparent temperature for each record,
>>>> then
>>>> finding the max value.
>>>>
>>>> In V4, this would not have even been attempted.
>>>>
>>>> The solution is to add appTemp to the database.
>>>>
>>>> On Thu, Jan 18, 2024 at 1:33 PM vince <vince...@gmail.com> wrote:
>>>>
>>>>> Tom - the NOAA stuff builds from this db ok for me v5 pip on pi3+.
>>>>> The Seasons skin NOAA file for 2024-01 'is' being updated every
>>>>> archive period.
>>>>> The historical files dating back to 2014 generated fine.
>>>>>
>>>>> Unrelated but interesting.....
>>>>>
>>>>> - this db shows the Belchertown taking too long issue
>>>>> - running htop I see it peg two cpus and StdReport aborts due to
>>>>> time vs. the Simulator 300 sec on a pi3+
>>>>> - Belchertown 'always' takes longer than 300 secs and weewx always
>>>>> aborts due to StdReport not completing in 300 secs
>>>>> - yet the same installation with 'my' db works fine with no
>>>>> issues, so it 'has' to be content of the db being used doesn't it ?
>>>>> - I have been unable to get Belchertown to succeed even at 600
>>>>> secs archive period.
>>>>>
>>>>> Question - why would the db be 'locked' if a skin is reading it in
>>>>> order to process its contents ??????
>>>>>
>>>>> last restart/failure...
>>>>>
>>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO __main__: Starting up weewx
>>>>> version 5.0.0
>>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Clock error
>>>>> is -0.21 seconds (positive is fast)
>>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Using binding
>>>>> 'wx_binding' to database 'weewx.sdb'
>>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.manager: Starting
>>>>> backfill of daily summaries
>>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.manager: Daily
>>>>> summaries up to date
>>>>> Jan 18 13:15:10 pi3plus weewxd[1756]: INFO weewx.engine: Starting main
>>>>> packet loop.
>>>>> Jan 18 13:20:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
>>>>> 2024-01-18 13:20:00 PST (1705612800) to database 'weewx.sdb'
>>>>> Jan 18 13:20:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
>>>>> 2024-01-18 13:20:00 PST (1705612800) to daily summary in 'weewx.sdb'
>>>>> Jan 18 13:20:19 pi3plus weewxd[1756]: INFO weewx.cheetahgenerator:
>>>>> Generated 8 files for report SeasonsReport in 3.96 seconds
>>>>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO weewx.imagegenerator:
>>>>> Generated 30 images for report SeasonsReport in 4.80 seconds
>>>>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO weewx.reportengine: Copied
>>>>> 5 files to /home/pi/weewx-data/public_html
>>>>> Jan 18 13:20:24 pi3plus weewxd[1756]: INFO user.belchertown: version
>>>>> 1.3.1
>>>>> Jan 18 13:30:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
>>>>> 2024-01-18 13:30:00 PST (1705613400) to database 'weewx.sdb'
>>>>> Jan 18 13:30:15 pi3plus weewxd[1756]: INFO weewx.manager: Added record
>>>>> 2024-01-18 13:30:00 PST (1705613400) to daily summary in 'weewx.sdb'
>>>>> Jan 18 13:30:20 pi3plus weewxd[1756]: INFO weewx.engine: Main loop
>>>>> exiting. Shutting engine down.
>>>>> Jan 18 13:30:20 pi3plus weewxd[1756]: INFO weewx.engine: Shutting down
>>>>> StdReport thread
>>>>> Jan 18 13:30:40 pi3plus weewxd[1756]: ERROR weewx.engine: Unable to
>>>>> shut down StdReport thread
>>>>> Jan 18 13:30:40 pi3plus weewxd[1756]: CRITICAL __main__: Database
>>>>> OperationalError exception: database is locked
>>>>> Jan 18 13:30:40 pi3plus weewxd[1756]: CRITICAL __main__: ****
>>>>> Waiting 2 minutes then retrying...
>>>>> Jan 18 13:30:42 pi3plus weewxd[1756]: INFO weewx.cheetahgenerator:
>>>>> Generated 12 files for report Belchertown in 617.64 seconds
>>>>> Jan 18 13:30:42 pi3plus weewxd[1756]: INFO weewx.reportengine: Copied
>>>>> 40 files to /home/pi/weewx-data/public_html/belchertown
>>>>>
>>>>>
>>>>> weewx related open files:
>>>>>
>>>>> pi@pi3plus:~/weewx-data $ lsof|grep weew
>>>>> python3 1633 pi mem REG 179,2
>>>>> 217360 272588
>>>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf
>>>>> python3 1633 pi mem REG 179,2
>>>>> 224592 272587
>>>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf
>>>>> python3 1633 pi mem REG 179,2
>>>>> 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>>> imaging.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 pi mem REG 179,2
>>>>> 2528660 271317
>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_
>>>>> libastro.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 pi mem REG 179,2
>>>>> 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>>> imagingmath.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 pi mem REG 179,2
>>>>> 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>>> imagingft.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 pi mem REG 179,2
>>>>> 45064 271812
>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_
>>>>> namemapper.cpython-311-arm-linux-gnueabihf.so
>>>>>
>>>>> When its cpus are pegged trying to process Belchertown:
>>>>>
>>>>> python3 1633 pi cwd DIR 179,2
>>>>> 4096 272808 /home/pi/weewx-data/skins/Belchertown
>>>>> python3 1633 pi mem REG 179,2
>>>>> 217360 272588
>>>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf
>>>>> python3 1633 pi mem REG 179,2
>>>>> 224592 272587
>>>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf
>>>>> python3 1633 pi mem REG 179,2
>>>>> 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>>> imaging.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 pi mem REG 179,2
>>>>> 2528660 271317
>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_
>>>>> libastro.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 pi mem REG 179,2
>>>>> 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>>> imagingmath.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 pi mem REG 179,2
>>>>> 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>>> imagingft.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 pi mem REG 179,2
>>>>> 45064 271812
>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_
>>>>> namemapper.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 pi 4uw REG 179,2
>>>>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb
>>>>> python3 1633 pi 5uw REG 179,2
>>>>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb
>>>>> python3 1633 pi 6u REG 179,2
>>>>> 87200 275903 /home/pi/weewx-data/archive/weewx.sdb-journal
>>>>> python3 1633 1853 python3 pi cwd DIR 179,2
>>>>> 4096 272808 /home/pi/weewx-data/skins/Belchertown
>>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>>> 217360 272588
>>>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Regular.ttf
>>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>>> 224592 272587
>>>>> /home/pi/weewx-data/skins/Seasons/font/OpenSans-Bold.ttf
>>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>>> 1630340 271670 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>>> imaging.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>>> 2528660 271317
>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/ephem/_
>>>>> libastro.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>>> 56296 271675 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>>> imagingmath.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>>> 93216 271673 /home/pi/weewx-venv/lib/python3.11/site-packages/PIL/_
>>>>> imagingft.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 1853 python3 pi mem REG 179,2
>>>>> 45064 271812
>>>>> /home/pi/weewx-venv/lib/python3.11/site-packages/Cheetah/_
>>>>> namemapper.cpython-311-arm-linux-gnueabihf.so
>>>>> python3 1633 1853 python3 pi 4uw REG 179,2
>>>>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb
>>>>> python3 1633 1853 python3 pi 5uw REG 179,2
>>>>> 163410944 41497 /home/pi/weewx-data/archive/weewx.sdb
>>>>> python3 1633 1853 python3 pi 6u REG 179,2
>>>>> 87200 275903 /home/pi/weewx-data/archive/weewx.sdb-journal
>>>>>
>>>>>
>>>>> On Thursday, January 18, 2024 at 12:18:18 PM UTC-8 Blaine wrote:
>>>>>
>>>>>> Thank you for the reply! Yes, I have tried this many times. It
>>>>>> continues to regenerate them with the missing data.
>>>>>>
>>>>>> On Wednesday, January 17, 2024 at 6:29:41 PM UTC-8 Tom Keffer wrote:
>>>>>>
>>>>>> Have you tried completely deleting all NOAA reports and letting weewx
>>>>>> regenerate them?
>>>>>>
>>>>>> On Wed, Jan 17, 2024 at 9:32 AM Blaine <bgri...@gmail.com> wrote:
>>>>>>
>>>>>> Way back in 2020 my NOAA reports stopped populating their daily
>>>>>> values. The average values for the month continue to be populated at the
>>>>>> bottom of the table. Since 2020 every year or so I have spent hours
>>>>>> banging
>>>>>> my head against the wall following any and all instructions on the wiki,
>>>>>> github, google and this user group to attempt to repair this issue. None
>>>>>> of
>>>>>> the published instructions seem to help. I have never been successful.
>>>>>>
>>>>>> If this is a problem that can be solved I would be more than happy to
>>>>>> pay someone to help fix this issue knowing full well there are no
>>>>>> guarantees recovery is possible. This data is important to me and having
>>>>>> essentially lost years worth with no chance of recovery is a real bummer.
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "weewx-user" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to weewx-user+...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/weewx-user/6be270a2-961f-4952-85b0-6074332671een%40googlegroups.com
>>>>>>
>>>>>> <https://groups.google.com/d/msgid/weewx-user/6be270a2-961f-4952-85b0-6074332671een%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "weewx-user" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to weewx-user+...@googlegroups.com.
>>>>>
>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/weewx-user/2bed5055-542d-4ef2-8e32-312e306af59cn%40googlegroups.com
>>>>>
>>>>> <https://groups.google.com/d/msgid/weewx-user/2bed5055-542d-4ef2-8e32-312e306af59cn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to weewx-user+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/weewx-user/d12128db-7ada-4373-9efb-cb5e75c65008n%40googlegroups.com.
