OK, understand. Ideally you want all of the processing of a loop packet (ie all of the services that do something with or to the loop packet, eg StdCalibrate, StdQC, StdArchive, your service etc - all those with a binding to NEW_LOOP_PACKET) to execute within a loop period. The loop period varies with station, in your case for your Vantage it is 2.5 odd seconds. If all of those services take longer than the loop period (remember they are executed sequentially) then you risk missing loop packets or other bad things occurring. The problem you have when you need to poll or otherwise query an external server is that issues beyond your control (eg network delays, latency, timeouts etc) may mean that your service blocks for some extended period while it obtains the external data or the request times out. This could mean your overall execution time for all services pushes out beyond the next loop packet arriving and it is missed.
You may recall in an earlier post I presented three options, the final one was a service running in its own thread. There is actually two parts to such a service, firstly some code is run in a separate thread that handles the external server/system and obtains the necessary data. Once this thread has some data it passes the data to a more traditional WeeWX type service (running in the same thread as WeeWX) via a python queue. When this traditional type service is run it takes the data it has received from the queue (if in fact it has received any) and performs its operation on the loop packet. If there is no data it may do nothing. This way any delays in dealing with the external server are confined the the non-WeeWX thread and the portion of your service running in the WeeWX thread can execute very quickly and not block. The forecast extension works in this way (it queries WU, NWS etc), as does the forecast aspects of my realtime SteelSeries Gauges extension (it queries WU and Darksky) as do a number of WeeWX drivers (they query the station). If it were me I would get your service running now as you have it. Do what you can to make sure it does not block WeeWX ie make sure you can handle the MQTT broker being unavailable, keep any timeouts to fairly low values compared to the loop interval. If you don't overload your system with a multitude of additional services you should have no problems. A longer term goal can be to rework your service to be a threaded service. You can take your time to build your python skills, research and understand how a threaded service works and ultimately write your own. It is not that hard to do but they can be a challenge to debug at times. Hope that helps. Gary On Friday, 5 October 2018 12:39:38 UTC+10, Thomas Carlin wrote: > I am augmenting the VP2 data, and looking for an idea what the maximum > time my code could run to without interrupting or missing data from my > VP2. Right now, on my 'production' system I expect additional code to run > about 3/4 second, with most of it spent connecting to MQTT, but I know that > I can optimize that by moving a couple things to the init function rather > than being part of the loop. Do you thing that 3/4 or even 1 second will > cause issues? I'm planning on setting my sensors up to report data every > 10 seconds, so every 4th cycle will be slightly longer. > > On Thursday, October 4, 2018 at 8:20:27 PM UTC-6, gjr80 wrote: >> >> On Friday, 5 October 2018 11:58:44 UTC+10, Thomas Carlin wrote: >>> >>> Perfect! thank you Gary! I knew it had to be something simple like >>> that. Everything is up and running now, and I'll just need to catch the >>> necessary exceptions, and optimize the collection as much as possible. Is >>> there a target time I should shoot for, or just try to make it as fast as >>> possible? >>> >> >> Not quite sure what you mean by a target time. If you are still using the >> Vantage as the heart of your system then the Vantage loop interval will set >> the rate at which NEW_LOOP_PACKETS events occur, ie every 2.5 odd seconds, >> so that will be the fastest rate at which WeeWX will receive data. If you >> are now subscribing to an MQTT topic to get your data that you augment the >> Vantage loop packets with then I guess how your service works with MQTT >> will depend on how you are using MQTT; are you subscribing to a topic, >> caching values etc. Perhaps if you show us your service as it stands now. >> >> Gary >> >> >> >>> >>> On Thursday, October 4, 2018 at 7:52:37 PM UTC-6, gjr80 wrote: >>>> >>>> Hi Thomas, >>>> >>>> It sounds like you should be using event.packet rather than >>>> event.record. Archive records are found in event.record, loop packets >>>> are found in event.packet. It's a subtle but important distinction. >>>> You have probably moved on from the code you posted above, and I probably >>>> should have mentioned it earlier, but I would avoid method names such as >>>> def >>>> new_archive_packet(). There is nothing wrong with using that name per >>>> se, but since we have loop packets and archive records it does tend to >>>> create a little confusion, something like def new_archive_record() >>>> and def new_loop_packet() would be better. >>>> >>>> Gary >>>> >>>> On Friday, 5 October 2018 10:30:15 UTC+10, Thomas Carlin wrote: >>>>> >>>>> Hi Gary, >>>>> >>>>> I've started working on this service, and I have run into an issue. >>>>> >>>>> A quick refresher so you don't have to read through the whole thread: >>>>> I am writing an MQTT service to add data to the loop packets. You >>>>> pointed >>>>> me in the right direction with the NEW_LOOP_PACKET, and I'm 99% of the >>>>> way >>>>> there. My service is pulling data with the loop packets, processing and >>>>> normalizing it, and I'm now to the point that I am ready to stuff it into >>>>> the loop packet. >>>>> >>>>> Since I am modifying an existing service that I wrote, I am trying to >>>>> use the line event.record[key] = value. that I found in the >>>>> documentation. When I try this, I get the following trackback. I'm sure >>>>> that this is because event.record is for archive packets, and I'm now >>>>> working with loop packets. >>>>> >>>>> Traceback (most recent call last): >>>>> File "/usr/bin/weewxd", line 64, in <module> >>>>> weewx.engine.main(options, args) >>>>> File "/usr/share/weewx/weewx/engine.py", line 877, in main >>>>> engine.run() >>>>> File "/usr/share/weewx/weewx/engine.py", line 191, in run >>>>> self.dispatchEvent(weewx.Event(weewx.NEW_LOOP_PACKET, >>>>> packet=packet)) >>>>> File "/usr/share/weewx/weewx/engine.py", line 224, in dispatchEvent >>>>> callback(event) >>>>> File "/usr/share/weewx/user/esp8266.py", line 105, in new_loop_packet >>>>> event.record[self.subscriptions[topic]] = value >>>>> AttributeError: 'Event' object has no attribute 'record' >>>>> >>>>> >>>>> >>>>> Any thoughts on what I can do to correct this? I have verified that >>>>> the variable self.subscriptions[topic] provides the expected value of >>>>> extraTemp1, and value contains 74.30. >>>>> >>>>> Also, related to this discussion, I would like to provide a simplified >>>>> version of code, and an explanatory blurb to be added to the >>>>> documentation >>>>> to hopefully help others in a similar situation. Is that something that >>>>> the development team would be open to? >>>>> >>>>> >>>>> -- You received this message because you are subscribed to the Google Groups "weewx-user" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
