Thanks for the explanation. I think I understand the failure mode now.
One thing we could do is put a check in ScalarStats.addSum() to make sure
the incoming value is, in fact, a number (and not a String, None, or NaN).
I don't think it's a good idea to just swallow and suppress the error, but
this way, it would be immediately obvious who and what the culprit is. The
patch would look something like this:
diff --git a/bin/weewx/accum.py b/bin/weewx/accum.py
index 26bf243..ee54db3 100644
--- a/bin/weewx/accum.py
+++ b/bin/weewx/accum.py
@@ -85,6 +85,8 @@
def addSum(self, val, weight=1):
"""Add a scalar value to my running sum and count."""
if val is not None:
+ if not isinstance(val, (float, int)):
+ raise ValueError("accum: Expected float or int, got %s" %
val)
self.sum += val
self.count += 1
self.wsum += val * weight
-tk
On Mon, Nov 20, 2017 at 9:30 AM, Thomas Carlin <[email protected]>
wrote:
> Tom,
> Your interpretation is exactly correct, and yes it is a coding error on my
> side. My thought was just a single sentence there about the Weewx engine
> not doing any sanitation, and that being the responsibility of the driver.
> If not, no worries, just a thought.
>
> Gary,
> No worries on the delay. You got me going in the right direction, and we
> got it figured out! It's not like you get paid for doing this. Until
> yesterday, the extent of the sanitation that I was doing on the incoming
> data was running float(data), and catching any ValueError exception. Now,
> because nan is a special 'string', when you run float('nan'), it returns
> NaN, but now it is the Python 'number' NaN. When you add a number to NaN,
> your result is always NaN. When this is stuffed into the database,
> apparently it is interpreted as SQL NULL. Next time the loop runs, Python
> queries the database, and interprets SQL NULL as None, and then we get the
> type mismatch.
>
> I have added some more sanitation for this, and we are back up and
> running, and it *should* conform to expected behavior.
>
> Does that make sense?
>
> Thank you for the input on the driver. I have a laundry list of things
> that I would like to add to make it more robust, and I'll add the loop
> output to that. I may be converting to a queue as well, simply due to the
> number of sensors that are being polled, and the "couple of seconds" note
> in the Second Datasource section of the docs. But that is a project for
> another day!
>
> On Mon, Nov 20, 2017 at 12:53 AM, gjr80 <[email protected]> wrote:
>
>> Thomas C,
>>
>> Sorry I did not respond to your earlier posts after mine, the extract you
>> provided does indeed show a daily summary record with sum and wsum
>> fields as None (well the db fields contain nothing but they will end up
>> as python None values within weeWX). At the moment I have still not been
>> able to fathom the circumstances that would cause this.
>>
>> One thing about your driver though. If you are using the default weeWX
>> schema, or a modified version of the default schema to which you have only
>> added numeric (read SQL type REAL) fields, then your driver must only emit
>> numeric values or the python None value. If you emit strings it will
>> cause an exception to be thrown within weeWX. The default weeWX behaviour
>> is if the sensor is present and emitting values then the driver should emit
>> those values as numeric values in the approriate loop packet/archive record
>> field, if the sensor is present but offline then the driver emits the
>> python None value in the field concerned and if the sensor does not
>> exist then the driver should not emit anything for that sensor ie the field
>> is excluded from the loop packet/archive record.
>>
>> Note. Technically there is no reason that you cannot add a string to a
>> loop packet or an archive record but if it is placed in a field that weeWX
>> tries to 'accumulate' it will cause an exception.
>>
>> I still don't see how your driver can cause the exception you first
>> posted (emitting a string will cause a similar but clearly different
>> exception), but it may be worthwhile reworking your driver to conform with
>> the weeWX expected behaviour. It might also be worthwhile instrumenting
>> your driver to drop the emitted loop packet to the log so we can see
>> exactly what is coming out of the driver in the leadup to the exception.
>>
>> Gary
>>
>>
>> On Monday, 20 November 2017 08:33:28 UTC+10, Thomas Carlin wrote:
>>>
>>> Hey Tom
>>>
>>> At this point in the code 'nan' is not equivalent to 'Not a Number', it
>>> is simply a string that is pulled out of an http page, not generated by
>>> python. Originally, it is generated on my Arduino when data cannot be
>>> pulled from the temperature sensor. Unfortunately in this case when you
>>> run float("nan") in python, it returns nan, and that was the extent of my
>>> sanitization. I'll write a condition into my driver to catch that, and
>>> should be good to go.
>>>
>>> On Sunday, November 19, 2017 at 2:14:59 PM UTC-7, Tom Keffer wrote:
>>>>
>>>> But, how did you create a NaN? They are not returned from any of the
>>>> Python mathematical functions.
>>>>
>>>> -tk
>>>>
>>>> On Sun, Nov 19, 2017 at 2:04 PM, Thomas Carlin <[email protected]>
>>>> wrote:
>>>>
>>>>> I'm pretty sure this is what is happening. Take a look at the output
>>>>> below:
>>>>>
>>>>> ...
>>>>> DB Entry: extraTemp3
>>>>> Value: nan
>>>>> DB Entry: extraHumid3
>>>>> Value: nan
>>>>> ...
>>>>> Unable to access http://192.168.2.105
>>>>> REC: 2017-11-19 12:38:00 MST (1511120280) ..., *extraHumid3: nan*,
>>>>> extraTemp1: 55.58, *extraTemp3: nan*, heatindex: 31.7143461695...
>>>>>
>>>>> The next attempt to add data to the database resulted in the program
>>>>> crashing due to a type mismatch.
>>>>>
>>>>> May I request that a blurb be added to the documentation here:
>>>>> http://www.weewx.com/docs/customizing.htm#Adding_2nd_source about
>>>>> sanitizing the data before it is fed to event.record()?
>>>>>
>>>>>
>>>>> On Sunday, November 19, 2017 at 10:59:29 AM UTC-7, Thomas Carlin wrote:
>>>>>>
>>>>>> The plot thickens. Right before I went to bed this morning, I
>>>>>> checked the current daily summary record for extraTemp3, and it looked as
>>>>>> you would expect it to. It had data in it for both sum and wsum, unlike
>>>>>> the one posted above. Roughly an hour later, Weewx crashed with the same
>>>>>> error, and this morning when I went to look into it, the current record,
>>>>>> same one as last night, no longer had data in it for the same fields.
>>>>>>
>>>>>> After looking a little closer, I noticed that the extraHumid3, which
>>>>>> I created, did exactly the same thing as extraTemp3. Since both these
>>>>>> pieces of data come from the same DHT11 sensor, my current theory is that
>>>>>> under the wrong conditions, my sensor is spitting out garbage that is not
>>>>>> being properly sanitized, which somehow empties those fields. Then next
>>>>>> time the loop runs, everything falls apart. I'm setting up a sandbox
>>>>>> right
>>>>>> now so I can play with it without screwing with my 'production' system.
>>>>>> Just hoping the sandbox crashes like the current system does!
>>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "weewx-user" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to [email protected].
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "weewx-user" group.
>> To unsubscribe from this topic, visit https://groups.google.com/d/to
>> pic/weewx-user/TE57NMTaIXw/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> [email protected].
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Thomas Carlin
> 970-401-3805 <(970)%20401-3805>
>
> --
> You received this message because you are subscribed to the Google Groups
> "weewx-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
