OMG that's bad!! For now I have restructured my logic so I can just deal with integers. Hopefully decimal support comes soon, though I am using sqlite for this...
On Nov 12, 3:03 am, Jonathan Lundell <jlund...@pobox.com> wrote: > On Nov 10, 2009, at 11:49 PM, Richard wrote: > > > I have some floating point numbers in my database which I need to test > > for equality. Numbers like "0.3" are failing an equality test. > > So, is there a way to store floating point numbers precisely? Or > > should I be using something like: > > offset = 0.000001 > > (db.table.fp > v - offset)(db.table.fp < v + offset) > > If you're using floating point, then yes, that's what you should do. > > You can write: (v-offset) < db.table.fp < (v+offset) > > or: abs(db.table.fp - v) < offset > > ...and stick it in a function if you use it much. > > Notice that for consistency, you need to use the same technique for < <= => > > as well. > > a < b becomes a < (b-offset), etc, or else you get cases where a==b and a<b > at the same time. > > Two other caveats. One is that the value of your epsilon (or offset) depends > on your domain of values. 0.000001 isn't appropriate if you're dealing with > (say) the mass of a proton in grams. > > The other, a minor one in most cases, is that the equality relationship we're > talking about here isn't transitive. That is, it's possible that a == b and b > == c but a != c. > > There's a large literature on the hazards of computing with floating point > numbers. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~----------~----~----~----~------~----~------~--~---
