random number generator thread safety
I have questions about thread safety in the 'random' module. When using the random.Random class (be it Mersenne Twister or Wichmann-Hill based), is it sufficiently thread-safe (preserving entropy and guarding against attack) to just have each thread work with its own random.Random instance? Or do I need to wrap my calls to each instance's methods to use locks? Wichmann-Hill in particular has the warning in its .random() vulnerability; do I need to make an exception for that case? What about when using the random.SystemRandom class (be it based on /dev/urandom or CryptGenRandom based)? Should I be doing anything in my code to ensure that /dev/urandom, for example, is polled serially? (I thought I read somewhere that this was advisable). SystemRandom claims it has no state, which may be true at the Python class level, but is that guaranteed to be true for the underlying RNG? Also, side note: The doc string for random.SystemRandom says "Not available on all systems (see os.urandom() for details)" ...but os.urandom()'s doc string provides no details. Actually, os.urandom() and random.SystemRandom are "available" on all systems (they exist and can be called/instantiated), but os.urandom() can raise a NotImplementedError (any time it is called, in theory), therefore random.SystemRandom's methods can fail with a bubbled-up NotImplementedError (on non-win32 OSes, at least). At the very least, this should be documented. I'm not in a position to submit a patch, but will submit a documentation bug report if so advised. Thanks, Mike -- http://mail.python.org/mailman/listinfo/python-list
Re: random number generator thread safety
Raymond Hettinger wrote: > Mike Brown wrote: > > I have questions about thread safety in the 'random' module. > > > > When using the random.Random class (be it Mersenne Twister or Wichmann-Hill > > based), is it sufficiently thread-safe (preserving entropy and guarding > > against attack) to just have each thread work with its own random.Random > > instance? Or do I need to wrap my calls to each instance's methods to use > > locks? Wichmann-Hill in particular has the warning in its .random() > > vulnerability; do I need to make an exception for that case? > > Thread-safety has nothing to do with preserving entropy or guarding > against attack. Well, the Wichmann-Hill implementation claims to be "not thread safe" but really, as Paul Rubin pointed out, it's about the risk (depending on the implementation) that when two threads have access to the same RNG, there is a loss of confidence in the 'randomness' of the results that each thread sees... e.g., can thread 2 manipulate the state of the RNG while thread 1 is using it? and can thread 2 see the same result as thread 1? I think both of these situations are alleviated just by using separate RNG instances, but I don't know enough to know if I should still be doing some blocking when calling some functions & methods, such as os.urandom and random.SystemRandom's methods. > Nothing in the random module provides cryptographic guarantees. I don't need cryptographic guarantees. I just need to expose some RNG functions in a multithreaded application that supports Python 2.2 and up (hence the concern about WH). [ I also intend to use stdlib and will work around whatever bugs/issues I know about, such as the undesirable NotImplementedError (i.e., I will fall back on Mersenne Twister if os.urandom or SystemRandom methods fail); the WH 'thread safety' issue (which affects my Py 2.2 users); the Py 2.3.0 'no milliseconds in default seed' issue (I will make sure whatever RNGs are available are better seeded by default); the Py 2.4.0-2.4.1 posix os.urandom filehandle caching/hold-open issue (I will use 2.4.2's os.urandom); and anything else that I should be concerned about. Since I'll be reimplementing os.urandom, I'll be making use of it wherever I can in 2.2-2.3, including as the preferred source over WH & MT, and as a better seed source than system clock. ] I am asking for advice on how to mitigate the risks related to multithreading -- I don't the functions to be any less 'random' or more vulnerable than they would be in a single-threaded app. I don't know enough about the subject to know for sure whether the answer to "should I block when polling a PRNG" is "no", or whether the answer to "should I block when polling a system RNG (likely a PRNG seeded often from a hash of multiple sources)" would be different. That's why I am asking here. The random module's docs suggest that I'll want to mitigate such risks by using my own instance(s) of random.Random, and I'm asking here if that alone is going to be sufficient -- I think that in Py 2.2 it will keep me from having to block when calling .random(), right? -- or if there are some risks that using separate random.Random instances doesn't take care of, and that would thus require me to use locks on more of my calls. I'm also asking whether the same answers would apply to the use of random.SystemRandom and os.urandom(); e.g., even if I don't need to block when calling random.Random methods, might it still be a good idea to block when accessing systemRandom/urandom? -Mike -- http://mail.python.org/mailman/listinfo/python-list
unicode(obj, errors='foo') raises TypeError - bug?
This works as expected (this is on an ASCII terminal):
>>> unicode('asdf\xff', errors='replace')
u'asdf\ufffd'
This does not work as I expect it to:
>>> class C:
... def __str__(self):
... return 'asdf\xff'
...
>>> o = C()
>>> unicode(o, errors='replace')
Traceback (most recent call last):
File "", line 1, in ?
TypeError: coercing to Unicode: need string or buffer, instance found
Shouldn't it work the same as calling unicode(str(self), errors='replace')?
It doesn't matter what value you use for 'errors' (ignore, replace, strict);
you'll get the same TypeError.
What am I doing wrong? Is this a bug in Python?
--
http://mail.python.org/mailman/listinfo/python-list
unicode surrogates in py2.2/win
In mid-October 2004, Jeff Epler helped me here with this string iterator:
def chars(s):
"""
This generator function helps iterate over the characters in a
string. When the string is unicode and a surrogate pair is
encountered, the pair is returned together, regardless of whether
Python was built with UCS-4 ('wide') or UCS-2 code values for
its internal representation of unicode. This function will raise a
ValueError if it detects an illegal surrogate pair.
"""
if isinstance(s, str):
for i in s:
yield i
return
s = iter(s)
for i in s:
if u'\ud800' <= i < u'\udc00':
try:
j = s.next()
except StopIteration:
raise ValueError("Bad pair: string ends after %r" % i)
if u'\udc00' <= j < u'\ue000':
yield i + j
else:
raise ValueError("Bad pair: %r (bad second half)" % (i+j))
elif u'\udc00' <= i < u'\ue000':
raise ValueError("Bad pair: %r (no first half)" % i)
else:
yield i
I have since discovered that I can't use it on Python 2.2 on Windows because
of some weird module import bug caused by the surrogate code values expressed
in the Python code as u'\ud800' and u'\udc00' -- apparently the string
literals are being coerced to UTF-8 internally, which results in an invalid
byte sequence upon import of the module containing this function.
A simpler test case demonstrates the symptom:
C:\dev\test>echo x = u'\ud800' > testd800.py
C:\dev\test>cat testd800.py
x = u'\ud800'
C:\dev\test>python -c "import testd800"
C:\dev\test>python -c "import testd800"
Traceback (most recent call last):
File "", line 1, in ?
UnicodeError: UTF-8 decoding error: unexpected code byte
C:\dev\test>python testd800.py
C:\dev\test>python testd800.py
Very strange how it only shows up after the 1st import attempt seems to
succeed, and it doesn't ever show up if I run the code directly or run the
code in the command-line interpreter.
The error does not occur with u'\ud800\udc00' or u'\ue000' or any other valid
sequence.
In my function I can use "if u'\ud7ff' > i ..." to work around the d800 case,
but I can't use the same trick for the dc00 case. I will have to go back to
calling ord(i) and comparing against integers. IIRC the explicit ord() call
slowed things down a bit, though, so I'd like to avoid it if I can.
Can anyone tell me what's causing this, or point me to a reference to show
when it was fixed? I'm using 2.2.1 and I couldn't find mention of it in any
release notes up through 2.3. Any other comments/suggestions (besides "stop
supporting narrow unicode builds of Py 2.2") would be appreciated, too. Thanks
:)
-Mike
--
http://mail.python.org/mailman/listinfo/python-list
Re: Do You Want To Know For Sure That You Are Going To Heaven? The reason some people don't know for sure if they are going to Heaven when they die is because they just don't know. The good news is that you can know for sure that you are going to Heaven wh
In article <[EMAIL PROTECTED]>, "thesonoftruth" <[EMAIL PROTECTED]> wrote: > I wonder if anyone yells VISHNU !! VISHNU ! when making > out, I might have. It's been so long that I've forgotten. MJRB -- http://mail.python.org/mailman/listinfo/python-list
Re: Ron Grossi: God is not a man
> 4) I doubt seriously whether God plays a guitar, since guitars are made by > men, for men. His Son could theoretically play a guitar. Perhaps He does. > Perhaps He doesn't. Only the Father and His Holy Angels know. Perlse. Do you really take this stuf seriously ? It's like the ancient discussion about how many angels can dance on the head of a pin, a lot of bullshit. Sorry in advance to the believers amongst us, but these God Botherers annoy the crap out of me. Believe what you want to, but leave me alone. MJRB -- http://mail.python.org/mailman/listinfo/python-list
Re: Ron Grossi: God is not a man
Just go away from RMMGA -- http://mail.python.org/mailman/listinfo/python-list
Re: Jesus said, "I am the way, the truth and the life: no one can come to the Father(God)(in Heaven), but by me." (John 14:6) This means that if you die without trusting in Jesus Christ as your Lord and Saviour you will die in your sins and be forever sepa
I don't give a feather or a fig. MJRB -- http://mail.python.org/mailman/listinfo/python-list
Re: Jesus said, "I am the way, the truth and the life: no one can come to the Father(God)(in Heaven), but by me." (John 14:6) This means that if you die without trusting in Jesus Christ as your Lord and Saviour you will die in your sins and be forever sepa
In article <[EMAIL PROTECTED]>, "Bubba" <[EMAIL PROTECTED]> wrote: > I'm so glad you've decided what everyone believes > Some of us don't. Believe that is. In anything in particular. MJRB -- http://mail.python.org/mailman/listinfo/python-list
