[issue27277] Fatal Python error: Segmentation fault in test_exceptions

[Rohit Mediratta]

New submission from Rohit Mediratta:

Fresh clone and running test_exceptions testcase caught a Seg fault.
This was being run on a Centos VM.

[/loc/rom/pyd/cpython] $ ./python 
Python 3.6.0a1+ (default:12d939477b4f, Jun  7 2016, 17:32:31) 
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 


[/loc/rom/pyd/cpython] $ ./python ../coveragepy/ run --pylib 
Lib/test/regrtest.py  test_exceptions 
Run tests sequentially
0:00:00 [1/1] test_exceptions
Fatal Python error: Segmentation fault

Current thread 0x7fc7324d1700 (most recent call first):
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exceptions.py", line 490 in 
f
  File "/local/romedira/pydev/cpython/Lib/test/test_exception

[issue27272] random.Random should not read 2500 bytes from urandom

[Christian Heimes]

Christian Heimes added the comment:

Tim,

you are saying that some methods (e.g. randint) of the MT don't fail security 
properties as bad as other. I'm sorry, that argument is not good enough for me. 
The seed() method of the MT is causing real-world problems, e.g. #25420 where 
'import random' in a VM blocks forever because the Kernel's RNG state hasn't 
been seeded yet.

I'm in favor of changing the default seed for the default random instance.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Martin Panter]

Martin Panter added the comment:

I dislike the “block” term for a different reason: it suggests raising EAGAIN 
(= BlockingIOError). But the proposal here is actually to generate data with 
low entropy.

In the long term, it sounds like two separate functions is the way to go. I 
would prefer os.urandom() to retain its 3.5.1 behaviour introduced by Issue 
22181: wait for sufficient entropy. This would be consistent with other 
platforms. It can continue to be used by SystemRandom, secrets, etc. The 
documentation should be clarified to say that os.urandom() either returns data 
that is supposed to be “unpredictable enough for cryptographic applications”, 
blocks waiting for entropy, or raises an NotImplementedError if an OS API (e.g. 
/dev/urandom) is missing.

The other function, say os.urandom_even_with_low_entropy(), would work like the 
old os.urandom() on Linux. On platforms like Free BSD, I guess it could fall 
back to the os.urandom() that waits for entropy. “Import random” would be 
changed to try this function.

I wonder what we should do on Linux if /dev/urandom is unavailable and 
getrandom() would block. Probably raise NotImplementedError without waiting for 
entropy.

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27278] py_getrandom() uses an int for syscall() result

[STINNER Victor]

New submission from STINNER Victor:

syscall() result type is long.

Moreover, long type might can smaller than the size type, so we may need to add:

n = Py_MIN(size, LONG_MAX);

--
messages: 267969
nosy: haypo
priority: normal
severity: normal
status: open
title: py_getrandom() uses an int for syscall() result
versions: Python 3.5, Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Christian Heimes]

Christian Heimes added the comment:

Can we please stop using the term "low entropy" here. It is wrong and makes my 
head hurt. "Running out of entropy" is an urban myth. It's about the 
initialization state of Kernel's internal RNG. The Kernel blocks the syscall as 
long as it is not confident enough about its RNG. It wants to collect a decent 
amount of events from different sources until it considers its RNG properly 
seeded. Once the RNG is seeded, you can pull as many data from the RNG as you 
like.

It's really block_on_boot_until_kernel_rng_has_been_seeded.

--
nosy: +christian.heimes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27279] Add random.cryptorandom() and random.pseudorandom, deprecate os.urandom()

[Marc-Andre Lemburg]

New submission from Marc-Andre Lemburg:

I propose to deprecate os.urandom() altogether due to all the
issues we've discussed on all those recent tickets, see e.g. 
#26839, #27250, #25420.

Unlike what we've told people for many years, it's clear that in the
age of VMs/containers getting booted/started every few seconds, it's
not longer the generic correct answer to people asking for random
data, since it doesn't make a difference between crypto random and 
pseudo random data.

By far most use cases only need pseudo random data and only very
few applications require crypto random data.

Instead, let's define something everybody can start to use correctly
and get sane behavior on most if not all platforms. As Larry
suggested in #27266, getrandom() is a good starting point for this,
since it's adoption is spreading fast and it provides the necessary
features we need for the two new APIs.

I propose these new APIs:

 * random.cryptorandom() for getting OS crypto random data

 * random.pseudorandom() for getting OS pseudo random data

Crypto applications will then clearly know that random.cryptorandom()
is the right choice for them and everyone else can use 
random.pseudorandom().

random.cryptorandom() will guarantee that the returned data
is safe for crypto applications on all platforms, blocking or
raising an exception if necessary to make sure only safe
data is returned. The API should get a parameter to determine
whether to raise or block.

random.pseudorandom() will guarantee to not block and always
return random data that can be used as seeds for simulations, 
games, tests, monte carlo, etc.

The APIs should use the getrandom() C API, where available,
with appropriate default settings, i.e. blocking or raising
for random.cryptorandom() and non-blocking, non-raising for
random.pseudorandom().

The existing os.urandom() would then be deprecated to guide
new developments to the these new APIs, getting rid of the
ambiguities and problems this interface has on several platforms
(see all the other tickets and https://en.wikipedia.org/wiki//dev/random
for details).

--
components: Library (Lib)
messages: 267970
nosy: lemburg
priority: normal
severity: normal
status: open
title: Add random.cryptorandom() and random.pseudorandom, deprecate os.urandom()
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Larry Hastings]

Larry Hastings added the comment:

> I wonder what we should do on Linux if /dev/urandom is unavailable and 
> getrandom() would block.

I don't think that happens.

getrandom() actually supports two flags.  The flag GRND_RANDOM tells it "behave 
like /dev/random".  If you don't pass in GRND_RANDOM, it behaves like 
"/dev/urandom".  So it's hard to imagine that you could have getrandom() and 
not have /dev/urandom.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

I propose to deprecate os.urandom() altogether due to all the
issues we've discussed on all those recent tickets.

See #27279 for details.

On 09.06.2016 02:04, Nick Coghlan wrote:
> I'd also *STRONGLY* request that we avoid adding any new APIs in relation to 
> this that mean "Use os.urandom" is no longer the preferred option to obtain 
> cryptographically strong random numbers in Python. Any such APIs can't be 
> used in single source Python 2/3 code, they invalidate existing third party 
> documentation like https://cryptography.io/en/latest/random-numbers/ and they 
> invalidate answers on Q&A sites like 
> http://stackoverflow.com/questions/20936993/how-can-i-create-a-random-number-that-is-cryptographically-secure-in-python

It's easy enough to write Python2/3 code to use the new
APIs for Python3, so I don't really buy that argument.

As for answers on SO: it's not a definite source for anything
anyway and life moves on, so eventually people will up the
new changes and update older answers to the new ones. The
deprecation notice will get people aware of the change.

--
nosy: +lemburg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[STINNER Victor]

STINNER Victor added the comment:

I played with select() on Linux on a VM:

* /dev/random: it works as expected
* /dev/urandom: the device is already seen as readable even before the urandom 
entropy pool is initialized. It is not really surprising since, yes, read() 
does not block in practice

To test Python before urandom is initialized, I used the init=/path/to/python 
trick in the boot loader.

By the way, I confirm that getrandom(GRND_NONBLOCK) fails with EAGAIN before 
urandom is initialized.

--

I'm now trying to get a non initialized /dev/urandom in a FreeBSD VM, but it 
seems harder... Even if single user mode, select() on /dev/urandom says that 
the device is ready and yes, it works to read a few bytes. Moreover, "sysctl 
kern.random.sys.seeded" returns 1, which confirms that urandom is already 
initialized.

If even in single mode urandom is already is initialized in a VM, I'm not sure 
that FreeBSD is really impacted by the issue #26839.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27250] Add os.urandom_block()

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

Please also see #27279 for an alternative plan.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27279] Add random.cryptorandom() and random.pseudorandom, deprecate os.urandom()

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

Fleshing out the API signatures and implementation details will have to be done 
in a PEP. 

The topic is (as all the related ticket show) too complex for discussions on a 
bug tracker.

I just opened this ticket for reference to the idea.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[STINNER Victor]

STINNER Victor added the comment:

> I dislike the “block” term for a different reason: it suggests raising EAGAIN 
> (= BlockingIOError). But the proposal here is actually to generate data with 
> low entropy.

Since os.urandom() is part of the os module, yes, I'm suggesting to raise 
BlockingIOError. The caller would be responsible to handle this error and make 
a choice. For example, random.py can reuse its fallback on time.time().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27279] Add random.cryptorandom() and random.pseudorandom, deprecate os.urandom()

[Christian Heimes]

Christian Heimes added the comment:

-1

os.urandom() is just fine. Let's not confuse users and make it even harder to 
write secure software.

--
nosy: +christian.heimes

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Cory Benfield]

Changes by Cory Benfield :


--
nosy: +Lukasa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27279] Add random.cryptorandom() and random.pseudorandom, deprecate os.urandom()

[Larry Hastings]

Larry Hastings added the comment:

I +1 on new functions that are designated the best-practice places to get your 
pseudo-random numbers.

(IDK if the best place for both is in random; maybe the crypto one should be in 
secrets?)

To be precise: on most OSes what you're calling "crypto random data" is 
actually "crypto-quality pseudo-random data".  Very few platforms provide 
genuine random data--rather, they seed a CPRNG and give you data from that.  
(And then the cryptographers have endless arguments about whether the CPRNG is 
really crypto-safe.)

I'm -1 on actually deprecating os.urandom().  It should be left alone, as a 
thin wrapper around /dev/urandom.  I imagine your cryptorandom() and 
pseudorandom() functions would usually be written in Python and just import and 
use the appropriate function on a platform-by-platform basis.

--
nosy: +larry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27274] [ctypes] Allow from_pointer creation

[SilentGhost]

Changes by SilentGhost :


--
nosy: +amaury.forgeotdarc, belopolsky, meador.inge
versions: +Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[STINNER Victor]

STINNER Victor added the comment:

Martin:
> I wonder what we should do on Linux if /dev/urandom is unavailable and 
> getrandom() would block.

os.urandom(block=False) should raise BlockingIOError if 
getrandom(GRND_NONBLOCK) fails with EAGAIN and /dev/urandom is not available.

Larry:
> I don't think that happens. getrandom() actually supports two flags.  The 
> flag GRND_RANDOM tells it "behave like /dev/random".  If you don't pass in 
> GRND_RANDOM, it behaves like "/dev/urandom".  So it's hard to imagine that 
> you could have getrandom() and not have /dev/urandom.

You can imagine a "badly configured" container or chroot where /dev or only 
/dev/urandom doesn't exist.

When I played with chroot, it was easy to forget /dev/urandom. Since Python 
uses getrandom() on Linux since Python 3.5, you can imagine that a Python 3.5 
user may not notice the lack of /dev/urandom in the common case (urandom 
initialized), but start to get errors when the container runs before urandom is 
initialized.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Christian Heimes]

Christian Heimes added the comment:

I'm -1 on a block=True/False parameter. It makes the API more awkward and will 
make people move away from os.urandom() to a self-made RNG because they 
perceive os.urandom() as potential blocking.

Victor, you can use sysctl on all BSD and Linux to check if the RNG has been 
seeded. On Linux it is kernel.random.entropy_avail >= 
kernel.random.read_wakeup_threshold. On BSD it is kern.random.sys.seeded == 1.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 09.06.2016 09:57, STINNER Victor wrote:
> 
> STINNER Victor added the comment:
> 
> I played with select() on Linux on a VM:
> 
> * /dev/random: it works as expected
> * /dev/urandom: the device is already seen as readable even before the 
> urandom entropy pool is initialized. It is not really surprising since, yes, 
> read() does not block in practice
> 
> To test Python before urandom is initialized, I used the init=/path/to/python 
> trick in the boot loader.

It's best to look at the code for this in Linux:

http://lxr.free-electrons.com/source/drivers/char/random.c

getrandom() is implemented directly on top of the two
devices:

http://lxr.free-electrons.com/source/drivers/char/random.c#L1601

> By the way, I confirm that getrandom(GRND_NONBLOCK) fails with EAGAIN before 
> urandom is initialized.

Right, and that's good, since it's better to let the application
control what to do than to simply block.

Here's a good reading on getrandom():

https://lwn.net/Articles/606141/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Larry Hastings]

Larry Hastings added the comment:

Fair enough.  But Theodore Ts'o said on the tracker: if you call getrandom() 
and don't pass in GRND_RANDOM, it's equivalent to /dev/urandom.  So, if 
getrandom is available, getrandom(flags=0) will always work and never block.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27279] Add random.cryptorandom() and random.pseudorandom, deprecate os.urandom()

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

Some resources:

 * getrandom() man page:
   http://man7.org/linux/man-pages/man2/getrandom.2.html

 * nice readup on how getrandom() was introduced:
   https://lwn.net/Articles/606141/

 * random devices implementation on Linux:
   http://lxr.free-electrons.com/source/drivers/char/random.c

 * getrandom() implementation on Linux:
   http://lxr.free-electrons.com/source/drivers/char/random.c#L1601
   (built straight on opt of the device APIs)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27275] KeyError thrown by optimised collections.OrderedDict.popitem()

[William Pitcock]

William Pitcock added the comment:

A frequent reproducer is to run the expiringdict tests on Python 3.5.1, 
unfortunately I cannot come up with a testcase.

Replacing use of popitem() with "del self[next(OrderedDict.__iter__(self))]" 
removes the KeyErrors and the structure otherwise works fine.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Cory Benfield]

Cory Benfield added the comment:

> But Theodore Ts'o said on the tracker: if you call getrandom() and don't pass 
> in GRND_RANDOM, it's equivalent to /dev/urandom.  So, if getrandom is 
> available, getrandom(flags=0) will always work and never block.

Can we please try to be clear about what kind of blocking we mean? 
getrandom(flags=0) absolutely *can* block: that's what the original issue was 
all about. To ensure it *never* blocks you need to call 
getrandom(GRND_NONBLOCK): that's why the flag exists.

Put another way:

- getrandom(flags=GRND_RANDOM) == /dev/random
- getrandom(flags=GRND_NONBLOCK) == /dev/urandom on Linux
- getrandom(flags=0) == /dev/urandom everywhere but Linux

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27278] py_getrandom() uses an int for syscall() result

[Martin Panter]

Martin Panter added the comment:

According to , 
getrandom() returns no more than 32 MiB as an int on Linux. Doesn’t that mean 
you can rely on syscall()’s long return value fitting in an int? Maybe just 
cast n = (int)sycall(...) to be explicit.

But it does make sense to cap n to LONG_MAX just in case there is some strange 
platform where it matters :)

--
nosy: +martin.panter

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[STINNER Victor]

STINNER Victor added the comment:

Marc-Andre Lemburg:
> I propose to deprecate os.urandom() altogether due to all the issues we've 
> discussed on all those recent tickets.

I'm sorry, but I don't understand the purpose of this change. Usually, when we 
deprecate something, it is in favor of a new better function. What do you 
propose?

I read that you proposed to expose getrandom() as os.getrandom(). It would be 
painful to write portable code if each OS provides its own RNG function.

Python has the habit of helping users by providing portables functions. Recent 
example: time.monotonic (PEP 418). Somehow related: non inheritable file 
descriptors by default (PEP 446) and retry system calls failing with EINTR (PEP 
475). These changes aim to simplify the life of Python developers to reduce the 
subtle differences between each operating system.

To me, os.urandom() is well defined. The corner case of not initialized urandom 
is really a corner case which only occurs in "catastrophic" cases like ("badly 
configured") VM or embedded devices without hardware RNG (nor RTC).

When it's hard to write a reliable behaviour on all platforms, the simple 
solution was always to document the subtle differences between each platforms. 
I started to do with documenting getrandom() and the fallback on /dev/urandom 
for Linux:
https://docs.python.org/dev/library/os.html#os.urandom

--

If we cannot agree on a technical solution, a PEP is required.

But please give me some time to investigate the different technical solutions 
before trying to take a decision.

Right now, I'm investigating the options to keep the Python startup "secure" in 
the "urandom not initialized" case and keep os.urandom() "blocking".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[STINNER Victor]

STINNER Victor added the comment:

If you didn't read it yet, I suggest you to read my summary of the issue #26839 
which contains a lot of information on RNG in the case of Python:
http://haypo-notes.readthedocs.io/pep_random.html

It can be used to write a real PEP.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27278] py_getrandom() uses an int for syscall() result

[Martin Panter]

Martin Panter added the comment:

Make that INT_MAX. Or change n from an int to a Py_ssize_t. Both Linux and 
Solaris versions or getrandom() are documented as accepting size_t buflen.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27181] Add geometric mean to `statistics` module

[Mark Dickinson]

Mark Dickinson added the comment:

Choice of algorithm is a bit tricky here. There are a couple of obvious 
algorithms that work mathematically but result in significant accuracy loss in 
an IEEE 754 floating-point implementation: one is `exp(mean(map(log, 
my_numbers)))`, where the log calls can introduce significant loss of 
information, and the other is `prod(x**(1./len(my_numbers)) for x in 
my_numbers)`, where the `**(1./n)` operation similarly discards information. A 
better algorithm numerically is `prod(my_numbers)**(1./len(my_numbers))`, but 
that's likely to overflow quickly for large datasets (and/or datasets 
containing large values).

I'd suggest something along the lines of 
`prod(my_numbers)**(1./len(my_numbers))`, but keeping track of the exponent of 
the product separately and renormalizing where necessary to avoid overflow.

There are also algorithms for improved accuracy in a product, along the same 
lines as the algorithm used in fsum. See e.g., the paper "Accurate 
Floating-Point Product and Exponentiation" by Stef Graillat. [1] (I didn't know 
about this paper: I just saw a reference to it in a StackOverflow comment [2], 
which reminded me of this issue.)

[1] http://www-pequan.lip6.fr/~graillat/papers/IEEE-TC-prod.pdf
[2] 
http://stackoverflow.com/questions/37715250/safe-computation-of-geometric-mean

--
nosy: +mark.dickinson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27279] Add random.cryptorandom() and random.pseudorandom, deprecate os.urandom()

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 09.06.2016 10:07, Larry Hastings wrote:
> 
> I +1 on new functions that are designated the best-practice places to get 
> your pseudo-random numbers.
> 
> (IDK if the best place for both is in random; maybe the crypto one should be 
> in secrets?)

All up for discussion. As long as we get the separation clear,
I'm fine with any location in the stdlib.

> To be precise: on most OSes what you're calling "crypto random data" is 
> actually "crypto-quality pseudo-random data".  Very few platforms provide 
> genuine random data--rather, they seed a CPRNG and give you data from that.  
> (And then the cryptographers have endless arguments about whether the CPRNG 
> is really crypto-safe.)

Yes, I know, this should be documented in the docs for
random.cryptorandom().

We might even make the available entropy available as
additional API, on platforms where this is possible,
or even add APIs to access the entropy daemon where available:

http://egd.sourceforge.net/

(the necessary API is available via OpenSSL:
http://linux.die.net/man/3/rand_egd)

Some crypto applications do need to know a bit more about where
the random data is coming from, e.g. for generation of root
certificates and secure one time pads.

> I'm -1 on actually deprecating os.urandom().  It should be left alone, as a 
> thin wrapper around /dev/urandom.  I imagine your cryptorandom() and 
> pseudorandom() functions would usually be written in Python and just import 
> and use the appropriate function on a platform-by-platform basis.

Fair enough. I don't feel strong about this part. The main idea
here was to move people away from thinking that we can fix a broken
system, which is not under our control (because it's a shim on an
OS device).

How we implement the functions is up to debate as well. I could
imaging that we expose the getrandom() function as e.g.
random._getrandom() and then use this from Python where available,
with fallbacks to other solutions where necessary. This would
also make it possible to have similar functionality on non-CPython
platforms and opens up the door for future changes without
breaking applications again.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27274] [ctypes] Allow from_pointer creation

[Eryk Sun]

Eryk Sun added the comment:

Probably adding from_pointer is a good idea. That said, there's a simpler way 
to go about getting a bytes copy for a pointer.

Say that you have a pointer p for the following array:

>>> a = (c_float * 3)(1, 2, 3)
>>> bytes(a)
b'\x00\x00\x80?\x00\x00\x00@\x00\x00@@'
>>> p = POINTER(c_float)(a)

IMO, the most straight-forward way to get a bytes copy is to call string_at:

>>> string_at(p, sizeof(p.contents) * 3)
b'\x00\x00\x80?\x00\x00\x00@\x00\x00@@

In 3.x string_at uses the FFI machinery to call the following function:

static PyObject *
string_at(const char *ptr, int size)
{
if (size == -1)
return PyBytes_FromStringAndSize(ptr, strlen(ptr));
return PyBytes_FromStringAndSize(ptr, size);
}

The first argument can be any type accepted by c_void_p.from_param, such as a 
ctypes pointer/array, str, bytes, or an integer address.

Alternatively, note that pointer instantiation is the same as setting the 
`contents` value, which accepts any ctypes data object. Here's the C code that 
implements this assignment:

dst = (CDataObject *)value;
*(void **)self->b_ptr = dst->b_ptr;

The b_ptr field points at the buffer of the ctypes data object. Thus you can 
cast p to a char pointer without even calling the cast() function, which avoids 
an FFI call:

>>> bp = POINTER(c_char)(p.contents)

Slicing c_char and c_wchar pointers is special cased to return bytes and str, 
respectively. So you can slice bp to get bytes:

>>> bp[:sizeof(p.contents) * 3]
b'\x00\x00\x80?\x00\x00\x00@\x00\x00@@'

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16132] ctypes incorrectly encodes .format attribute of memory views

[Mark Lawrence]

Changes by Mark Lawrence :


--
nosy:  -BreamoreBoy

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Larry Hastings]

Larry Hastings added the comment:

> Can we please try to be clear about what kind of blocking we mean? 
> getrandom(flags=0) absolutely *can* block: that's what the original issue was 
> all about. To ensure it *never* blocks you need to call 
> getrandom(GRND_NONBLOCK): that's why the flag exists.

Thanks, I was actually confused on this issue.  I thought CPython was using 
getrandom(GRND_RANDOM) and that's why it was blocking.  But to be clear, you're 
right: 3.5.1 is calling getrandom(0) in all circumstances.  It never passes in 
GRND_RANDOM and it never passes in GRND_NOBLOCK.  And according to the manpage 
for getrandom(), getrandom(0) "blocks if the entropy pool has not yet been 
initialized".

What I don't understand is this, from the manpage for urandom:

> A read from the /dev/urandom device will not block waiting for more entropy.  
> If there is not sufficient entropy, a pseudorandom number generator is used 
> to create the requested bytes.

If both sources are right, then /dev/urandom behaves quite differently from 
getrandom(0).

Imagine how confused I was when Theodore Ts'o said:

> First of all, if you are OK with reading from /dev/urandom, then you might as 
> well use getrandom's GRND_NONBLOCK flag.  They are logically equivalent.

He wrote it.  But what he said there doesn't jibe with what the manpages say.  
Those say that if you call getrandom(GRND_NONBLOCK) before the entropy pool has 
been initialized, it will return EAGAIN, but any time you read from 
/dev/urandom you will always get random data.

... the more I learn about this, the less I think I understand it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[STINNER Victor]

STINNER Victor added the comment:

I also tested to call os.urandom() "early" in the Windows boot process in a 
Windows VM. Using a startup entry in the 
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run registry key, 
os.urandom() works immediatly (it is able to produce 16 bytes of entropy), 
whereas I didn't touch the mouse, keyboard, or anything. The VM is running with 
qemu without RNG device (I got one, I removed it).

"early" is not correct: it's late in fact, only when the desktop is opened. I 
should use GPEDIT.MSC to start a task before the desktop, but this command 
doesn't seem to be available on my Windows 8?
http://stackoverflow.com/questions/614766/run-a-script-on-windows-startup-without-a-user-logged-on

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27279] Add random.cryptorandom() and random.pseudorandom, deprecate os.urandom()

[Christian Heimes]

Christian Heimes added the comment:

On 2016-06-09 10:30, Marc-Andre Lemburg wrote:
> 
> Marc-Andre Lemburg added the comment:
> 
> On 09.06.2016 10:07, Larry Hastings wrote:
>>
>> I +1 on new functions that are designated the best-practice places to get 
>> your pseudo-random numbers.
>>
>> (IDK if the best place for both is in random; maybe the crypto one should be 
>> in secrets?)
> 
> All up for discussion. As long as we get the separation clear,
> I'm fine with any location in the stdlib.
> 
>> To be precise: on most OSes what you're calling "crypto random data" is 
>> actually "crypto-quality pseudo-random data".  Very few platforms provide 
>> genuine random data--rather, they seed a CPRNG and give you data from that.  
>> (And then the cryptographers have endless arguments about whether the CPRNG 
>> is really crypto-safe.)
> 
> Yes, I know, this should be documented in the docs for
> random.cryptorandom().
> 
> We might even make the available entropy available as
> additional API, on platforms where this is possible,
> or even add APIs to access the entropy daemon where available:

EDG has died about 15 years ago. Please don't reanimate it.

> Some crypto applications do need to know a bit more about where
> the random data is coming from, e.g. for generation of root
> certificates and secure one time pads.

No, that is not how applications deal with certificates or OTPs. When an
application is really, REALLY concerned with RNG source on that level,
it will never ever use Python or even a Kernel CSPRNG to generate
private keys. Instead it will use a certified, industrial grade HSM
(hardware security model) to offload all cryptographic operations on a
secure device.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 09.06.2016 10:16, STINNER Victor wrote:
> 
> STINNER Victor added the comment:
> 
> Marc-Andre Lemburg:
>> I propose to deprecate os.urandom() altogether due to all the issues we've 
>> discussed on all those recent tickets.
> 
> I'm sorry, but I don't understand the purpose of this change. Usually, when 
> we deprecate something, it is in favor of a new better function. What do you 
> propose?

It's mainly to get people stop thinking that we can fix os.urandom()
in a way so that it works the same across platforms and to
draw some attention to the main proposal in #27279 :-)

I don't feel strong about the deprecation.

> I read that you proposed to expose getrandom() as os.getrandom(). It would be 
> painful to write portable code if each OS provides its own RNG function.

This would be done in the two new functions, in Python and
based on whatever is available on the platform. See #27279.

Application writers should not have to worry about the
different mechanisms used on the different platforms.

> Python has the habit of helping users by providing portables functions. 
> Recent example: time.monotonic (PEP 418). Somehow related: non inheritable 
> file descriptors by default (PEP 446) and retry system calls failing with 
> EINTR (PEP 475). These changes aim to simplify the life of Python developers 
> to reduce the subtle differences between each operating system.

Exactly.

> To me, os.urandom() is well defined. The corner case of not initialized 
> urandom is really a corner case which only occurs in "catastrophic" cases 
> like ("badly configured") VM or embedded devices without hardware RNG (nor 
> RTC).

It is a corner case, but one we'll see hit us more often going
forward, since booting up VMs/containers no longer is a rather
rare incident. It's being used as part of the architecture of
system nowadays.

> When it's hard to write a reliable behaviour on all platforms, the simple 
> solution was always to document the subtle differences between each 
> platforms. I started to do with documenting getrandom() and the fallback on 
> /dev/urandom for Linux:
> https://docs.python.org/dev/library/os.html#os.urandom
> 
> --
> 
> If we cannot agree on a technical solution, a PEP is required.

Right. The topic has too many aspects to really address in a bug
tracker ticket. It's better to start writing these things down
in a PEP. I'm using #27279 as scratch pad for this.

> But please give me some time to investigate the different technical solutions 
> before trying to take a decision.

Sure, Python 3.6 is still in the making...

> Right now, I'm investigating the options to keep the Python startup "secure" 
> in the "urandom not initialized" case and keep os.urandom() "blocking".

The Python startup definitely has to be fixed, regardless of what
we do with os.urandom(). This means: getting hash randomization
and the random module initialized without blocking even when
the entropy pool is not yet initialized.

I guess the easiest way to solve this for the random module
is by making the initialization lazy (as was proposed on the
resp. ticket).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Christian Heimes]

Christian Heimes added the comment:

On 2016-06-09 10:49, Marc-Andre Lemburg wrote:
> It is a corner case, but one we'll see hit us more often going
> forward, since booting up VMs/containers no longer is a rather
> rare incident. It's being used as part of the architecture of
> system nowadays.

Containers are not a problem. By definition they share the Kernel with
the host and therefore use the CPRNG of the host.

For early boot and VM Python needs to address the problem. Neither
Py_HashSecret nor the init vector of random's MT need strong
cryptographic keys. Python can ask the kernel if it has initialized its
RNG and then fall back to a different RNG source (e.g. srandom(3) /
random(3)).

For os.urandom() let's define it as non-blocking and raise an exception
when it would blocks. We can safely assume that the Kernel will never
block reads from its entropy source once the entropy source has been
initialized. Like Py_HashSecret we can ask the Kernel for its state (on
Linux: use non-blocking getrandom()).

The design is simple, secure and well specified. Python's startup works
under all circumstancesm os.urandom() either succeeds or fails properly
and 'import random' never blocks.

Christian

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[STINNER Victor]

STINNER Victor added the comment:

> "early" is not correct: it's late in fact, only when the desktop is opened. I 
> should use GPEDIT.MSC to start a task before the desktop, but this command 
> doesn't seem to be available on my Windows 8?

Oh, I found how to start a task before the user login, and os.urandom() still 
works: it produces 16 bytes immediatly (in 0.0 second).

As FreeBSD, I'm not sure that Windows is really impacted by the issue #26839.

Latest article about Windows RNG: http://eprint.iacr.org/2007/419
"Cryptanalysis of the Random Number Generator of the Windows Operating System" 
(2007) by Leo Dorrendorf and Zvi Gutterman and Benny Pinkas

See also "Where does a Hyper-V guest get its entropy when generating a 
certificate authority key pair?":
http://security.stackexchange.com/questions/51538/where-does-a-hyper-v-guest-get-its-entropy-when-generating-a-certificate-authori

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27280] Paste fail in ipaddress documentation

[Arthur Carcano]

New submission from Arthur Carcano:

There is a type in the `IPv6Network' constructor doc, fixed in attached patch.

--
assignee: docs@python
components: Documentation
files: patch_doc_ipaddress.patch
keywords: patch
messages: 267999
nosy: NougatRillettes, docs@python
priority: normal
severity: normal
status: open
title: Paste fail in ipaddress documentation
type: enhancement
versions: Python 3.5
Added file: http://bugs.python.org/file43317/patch_doc_ipaddress.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Larry Hastings]

Larry Hastings added the comment:

> Oh, I found how to start a task before the user login, and os.urandom() still 
> works: it produces 16 bytes immediatly (in 0.0 second).

Just to confirm: that's a fresh Windows VM, never been booted before ever?  If 
it had ever been booted before, it might be saving its entropy pools to the 
hard disk at shutdown.

If you do the experiment a second time with another copy of the same fresh VM, 
does it generate the same 16 bytes?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[STINNER Victor]

STINNER Victor added the comment:

Christian Heimes added the comment:
> I'm -1 on a block=True/False parameter. It makes the API more awkward and 
> will make people move away from os.urandom() to a self-made RNG because they 
> perceive os.urandom() as potential blocking.

Are you against the feature of the API (function name/parameter name)?
If it's just the API, could you please propose something better?

> Victor, you can use sysctl on all BSD and Linux to check if the RNG has been 
> seeded. On Linux it is kernel.random.entropy_avail >= 
> kernel.random.read_wakeup_threshold. On BSD it is kern.random.sys.seeded == 1.

I know for BSD, but I failed to find a way to get access to a FreeBSD
where kern.random.sys.seeded is still zero.

For Linux, is kernel.random.entropy_avail related to /dev/random? In
my tests, reading from /dev/urandom never blocks even before urandom
is initialized.

=> see msg267974

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Larry Hastings]

Larry Hastings added the comment:

> In my tests, reading from /dev/urandom never blocks even before urandom is 
> initialized.

That's correct, and is the big difference between getrandom(0) and reading from 
/dev/urandom.  If "the entropy pool has not been initialized" (terminology from 
the man pages), getrandom(0) will block, but read(/dev/urandom) will return 
bytes from the urandom CPRNG before it's been initialized.  Which means they 
are some seriously low-quality not-very-random numbers.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Cory Benfield]

Cory Benfield added the comment:

> Those say that if you call getrandom(GRND_NONBLOCK) before the entropy
> pool has been initialized, it will return EAGAIN, but any time you read
> from /dev/urandom you will always get random data.


Yeah, so this is why the crypto folks were all up in arms about falling back to 
the /dev/urandom behaviour on Linux: Linux's /dev/urandom behaviour is really 
pretty dangerous.

In essence, on Linux, /dev/urandom will *always* return you some bytes, and 
their actual quality is entirely uncertain. This means that fundamentally 
without interrogating the kernel state before you read, you can't really be 
sure that /dev/urandom is safe for what you want it to be safe for.

But /dev/random isn't a suitable replacement in most cases, because /dev/random 
goes too far the other way: it has this hyper-paranoid notion about "entropy" 
that isn't really well-founded, and so sometimes it'll go on strike for a while.

This is why we've been pushing to keep hold of the os.urandom() implementation 
that CPython 3.5 now has: it irons out one of the most annoying warts in Linux 
RNGs. As Ted Tso has said elsewhere in this thread, /dev/urandom only exhibits 
its current behaviour for backward compatibility reasons: getrandom(flags=0) is 
really the ideal behaviour for almost any case *except* what Python is doing at 
startup.

Not that it needs saying again, but I'm still in favour of doing something like 
what Christian is suggesting, or like I suggested earlier (have os.urandom 
remain good, have Python internally fall back to weaker seeds for random and 
SipHash if it's run so early that the system hasn't started up fully yet).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27279] Add random.cryptorandom() and random.pseudorandom, deprecate os.urandom()

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

Some more resources for FreeBSD:

 * /dev/random and /dev/urandom (symlink to /dev/random) ma page:
   https://www.freebsd.org/cgi/man.cgi?query=random&sektion=4

 * Developer discussion about /dev/random and its future from 2013:
   https://wiki.freebsd.org/201308DevSummit/Security/DevRandom
 
 * FreeBSD uses the Yarrow CPRNG for /dev/random:
   https://www.usenix.org/legacy/events/bsdcon02/full_papers/murray/murray_html/
   https://www.schneier.com/academic/archives/2000/01/yarrow-160.html

 * FreeBSD will likely switch to the new Fortuna successor of Yarrow in an 
upcoming release:
   https://www.schneier.com/academic/fortuna/

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[STINNER Victor]

STINNER Victor added the comment:

Christian Heimes:
> For os.urandom() let's define it as non-blocking and raise an exception
> when it would blocks.

There is a lot of discussion around random in the bug tracker, it's
really hard to follow :-( Please use specific issues for each idea.
Making os.urandom() non-blocking and always raise an exception is not
the goal of this issue. This idea explicitly makes os.urandom()
blocking on Linux. I suggest you to open a new issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[STINNER Victor]

STINNER Victor added the comment:

> Not that it needs saying again, but I'm still in favour of doing something 
> like what Christian is suggesting, or like I suggested earlier (have 
> os.urandom remain good, have Python internally fall back to weaker seeds for 
> random and SipHash if it's run so early that the system hasn't started up 
> fully yet).

It looks like people don't read what I'm writing :-( You are just
repeating the title of this issue :-) (So yes, I agree with you, since
I'm proposing the same thing.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27181] Add geometric mean to `statistics` module

[Mark Dickinson]

Mark Dickinson added the comment:

On the other hand, apparently `exp(mean(log(...)))` is good enough for SciPy: 
its current implementation looks like this:

def gmean(a, axis=0):
a, axis = _chk_asarray(a, axis)
log_a = ma.log(a)
return ma.exp(log_a.mean(axis=axis))

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Cory Benfield]

Cory Benfield added the comment:

> It looks like people don't read what I'm writing :-( 

I'm reading you, Victor, but you and I aren't disagreeing, so I'm not trying to 
convince you. =) I'm trying to convince Larry.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[STINNER Victor]

STINNER Victor added the comment:

Larry Hastings:
> Just to confirm: that's a fresh Windows VM, never been booted before ever?  
> If it had ever been booted before, it might be saving its entropy pools to 
> the hard disk at shutdown.

The VM was booted before. I don't see how I could schedule a task at boot, and 
then reboot... the new boot will obviously not be a "fresh VM".

Maybe it's possible to skip entropy written on disk on FreeBSD or Windows? If 
not, it confirms that the issue doesn't really affect FreeBSD and Windows in 
practice.

I read that OpenBSD is able to pass the entropy file through the boot loader. 
It is done before the kernel is loaded, so it doesn't matter when Python 3.5 is 
started, urandom will always be initialized after the first boot on OpenBSD, 
no? (If the first boot was able to produce enough entropy.) Maybe it's the same 
thing for FreeBSD.

Linux has a different design, loading the entropy file from the disk comes 
"later" in the init process, after the kernel booted. It's not done (currently) 
by the boot loader. It was discussed at:
http://bugs.python.org/issue26839#msg267853


> If you do the experiment a second time with another copy of the same fresh 
> VM, does it generate the same 16 bytes?

>From what I read, Windows is vulnerable the "reset" attack on the RNG when 
>using a VM. So you can expect the same random numbers with your scenario.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27279] Add random.cryptorandom() and random.pseudorandom, deprecate os.urandom()

[Larry Hastings]

Larry Hastings added the comment:

>  * FreeBSD will likely switch to the new Fortuna successor of Yarrow in an 
> upcoming release:

A little more information about that.

FreeBSD did a major refactoring of their /dev/urandom (etc) support, which 
landed in October 2014:

https://svnweb.freebsd.org/base?view=revision&revision=273872

This kept Yarrow but also added Fortuna.  You can switch between them with a 
kernel option.

FreeBSD 10 shipped in January 2014, so clearly this rework didn't make it in.

I see several references to "let's make Fortuna the default in FreeBSD 11".  
FreeBSD 11 hasn't shipped yet.  However, the "what's new in FreeBSD 11" wiki 
page doesn't mention changing this default.  So I don't know whether or not 
it's happening for 11.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26826] Expose new copy_file_range() syscall in os module.

[Marcos Dione]

Marcos Dione added the comment:

Fixed the last comments, including comparing what was written to the original 
data, but only to the length of what was actually written. I'm just not sure if 
the way to handle the syscall not existing is ok.

--
Added file: http://bugs.python.org/file43318/copy_file_range.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27272] random.Random should not read 2500 bytes from urandom

[Christian Heimes]

Christian Heimes added the comment:

New plan:

* Add a new uint64 _Py_HashSecret.random_seed
* On 'import random' seed random.Random() from _Py_HashSecret.random_seed + 
gettimeofday().tv_sec + gettimeofday().tv_usec + id(self). That way 
subinterpreters get a different init state.

On systems with a good entropy source, _Py_HashSecret will be filled with 
cryptographically strong material. On early boot / VM scenarios the situation 
is a bit degraded but still properly good enough. It's better than to seed from 
a bad RNG or just from time.time().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27280] Paste fail in ipaddress documentation

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
nosy: +pmoody

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27277] Fatal Python error: Segmentation fault in test_exceptions

[SilentGhost]

Changes by SilentGhost :


--
components: +Tests

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27272] random.Random should not read 2500 bytes from urandom

[STINNER Victor]

STINNER Victor added the comment:

> On 'import random' seed random.Random() from _Py_HashSecret.random_seed + 
> gettimeofday().tv_sec + gettimeofday().tv_usec + id(self). That way 
> subinterpreters get a different init state.

Can we use os.urandom() was random.Random is instanciated manually,
but use the "secret random seed" when the random module is imported?

You can easily restrict the feature to workaround blocking "import
random" on VM: consume the secret in the C code, and ensure that it's
only used once.

For example, reimplement random.Random.seed() in C (remove the Python
implementation) and use the secret in C, but only the first user of
seed(None) will get it. WDYT?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27279] Add random.cryptorandom() and random.pseudorandom, deprecate os.urandom()

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

Resources for entropy gathering sources:

 * Kernel based devices such as /dev/random:
   https://en.wikipedia.org/wiki//dev/random

 * EGD - old entropy gathering daemon; blocks when out of
   entropy
   http://egd.sourceforge.net/
   (not maintained anymore)

   Important here is not the original implementation, but
   the Unix domain socket interface, which many applications
   support.

 * PRNG - provides the EGD interface, but feeds entropy into
   the OpenSSL pool; essentially a CPRNG with EGD interface.
   http://prngd.sourceforge.net/
   
 * Virtio RNG - paravirtualized device for passing host RNG
   to guest VMs (running under KVM)
   https://fedoraproject.org/wiki/Features/Virtio_RNG
   
 * haveged - entropy daemon which feeds entropy into the
   Linux /dev/random pool
   http://www.issihosts.com/haveged/
   https://wiki.archlinux.org/index.php/Haveged
   
   Whether this is useful on VMs, is contested, due to the way
   haveged works (reliance on rdtsc instructions which don't work
   well in VMs)
   
http://security.stackexchange.com/questions/34523/is-it-appropriate-to-use-haveged-as-a-source-of-entropy-on-virtual-machines
   
 * Hardware RNG in Raspberry Pi:
   
https://sites.google.com/site/astudyofentropy/project-definition/raspberry-pi-internal-hardware-random-number-generator
   
 * rng-tools - provides the rngd daemon to feed entropy from
   hardware RNGs into the Linux /dev/random pool
   https://wiki.archlinux.org/index.php/Rng-tools
   http://linux.die.net/man/8/rngd

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26826] Expose new copy_file_range() syscall in os module.

[Martin Panter]

Martin Panter added the comment:

It’s a bit ugly, but I would write the test so that it is recorded as skipped:

try:
os.copy_file_range(...)
except OSError as err:
if err.errno != ENOSYS:
raise  # We get to see the full exception details
self.skipTest(err)  # Test is recorded as skipped, not passed

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27272] random.Random should not read 2500 bytes from urandom

[Donald Stufft]

Donald Stufft added the comment:

If seeding from urandom was causing no problems, then I would not care if 
random.Random() wanted to seed from urandom, even though it doesn't need to. 
However it is causing problems, and thus it shouldn't.

Here's another script, this one runs on Python 3.5.1 that can clone MT using 
nothing but it's output: 
https://gist.github.com/dstufft/394ea7dd8f64159df10e25a75865c03c.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Donald Stufft]

Donald Stufft added the comment:

Having os.urandom raise an error instead of blocking is OK with me. It turns an 
implicit error into an explicit one. However, I prefer to have it block until 
it has initialized it's entropy pool because that makes Linux behave similarly 
to all of the other major, modern OSs instead of making Linux continue to be a 
weird edge case.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26839] Python 3.5 running on Linux kernel 3.17+ can block at startup or on importing the random module on getrandom()

[Larry Hastings]

Larry Hastings added the comment:

I just posted to python-dev and asked Guido to make a BDFL ruling.  I only 
represented my side, both because I worried I'd do a bad job of representing 
*cough* literally everybody else *cough*, and because it already took me so 
long to write the email.  All of you who disagree with me, I'd appreciate it if 
you'd reply to my python-dev posting and state your point of view.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26826] Expose new copy_file_range() syscall in os module.

[Marcos Dione]

Marcos Dione added the comment:

ENOSYS catching fixed.

--
Added file: http://bugs.python.org/file43319/copy_file_range.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27181] Add geometric mean to `statistics` module

[Steven D'Aprano]

Steven D'Aprano added the comment:

On Thu, Jun 09, 2016 at 09:24:04AM +, Mark Dickinson wrote:

> On the other hand, apparently `exp(mean(log(...)))` is good enough for SciPy:

Hmm, well, I don't have SciPy installed, but I've found that despite 
their (well-deserved) reputation, numpy (and presumably scipy) often 
have rather naive algorithms that can lose accuracy rather 
spectacularly.

py> statistics.mean([1e50, 2e-50, -1e50, 2e-50])
1e-50
py> np.mean(np.array([1e50, 2e-50, -1e50, 2e-50]))
5e-51

py> statistics.mean([1e50, 2e-50, -1e50, 2e-50]*1000)
1e-50
py> np.mean(np.array([1e50, 2e-50, -1e50, 2e-50]*1000))
5.0002e-54

On the other hand, np is probably a hundred times (or more) faster, so I 
suppose accuracy/speed makes a good trade off.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27140] Opcode for creating dict with constant keys

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Could anyone please make a review?

--
keywords: +needs review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17611] Move unwinding of stack for "pseudo exceptions" from interpreter to compiler.

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
stage: patch review -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23867] Argument Clinic: inline parsing code for 1-argument functions

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
stage: patch review -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26305] Make Argument Clinic to generate PEP 7 conforming code

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
assignee:  -> serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27181] Add geometric mean to `statistics` module

[Mark Dickinson]

Mark Dickinson added the comment:

> Hmm, well, I don't have SciPy installed, but I've found that despite 
> their (well-deserved) reputation, numpy (and presumably scipy) often 
> have rather naive algorithms that can lose accuracy rather 
> spectacularly.

Agreed. And as Ram Rachum hinted, there seems little point aiming to duplicate 
things that already exist in the de facto standard scientific libraries. So I 
think there's a place for a non-naive carefully computed geometric mean in the 
std. lib. statistics module, but I wouldn't see the point of simply adding an 
exp-mean-log implementation (not that anyone is advocating that).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26305] Make Argument Clinic to generate PEP 7 conforming code

[Roundup Robot]

Roundup Robot added the comment:

New changeset eeb742d8bf9c by Serhiy Storchaka in branch '3.5':
Issue #26305: Argument Clinic now escapes braces. No need to double them.
https://hg.python.org/cpython/rev/eeb742d8bf9c

New changeset d983c313b8f1 by Serhiy Storchaka in branch 'default':
Issue #26305: Argument Clinic now escapes braces. No need to double them.
https://hg.python.org/cpython/rev/d983c313b8f1

New changeset 2d7dcd8cf928 by Serhiy Storchaka in branch 'default':
Issue #26305: Argument Clinic now uses braces in C code as required by PEP 7.
https://hg.python.org/cpython/rev/2d7dcd8cf928

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26305] Make Argument Clinic to generate PEP 7 conforming code

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue23026] Winreg module doesn't support REG_QWORD, small DWORD doc update

[Roundup Robot]

Roundup Robot added the comment:

New changeset 5306f27c53aa by Serhiy Storchaka in branch 'default':
Regenerate Argument Clinic code for issue #23026.
https://hg.python.org/cpython/rev/5306f27c53aa

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26243] zlib.compress level as keyword argument

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

The original request was for supporting level as keyword argument. Making the 
first argument a keyword argument was unintentional side effect (due a to the 
limitation of argument parsing functions). Now it is possible to support 
positional-only and keyword arguments in one function (see issue26282). 
Proposed patch makes the first argument positional-only again.

--
nosy: +serhiy.storchaka
resolution: fixed -> 
stage: resolved -> patch review
status: closed -> open
Added file: 
http://bugs.python.org/file43320/zlib_compress_positional_only_data.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27264] python 3.4 vs. 3.5 strftime same locale different output on Windows

[Eryk Sun]

Changes by Eryk Sun :


--
resolution:  -> third party
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18027] distutils should access stat_result timestamps via .st_*time attributes

[Berker Peksag]

Berker Peksag added the comment:

This is a duplicate of issue 13420.

--
nosy: +berker.peksag
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed
superseder:  -> newer() function in dep_util.py discard changes in the same 
second

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue13420] newer() function in dep_util.py discard changes in the same second

[Jakub Wilk]

Changes by Jakub Wilk :


--
nosy: +jwilk

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27270] 'parentheses-equality' warnings when building with clang and ccache

[STINNER Victor]

STINNER Victor added the comment:

clang_ccache.patch LGTM.

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27281] unpickling an xmlrpc.client.Fault raises TypeError

[Uri Okrent]

New submission from Uri Okrent:

Attempting to unpickle an xmlrpc.client.Fault will raise a TypeError:

>>> import xmlrpc.client as xmlrpclib
>>> f = xmlrpclib.Fault(42, 'Test Fault')
>>> import pickle
>>> s = pickle.dumps(f)
>>> pickle.loads(s)
Traceback (most recent call last):
  File "", line 1, in 
TypeError: __init__() missing 2 required positional arguments: 'faultCode' and 
'faultString'

Apparently unpickle relies on BaseException's args attribute when 
reconstructing an Exception class that inherits from BaseException (Fault 
inherits from Exception).  Unfortunately, Fault implements __init__() and does 
call the base class constructor, but does not pass its arguments, so Fault.args 
is always an empty tuple.

Simply passing Fault's arguments to the base class constructor allows it to be 
unpickled.

I've included a patch for 3.x but the issue is present in 2.x as well (the code 
and fix are exactly the same except in xmlrpclib.py).

--
components: Library (Lib)
files: 0001-xmlrpc.client-make-Fault-pickleable.patch
keywords: patch
messages: 268028
nosy: Uri Okrent
priority: normal
severity: normal
status: open
title: unpickling an xmlrpc.client.Fault raises TypeError
type: behavior
versions: Python 2.7, Python 3.2, Python 3.3, Python 3.4, Python 3.5, Python 3.6
Added file: 
http://bugs.python.org/file43321/0001-xmlrpc.client-make-Fault-pickleable.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27265] Hash of different, specific Decimals created from str is the same

[Radosław Szalski]

Radosław Szalski added the comment:

Thanks for the reply and analysis, Mark.

My motivation was that as a "clueless user", I shouldn't worry about how 
Decimals are created. Given two equal numbers, I would expect their behavior 
(e.g., result of a hash) to be the same as well. In this example, the behavior 
differs simply based on whether the Decimal was created from a string vs a 
float.

However, since there are bound to be collisions, and the performance overhead 
is negligible (in case of a collision), and Python 3 solves this problem I 
agree this can be closed as "won't-fix".

--
status: pending -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26985] Information about CodeType in inspect documentation is outdated

[Xiang Zhang]

Xiang Zhang added the comment:

So maybe remove the docstring entirely?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27271] asyncio lost udp packets

[valdemar pavesi]

valdemar pavesi added the comment:

thanks Guido and Yury

I am new on python world. I was working with automation tests, sw implemented 
in Delphi in 199x.

this year I got a python certification from University Texas Arlington 
University by EDX.

and I already wrote 4 projects in python3 ,handling heavy traffic , one related 
to voice over lte simulation.

thanks for all your contribution to the world without asking nothing back.


regards!
valdemar

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27274] [ctypes] Allow from_pointer creation

[Memeplex]

Memeplex added the comment:

Thank you for the great tips, Eryk, somehow I overlooked string_at while 
reading the docs.

Now, given that the address parameter of string_at is pretty overloaded, 
wouldn't it be reasonable to overload from_address the same instead of 
introducing from_pointer? That is, everywhere an address is expected, an 
address-like ctypes object would be ok.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26985] Information about CodeType in inspect documentation is outdated

[Berker Peksag]

Berker Peksag added the comment:

"Return true if the object is a code object." should stay. We can add a short 
sentence to refer people to the inspect documentation for the list of co_* 
attributes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27274] [ctypes] Allow from_pointer creation

[Memeplex]

Memeplex added the comment:

> The first argument can be any type accepted by c_void_p.from_param, such as a 
> ctypes pointer/array, str, bytes, or an integer address.

Now I see why you suggested ptr.as_void in 26565. Both issues are very related. 
Some functions are overloaded in the sense they automatically call 
c_void_p.from_param on their arguments; other functions aren't. Uniformity 
would dictate to treat all functions expecting an address the same way. Anyway 
this would not cover all use cases for ptr.toaddress or ptr.as_void or whatever 
it gets called.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27243] __aiter__ should return async iterator instead of awaitable

[Nick Coghlan]

Nick Coghlan added the comment:

+1 from me - my only comments were on the docs updates and one of the 
explanatory comments in the code.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27281] unpickling an xmlrpc.client.Fault raises TypeError

[SilentGhost]

Changes by SilentGhost :


--
nosy: +loewis
stage:  -> patch review
versions:  -Python 2.7, Python 3.2, Python 3.3, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27272] random.Random should not read 2500 bytes from urandom

[Tim Peters]

Tim Peters added the comment:

Donald, your script appears to recreate the state from some hundreds of 
consecutive outputs of getrandbits(64).  Well, sure - but what of it?  That 
just requires inverting the MT's tempering permutation.  You may as well note 
that the state can be recreated from the output of random.getstate() - in which 
case, your script could be a lot shorter too ;-)

But that's not what real-life programs expose.  All the flaws in the PHP paper 
related to deducing PRNG state were found in real-life code using idiomatic PHP 
ways of spelling choice() or randrange(), with relatively few possible outputs.

Produce a program that can deduce the state, in Python 3, from - say - a 
million consecutive outputs of randrange(256), and _that_ would be interesting, 
because that would be relevant.  It's easy in Python 2.  But in Python 3, you 
can't tell from the outputs how many times MT was invoked under the covers 
(but, of course, you can from your contrived getrandbits(64) outputs - the 
C-level MT is called exactly twice for each of those outputs).

In any case, the vast bulk of the PHP flaws were found by out-brute-forcing 
dumb PRNG initialization, which requires nothing in the way of reproducing 
state via reverse-engineering outputs (see Figure 13).  Noting that idiomatic 
use of Python 3's choice() (etc) is resistant to the paper's equation-solving 
state-deducing approach is really just a footnote - the _point_ is that lame 
seeding is, all by itself, a Bad Idea.  That alone was enough to crack most of 
the PHP programs the paper covered.

As to the rest, there are already too many massive bug reports arguing about 
urandom()-in-general.  The title of _this_ bug report suggested it may be good 
enough to reduce the _number_ of urandom() bytes MT initialization uses.  But, 
so far, Victor & I appear to be the only ones who made an on-topic comment 
about that ;-)

What do you believe?  For example, do you believe it would remain a disaster if 
MT initialization wanted only 1 byte from urandom()?  Or is 0 the only value 
you can live with?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27271] asyncio lost udp packets

[Guido van Rossum]

Guido van Rossum added the comment:

You're welcome Valdemar. It's a wonderful world, there's so much to learn!
Sounds like you're on the right path. Good luck!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27279] Add random.cryptorandom() and random.pseudorandom, deprecate os.urandom()

[Nick Coghlan]

Nick Coghlan added the comment:

As with other proposals to add new APIs, I think this is an overreaction to a 
Linux specific problem. Linux system boot could deadlock with 3.5.0 and 3.5.1 
due to:

- CPython startup using os.urandom() when it wasn't necessary
- systemd invoking a Python script before the OS entropy pool had been 
initialised (which then deadlocked until the Python invocation timed out)

As long as we switch the internal hash algorithm to seeding from a non-blocking 
random source, and also ensure that importing the random module doesn't 
implicitly call os.urandom, then any other software that only needs 
pseudorandom data can just use the random module APIs.

--
nosy: +ncoghlan

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27281] unpickling an xmlrpc.client.Fault raises TypeError

[Berker Peksag]

Berker Peksag added the comment:

Looks good to me, thanks.

--
nosy: +berker.peksag

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27272] random.Random should not read 2500 bytes from urandom

[Donald Stufft]

Donald Stufft added the comment:

> But that's not what real-life programs expose.

Are you sure? Searching github pulls up a number of results of people calling 
it, but I haven't looked through them to see how/why they're calling it.

> What do you believe?  For example, do you believe it would remain a disaster 
> if MT initialization wanted only 1 byte from urandom()?  Or is 0 the only 
> value you can live with?

I don't really care that much what random.Random initialized with except as it 
related to what os.urandom does by default. Here's a copy/paste from my email 
to python-dev about it:

* Use getrandom(GRND_NONBLOCK) for random.Random since it doesn't matter if we
  get cryptographically secure random numbers or not.
* Switch it to use something other than a CSPRNG by default since it doesn't
  need that.
* Instead of seeding itself from os.urandom on import, have it lazily do that
  the first time one of the random.rand* functions are called.
* Do nothing, and say that ``import random`` relies on having the kernel's
  urandom pool initialized.

Between these options, I have a slight preference for switching it to use a non 
CSPRNG, but I really don't care that much which of these options we pick. Using 
random.Random is not secure and none of the above options meaningfully change 
the security posture of something that accidently uses it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27282] Raise BlockingIOError in os.urandom if kernel is not ready

[Nick Coghlan]

New submission from Nick Coghlan:

This proposal competes directly with #27250, #27266, and #27279 as possible 
long term solutions to the Linux/systemd os.urandom deadlock bug described in 
#26839

Rather than adding new APIs, or making os.urandom potentially blocking on Linux 
(as it was in 3.5.0 and 3.5.1), it instead proposes that os.urandom immediately 
raise BlockingIOError if the kernel entropy pool has not yet been initialised.

This behaviour will mean that users attempting to gather strong entropy too 
early in the Linux boot process will fail rather than block, so affected 
scripts and programs can readily fall back to reading from /dev/urandom or 
using the random module APIs if they don't need cryptographically strong random 
data. Scripts that do need cryptographically strong random data can either poll 
os.urandom until it succeeds, or else fail explicitly and let their caller 
resolve the problem.

--
messages: 268041
nosy: ncoghlan
priority: normal
severity: normal
stage: needs patch
status: open
title: Raise BlockingIOError in os.urandom if kernel is not ready
type: enhancement
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27282] Raise BlockingIOError in os.urandom if kernel is not ready

[Donald Stufft]

Donald Stufft added the comment:

This proposal is reasonable to me and solves any problems I have with the 
default behavior of os.urandom.

--
nosy: +dstufft

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27266] Always use getrandom() in os.random() on Linux and add block=False parameter to os.urandom()

[Nick Coghlan]

Nick Coghlan added the comment:

Since Victor requested it, I filed #27282 to track the "raise BlockingIOError 
if the kernel would block" design option.

The key advantage that particular model offers is that it's trivial to build a 
blocking version as a busy loop around the non-blocking version:

def urandom_wait_for_entropy(num_bytes):
while True:
try:
return os.urandom(num_bytes)
except BlockingIOError:
pass

And if you ignore the problem and just call os.urandom(), you'll almost 
certainly be fine unless you're working with Linux boot scripts or embedded ARM 
devices (in which case, this point will be minor compared to the other arcana 
you're dealing with).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27282] Raise BlockingIOError in os.urandom if kernel is not ready

[Nick Coghlan]

Nick Coghlan added the comment:

Quoting http://bugs.python.org/issue27266#msg268043:

The key advantage the BlockingIOError model offers is that it's trivial to 
build a blocking version as a busy loop around the non-blocking version:

def urandom_wait_for_entropy(num_bytes):
while True:
try:
return os.urandom(num_bytes)
except BlockingIOError:
pass

And if you ignore the problem and just call os.urandom(), you'll almost 
certainly be fine unless you're working with Linux boot scripts or embedded ARM 
devices (in which case, this point will be minor compared to the other arcana 
you're dealing with).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27265] Hash of different, specific Decimals created from str is the same

[Mark Dickinson]

Mark Dickinson added the comment:

> the behavior differs simply based on whether the Decimal was created from a 
> string vs a float

That's not quite right: a Decimal object keeps no knowledge of how it was 
created. The behaviour differs depending on whether the value of the Decimal 
happens to be exactly representable as a Python float or not. That's necessary 
to ensure the invariant `x == y` implies `hash(x) == hash(y)` continues to hold 
across types (though Python 3 has a better way of doing this).

So for example `Decimal('0.375')` was created from a string, but will hash 
equal to the exactly equal float `0.375`:

noether:float-proofs mdickinson$ python2
Python 2.7.11 (default, May  1 2016, 08:20:00) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from decimal import Decimal
>>> hash(Decimal('0.375')), hash(Decimal(0.375))
(1610579968, 1610579968)
>>> hash(0.375)
1610579968

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27281] unpickling an xmlrpc.client.Fault raises TypeError

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27272] random.Random should not read 2500 bytes from urandom

[Tim Peters]

Tim Peters added the comment:

> Searching github pulls up a number of results of people
> calling it, but I haven't looked through them to see
> how/why they're calling it.

Sorry, I don't know what "it" refers to.  Surely not to a program exposing the 
output of .getstate()?!

Regardless, there was a long discussion about the `secrets` module at the time, 
and nobody found any real code vulnerable to the approaches in the PHP paper 
under Python 3 (contrived code, certainly - that's easy).  Again, exploiting 
lame seeding alone sufficed to crack most of their examples, and Python's use 
of urandom() for seeding eliminated that approach (in Python 2 too).  

Examples potentially vulnerable to state equation-solving were "just like" what 
the PHP coders rolled by hand:  uses of things like .choice() and .randrange() 
to build "random" strings (passwords, session tokens, ...), from relatively 
small alphabets.  The smaller the alphabet, the more resistant Python 3 is to 
this approach, because the more likely ._randbelow() will invisibly skip over 
MT outputs.

For a while an incorrect claim was mistakenly accepted:  that when 
len(alphabet) was a power of 2, choice(alphabet) made an always-known number of 
MT calls.  If that were true, the equation solver could deduce the state 
quickly in such cases, which are relatively common.  But it's false - 
._randbelow() is actually _most_ likely to skip over MT outputs when it's 
making a choice from a power-of-2 number of possibilities.  That's not obvious 
from a glance at the code.

I remain -1 on making seeding "dumb" again.  But I don't care whether urandom() 
returns low-quality bytes in the boot-time edge cases people are upset about.  
They're still likely to be "better" than anything spun out of stuff like 
time.time().

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27272] random.Random should not read 2500 bytes from urandom

[Donald Stufft]

Donald Stufft added the comment:

> Sorry, I don't know what "it" refers to.  Surely not to a program exposing 
> the output of .getstate()?!

random.getrandbits()

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27281] unpickling an xmlrpc.client.Fault raises TypeError

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Why you need to pickle Fault?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27281] unpickling an xmlrpc.client.Fault raises TypeError

[Uri Okrent]

Uri Okrent added the comment:

I'm not pickling/unpickling it directly, I'm using multiprocessing to handle 
queries to my server in worker processes which is using pickle to propagate 
exceptions raised in the worker to the parent.

I could instead raise a different exception and wrap it in a Fault later (which 
is what I ended up doing to avoid the issue), but it seems like a case of 
"to-may-to" vs. "to-mah-to".  Either way an Exception should be able to be 
propagated up to the parent in multiprocessing, whether it's my own or Fault.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27272] random.Random should not read 2500 bytes from urandom

[Tim Peters]

Tim Peters added the comment:

Ah!  Yes, .getrandbits(N) outputs remain vulnerable to equation-solving in 
Python 3, for any value of N.  I haven't seen any code where that matters (may 
be "a security hole"), but would bet some _could_ be found.

There's no claim of absolute security here.  To the contrary.  What I'm opposed 
to is making _all_ naive code vulnerable to easy script-kiddie brute force 
attacks against lame seeding.

The kinds of things people _were_ jumping up & down about were the many 
instances of stuff like this on the web:

https://stackoverflow.com/questions/3854692/generate-password-in-python

Again, I'd be impressed if you could write code under Python 3 to deduce the MT 
state from any number of outputs from his naive approach in reasonable time.  
Of course he should be using urandom() instead (as an unaccepted answer urges) 
- but much code plain doesn't, and in Python 3 it's resistant to any attack the 
PHP paper exposed.

Make seeding lame again, and the easiest attacks can succeed again (the 
equation-solving stuff remains a footnote to me).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27265] Hash of different, specific Decimals created from str is the same

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Note that Decimal(0.05) != Decimal('0.05').

>>> Decimal(0.05)
Decimal('0.05000277555756156289135105907917022705078125')
>>> hash(Decimal(0.05))
966367654
>>> hash(Decimal('0.05000277555756156289135105907917022705078125'))
966367654
>>> hash(0.05)
966367654

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com