Changes by Richard Oudkerk :
--
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue16454>
___
___
Python-bugs-list mailing list
Unsubscri
Richard Oudkerk added the comment:
In Torsten's example
from . import moduleX
can be replaced with
moduleX = importlib.import_module('.moduleX', __package__) (*)
or
moduleX = importlib.import_module('package.moduleX')
If that is not pretty en
New submission from Richard Oudkerk:
On Windows the handle for a child process is not closed when the popen/process
object is garbage collected.
--
messages: 175629
nosy: sbt
priority: normal
severity: normal
stage: needs patch
status: open
title: process handle leak on windows in
Richard Oudkerk added the comment:
Fixed in c574ce78cd61 and cb612c5f30cb.
--
resolution: -> fixed
stage: needs patch -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.or
Richard Oudkerk added the comment:
pthread_atfork() allows the registering of three types of callbacks:
1) prepare callbacks which are called before the fork,
2) parent callbacks which are called in the parent after the fork
3) child callbacks which are called in the child after the fork.
I
Richard Oudkerk added the comment:
Note that Gregory P. Smith has written
http://code.google.com/p/python-atfork/
I also started a pure python patch but did not get round it posting it. (It
also implements the fork lock idea.) I'll attach it here.
How do you intend to handl
Richard Oudkerk added the comment:
> IFF we are going to walk the hard and rocky road of exception handling,
> then we are going to need at least four hooks and a register function that
> takres four callables as arguments: register(prepare, error, parent,
> child). Each prepare() ca
Richard Oudkerk added the comment:
http://hg.python.org/sandbox/sbt#spawn now contains support for starting
processes via a separate server process. This depends on fd passing support.
This also solves the problem of mixing threads and processes, but is much
faster than using fork+exec. It
Richard Oudkerk added the comment:
I am reopening this issue because 26bbff4562a7 only dealt with objects which
cannot be pickled. But CalledProcessError instances *can* be pickled: the
problem is that the resulting data cannot be unpickled.
Note that in Python 3.3 CalledProcessError can be
Richard Oudkerk added the comment:
The example works correctly on 3.3 because of #1692335. I am not sure if it is
appropriate to backport it though.
This is a duplicate of #9400 which I have assigned to myself. (I had thought
it was already fixed.)
--
resolution: -> duplic
Richard Oudkerk added the comment:
The patch is liable to break programs which explicitly call base constructors
because the constructor will be called more than once.
It also assumes that the __init__() method of all base classes should be called
with no arguments (other than self). That is
Richard Oudkerk added the comment:
> But I think the problem remains: do you agree that Classes should include
> a super() call in their __init__ ?
No, I don't.
I think super() is an attractive nuisance which is difficult to use correctly
in an __init__() method, except in the t
Changes by Richard Oudkerk :
--
nosy: +sbt
___
Python tracker
<http://bugs.python.org/issue16579>
___
___
Python-bugs-list mailing list
Unsubscribe:
Richard Oudkerk added the comment:
Attached is an alternative patch which only touches selectmodule.c. It still
does not support WinXP.
Note that in this version register() and modify() do not ignore the POLLPRI
flag if it was *explicitly* passed. But I am not sure how best to deal with
Richard Oudkerk added the comment:
Here is a version which loads WSAPoll at runtime. Still no tests or docs.
--
Added file: http://bugs.python.org/file28207/runtime_wsapoll.patch
___
Python tracker
<http://bugs.python.org/issue16
New submission from Richard Oudkerk:
PollTests.poll_unit_tests() is not run because its method name does not begin
with "test". It looks it was accidentally disabled when test_poll() was
converted to unittest in f56b25168142.
Renaming it test_poll_unit_tests() makes it run succes
Richard Oudkerk added the comment:
It seems that the return code of WSAPoll() does not include the count of array
items with revents == POLLNVAL. In the case where all of them are POLLNVAL,
instead of returning 0 (which usually indicates a timeout) it returns -1 and
WSAGetLastError
Changes by Richard Oudkerk :
--
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue16616>
___
___
Python-bugs-list mailing list
Unsubscri
Changes by Richard Oudkerk :
--
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue15526>
___
___
Python-bugs-list mailing list
Unsubscri
Richard Oudkerk added the comment:
Here is a new version with tests and docs.
Note that the docs do not mention the bug mentioned in
http://daniel.haxx.se/blog/2012/10/10/wsapoll-is-broken/
Maybe they should?
Note that that bug makes it a bit difficult to use poll with tulip on Windows
Richard Oudkerk added the comment:
When you run wy.py the wow module gets partially imported, and then garbage
collected because it fails to import successfully. The destructor for the
module replaces values in the module's __dict__ with None. So when the cleanup
function runs you ge
Richard Oudkerk added the comment:
> Things should be better in the future, when modules are cleared with true
> garbage collection.
When is this future of which you speak?
I am not sure whether it would affect performance, but a weakrefable subclass
of dict could be used for module
Richard Oudkerk added the comment:
> See issue812369 for the shutdown procedure and modules cleanup.
I am aware of that issue, but the original patch is 9 years old. Which is why
I ask if/when it will actually happen.
--
___
Python tracker
&l
New submission from Richard Oudkerk:
Relevant code:
int timeout = 0, poll_result, i, j;
...
tout = PyNumber_Long(tout);
if (!tout)
return NULL;
timeout = PyLong_AsLong(tout); <-- implicit cast to int
--
messages: 177811
nosy: sbt
prior
Richard Oudkerk added the comment:
Thanks. I will close.
--
stage: -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.org/i
Richard Oudkerk added the comment:
Perhaps the simplest thing would be to stop doing anything special when a
module is garbage collected: the garbage collector can take care of any
orphaned ref-cycles involving the module dict. Then at shutdown the remaining
modules in sys.modules could have
Richard Oudkerk added the comment:
I suspect that the size of the 5GB file is originally a 64 bit quantity, but
gets cast unsafely to a 32 bit size_t to give 1GB. This is causing the
miscalculations.
There is no way to map all of a 5GB file in a 32 bit process -- 4GB is the
maximum -- so
Richard Oudkerk added the comment:
This bit looks wrong to me:
if (offset - size > PY_SSIZE_T_MAX)
/* Map area too large to fit in memory */
m_obj->size = (Py_ssize_t) -1;
Should it not be "size - offset" instead of "offset - siz
Richard Oudkerk added the comment:
On 32 bit Unix mmap() will raise ValueError("mmap length is too large") in
Marc's example. This is correct since Python's sequence protocol does not
support indexes larger than sys.maxsize.
But on 32 bit Windows, if length == 0 then t
Richard Oudkerk added the comment:
> This change is not backward compatible. Now user can mmap a larger file
> and safely access lower 2 GiB. With the patch it will fail.
They should specify length=2GiB-1 if that is what they want.
With length=0 you can only access the lower 2
Richard Oudkerk added the comment:
New patch with same check for Unix.
--
Added file: http://bugs.python.org/file28446/mmap.patch
___
Python tracker
<http://bugs.python.org/issue16
Richard Oudkerk added the comment:
> Isn't 2 GiB + 1 bytes mmap file enough for testing?
Yes.
But creating multigigabyte files is very slow on Windows. On Linux/FreeBSD
test_mmap takes a fraction of a second, whereas on Windows it takes over 2
minutes. (Presumably Linux/Fr
Changes by Richard Oudkerk :
--
title: mmap accepts files > 1 GB, but processes only 1 GB -> mmap on Windows
can mishandle files larger than sys.maxsize
type: enhancement -> behavior
___
Python tracker
<http://bugs.python.or
Richard Oudkerk added the comment:
> Richard, apart from performance, what's the advantage of this approach over
> the
> fork+exec version?
It is really just performance. For context running the unittests in a 1 cpu
linux VM gives me
fork:
real0m53.868s
user
Richard Oudkerk added the comment:
Numbers when running on Linux on a laptop with 2 cores + hyperthreading.
RUNNING UNITTESTS:
fork:
real0m50.687s
user0m9.213s
sys 0m4.012s
fork+exec:
real1m9.062s
user0m48.579s
sys 0m6.648s
forkserver
Changes by Richard Oudkerk :
Added file: http://bugs.python.org/file28461/8f08d83264a0.diff
___
Python tracker
<http://bugs.python.org/issue8713>
___
___
Python-bug
Richard Oudkerk added the comment:
> The safest default would be fork+exec though we need to implement the
> fork+exec code as a C extension module or have it use subprocess (as I
> noted in the mb_fork_exec.patch review).
That was an old version of the patch.
In the branch
New submission from Richard Oudkerk:
The actual signature is
socket.socket(family=AF_INET, type=SOCK_STREAM, proto=0, fileno=None)
but the documented signature is
socket.socket([family[, type[, proto]]])
Should the fileno argument be documented or is it considered an implementation
Richard Oudkerk added the comment:
> The "fileno" argument looks like an implementation detail to me.
It has at least one potential use. On Windows socket.detach() returns a socket
handle but there is no documented way to close it -- os.close() will not work.
The only way to cl
Richard Oudkerk added the comment:
> There is an alternative (documented) interface:
>
> socket.fromfd(handle, socket.AF_INET, socket.SOCK_STREAM).close()
socket.fromfd() duplicates the handle, so that does not close the origina
Changes by Richard Oudkerk :
--
resolution: -> fixed
stage: -> committed/rejected
status: open -> closed
___
Python tracker
<http://bugs.python.o
Richard Oudkerk added the comment:
Note that on Windows there is an O_NOINHERIT flag which almost corresponds to
O_CLOEXEC on Linux.
I don't think there is a need to use the win32 api.
--
nosy: +sbt
___
Python tracker
<http://bugs.py
Richard Oudkerk added the comment:
A while ago I did write a PipeIO class which subclasses io.RawIOBase and works
for overlapped pipe handles. (It was intended for multiprocessing and doing
asynchronous IO with subprocess.)
As it is it would not work with normal files because when you do
Richard Oudkerk added the comment:
Attached is a module for Python 3.3+ which subclasses io.RawIOBase. The
constructor signature is
WinFileIO(handle, mode="r", closehandle=True)
where mode is "r", "w", "r+" or "w+". Handles can be
Changes by Richard Oudkerk :
Added file: http://bugs.python.org/file28545/test_winfileio.py
___
Python tracker
<http://bugs.python.org/issue12939>
___
___
Python-bug
Richard Oudkerk added the comment:
Is this actually a problem?
If events are arranged in a queue and epoll_wait() just removes the oldest
events (up to maxevents) from that queue then there would be no problem with
using a small value for maxevents.
I don't *know* if that is the case,
Richard Oudkerk added the comment:
Attached is a patch which adds a winio module which is a replacement for io,
but uses windows handles instead of fds.
It reimplements FileIO and open(), and provides openhandle() and closehandle()
as replacements for os.open() and os.close().
test_io has
Richard Oudkerk added the comment:
> I actually wrote a script to reproduce this issue:
The program does *not* demonstrate starvation because you are servicing the
resource represented by the "starved" duplicate fds before calling poll() again.
You are creating thousands of dupl
Richard Oudkerk added the comment:
> The fact that that the FDs are duped shouldn't change anything to the
> events reported: it works while the number of FDs is less than
> FD_SETSIZE (epoll_wait() maxevents argument).
That assumes that epoll_wait() is supposed to return *all* r
Richard Oudkerk added the comment:
Here is a version which uses epoll to service a number of pipes which is larger
than maxevents. (If NUM_WRITERS is too large then I get "OSError: [Errno 24]
Too many open files".)
All pipes get serviced and the output is:
Working with 20 FDs, 5
Richard Oudkerk added the comment:
> Yes, but the problem is that between two epoll_wait() calls, the
> readiness of the FDs can have changed: and if that happens, you'll get
> the same list over and over.
If an fd *was* ready but isn't anymore then why would you want to know
Richard Oudkerk added the comment:
> I don't like the idea of a specific I/O module for an OS. Is the public API
> different?
It was partly to make integration with the existing tests easier: _io, _pyio
and winio are tested in parallel.
> Can't you reuse the io module?
Richard Oudkerk added the comment:
> Richard, in Tulip's WSAPoll code, it reads:
>
> class WindowsPollPollster(PollPollster):
> """Pollster implementation using WSAPoll.
>
> WSAPoll is only available on Windows Vista and later. Pytho
Richard Oudkerk added the comment:
> try:
> _MAXFD = os.sysconf("SC_OPEN_MAX")
> -except:
> +except ValueError:
> _MAXFD = 256
os.sysconf() might raise OSError. I think ValueError is only raised if
_SC_OPEN_MAX was undefined when the module was compile
Richard Oudkerk added the comment:
Why are you connecting to a multiprocessing listener with a raw socket? You
should be using multiprocessing.connection.Client to create a client connection.
Connection.send(obj) writes a 32 bit unsigned int (in network order) to the
socket representing the
Richard Oudkerk added the comment:
Attached is a new patch which is implemented completely in C.
It adds a WinFileIO class to the io module, which has the same API
as FileIO except that:
* It has a handle attribute instead of a fileno() method.
* It has staticmethods openhandle() and
Changes by Richard Oudkerk :
Removed file: http://bugs.python.org/file28544/winfileio.c
___
Python tracker
<http://bugs.python.org/issue12939>
___
___
Python-bugs-list m
Changes by Richard Oudkerk :
Removed file: http://bugs.python.org/file28545/test_winfileio.py
___
Python tracker
<http://bugs.python.org/issue12939>
___
___
Python-bug
Changes by Richard Oudkerk :
Removed file: http://bugs.python.org/file28590/winfileio.patch
___
Python tracker
<http://bugs.python.org/issue12939>
___
___
Python-bug
Richard Oudkerk added the comment:
Forgot to mention, the handles are non-inheritable.
You can use _winapi.DuplicateHandle() to create an inheritable duplicate handle
if you really need to.
--
___
Python tracker
<http://bugs.python.org/issue12
Richard Oudkerk added the comment:
> If someone used regular sockets deliberately, they could crash
> multiprocessing server code deliberately. Any chance of doing a real message
> length check against the embedded message length check?
You can do
message = conn.recv_bytes(maxle
Richard Oudkerk added the comment:
Thanks for the report.
It seems to only affect Windows, and only when using sockets rather than pipes.
Till this is fixed you could use
temp = bool(multiprocessing.connection.wait([cl], 1))
instead of
temp = cl.poll(1)
As I mentioned on the other
Richard Oudkerk added the comment:
The commits did not have the intended effect.
They just define a _poll() function (and only on Windows) and it is not
referenced anywhere else.
I will look in to fixing this -- on 2.7 and 3.2 this will need to be done in
the C code.
--
resolution
Richard Oudkerk added the comment:
This should be fixed now.
--
resolution: -> fixed
stage: -> committed/rejected
status: open -> closed
type: -> behavior
___
Python tracker
<http://bugs.python
Richard Oudkerk added the comment:
> What do you mean? The intent was to use poll() instead of select()
> anywhere available in order to avoid running out of fds.
> The change didn't affect Windows because as of right now select() is
> the only thing available.
The chan
Richard Oudkerk added the comment:
It looks like the change to multiprocessing/connection.py committed does not
match the one uploaded as issue10527-3.patch
changeset 81174:e971a70984b8
1.1 --- a/Lib/multiprocessing/connection.py
1.2 +++ b/Lib/multiprocessing/connection.py
1.3
Richard Oudkerk added the comment:
> Added some comments on Rietveld.
> The .fileno() method is missing. Can this cause a problem when the file
> is passed to stdlib functions? subprocess for example?
Thanks. An older version of the patch had a fileno() method which returned th
Richard Oudkerk added the comment:
> What does this proposal bring exactly?
Unless we are willing to completely replace fds with handles on Windows,
perhaps not too much. (At one point I had assumed that that was the plan for
py3k.)
Although not advertised, openhandle() does hav
Richard Oudkerk added the comment:
If you want to communicate between processes of the same progam, you are best
off calling multiprocessing.Pipe() or multiprocessing.Queue() in the main
process. Queues or connections can then be inherited by the child processes.
Usually all communication
Richard Oudkerk added the comment:
For the reasons I wrote in the other issue, I don't think this an approach to
encourage.
There was no need to create a new issue: if you post to a closed issue then
people on the nosy list will still see your message.
So I will close this issue.
(
Richard Oudkerk added the comment:
New patch reflecting Amaury's comments.
--
Added file: http://bugs.python.org/file28737/winfileio.patch
___
Python tracker
<http://bugs.python.org/is
Changes by Richard Oudkerk :
Removed file: http://bugs.python.org/file28707/winfileio.patch
___
Python tracker
<http://bugs.python.org/issue12939>
___
___
Python-bug
Richard Oudkerk added the comment:
> I thought that access to the value field of Value instances was
> protected by locks to avoid lost updates.
Loads and stores are both atomic. But "+=" is made up of two operations, a
load followed by a store, and the lock is dropped betwe
Richard Oudkerk added the comment:
> I see. Then this is a documentation bug. The examples in the
> documentation use such non-thread-safe assignments (combined with the
> statement "These shared objects will be process and thread safe.").
Are you talking about this d
Richard Oudkerk added the comment:
> That compiles (after hacking the line endings). One Tulip test fails,
> PollEventLooptests.testSockClientFail. But that's probably because the
> PollSelector class hasn't been adjusted for Windows yet (need to dig this
> out of the
Richard Oudkerk added the comment:
On 21/01/2013 5:38pm, Guido van Rossum wrote:
> This is a very good question to which I have no good answer.
> If it weren't for this, we could probably do away with the distinction
> between add_writer and add_connector, and a lot of code coul
Richard Oudkerk added the comment:
On 21/01/2013 7:00pm, Guido van Rossum wrote:
> Regarding your IOCP changes, that sounds pretty exciting. Richard,
> could you check those into the Tulip as a branch? (Maybe a new branch
> named 'iocp'.)
OK. It may take me a w
Richard Oudkerk added the comment:
I have created an iocp branch.
--
___
Python tracker
<http://bugs.python.org/issue16507>
___
___
Python-bugs-list mailin
Richard Oudkerk added the comment:
It appears that Linux's "spurious readiness notifications" are a deliberate
deviation from the POSIX standard. (They are mentioned in the BUGS section of
the man page for select.)
Should I just apply the following patch to the default b
Richard Oudkerk added the comment:
According to Alan Cox
It's a design decision and a huge performance win. It's one of the areas
where POSIX read in its strictest form cripples your performance.
See https://lkml.org/lkml/2011/6/18/103
> (For write ready, you can o
Richard Oudkerk added the comment:
> For SOCK_STREAM, yes, not for SOCK_DGRAM
I thought SOCK_DGRAM messages just got truncated at the receiving end.
--
___
Python tracker
<http://bugs.python.org/issu
Changes by Richard Oudkerk :
--
nosy: +sbt
___
Python tracker
<http://bugs.python.org/issue17018>
___
___
Python-bugs-list mailing list
Unsubscribe:
Richard Wall added the comment:
This bug also causes problems when you try to install Python in a Linux chroot
environment or systemd-nspawn - before mounting devtmpfs.
For example, this Redhat bug "yum traceback with python-2.6.6-29.el6_2.2 and
higher + missing /dev/urandom"
Richard Oudkerk added the comment:
On 27/01/2013 8:27pm, Terry J. Reedy wrote:
> I agree we do not need to retain unpredictable 'dumb luck' -- in
> future versions. But in the absence of a clear discrepancy
> between doc and behavior (the definition of a bug) I believe
> b
Richard Oudkerk added the comment:
On 27/01/2013 9:06pm, Serhiy Storchaka wrote:
> Every bugfix breaks some code. As a compromise I propose set
> m_obj->size = PY_SSIZE_T_MAX in case of overflow and emit a warning.
Trying to allocate PY_SSIZE_T_MAX bytes always seems to
Changes by Richard Oudkerk :
--
nosy: +sbt
___
Python tracker
<http://bugs.python.org/issue17097>
___
___
Python-bugs-list mailing list
Unsubscribe:
Richard Oudkerk added the comment:
There are lots of things that behave differently depending on the currently set
start method: Lock(), Semaphore(), Queue(), Value(), ... It is not just when
creating a Process or Pool that you need to know the start method.
Passing a context or start_method
Richard Oudkerk added the comment:
I'll review the patch. (According to http://www.python.org/dev/peps/pep-0429/
feature freeze is expected in late November, so there is not too much of rush.)
--
___
Python tracker
<http://bugs.py
Richard Oudkerk added the comment:
An alternative would be to have separate files NEWS-3.2, NEWS-3.3, NEWS-3.4
etc. If a fix is added to 3.2 and will be merged to 3.3 and 3.4 then you add
an entry to NEWS-3.2 and append some sort of tags to indicate merges:
- Issue #1234: Fix something
Richard Oudkerk added the comment:
Thanks for the doc cleanup -- I am rather busy right now.
Note that stuff does still get replaced by None at shutdown, and this can still
produce errors, even if they are much harder to trigger. If I run the
following program
import _weakref
import
Richard Oudkerk added the comment:
The clearing of modules at shutdown has been substantially changed in 3.4. Now
a best effort is made to let the module go away purely by gc. Those modules
which survive get purged in random order.
In 3.3 all modules were purged, but builtins was special
Richard Oudkerk added the comment:
An alternative would be to use weakref.finalize() which would guarantee that
cleanup happens before any purging occurs. That would allow the use of shutil:
class TemporaryDirectory(object):
def __init__(self, suffix="", prefix=template
Changes by Richard Oudkerk :
--
nosy: +sbt
___
Python tracker
<http://bugs.python.org/issue19112>
___
___
Python-bugs-list mailing list
Unsubscribe:
Richard Oudkerk added the comment:
See
http://bugs.python.org/issue436259
This is a problem with Window's implementation of spawn*() and exec*(). Just
use subprocess instead which gets this stuff right.
Note that on Windows exec*() is useless: it just starts a subprocess and exits
Richard Oudkerk added the comment:
> I am not sure that I should see there. There is discussion of DOS,
> which is not supported, also some complain about Windows execv
> function, which deprecated since VC++ 2005 (which I hope also not
> supported). Can you be more specific?
_
Richard Oudkerk added the comment:
As I wrote in http://bugs.python.org/issue19066, on Windows execv() is
equivalent to
os.spawnv(os.P_NOWAIT, ...)
os._exit(0)
This means that control is returned to cmd when the child process *starts* (and
afterwards you have cmd and the child
Richard Oudkerk added the comment:
> It is said that execv() is deprecated, but it is not said that it is
> alias of _execv(). It is only said that _execv() is C++ compliant.
> http://msdn.microsoft.com/en-us/library/ms235416(v=vs.90).aspx
Microsoft seems to have decided that all fun
Richard Oudkerk added the comment:
> Where did you get that info? MSDN is silent about that.
> http://msdn.microsoft.com/en-us/library/886kc0as(v=vs.90).aspx
Reading the source code for the C runtime included with Visual Studio.
> The problem is not in what I should or should not
Richard Oudkerk added the comment:
> Hey. This ticket is about os.execv failing on spaced paths on Windows. It
> is not a duplicate of issue19124.
It is a duplicate of #436259 "[Windows] exec*/spawn* problem with spaces in
args".
--
__
Richard Oudkerk added the comment:
> Visual Studio 10+ ? Is it available somewhere for a reference?
Old versions of the relevant files are here:
http://www.controllogics.com/software/VB6/VC98/CRT/SRC/EXECVE.C
http://www.controllogics.com/software/VB6/VC98/CRT/SRC/SPAWNVE.C
h
601 - 700 of 1064 matches
Mail list logo
