Charles-Francois Natali added the comment:
Actually, it's possible that select(2) incorrectly reports sockets as ready for
reading : for example if the socket receives data, but then discards it because
of an invalid checksum (and I guess you're more likely to get this type of
pr
Charles-Francois Natali added the comment:
Actually, looking at the man page, this seems to be normal:
---
In order to change the owner or group, you must own the file and have the CHOWN
privilege (see setprivgrp(1M)). [...] Note that a given user's or group's
ability to use this c
Charles-Francois Natali added the comment:
Unfortunately, select doesn't necessarily update the timeout variable with the
remaining time, so we can't rely on this. This would mean having the select
enclosed within gettimeofday and friends, which seems a bi
Charles-Francois Natali added the comment:
Yes, I was concerned with the runtime overhead (especially under VMs, where
clocks are emulated) and strace output ;-)
As for the dull code writting, I could give it a shot unless someone else wants
to go ahead
Charles-Francois Natali added the comment:
Any thoughts on this?
--
___
Python tracker
<http://bugs.python.org/issue2643>
___
___
Python-bugs-list mailin
Charles-Francois Natali added the comment:
@ehohenstein: sorry, I hadn't seen you'd already submitted a patch, so you may
as well update it to take into account what was said above. Also, please note
that this issue is not specific to windows, so it would be nice to fix it for
Charles-Francois Natali added the comment:
Following EAFP principle, it would be better - cleaner and more efficient - to
put the stuct.unpack inside a try/except clause than checking the lengths
beforehand.
--
nosy: +neologix
___
Python tracker
Charles-Francois Natali added the comment:
Skimming through the code, the only place where we can reasonably block is
inside HTTPConnection._send_output(), when the data is actually sent to the
socket.
What probably happens is the following:
- connect() call succeeded, we have an established
Charles-Francois Natali added the comment:
I tested it on a Windows XP box, and encountered the same problem.
The error is raised because Windows XP requires the socket to be bound before
calling setsockopt(IPPROTO_IP, IP_ADD_MEMBERSHIP, mreq).
So calling bind() before setsockopt() solves this
Charles-Francois Natali added the comment:
> Are you sure of that? I don't see how Python behaviour would be different to
> a the same program written in C. Could you write a short example written in C
> to prove that?
I also found this surprising, so I wroke a quick C progra
Charles-Francois Natali added the comment:
> .. even with a self-compiled 1.2.3, INT_MAX/1000 ... nothing.
> The problem is not crc32(), but the buffer itself:
>
>if (pbuf.len > 1024*5) {
> unsigned char *buf = pbuf.buf;
> Py_ssize_t len = pbuf.len;
&
Changes by Charles-Francois Natali :
Removed file: http://bugs.python.org/file20815/urllib_ftp_close_27.diff
___
Python tracker
<http://bugs.python.org/issue11
Changes by Charles-Francois Natali :
Added file: http://bugs.python.org/file20847/urllib_ftp_close_27.diff
___
Python tracker
<http://bugs.python.org/issue11199>
___
___
Changes by Charles-Francois Natali :
Removed file: http://bugs.python.org/file20814/urllib_ftp_close.diff
___
Python tracker
<http://bugs.python.org/issue11199>
___
___
Charles-Francois Natali added the comment:
Attached are two new versions which don't wait for the end of transfer.
--
Added file: http://bugs.python.org/file20848/urllib_ftp_close.diff
___
Python tracker
<http://bugs.python.org/is
Charles-Francois Natali added the comment:
To elaborate on this, to my knowledge, there's no portable and reliable way to
close all open file descriptors.
Even with the current code, it's still possible that some FD aren't properly
closed, since getconf(SC_OPEN_MAX
Charles-Francois Natali added the comment:
Alright, it's normal behaviour, but since it doesn't seem to be documented, it
can be quite surprising.
A queue works like this:
- when you call queue.put(data), the data is added to a deque, which can grow
and shrink forever
- then a t
Charles-Francois Natali added the comment:
> If you stop to think about it, though, this is actually a shockingly huge
> percent increase. In any process creation scenario I'm familiar with, its
> overhead should be so small that you could bump it up several orders of
> ma
Charles-Francois Natali added the comment:
Could you try with this:
def setUp(self):
with open(support.TESTFN, "wb+") as f:
f.seek(_4G)
f.write(b"asdf")
f.flush()
+ os.fsync(f.fileno())
self
Charles-Francois Natali added the comment:
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0018
Crashed Thread: 0 Dispatch queue: com.apple.main-thread
Python crashes when dereferencing 0x0018, which is NULL + 24
This means that
Charles-Francois Natali added the comment:
True with the following:
import time
time.asctime((2011, 2, 26, -1, 0, 0, 0, 0, 0))
You'll get a segfault.
--
___
Python tracker
<http://bugs.python.org/is
Charles-Francois Natali added the comment:
This explains why you don't get a segfault: your libc is broken ;-)
--
___
Python tracker
<http://bugs.python.org/is
Charles-Francois Natali added the comment:
> If it's broken, complain to Apple.
Actually, I checked glibc's asctime, and it has the same behaviour. But the
point is that asctime can return NULL.
> That still doesn't explain the OP's crash on OS X 10.6.6.
Yes it doe
Charles-Francois Natali added the comment:
I updated my local svn checkout, and the code has been fixed recently:
r87648 | alexander.belopolsky | 2011-01-02 15:48:22 -0500 (Sun, 02 Jan 2011)
| 1 line
Issue #8013: Fixed time.asctime segfault when OS's asctime fails
p = as
Charles-Francois Natali added the comment:
> Errno 75 is EOVERFLOW here (Linux), which doesn't seem to be mentioned in
> sendfile's man page.
> Can you describe your system? (CPU architecture, bitness, endianness,
> glibc/kernel version, etc.)
Just checked
Charles-Francois Natali added the comment:
> Aaron Sherman added the comment:
>
> "That's why I asked for absolute numbers for the overhead difference."
>
> Did you not follow the link in my first post? I got pretty detailed, there.
>
By the way, strace ha
Charles-Francois Natali added the comment:
> If niceness is high, then maybe this test could try to decrease it instead of
> increasing.
You must be root or have CAP_SYS_NICE to do that.
--
nosy: +neologix
___
Python tracker
Charles-Francois Natali added the comment:
Alright, digging a little further, it's a kernel bug introduced by this commit:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=4a3956c790290efeb647bbb0c3a90476bb57800e
Note that this should only affect 64-bit sy
Charles-Francois Natali added the comment:
@Eric.Wolf
Could you try with this:
# Read in anohter chunk of the file
# NOTE: It's possible that an XML tag will be greater than buffsize
# This will break in that situation
-newb = self.fp
Charles-Francois Natali added the comment:
> I've attached the strace output. I was getting an error with the sbrk
> parameter, so I left it out.
Yeah, sbrk is not a syscall ;-)
> Let me know if there's anything else I can provide.
Stupid questions:
- have you checke
Charles-Francois Natali added the comment:
After running this under gdb, it turns out that it's actually bzlib's bzRead
that's returning a BZ_STREAM_END after only 900k bytes.
So it confims what I've been suspecting, i.e. that the file is corrupt (I got
the error at exactl
Charles-Francois Natali added the comment:
> Stupid questions are always worth asking. I did check the MD5 sum earlier
> and just checked it again (since I copied the file from one machine to
> another):
>
> ebwolf@ubuntu:/opt$ md5sum /host/full-planet-1101
Charles-Francois Natali added the comment:
> Antoine Pitrou added the comment:
>
> Perhaps your bz2 files are simply multi-stream files? The bz2 module
> currently doesn't support them (it only decompresses the first stream); see
> issue1625 for a patch.
That explains why
Charles-Francois Natali added the comment:
2011/3/2 Eric Wolf :
>
> Eric Wolf added the comment:
>
> I just got confirmation that OSM is using pbzip2 to generate these files. So
> they are multi-stream. At least that gives a final answer but doesn't solve
> my probl
Charles-Francois Natali added the comment:
> pitrou> I think your analysis is wrong. These mmap() calls are for
> pitrou> anonymous memory, most likely they are emitted by the libc's
> pitrou> malloc() to get some memory from the kernel. In other words
> pitrou>
Charles-Francois Natali added the comment:
> So, even though implemented in C, the file descriptor closing logic is still
> quite costly!
Yes, see this recent issue: http://bugs.python.org/issue11284
In the reporter's case, it's much worse, because FreeBSD (at least the ver
Charles-Francois Natali added the comment:
Attached is a patch adding os.closefrom.
If closefrom(2) is available, it's used.
Otherwise, two options:
- if sysconf and _SC_OPEN_MAX are defined, we close each file descriptor up to
_SC_OPEN_MAX
- if not, we choose a default value (256), and
Changes by Charles-Francois Natali :
Removed file: http://bugs.python.org/file20979/py3k_closefrom.diff
___
Python tracker
<http://bugs.python.org/issue11284>
___
___
Changes by Charles-Francois Natali :
Added file: http://bugs.python.org/file20980/py3k_closefrom.diff
___
Python tracker
<http://bugs.python.org/issue11284>
___
___
Pytho
Charles-Francois Natali added the comment:
Attached is a new version falling back to /proc/self/fd when closefrom(2) is
not available (on Unix), working on Linux.
It's indeed much faster than the current approach.
Note that it's only used if _posixsubprocess is not available, becau
Changes by Charles-Francois Natali :
Removed file: http://bugs.python.org/file20980/py3k_closefrom.diff
___
Python tracker
<http://bugs.python.org/issue11284>
___
___
Charles-Francois Natali added the comment:
> Your posix_closefrom() implementation as written today is not safe to call
> between fork() and exec() due to the opendir/readdir implementation. It can
> and will hang processes at unexpected times.
Yeah, I remove the patch when I real
Changes by Charles-Francois Natali :
--
nosy: neologix
priority: normal
severity: normal
status: open
title: some posix module functions
___
Python tracker
<http://bugs.python.org/issue11
New submission from Charles-Francois Natali :
Some posix module functions unnecessarily release the GIL.
For example, posix_dup, posix_dup2 and posix_pipe all release the GIL, but
those are non-blocking syscalls (the don't imply any I/O, only modifying the
process file descriptors table).
Charles-Francois Natali added the comment:
I didn't even know that Windows had such calls.
But anyway, if we start releasing the GIL around each malloc call, then it's
going to get really complicated:
static PyObject *
posix_geteuid(PyObject *self, PyObject *noargs)
{
Charles-Francois Natali added the comment:
Well, those are contrived examples showing the effect of the convoy effect
induced by those unneeded GIL release/acquire: releasing and re-acquiring the
GIL comes with a cost (e.g. under Linux, futex are really fast in the
uncontended case since
Charles-Francois Natali added the comment:
> Do you want to propose a patch?
Sure, if removing those calls to Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS
seems reasonable (I might haved missed something obvious).
Just to be clear, I'm not at all criticizing the current GIL implem
Charles-Francois Natali added the comment:
2011/3/3 Antoine Pitrou :
>
> Antoine Pitrou added the comment:
>
>> Just to be clear, I'm not at all criticizing the current GIL
>> implementation, there's been a great work done on it.
>> I'm just sayi
Charles-Francois Natali added the comment:
Attached is a patch removing useless calls to
Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS for several posix
functions.
It's straigthforward, but since I only have Linux boxes, I couldn't
test it under Windows.
--
keywords: +patch
New submission from Charles-Francois Natali :
$ cat /tmp/test_mmap.py
import mmap
m = mmap.mmap(-1, 1024, prot=mmap.PROT_READ|mmap.PROT_EXEC)
m[0] = 0
$ ./python /tmp/test_mmap.py
Segmentation fault
When trying to perform a write, is_writable is called to check that we can
indeed write to
Charles-Francois Natali added the comment:
> Patch looks mostly good. Why do you use ~PROT_WRITE instead of
> PROT_READ|PROT_EXEC as in your example?
Because I'm not sure that PROT_EXEC is supported by all platforms. See
http://pubs.opengroup.org/onlinepubs/007908799/xsh/mmap.
Charles-Francois Natali added the comment:
It's probably a Windows limitation regarding the number of bytes that can be
written to stdout in one write.
As for the difference between python versions, what does
python -c "import sys; print(sys.getsizeof('a'))"
Charles-Francois Natali added the comment:
Do you have a coredump ?
It'd be curious to see this faulting address.
I didn't notice the first time, but in the OP case the address is definitely
wrong: 0xecc778b7 is above PAGE_OFFSET (0xc000 on x86), so unless he's
using a kern
Charles-Francois Natali added the comment:
> The code that is segfaulting is using pycrypto and sqlite3, so it may be that
> a bug in one of these is trampling on something. No idea how to investigate
> any further.
You could try valgrind:
$ valgrind --tool=memcheck -o /tmp/o
New submission from Charles-Francois Natali :
While tracing a program using multiprocessing queues, I noticed that there were
many calls to gettimeofday.
It turns out that acquire_timed, used by lock_PyThread_acquire_lock and
rlock_acquire, always call gettimeofday, even if no timeout argument
Charles-Francois Natali added the comment:
> Big dirs are really slow to read at once. If user wants to read items one by
> one like here
The problem is that readdir doesn't read a directory entry one at a time.
When you call readdir on an open DIR * for the first time, the lib
Charles-Francois Natali added the comment:
If test_wait3 and test_fork1 pass, then yes, it's probably an issue with AIX's
wait4.
See http://fixunix.com/aix/84872-sigchld-recursion.html:
"""
Replace the wait4() call with a waitpid() call...
like this:
for(n=0;wai
Charles-Francois Natali added the comment:
> wait4 without WNOHANG works fine. waitpid works fine even with WNOHANG.
> I don't know which workaround is the better.
As far as the test is concerned, it's of course better to use wait4
without WNOHANG in a test names test_wait4 (
Charles-Francois Natali added the comment:
The problem lies here:
/* Close pipe fds. Make sure we don't close the same fd more than */
/* once, or standard fds. */
if (p2cread > 2) {
POSIX_CALL(close(p2cread));
}
(c2pwrite > 2) {
POSIX_CALL(close(c2pwrite));
}
if (errwrite
Charles-Francois Natali added the comment:
Attached is a patch checking that no FD is closed more once when
closing pipe FDs, along with an update for test_subprocess.
--
keywords: +patch
Added file: http://bugs.python.org/file21053/subprocess_same_fd.diff
Charles-Francois Natali added the comment:
The check is done in py3k:
Traceback (most recent call last):
File "/home/cf/test_zip.py", line 7, in
print(z.read("secretfile.txt"))
File "/home/cf/py3k/Lib/zipfile.py", line 889, in read
with self.open
Charles-Francois Natali added the comment:
Could you try with the attached patch ?
The problem is that subprocess silently replaces bufsize=0, so child.stdout is
actually buffered, and when you read just one byte, everything that's available
for reading is read into the python's obj
Charles-Francois Natali added the comment:
Could you try with Python 3.2 ?
In 3.1, the only available pickle implementation was in pure python: with
cPickle (2.7) or _pickle (3.2), it should be much faster.
--
nosy: +neologix
___
Python tracker
Charles-Francois Natali added the comment:
What's the problem here ?
CTRL-Z causes the controlling terminal to send a SIGTSTP to the process, and
the default handler stops the process, pretty much like a SIGSTOP.
If you don't want that to happen:
import signal
signal.signal(sign
Charles-Francois Natali added the comment:
I'm still not sure I understand the problem.
- when you hit CTRL-Z, the process is put in background, since it receives a
SIGTSTP : normal
- when you put it in foreground with 'fg', it doesn't resume ? Did you try to
Charles-Francois Natali added the comment:
In that case, it's likely due to the way OS-X handles interrupted syscalls.
Under Linux, getchar and friends (actually read with default SA_RESTART) won't
return EINTR on (SIGSTOP|SIGTSTP)/SIGCONT.
Under OS-X, it seems that e.g. getchar (
Charles-Francois Natali added the comment:
my_fgets in Parser/myreadline.c is broken:
There's a comment saying that a fgets is retried on EINTR, but the code doesn't
retry. It used to in older cPython versions, but there was also a bug, so my
guess is that this bug has been here
Charles-Francois Natali added the comment:
If you're suggesting to set FDs CLOEXEC by default, I think it's neither
possible nor reasonable:
- you have to take into account not only files, but also pipes, sockets, etc
- there's no portable way to e.g. open a file and set it CLO
Charles-Francois Natali added the comment:
Ooops, it's of course not going to break code containing accept + fork or pipe
+ fork, you obviously also need an execve ;-)
But the point is that you can't change the semantics of FDs being inheritable
across an execve (think about inetd f
Changes by Charles-Francois Natali :
Removed file: http://bugs.python.org/file17081/base_http_server_fqdn_lag.diff
___
Python tracker
<http://bugs.python.org/issue6
Charles-Francois Natali added the comment:
> I wonder whether the Java people are simply unaware of the potential problem?
> Or perhaps they have checked the Linux and Solaris implementations of
> readdir()
> and confirmed that it is in fact safe on those platforms. Even if this i
Charles-Francois Natali added the comment:
> There is something interesting in this output: the test uses a subprocess and
> we only have the traceback of the parent. It may be nice to have the trace of
> the child process. It might be possible by sending a signal to the child
>
Charles-Francois Natali added the comment:
This test assumes that send will necessarily return if interrupted by a signal,
but the kernel can automatically restart the syscall when no data has been
committed (instead of returning -1 with errno set to EINTR).
And, AFAIK, that's exactly
Charles-Francois Natali added the comment:
_remaining_time doesn't check that endtime > current time and can return a
negative number, which would trigger an EINVAL when passed to select
(select_select doesn't seem to check for negative double).
Note that a check is perf
Charles-Francois Natali added the comment:
Is the SIGBUS generated on the first page access ?
How much memory does this buildbot have ?
--
___
Python tracker
<http://bugs.python.org/issue11
Charles-Francois Natali added the comment:
Does this only happen on Cygwin buildbots ?
If yes, then it might simply be an issue with Cygwin's fork implementation,
which is much slower than natively.
Right now, the test waits 0.5s before checking that the processes are started,
Charles-Francois Natali added the comment:
It seems to have fixed the failure, no ?
I don't know what's the policy regarding syscall parameters check, but
I think it'd be better to check that the timeout passed to select is
not negative, and raise an exception otherwise, inst
Charles-Francois Natali added the comment:
> You may also patch poll_poll().
>
Poll accepts negative timeout values, since it's the only way to
specify an infinite wait (contrarily to select which can be passed
NULL).
--
___
Python tra
Charles-Francois Natali added the comment:
> Check also this:
>
> http://bugs.python.org/issue11740
You should indicate it as duplicate.
--
___
Python tracker
<http://bugs.python.o
Charles-Francois Natali added the comment:
> Oh, I didn't know. In this case, is my commit 3664fc29e867 correct? I
> think that it is, because without the patch, subprocess may call poll()
> with a negative timeout, and so it is no more a timeout at all.
>
Yes, it looks cor
Charles-Francois Natali added the comment:
I think those lockups are due to a race in the Pool shutdown code.
In Lib/multiprocessing/pool.py:
def close(self):
debug('closing pool')
if self._state == RUN:
self._state = CLOSE
self._work
Charles-Francois Natali added the comment:
Attached is a patch fixing this race, and a similar one in Pool's terminate.
--
keywords: +patch
Added file: http://bugs.python.org/file21608/pool_shutdown_race.diff
___
Python tracker
Charles-Francois Natali added the comment:
One possible cause for those intermittent failures is the preemtion of a thread
while waiting on the condition:
def wait(self, timeout=None):
233 assert self._lock._semlock._is_mine(), \
234'must acquire() cond
Charles-Francois Natali added the comment:
Sorry, wrong copy-paste, the failing assertion will of course be this one:
773 self.assertReturnsIfImplemented(6, get_value, woken)
since woken.get_value() == 5
--
___
Python tracker
<h
Charles-Francois Natali added the comment:
This problem arises because the pool's close method is called before all the
tasks have completed. Putting a sleep(1) before pool.close() won't exhibit this
lockup.
The root cause is that close makes the workers handler thread exit:
Charles-Francois Natali added the comment:
It's documented in
http://docs.python.org/library/multiprocessing.html#multiprocessing-programming
:
"""
Joining processes that use queues
Bear in mind that a process that has put items in a queue will wait before
terminating un
Charles-Francois Natali added the comment:
It's probably a duplicate of http://bugs.python.org/issue8428
It would be nice if you could try to reproduce it with a py3k snapshot though,
just to be sure.
--
nosy: +neologix
___
Python tracker
Charles-Francois Natali added the comment:
I'm not sure whether POSIX warrants anything about this behavior, but nothing
prevents a process from running with a UID not listed in /etc/passwd (or NIS,
whatever). For example, sudo allows running a command with a UID not listed in
the pas
Charles-Francois Natali added the comment:
This is due to a bug in the TLS key management when mixed with fork.
Here's what happens:
When a thread is created, a tstate is allocated and stored in the thread's TLS:
thread_PyThread_start_new_thread -> t_bootstrap -> _Py
Changes by Charles-Francois Natali :
Added file: http://bugs.python.org/file21677/test_specific.c
___
Python tracker
<http://bugs.python.org/issue10517>
___
___
Python-bug
Changes by Charles-Francois Natali :
--
keywords: +patch
Added file: http://bugs.python.org/file21678/thread_invalid_key.diff
___
Python tracker
<http://bugs.python.org/issue10
Charles-Francois Natali added the comment:
Note: this seems to be fixed in RHEL6.
(Sorry for the noise).
--
___
Python tracker
<http://bugs.python.org/issue10
Charles-Francois Natali added the comment:
The "problem" is not with Python, but with your libc.
When a program - such as Python - returns memory, it uses the free(3) library
call.
But the libc is free to either return the memory immediately to the kernel
using the relevant sy
Changes by Charles-Francois Natali :
--
keywords: +patch
Added file: http://bugs.python.org/file21696/gc_trim.diff
___
Python tracker
<http://bugs.python.org/issue11
Charles-Francois Natali added the comment:
> kaifeng added the comment:
>
> I added 'malloc_trim' to the test code and rerun the test with Python 2.5 /
> 3.2 on CentOS 5.3. The problem still exists.
>
Well, malloc_trim can fail, but how did you "add" it ?
Charles-Francois Natali added the comment:
> IMO, it would be nice if I could ask my queue, "Just what is your capacity
(in bytes, not entries) anyways? I want to know how much I can put in here
without worrying about whether the remote side is dequeueing." I guess I'd
s
Charles-Francois Natali added the comment:
> BTW, after utilize lxml instead of ElementTree, such phenomenon of increasing
> memory usage disappeared.
If you looked at the link I posted, you'll see that lxml had some similar
issues and solved it by calling malloc_trim systemat
Charles-Francois Natali added the comment:
I know that POSIX makes no guarantee regarding durable writes, but IMHO that's
definitely wrong, in the sense that when one calls fsync, he expects the data
to be committed to disk and be durable.
Fixing this deficiency through Python's exp
Charles-Francois Natali added the comment:
> in particular: linux doesn't guarantee that data gets writting to the disk
> when you call fsync, only that the data gets pushed to the storage device.
Barriers are now enable by default in EXT4, and Theodore Tso has been
favourable
Charles-Francois Natali added the comment:
Is there anything I can do to help this move forward ?
--
___
Python tracker
<http://bugs.python.org/issue11
1 - 100 of 207 matches
Mail list logo
