Nir Aides added the comment:
For the record, turns out there was a bit of misunderstanding.
I used the term deprecate above to mean "warn users (through documentation)
that they should not use (a feature)" and not in its Python-dev sense of
"remove (a feature) after a period
Nir Aides added the comment:
Hi Gregory,
> Gregory P. Smith added the comment:
> No Python thread is ever fork safe because the Python interpreter itself can
> never be made fork safe.
> Nor should anyone try to make the interpreter itself safe. It is too complex
>
Nir Aides added the comment:
> (BTW: there are religions without "god", so whom shall e.g. i praise for the
> GIL?)
Guido? ;)
--
___
Python tracker
<http://bugs.p
Nir Aides added the comment:
> then multiprocessing is completely brain-damaged and has been
> implemented by a moron.
Please do not use this kind of language.
Being disrespectful to other people hurts the discussion.
--
___
Python tracker
Nir Aides added the comment:
Here is a morning reasoning exercise - please help find the flaws or refine it:
5) Sanitizing worker threads in the multiprocessing module
Sanitizing a worker thread in the context of this problem is to make sure it
can not create a state that may deadlock
Nir Aides added the comment:
Well, my brain did not deadlock, but after spinning on the problem for a while
longer, it now thinks Tomaž Šolc and Steffen are right.
We should try to fix the multiprocessing module so it does not deadlock
single-thread programs and deprecate fork in multi
Changes by Nir Aides :
--
title: malloc -> threading module can deadlock after fork
___
Python tracker
<http://bugs.python.org/issue874900>
___
___
Python-
Changes by Nir Aides :
--
title: threading module can deadlock after fork -> malloc
___
Python tracker
<http://bugs.python.org/issue874900>
___
___
Python-
Nir Aides added the comment:
> Would you like to work on a patch to add an atfork mechanism?
I will start with an attempt to generate a summary "report" on this rabbit hole
of a problem, the views and suggestions raised by people here and what we may
expect from atfork
Nir Aides added the comment:
> Sorry, I fail to see how the "import graph" is related to the correct
> lock acquisition order. Some locks are created dynamically, for
> example.
Import dependency is a reasonable heuristic to look into for inter-module
locking order
Nir Aides added the comment:
> - what would be the API of this atfork() mechanism (with an example of how it
> would be used in the library)?
The atfork API is defined in POSIX and Gregory P. Smith proposed a Python one
above that we can look into.
http://pubs.opengroup.org/onli
Nir Aides added the comment:
> I believe that the comp.programming.threads post from
> David Butenhof linked above explains why adding atfork()
> handlers isn't going to solve this.
In Python atfork() handlers will never run from signal handlers, and if I
understood corr
Nir Aides added the comment:
Well, I ping my view that we should:
1) Add general atfork() mechanism.
2) Dive into the std lib and add handlers one by one, that depending on case,
either do the lock/init thing or just init the state of the library to some
valid state in the child.
Once this
Changes by Nir Aides :
Removed file: http://bugs.python.org/file22087/cpython-bz2-streams.patch
___
Python tracker
<http://bugs.python.org/issue1625>
___
___
Python-bug
Nir Aides added the comment:
Right! I updated the patch and added a test for the aligned stream/buffer case.
--
Added file: http://bugs.python.org/file22114/cpython-bz2-streams.patch
___
Python tracker
<http://bugs.python.org/issue1
Nir Aides added the comment:
False alarm; go ahead with the review. I took a look too early in the morning
before caffeine kicked in.
Note Lib/test/test_bz2.py was directly upgraded from bz2ms.patch.
A note on bz2 behavior: A BZ2Decompressor object is only good for one stream;
after that
Nir Aides added the comment:
Wait, the tests seem wrong. I'll post an update later today.
--
___
Python tracker
<http://bugs.python.org/issue1625>
___
___
Nir Aides added the comment:
Hi, I attach a patch to Python 3.3 Lib/bz2.py with updated tests:
cpython-bz2-streams.patch
--
keywords: +needs review
stage: needs patch -> patch review
Added file: http://bugs.python.org/file22087/cpython-bz2-streams.pa
Changes by Nir Aides :
--
assignee: niemeyer -> nirai
___
Python tracker
<http://bugs.python.org/issue1625>
___
___
Python-bugs-list mailing list
Unsubscri
Nir Aides added the comment:
Steffen, can you explain in layman's terms?
On Sun, May 15, 2011 at 8:03 PM, Steffen Daode Nurpmeso
wrote:
>
> @ Charles-François Natali wrote (2011-05-15 01:14+0200):
>> So if we really wanted to be safe, the only solution would be to
>> f
Nir Aides added the comment:
Is it possible the following issue is related to this one?
http://bugs.python.org/issue10037 - "multiprocessing.pool processes started by
worker handler stops working"
--
___
Python tracker
<http://bu
Nir Aides added the comment:
I think that generally it is better to deadlock than corrupt data.
> 2) acquiring locks just before fork is probably one of the best way to
> deadlock (acquiring a lock we already hold, or acquiring a lock needed
> by another thread before it releases its
Nir Aides added the comment:
Hi,
There seem to be two alternatives for atfork handlers:
1) acquire locks during prepare phase and unlock them in parent and child after
fork.
2) reset library to some consistent state in child after fork.
http://pubs.opengroup.org/onlinepubs/009695399
Changes by Nir Aides :
--
nosy: +nirai
___
Python tracker
<http://bugs.python.org/issue6721>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.pyth
Changes by Nir Aides :
--
nosy: +nirai
___
Python tracker
<http://bugs.python.org/issue9971>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.pyth
Changes by Nir Aides :
--
nosy: +nirai
___
Python tracker
<http://bugs.python.org/issue11743>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.pyth
Changes by Nir Aides :
--
nosy: +nirai
___
Python tracker
<http://bugs.python.org/issue9504>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.pyth
Changes by Nir Aides :
--
nosy: +nirai
___
Python tracker
<http://bugs.python.org/issue1625>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.pyth
Changes by Nir Aides :
--
nosy: +nirai
___
Python tracker
<http://bugs.python.org/issue10037>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.pyth
Nir Aides added the comment:
Right, I missed the change from self.max_read_chunk to 1024 (read_size). Should
not peek() limit to self.max_read_chunk as read() does?
--
___
Python tracker
<http://bugs.python.org/issue9
Nir Aides added the comment:
Should be min(n, 1024) instead of max(...)
--
___
Python tracker
<http://bugs.python.org/issue9962>
___
___
Python-bugs-list mailin
Nir Aides added the comment:
Hi Antoine,
BufferedIOBase is not documented to have peek():
http://docs.python.org/dev/py3k/library/io.html
Small note about patch:
1) IOError string says "read() on write-only...", should be "peek() on
write-only..." ?
2) Should be min
Nir Aides added the comment:
But you answered my question with code :) self._file_size is now unused and may
be removed.
--
___
Python tracker
<http://bugs.python.org/issue7
Nir Aides added the comment:
I think patch may be simplified. Instead of keeping track of CRC offset, invoke
it directly on the 'data' variable being added to _readbuffer.
Also the call to _update_crc() before the return from read1() looks redundant.
Finally, is it possible to det
Nir Aides added the comment:
Updated bfs.patch with BSD license and copyright notice.
! Current version patches cleanly and builds with Python revision svn r81201.
Issue 7946 and proposed patches were put on hold indefinitely following this
python-dev discussion:
http://mail.python.org
Changes by Nir Aides :
Removed file: http://bugs.python.org/file17356/bfs.patch
___
Python tracker
<http://bugs.python.org/issue7946>
___
___
Python-bugs-list mailin
Nir Aides added the comment:
> Nir, would you be interested in looking at this?
yes, I'll take a look, but will take me a few days.
--
___
Python tracker
<http://bugs.python.or
Nir Aides added the comment:
A link to ccbench results comparing old GIL, old GIL with long check interval,
new GIL and BFS:
http://bugs.python.org/file17370/nir-ccbench-linux.log
Summary:
Results for ccbench latency and bandwidth test run on Ubuntu Karmic 64bit,
q9400 2.6GHz, all Python
Changes by Nir Aides :
Added file: http://bugs.python.org/file17370/nir-ccbench-linux.log
___
Python tracker
<http://bugs.python.org/issue7946>
___
___
Python-bugs-list m
Nir Aides added the comment:
Updated bfs.patch to patch cleanly updated py3k branch. Use:
$ patch -p1 < bfs.patch
--
Added file: http://bugs.python.org/file17356/bfs.patch
___
Python tracker
<http://bugs.python.org/iss
Changes by Nir Aides :
Removed file: http://bugs.python.org/file17330/bfs.patch
___
Python tracker
<http://bugs.python.org/issue7946>
___
___
Python-bugs-list mailin
Nir Aides added the comment:
Duck, here comes another update to bfs.patch.
This one with some cleanups which simplify the code and improve behavior (on
Windows XP), shutdown code, comments, and "experimental" use of TSC for
timestamps, which eliminates timestamp reading overhead.
Changes by Nir Aides :
Removed file: http://bugs.python.org/file17195/bfs.patch
___
Python tracker
<http://bugs.python.org/issue7946>
___
___
Python-bugs-list mailin
Nir Aides added the comment:
I updated bfs.patch with improvements on Windows XP.
The update disables priority boosts associated with the scheduler condition on
Windows for CPU bound threads.
Here is a link to ccbench results:
http://bugs.python.org/file17194/nir-ccbench-xp32.log
Summary
Changes by Nir Aides :
Removed file: http://bugs.python.org/file16967/bfs.patch
___
Python tracker
<http://bugs.python.org/issue7946>
___
___
Python-bugs-list mailin
Changes by Nir Aides :
Added file: http://bugs.python.org/file17194/nir-ccbench-xp32.log
___
Python tracker
<http://bugs.python.org/issue7946>
___
___
Python-bugs-list m
Nir Aides added the comment:
Dave,
The behavior of your patch on Windows XP/2003 (and earlier) might be related to
the way Windows boosts thread priority when it is signaled.
Try to increase priority of monitor thread and slice size. Another thing to
look at is how to prevent Python CPU
Nir Aides added the comment:
On Thu, Apr 29, 2010 at 2:03 AM, David Beazley wrote:
> Wow, that is a *really* intriguing performance result with radically
> different behavior than Unix. Do you have any ideas of what might be causing
> it?
Instrument the code and I'll se
Nir Aides added the comment:
Dave, there seems to be some problem with your patch on Windows:
F:\dev>z:\dabeaz-wcg\PCbuild\python.exe y:\ccbench.py -b
== CPython 3.2a0.0 (py3k) ==
== x86 Windows on 'x86 Family 6 Model 23 Stepping 10, GenuineIntel' ==
--- I/O bandwidth ---
Backgr
Nir Aides added the comment:
On Wed, Apr 28, 2010 at 12:41 AM, Larry Hastings wrote:
> The simple solution: give up QPC and use timeGetTime() with
> timeBeginPeriod(1), which is totally
> reliable but only has millisecond accuracy at best.
It is preferable to use a high precision
Nir Aides added the comment:
On Tue, Apr 27, 2010 at 12:23 PM, Charles-Francois Natali wrote:
> @nirai
> I have some more remarks on your patch:
> - /* Diff timestamp capping results to protect against clock differences
> * between cores. */
> _LOCAL(long double) _bfs_diff_ts(
Nir Aides added the comment:
Dave, there seems to be a bug in your patch on Windows XP. It crashes in
ccbench.py with the following output:
>python_d.exe y:\ccbench.py
== CPython 3.2a0.0 (py3k) ==
== x86 Windows on 'x86 Family 6 Model 23 Stepping 10, GenuineIntel' ==
--- Throu
Nir Aides added the comment:
Yet another update to bfs.patch.
I upload a variation on Florent's write test which prints progress of
background CPU bound threads as: thread-name timestamp progress
Here are some numbers from Windows XP 32bit with Intel q9400 (4 cores). Builds
produced wi
Changes by Nir Aides :
Added file: http://bugs.python.org/file16967/bfs.patch
___
Python tracker
<http://bugs.python.org/issue7946>
___
___
Python-bugs-list mailin
Changes by Nir Aides :
Removed file: http://bugs.python.org/file16947/bfs.patch
___
Python tracker
<http://bugs.python.org/issue7946>
___
___
Python-bugs-list mailin
Nir Aides added the comment:
> the scheduling function bfs_find_task returns the first task that
> has an expired deadline. since an expired deadline probably means
> that the scheduler hasn't run for a while, it might be worth it to
> look for the thread with the oldest d
Nir Aides added the comment:
I uploaded an update to bfs.patch which improves behavior in particular on
non-Linux multi-core (4+) machines.
Hi Charles-Francois, Thanks for taking the time to review this patch!
> - nothing guarantees that you'll get a msec resolution
Right, the cod
Changes by Nir Aides :
Removed file: http://bugs.python.org/file16830/bfs.patch
___
Python tracker
<http://bugs.python.org/issue7946>
___
___
Python-bugs-list mailin
Nir Aides added the comment:
Uploaded an update.
--
Added file: http://bugs.python.org/file16830/bfs.patch
___
Python tracker
<http://bugs.python.org/issue7
Changes by Nir Aides :
Removed file: http://bugs.python.org/file16710/bfs.patch
___
Python tracker
<http://bugs.python.org/issue7946>
___
___
Python-bugs-list mailin
Nir Aides added the comment:
I upload a new update to bfs.patch which improves scheduling and reduces
overhead.
--
Added file: http://bugs.python.org/file16710/bfs.patch
___
Python tracker
<http://bugs.python.org/issue7
Changes by Nir Aides :
Removed file: http://bugs.python.org/file16680/bfs.patch
___
Python tracker
<http://bugs.python.org/issue7946>
___
___
Python-bugs-list mailin
Changes by Nir Aides :
Added file: http://bugs.python.org/file16680/bfs.patch
___
Python tracker
<http://bugs.python.org/issue7946>
___
___
Python-bugs-list mailin
Changes by Nir Aides :
Removed file: http://bugs.python.org/file16679/bfs.patch
___
Python tracker
<http://bugs.python.org/issue7946>
___
___
Python-bugs-list mailin
Nir Aides added the comment:
I update bfs.patch. It now builds on Windows (and Posix).
--
Added file: http://bugs.python.org/file16679/bfs.patch
___
Python tracker
<http://bugs.python.org/issue7
Changes by Nir Aides :
Removed file: http://bugs.python.org/file16663/bfs.patch
___
Python tracker
<http://bugs.python.org/issue7946>
___
___
Python-bugs-list mailin
Nir Aides added the comment:
gilinter.patch has good IO latency in UDP test on my system when built with
--with-computed-gotos:
In [34]: %timeit -n3 client.work()
0.320 seconds (32782026.509 bytes/sec)
0.343 seconds (30561727.443 bytes/sec)
0.496 seconds (21154075.417 bytes/sec)
0.326 seconds
Nir Aides added the comment:
> But on a busy system, won't measuring wall clock time rather than CPU time
> give bogus results?
This was the motivation for using clock_gettime(). I tried the wall clock
version under load (including on single core system) and it seems to beha
Nir Aides added the comment:
Uploaded an updated bfs.patch
The latency problem was related to the --with-computed-gotos flag. I fixed it
and it seems to work fine now.
I also switched to gettimeofday() so it should work now on all Posix with high
resolution timer.
--
Added file
Changes by Nir Aides :
Removed file: http://bugs.python.org/file16644/bfs.patch
___
Python tracker
<http://bugs.python.org/issue7946>
___
___
Python-bugs-list mailin
Nir Aides added the comment:
Well, on initial check the scheduler seems to work well with regular
gettimeofday() wall clock instead of clock_gettime().
:)
/* Return thread running time in seconds (with nsec precision). */
static inline long double get_thread_timestamp(void) {
return
Nir Aides added the comment:
> It's a dual-core Linux x86-64 system. But, looking at the patch again, the
> reason is obvious:
>
> #define CHECK_SLICE_DEPLETION(tstate) (bfs_check_depleted || (tstate
> >tick_counter % 1000 == 0))
>
> `tstate->tick_counter % 10
Nir Aides added the comment:
> Ouch. CLOCK_THREAD_CPUTIME_ID is not a required part of the standard. Only
> CLOCK_REALTIME is guaranteed to exist.
Right, however the man page at kernel.org says the following on
CLOCK_THREAD_CPUTIME_ID:
"Sufficiently recent versions of glibc an
Changes by Nir Aides :
Removed file: http://bugs.python.org/file16567/linux-7946.patch
___
Python tracker
<http://bugs.python.org/issue7946>
___
___
Python-bugs-list m
Changes by Nir Aides :
Removed file: http://bugs.python.org/file16634/bfs.patch
___
Python tracker
<http://bugs.python.org/issue7946>
___
___
Python-bugs-list mailin
Nir Aides added the comment:
I upload an updated bfs.patch. Apply to updated py32 and ignore the error with:
$ patch -fp1 < bfs.patch
$ ./configure
> Please give understandable benchmark numbers, including an explicit
> comparison with baseline 3.2, and patched 3.2 (e.g. gilin
Nir Aides added the comment:
I upload bfs.patch
To apply the patch use the following commands on updated python 3.2:
$ patch -fp1 < bfs.patch
$ ./configure
The patch replaces the GIL with a scheduler. The scheduler is a simplified
implementation of the recent kernel Brain F**k Scheduler
Nir Aides added the comment:
I updated the patch with a small fix and increased the ticks
countdown-to-release considerably. This seems to help the OS classify CPU bound
threads as such and actually improves IO performance.
--
Added file: http://bugs.python.org/file16570/linux-7946
Nir Aides added the comment:
I tried Florent's modification to the write test and did not see the effect on
my machine with an updated revision of Python32.
I am running Ubuntu Karmic 64 bit.
7s - no background threads.
20s - one background thread.
According to the following document
Changes by Nir Aides :
--
nosy: +nirai
___
Python tracker
<http://bugs.python.org/issue7946>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.pyth
Nir Aides added the comment:
I actually meant how would you simulate zlib's absence on a system in which it
is present?
--
___
Python tracker
<http://bugs.python.org/i
Nir Aides added the comment:
Unconsumed data is compressed data. If the part which handles unconsumed data
does not work when zlib is available, then the existing tests would fail. In
any case the unconsumed buffer is an implementation detail of zipfile.
I see a point in adding a test to
Nir Aides added the comment:
The related scenario is a system without zlib. How do you suggest simulating
this in test?
--
___
Python tracker
<http://bugs.python.org/issue7
Nir Aides added the comment:
Right, removed MAX_N from read(); remains in read1().
If good, what versions of Python is this patch desired for?
--
Added file: http://bugs.python.org/file15949/zipfile_7610_py27_v6.diff
___
Python tracker
<h
Nir Aides added the comment:
Uploaded an updated patch with read() which calls underlying stream enough
times to satisfy required read size.
--
Added file: http://bugs.python.org/file15941/zipfile_7610_py27_v5.diff
___
Python tracker
<h
Nir Aides added the comment:
> I do not find the existing phrasing in the IO docs ambiguous, but since
> it is obviously possible to misinterpret it it would be good to clarify
> it. Can you suggest an alternate phrasing that would be clearer?
Replace 'may' with 'wil
Nir Aides added the comment:
May be a good idea to clear this up in the documentation.
http://en.wiktionary.org/wiki/may#Verb
"(modal auxiliary verb, defective) To have permission to. Used in granting
permission and in questions to make polite req
Nir Aides added the comment:
I uploaded an update for Python 2.7.
> * you should probably write `n = sys.maxsize` instead of `n = 1 << 31 - 1`
sys.maxsize is 64 bit number on my system but the maximum value accepted by
zlib's decompress() seems to be INT_MAX defined in pyport.
Nir Aides added the comment:
If the sequence of readaheads is ['a\r', '\nb\n'], the first use of the pattern
will consume 'a', then the peek(2) will trigger a read() and the next use of
the pattern will consume '\r\n'.
I updated the patch and enhanc
Nir Aides added the comment:
> Thank you. Are you sure the "Shortcut common case" in readline()
> is useful? BufferedIOBase.readline() in itself should be rather fast.
On my dataset the shortcut speeds up readline() 400% on top of the default C
implementation.
I can take a
Nir Aides added the comment:
Right, I was reading the 3.1 docs by mistake.
I updated the patch. This time universal newlines are supported.
On my dataset (75MB 650K lines log file) the readline() speedup is x40 for 'r'
mode and x8 for 'rU' mode, and you can get an extra
Nir Aides added the comment:
I uploaded a possible patch for Python 2.7.
The patch converts ZipExtFile into subclass of io.BufferedIOBase and drops most
of the original implementation.
However, the patch breaks current newline behavior of ZipExtFile. I figured
this was reasonable because
Changes by Nir Aides :
--
nosy: +nirai
___
Python tracker
<http://bugs.python.org/issue7610>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.pyth
Nir Aides added the comment:
Memory can be pre-allocated by thread_PyThread_start_new_thread() before
thread is spawned.
--
___
Python tracker
<http://bugs.python.org/issue7
Changes by Nir Aides :
--
nosy: +nirai
___
Python tracker
<http://bugs.python.org/issue7544>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.pyth
Changes by Nir Aides :
--
nosy: +nirai
___
Python tracker
<http://bugs.python.org/issue6452>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.pyth
Changes by Nir Aides :
--
nosy: +nirai
___
Python tracker
<http://bugs.python.org/issue7481>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.pyth
Changes by Nir Aides :
--
nosy: +nirai
___
Python tracker
<http://bugs.python.org/issue7503>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.pyth
Changes by Nir Aides :
--
nosy: +nirai
___
Python tracker
<http://bugs.python.org/issue7519>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.pyth
Changes by Nir Aides :
--
nosy: +nirai
___
Python tracker
<http://bugs.python.org/issue7564>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.pyth
1 - 100 of 108 matches
Mail list logo
