[ python-Bugs-1575945 ] from_param and _as_parameter_ truncating 64-bit value
Bugs item #1575945, was opened at 2006-10-12 16:25
Message generated for change (Comment added) made by theller
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1575945&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Albert Strasheim (albertstrasheim)
Assigned to: Thomas Heller (theller)
Summary: from_param and _as_parameter_ truncating 64-bit value
Initial Comment:
There seems to be something strange going on with
ctypes' _as_parameter_ on 64-bit machines. I haven't
been able to replicate this problem without NumPy, but
it looks like a ctypes issue since NumPy's
_as_parameter_ contains a valid value but the value
that arrives in the C function has had its upper 4
bytes zeroed.
SConstruct to build library:
env = Environment()
env.Replace(CCFLAGS=['-O0','-ggdb','-Wall','-ansi','-pedantic'])
env.SharedLibrary('spfuncs',['spfuncs.c'])
C code:
#include
void nnz(double *ary) {
printf("ary = %p\n", (void*)ary);
}
Python code:
import numpy as N
from ctypes import *
from numpy.ctypeslib import ndpointer
_libspfuncs = N.ctypeslib.load_library('libspfuncs',
__file__) _libspfuncs.nnz.restype = None A =
N.eye((128)) print 'data_as',
A.ctypes.data_as(c_void_p) print 'array interface',
hex(A.__array_interface__['data'][0])
_libspfuncs.nnz.argtypes = [POINTER(c_double)]
_libspfuncs.nnz(A.ctypes.data_as(POINTER(c_double)))
_libspfuncs.nnz.argtypes = [ndpointer(dtype = N.float64)]
_libspfuncs.nnz(A)
print '_as_parameter', hex(A.ctypes._as_parameter_)
Output on 32-bit system:
data_as c_void_p(-1212006392)
array interface -0x483dbff8
ary = 0xb7c24008
ary = 0xb7c24008
_as_parameter -0x483dbff8
Output on 64-bit system:
data_as c_void_p(46912559644688)
array interface 0x2e740010
ary = 0x2e740010
ary = 0xae740010
_as_parameter 0x2e740010
--
>Comment By: Thomas Heller (theller)
Date: 2006-10-16 17:05
Message:
Logged In: YES
user_id=11105
This is not a ctypes bug.
It seems that A.ctypes._as_parameter_ is a Python integer.
These are passed as 'C int' type to foreign function calls.
(The C int type typically has 32 bits on 64-bit platforms,
while a pointer has 64 bit.)
To pass a pointer, A.ctypes._as_parameter_ should be a
ctypes c_void_p instance, not a Python integer.
--
Comment By: Martin v. Löwis (loewis)
Date: 2006-10-14 22:07
Message:
Logged In: YES
user_id=21627
Thomas, can you take a look? If not, please unassign.
--
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1575945&group_id=5470
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Feature Requests-1578269 ] Add os.link() and os.symlink() support for Windows
Feature Requests item #1578269, was opened at 2006-10-16 17:21 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1578269&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Library Group: Python 2.6 Status: Open Resolution: None Priority: 5 Submitted By: M.-A. Lemburg (lemburg) Assigned to: Nobody/Anonymous (nobody) Summary: Add os.link() and os.symlink() support for Windows Initial Comment: NTFS version 5.0 and up has all the needed APIs for creating hard links and symlinks (junctions), so it should be possible to add support for both hard links and symlinks to the posixmodule. Here are a few references: http://schinagl.priv.at/nt/hardlinkshellext/hardlinkshellext.html (see list of links at the end of the page) http://www.codeproject.com/w2k/junctionpoints.asp (junction points only, but includes analysis and source code) Note: NTFS 5.0 is supported in Win 2k, XP, 2003 and later. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1578269&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1575945 ] from_param and _as_parameter_ truncating 64-bit value
Bugs item #1575945, was opened at 2006-10-12 16:25
Message generated for change (Comment added) made by albertstrasheim
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1575945&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Albert Strasheim (albertstrasheim)
Assigned to: Thomas Heller (theller)
Summary: from_param and _as_parameter_ truncating 64-bit value
Initial Comment:
There seems to be something strange going on with
ctypes' _as_parameter_ on 64-bit machines. I haven't
been able to replicate this problem without NumPy, but
it looks like a ctypes issue since NumPy's
_as_parameter_ contains a valid value but the value
that arrives in the C function has had its upper 4
bytes zeroed.
SConstruct to build library:
env = Environment()
env.Replace(CCFLAGS=['-O0','-ggdb','-Wall','-ansi','-pedantic'])
env.SharedLibrary('spfuncs',['spfuncs.c'])
C code:
#include
void nnz(double *ary) {
printf("ary = %p\n", (void*)ary);
}
Python code:
import numpy as N
from ctypes import *
from numpy.ctypeslib import ndpointer
_libspfuncs = N.ctypeslib.load_library('libspfuncs',
__file__) _libspfuncs.nnz.restype = None A =
N.eye((128)) print 'data_as',
A.ctypes.data_as(c_void_p) print 'array interface',
hex(A.__array_interface__['data'][0])
_libspfuncs.nnz.argtypes = [POINTER(c_double)]
_libspfuncs.nnz(A.ctypes.data_as(POINTER(c_double)))
_libspfuncs.nnz.argtypes = [ndpointer(dtype = N.float64)]
_libspfuncs.nnz(A)
print '_as_parameter', hex(A.ctypes._as_parameter_)
Output on 32-bit system:
data_as c_void_p(-1212006392)
array interface -0x483dbff8
ary = 0xb7c24008
ary = 0xb7c24008
_as_parameter -0x483dbff8
Output on 64-bit system:
data_as c_void_p(46912559644688)
array interface 0x2e740010
ary = 0x2e740010
ary = 0xae740010
_as_parameter 0x2e740010
--
>Comment By: Albert Strasheim (albertstrasheim)
Date: 2006-10-16 18:18
Message:
Logged In: YES
user_id=945096
Changing NumPy's _as_parameter_ to return the pointer as a
c_void_p causes ctypes to raise the following erorr:
ctypes.ArgumentError: argument 1: exceptions.TypeError:
Don't know how to convert parameter 1
--
Comment By: Thomas Heller (theller)
Date: 2006-10-16 17:05
Message:
Logged In: YES
user_id=11105
This is not a ctypes bug.
It seems that A.ctypes._as_parameter_ is a Python integer.
These are passed as 'C int' type to foreign function calls.
(The C int type typically has 32 bits on 64-bit platforms,
while a pointer has 64 bit.)
To pass a pointer, A.ctypes._as_parameter_ should be a
ctypes c_void_p instance, not a Python integer.
--
Comment By: Martin v. Löwis (loewis)
Date: 2006-10-14 22:07
Message:
Logged In: YES
user_id=21627
Thomas, can you take a look? If not, please unassign.
--
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1575945&group_id=5470
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Feature Requests-1509798 ] replace dist/src/Tools/scripts/which.py with tmick's which
Feature Requests item #1509798, was opened at 2006-06-21 08:47 Message generated for change (Comment added) made by tmick You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1509798&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: wrstl prmpft (wrstlprmpft) Assigned to: Nobody/Anonymous (nobody) Summary: replace dist/src/Tools/scripts/which.py with tmick's which Initial Comment: see http://starship.python.net/~tmick/#which The author agrees with me in his README. Even better: include which as a module in the standard library so it can be used programmatically and as:: python -m which ... or maybe:: python -m os.which ... cheers -- >Comment By: Trent Mick (tmick) Date: 2006-10-16 16:37 Message: Logged In: YES user_id=34892 I'm happy to contribute this to the Python core. There are some TODOs here that I think should be worked through: http://trentm.com/projects/which/TODO.txt I'll see if wrstlprmpft and I can get through those. Help and opinions from others very welcome. -- Comment By: Martin v. Löwis (loewis) Date: 2006-10-15 13:46 Message: Logged In: YES user_id=21627 Can you please work with Trent on contributing it? Determine whether it should be targeted for the standard library or for Tools/scripts; if for the standard library, somebody needs to provide documentation and test cases also (and there should be an API review by potential users). -- Comment By: wrstl prmpft (wrstlprmpft) Date: 2006-10-15 13:31 Message: Logged In: YES user_id=801589 Current location: http://trentm.com/projects/which/ About getting Trent to actively contribute, I'll cite the linked page: "I also would be happy to have this be a replacement for the which.py in the Python CVS tree at dist/src/Tools/ scripts/which.py which is Unix-specific and not usable as a module; and perhaps for inclusion in the stdlib." So I do not think this will be a problem. (No way of cc:ing someone for sf.net replies? I will try contacting him directly, cc:ing you.) cheers -- Comment By: Martin v. Löwis (loewis) Date: 2006-10-14 22:38 Message: Logged In: YES user_id=21627 The URL is now bad... Do you have a current location of the file? In any case, we cannot really replace it unless Trent actively contributes it. So you have to pursudate Trent to contribute it first. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1509798&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1578513 ] 2.4.4c1 will not build when cross compiling
Bugs item #1578513, was opened at 2006-10-16 17:27 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1578513&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Build Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: smithj (smithj_rpath) Assigned to: Nobody/Anonymous (nobody) Summary: 2.4.4c1 will not build when cross compiling Initial Comment: When trying to build 2.4.4c1 with cross-compiling, I get the following error. checking for /dev/ptmx... configure: error: cannot check for file existence when cross compiling ./config.log:configure:20566: checking for /dev/ptmx ./config.log:configure:20572: error: cannot check for file existence when cross compiling This does not occur with 2.4.3. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1578513&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Feature Requests-1572968 ] release GIL while doing I/O operations in the mmap module
Feature Requests item #1572968, was opened at 2006-10-07 17:26 Message generated for change (Comment added) made by josiahcarlson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1572968&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Lukas Lalinsky (luks) Assigned to: Nobody/Anonymous (nobody) Summary: release GIL while doing I/O operations in the mmap module Initial Comment: There is a few system I/O calls in the mmap module that can take some time. Would be be possible to put Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS around these to release the GIL and allow other Python threads to do their work? -- Comment By: Josiah Carlson (josiahcarlson) Date: 2006-10-16 15:12 Message: Logged In: YES user_id=341410 Can you come up with the listing of code blocks where it would make sense to allow threads, or even provide a patch? -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1572968&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1560179 ] Better/faster implementation of os.path.basename/dirname
Bugs item #1560179, was opened at 2006-09-17 07:55
Message generated for change (Comment added) made by josiahcarlson
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1560179&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: None
Status: Closed
Resolution: Accepted
Priority: 5
Submitted By: Michael Gebetsroither (einsteinmg)
Assigned to: Nobody/Anonymous (nobody)
Summary: Better/faster implementation of os.path.basename/dirname
Initial Comment:
hi,
basename/dirname could do better (especially on long
pathnames)
def basename(p):
return split(p)[1]
def dirname(p):
return split(p)[0]
both construct base and dirname and discard the unused
one.
what about that?
def basename(p):
i = p.rfind('/') + 1
return p[i:]
def dirname(p):
i = p.rfind('/') + 1
return p[:i]
greets,
michael
--
Comment By: Josiah Carlson (josiahcarlson)
Date: 2006-10-16 15:23
Message:
Logged In: YES
user_id=341410
I note that in the current SVN, dirname uses a test of "if
head and head != '/'*len(head):" to check for the path being
all /, could be replaced by "if head and head.count('/') !=
len(head):", but it probably isn't terribly important.
--
Comment By: Georg Brandl (gbrandl)
Date: 2006-10-12 06:08
Message:
Logged In: YES
user_id=849994
Committed in rev. 52316.
--
Comment By: Michael Gebetsroither (einsteinmg)
Date: 2006-09-18 05:42
Message:
Logged In: YES
user_id=1600082
posixpath with this patch passes all test from
test_posixpath cleanly.
benchmark:
basename( 310 ) means basename called with a 310 chars long
path
sum = 0.0435626506805 min = 4.19616699219e-05
posixpath.basename( 310 )
sum = 0.152147769928 min = 0.00014591217041
posixpath_orig.basename( 310 )
sum = 0.0436658859253 min = 4.07695770264e-05
posixpath.basename( 106 )
sum = 0.117312431335 min = 0.000112771987915
posixpath_orig.basename( 106 )
sum = 0.0426909923553 min = 4.07695770264e-05
posixpath.basename( 21 )
sum = 0.113305330276 min = 0.000110864639282
posixpath_orig.basename( 21 )
sum = 0.12392115593 min = 0.000121831893921
posixpath.dirname( 310 )
sum = 0.152860403061 min = 0.00014591217041
posixpath_orig.dirname( 310 )
sum = 0.0942873954773 min = 9.08374786377e-05
posixpath.dirname( 106 )
sum = 0.114937067032 min = 0.000111818313599
posixpath_orig.dirname( 106 )
sum = 0.0918889045715 min = 8.79764556885e-05
posixpath.dirname( 21 )
sum = 0.114675760269 min = 0.000109910964966
posixpath_orig.dirname( 21 )
greets
--
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1560179&group_id=5470
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1578513 ] 2.4.4c1 will not build when cross compiling
Bugs item #1578513, was opened at 2006-10-16 16:27 Message generated for change (Comment added) made by montanaro You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1578513&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Build Group: Python 2.4 Status: Open Resolution: None >Priority: 7 Submitted By: smithj (smithj_rpath) >Assigned to: Anthony Baxter (anthonybaxter) Summary: 2.4.4c1 will not build when cross compiling Initial Comment: When trying to build 2.4.4c1 with cross-compiling, I get the following error. checking for /dev/ptmx... configure: error: cannot check for file existence when cross compiling ./config.log:configure:20566: checking for /dev/ptmx ./config.log:configure:20572: error: cannot check for file existence when cross compiling This does not occur with 2.4.3. -- >Comment By: Skip Montanaro (montanaro) Date: 2006-10-16 17:27 Message: Logged In: YES user_id=44345 Boosting this and at least provisionally assigning to Anthony since it's related to a change between 2.4.3 and 2.4.4c1. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1578513&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1573394 ] struct module doesn't use weakref for cache
Bugs item #1573394, was opened at 2006-10-08 16:37 Message generated for change (Comment added) made by josiahcarlson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1573394&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Library Group: Python 2.5 Status: Closed Resolution: Wont Fix Priority: 5 Submitted By: Mark Flacy (markaflacy) Assigned to: Bob Ippolito (etrepum) Summary: struct module doesn't use weakref for cache Initial Comment: At the moment, when you attempt to add your 101st different Struct object to the cache, all the other 100 entries are tossed into the garbage. That seems a trifle odd. The Struct cache would appear to be a perfect use for a weakref dictionary. -- Comment By: Josiah Carlson (josiahcarlson) Date: 2006-10-16 15:41 Message: Logged In: YES user_id=341410 Add a deque and a bit of logic, and one would get a fifo implementation of the cache. from collections import deque _toss = deque() def _compile(fmt): # Internal: compile struct pattern if len(_cache) >= _MAXCACHE: del _cache[_toss.popleft()] s = Struct(fmt) _cache[fmt] = s _toss.append(fmt) return s Race conditions from multiple threads could result in #threads-1 more entries in the cache than _MAXCACHE, but this could be trimmed in later calls by replacing 'if' with 'while'. -- Comment By: Bob Ippolito (etrepum) Date: 2006-10-12 10:54 Message: Logged In: YES user_id=139309 Yes, that code is different. You haven't shown that it's better though. Using popitem probably doesn't have very good cache effects. -- Comment By: �iga Seilnacht (zseil) Date: 2006-10-12 10:36 Message: Logged In: YES user_id=1326842 I was thinking abot something like this: Index: Lib/struct.py === --- Lib/struct.py (revision 52316) +++ Lib/struct.py (working copy) @@ -35,7 +35,7 @@ def _compile(fmt): # Internal: compile struct pattern if len(_cache) >= _MAXCACHE: -_cache.clear() +_cache.popitem() s = Struct(fmt) _cache[fmt] = s return s (sorry, I don't have the rights to attach a file) -- Comment By: Bob Ippolito (etrepum) Date: 2006-10-12 09:56 Message: Logged In: YES user_id=139309 Weakrefs would slow it down.. probably to the point where the cache wouldn't be an improvement at all. The re module does the same thing with the regular expression cache. This isn't a bug until someone presents a patch that proves another strategy is better for performance. -- Comment By: �iga Seilnacht (zseil) Date: 2006-10-12 08:52 Message: Logged In: YES user_id=1326842 WeakValueDictionary would be useless here; the cache is the only thing that holds references to Struct objects. Replacing the _cache.clear() call with _cache.popitem() seems like a better solution. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1573394&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1573394 ] struct module doesn't use weakref for cache
Bugs item #1573394, was opened at 2006-10-08 19:37 Message generated for change (Comment added) made by etrepum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1573394&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Library Group: Python 2.5 Status: Closed Resolution: Wont Fix Priority: 5 Submitted By: Mark Flacy (markaflacy) Assigned to: Bob Ippolito (etrepum) Summary: struct module doesn't use weakref for cache Initial Comment: At the moment, when you attempt to add your 101st different Struct object to the cache, all the other 100 entries are tossed into the garbage. That seems a trifle odd. The Struct cache would appear to be a perfect use for a weakref dictionary. -- >Comment By: Bob Ippolito (etrepum) Date: 2006-10-16 19:57 Message: Logged In: YES user_id=139309 The flush strategy is a silly thing to rattle on about. The idea is just to pick a cache size that's big enough that any sane application will never have to flush it. The current cache size is plenty big enough for anything I threw at it. If there's an application out there that needs a bigger cache I'd like to see it, and then perhaps we could adjust the size in some minor release to accommodate it. A FIFO is probably a bad strategy because there are a lot of common structs that are used a lot (the ones with just a few characters). If there was an obviously better solution here, surely it would have already been proposed or implemented for the re module. -- Comment By: Josiah Carlson (josiahcarlson) Date: 2006-10-16 18:41 Message: Logged In: YES user_id=341410 Add a deque and a bit of logic, and one would get a fifo implementation of the cache. from collections import deque _toss = deque() def _compile(fmt): # Internal: compile struct pattern if len(_cache) >= _MAXCACHE: del _cache[_toss.popleft()] s = Struct(fmt) _cache[fmt] = s _toss.append(fmt) return s Race conditions from multiple threads could result in #threads-1 more entries in the cache than _MAXCACHE, but this could be trimmed in later calls by replacing 'if' with 'while'. -- Comment By: Bob Ippolito (etrepum) Date: 2006-10-12 13:54 Message: Logged In: YES user_id=139309 Yes, that code is different. You haven't shown that it's better though. Using popitem probably doesn't have very good cache effects. -- Comment By: �iga Seilnacht (zseil) Date: 2006-10-12 13:36 Message: Logged In: YES user_id=1326842 I was thinking abot something like this: Index: Lib/struct.py === --- Lib/struct.py (revision 52316) +++ Lib/struct.py (working copy) @@ -35,7 +35,7 @@ def _compile(fmt): # Internal: compile struct pattern if len(_cache) >= _MAXCACHE: -_cache.clear() +_cache.popitem() s = Struct(fmt) _cache[fmt] = s return s (sorry, I don't have the rights to attach a file) -- Comment By: Bob Ippolito (etrepum) Date: 2006-10-12 12:56 Message: Logged In: YES user_id=139309 Weakrefs would slow it down.. probably to the point where the cache wouldn't be an improvement at all. The re module does the same thing with the regular expression cache. This isn't a bug until someone presents a patch that proves another strategy is better for performance. -- Comment By: �iga Seilnacht (zseil) Date: 2006-10-12 11:52 Message: Logged In: YES user_id=1326842 WeakValueDictionary would be useless here; the cache is the only thing that holds references to Struct objects. Replacing the _cache.clear() call with _cache.popitem() seems like a better solution. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1573394&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[ python-Bugs-1573394 ] struct module doesn't use weakref for cache
Bugs item #1573394, was opened at 2006-10-08 16:37 Message generated for change (Comment added) made by josiahcarlson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1573394&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Python Library Group: Python 2.5 Status: Closed Resolution: Wont Fix Priority: 5 Submitted By: Mark Flacy (markaflacy) Assigned to: Bob Ippolito (etrepum) Summary: struct module doesn't use weakref for cache Initial Comment: At the moment, when you attempt to add your 101st different Struct object to the cache, all the other 100 entries are tossed into the garbage. That seems a trifle odd. The Struct cache would appear to be a perfect use for a weakref dictionary. -- Comment By: Josiah Carlson (josiahcarlson) Date: 2006-10-16 17:08 Message: Logged In: YES user_id=341410 I agree, just pointing out that there exists a cache replacement strategy. Also, there is an LRU implementation in the Python cookbook, but unless it is rewritten in C (a serious overkill for the 2? uses in the stdlib), it would necessitate a lock, which would be a waste. But we've talked enough, and I probably shouldn't have bothered posting *this* message. I'll let the thread die this time, really! -- Comment By: Bob Ippolito (etrepum) Date: 2006-10-16 16:57 Message: Logged In: YES user_id=139309 The flush strategy is a silly thing to rattle on about. The idea is just to pick a cache size that's big enough that any sane application will never have to flush it. The current cache size is plenty big enough for anything I threw at it. If there's an application out there that needs a bigger cache I'd like to see it, and then perhaps we could adjust the size in some minor release to accommodate it. A FIFO is probably a bad strategy because there are a lot of common structs that are used a lot (the ones with just a few characters). If there was an obviously better solution here, surely it would have already been proposed or implemented for the re module. -- Comment By: Josiah Carlson (josiahcarlson) Date: 2006-10-16 15:41 Message: Logged In: YES user_id=341410 Add a deque and a bit of logic, and one would get a fifo implementation of the cache. from collections import deque _toss = deque() def _compile(fmt): # Internal: compile struct pattern if len(_cache) >= _MAXCACHE: del _cache[_toss.popleft()] s = Struct(fmt) _cache[fmt] = s _toss.append(fmt) return s Race conditions from multiple threads could result in #threads-1 more entries in the cache than _MAXCACHE, but this could be trimmed in later calls by replacing 'if' with 'while'. -- Comment By: Bob Ippolito (etrepum) Date: 2006-10-12 10:54 Message: Logged In: YES user_id=139309 Yes, that code is different. You haven't shown that it's better though. Using popitem probably doesn't have very good cache effects. -- Comment By: �iga Seilnacht (zseil) Date: 2006-10-12 10:36 Message: Logged In: YES user_id=1326842 I was thinking abot something like this: Index: Lib/struct.py === --- Lib/struct.py (revision 52316) +++ Lib/struct.py (working copy) @@ -35,7 +35,7 @@ def _compile(fmt): # Internal: compile struct pattern if len(_cache) >= _MAXCACHE: -_cache.clear() +_cache.popitem() s = Struct(fmt) _cache[fmt] = s return s (sorry, I don't have the rights to attach a file) -- Comment By: Bob Ippolito (etrepum) Date: 2006-10-12 09:56 Message: Logged In: YES user_id=139309 Weakrefs would slow it down.. probably to the point where the cache wouldn't be an improvement at all. The re module does the same thing with the regular expression cache. This isn't a bug until someone presents a patch that proves another strategy is better for performance. -- Comment By: �iga Seilnacht (zseil) Date: 2006-10-12 08:52 Message: Logged In: YES user_id=1326842 WeakValueDictionary would be useless here; the cache is the only thing that holds references to Struct objects. Replacing the _cache.clear() call with _cache.popitem() seems like a better solution. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1573394&group_id=5470 __
[ python-Bugs-1558223 ] apache2 - mod_python - python2.4 core dump
Bugs item #1558223, was opened at 2006-09-14 00:05 Message generated for change (Comment added) made by thurnerrupert You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1558223&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Parser/Compiler Group: 3rd Party Status: Closed >Resolution: Duplicate Priority: 5 Submitted By: ThurnerRupert (thurnerrupert) Assigned to: Nobody/Anonymous (nobody) Summary: apache2 - mod_python - python2.4 core dump Initial Comment: $ gdb bin/httpd GNU gdb 6.0 Copyright 2003 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "sparc-sun-solaris2.8"... (gdb) core core Core was generated by `/usr/local/bin/httpd -k restart'. Program terminated with signal 11, Segmentation fault. Reading symbols from /usr/local/lib/libssl.so.0.9.8...done. Loaded symbols for /usr/local/lib/libssl.so.0.9.8 <... deleted the whole loaded libraries> #0 0xfebb3218 in strlen () from /usr/lib/libc.so.1 (gdb) bt #0 0xfebb3218 in strlen () from /usr/lib/libc.so.1 #1 0xfda6ac4c in PyString_FromString ( str=0xfef65ec0 "unexpected parser state - please send a bug report") at Objects/stringobject.c:106 #2 0xfdac9b50 in PyModule_AddStringConstant (m=0x594cd0, name=0xfe5a5478 "XML_ERROR_ENTITY_DECLARED_IN_PE", value=0x0) at Python/modsupport.c:589 #3 0xfe57cec4 in initpyexpat () at /usr/local/Python-2.4.3/Modules/pyexpat.c:1973 this happens when running moinmoin 1.5.4, and doing a gui-edit. mod_python-3.2.10, httpd-2.2.3, solaris 8, compile with gcc-3.4.6. -- >Comment By: ThurnerRupert (thurnerrupert) Date: 2006-10-17 07:32 Message: Logged In: YES user_id=1597584 thanks a lot, zseil! with python-2.5 it works, so marking as duplicate of http://www.python.org/sf/1295808. -- Comment By: �iga Seilnacht (zseil) Date: 2006-10-04 00:47 Message: Logged In: YES user_id=1326842 This looks like another problem with pyexpat getting the export symbols from an older expat version. See: http://www.python.org/sf/1295808 and http://www.python.org/sf/1075984 for details. -- Comment By: ThurnerRupert (thurnerrupert) Date: 2006-10-03 23:54 Message: Logged In: YES user_id=1597584 ok ... i will. we noticed similar errors when running: * edgewall trac 0.11-dev (which uses edgewall ghenshi) * xml-rpc plugin for edgewall trac also, i filed the bug report after changing to newest mod_python. the old mod_python with apache 2.0.54 did not give a core, just segfaulted. -- Comment By: Neal Norwitz (nnorwitz) Date: 2006-09-14 05:34 Message: Logged In: YES user_id=33168 Please file this bug report with mod_python. That's typically the cause and it will likely be very hard for any Python developer to create this setup and much less try to reproduce the error. If you can provoke the same error without mod_python or other third party C extensions, please file a bug report with the minimal test case to reproduce. If you need a work-around, I would suggest changing to a different version of mod_python. -- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1558223&group_id=5470 ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
