Re: [Python-Dev] [RELEASE] Python 2.7.4 release candidate 1
Any plans backport decimal C implementation from 3.3? Thanks. Andriy Kornatskyy > Date: Tue, 26 Mar 2013 16:18:34 -0700 > From: [email protected] > To: [email protected] > CC: [email protected] > Subject: Re: [Python-Dev] [RELEASE] Python 2.7.4 release candidate 1 > > On Tue, Mar 26, 2013 at 3:26 PM, Hynek Schlawack wrote: > > Speakerdeck is slides only. The video is here: > > http://pyvideo.org/video/1730/python-33-trust-me-its-better-than-27 > > Sweet thanks! > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/andriy.kornatskyy%40live.com > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASE] Python 2.7.4 release candidate 1
No. _decimal is new functionality that will never be backported. On Wed, Mar 27, 2013 at 9:40 AM, Andriy Kornatskyy wrote: > Any plans backport decimal C implementation from 3.3? > > Thanks. > > Andriy Kornatskyy > > > >> Date: Tue, 26 Mar 2013 16:18:34 -0700 >> From: [email protected] >> To: [email protected] >> CC: [email protected] >> Subject: Re: [Python-Dev] [RELEASE] Python 2.7.4 release candidate 1 >> >> On Tue, Mar 26, 2013 at 3:26 PM, Hynek Schlawack wrote: >> > Speakerdeck is slides only. The video is here: >> > http://pyvideo.org/video/1730/python-33-trust-me-its-better-than-27 >> >> Sweet thanks! >> ___ >> Python-Dev mailing list >> [email protected] >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/andriy.kornatskyy%40live.com > ___ > Python-Dev mailing list > [email protected] > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com -- Thanks, Andrew Svetlov ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASE] Python 2.7.4 release candidate 1
Andrew, Thank you for the prompt response back. > will never be backported. Who knows? Is python 3.3 _decimal interface 100% compatible with one pure python implementation in 2.7? I suppose code written using decimals in python 2.7 should work in python 3 with no changes, or there should be taken caution for something? According to presentation slides, 30x performance boost is something definitely valuable for 2.7 crowd. Thanks. Andriy > Date: Wed, 27 Mar 2013 10:24:47 +0200 > Subject: Re: [Python-Dev] [RELEASE] Python 2.7.4 release candidate 1 > From: [email protected] > To: [email protected] > CC: [email protected] > > No. _decimal is new functionality that will never be backported. > > On Wed, Mar 27, 2013 at 9:40 AM, Andriy Kornatskyy > wrote: > > Any plans backport decimal C implementation from 3.3? > > > > Thanks. > > > > Andriy Kornatskyy > > > > > > > >> Date: Tue, 26 Mar 2013 16:18:34 -0700 > >> From: [email protected] > >> To: [email protected] > >> CC: [email protected] > >> Subject: Re: [Python-Dev] [RELEASE] Python 2.7.4 release candidate 1 > >> > >> On Tue, Mar 26, 2013 at 3:26 PM, Hynek Schlawack wrote: > >> > Speakerdeck is slides only. The video is here: > >> > http://pyvideo.org/video/1730/python-33-trust-me-its-better-than-27 > >> > >> Sweet thanks! > >> ___ > >> Python-Dev mailing list > >> [email protected] > >> http://mail.python.org/mailman/listinfo/python-dev > >> Unsubscribe: > >> http://mail.python.org/mailman/options/python-dev/andriy.kornatskyy%40live.com > > ___ > > Python-Dev mailing list > > [email protected] > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com > > > > -- > Thanks, > Andrew Svetlov ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASE] Python 2.7.4 release candidate 1
2013/3/27 Andrew Svetlov > No. _decimal is new functionality that will never be backported. > > On Wed, Mar 27, 2013 at 9:40 AM, Andriy Kornatskyy > wrote: > > Any plans backport decimal C implementation from 3.3? > No. 2.7 does not accept new features anymore, but you can install this backport from PyPI: https://pypi.python.org/pypi/cdecimal/2.3 -- Amaury Forgeot d'Arc ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASE] Python 2.7.4 release candidate 1
> No. 2.7 does not accept new features anymore, It is clear now. > but you can install this backport from PyPI: > https://pypi.python.org/pypi/cdecimal/2.3 This is what I was looking for. Thanks. Andriy > Date: Wed, 27 Mar 2013 10:54:37 +0100 > Subject: Re: [Python-Dev] [RELEASE] Python 2.7.4 release candidate 1 > From: [email protected] > To: [email protected] > CC: [email protected]; [email protected] > > 2013/3/27 Andrew Svetlov > mailto:[email protected]>> > No. _decimal is new functionality that will never be backported. > > On Wed, Mar 27, 2013 at 9:40 AM, Andriy Kornatskyy > mailto:[email protected]>> wrote: > > Any plans backport decimal C implementation from 3.3? > > No. 2.7 does not accept new features anymore, but you can install > this backport from PyPI: https://pypi.python.org/pypi/cdecimal/2.3 > > -- > Amaury Forgeot d'Arc ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [RELEASE] Python 2.7.4 release candidate 1
Andriy Kornatskyy wrote: > > but you can install this backport from PyPI: > > https://pypi.python.org/pypi/cdecimal/2.3 > > This is what I was looking for. Note that for numerical work _decimal from Python3.3 is vastly faster than cdecimal. I've added two major speedups (credit for one of them goes to Antoine), so these are the numbers for the pi benchmark: Python3.3 (_decimal):time: 0.15s Python3.3 (decimal.py): time: 17.22s = Python2.7 (cdecimal):time: 0.29s Python2.7 (decimal.py): time: 17.74s = For database work and such the numbers should be about the same. Stefan Krah ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Safely importing zip files with C extensions
> This quote is here to stop GMane complaining that I'm top-posting. Ignore.
I've already posted this to distutils-sig, but thought that it might be of
interest to readers here as it relates to importing C extensions ...
zipimport is great, but there can be issues importing software that contains
C extensions. But the new wheel format (PEP 427) may give us a better way of
importing zip files containing C extensions. Since wheels are .zip files, they
can sometimes be used to provide functionality without needing to be installed.
But whereas .zip files contain no convention for indicating compatibility with
a particular Python, wheels do contain this compatibility information. Thus, it
is possible to check if a wheel can be directly imported from, and the wheel
support in distlib allows you to take advantage of this using the mount() and
unmount() methods. When you mount a wheel, its absolute path name is added to
sys.path, allowing the Python code in it to be imported. (A DistlibException is
raised if the wheel isn't compatible with the Python which calls the mount()
method.)
You don't need mount() just to add the wheel's name to sys.path, or to import
pure-Python wheels, of course. But the mount() method goes further than just
enabling Python imports - any C extensions in the wheel are also made available
for import. For this to be possible, the wheel has to be built with additional
metadata about extensions - a JSON file called EXTENSIONS which serialises an
extension mapping dictionary. This maps extension module names to the names in
the wheel of the shared libraries which implement those modules.
Running unmount() on the wheel removes its absolute pathname from sys.path and
makes its C extensions, if any, also unavailable for import.
Wheels built with the new "distil" tool contain the EXTENSIONS metadata, so can
be mounted complete with C extensions:
$ distil download -d /tmp simplejson
Downloading simplejson-3.1.2.tar.gz to /tmp/simplejson-3.1.2
63KB @ 73 KB/s 100 % Done: 00:00:00
Unpacking ... done.
$ distil package --fo=wh -d /tmp /tmp/simplejson-3.1.2/
The following packages were built:
/tmp/simplejson-3.1.2-cp27-none-linux_x86_64.whl
$ python
Python 2.7.2+ (default, Jul 20 2012, 22:15:08)
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from distlib.wheel import Wheel
>>> w = Wheel('/tmp/simplejson-3.1.2-cp27-none-linux_x86_64.whl')
>>> w.mount()
>>> import simplejson._speedups
>>> dir(simplejson._speedups)
['__doc__', '__file__', '__loader__', '__name__', '__package__',
'encode_basestring_ascii', 'make_encoder', 'make_scanner', 'scanstring']
>>> simplejson._speedups.__file__
'/home/vinay/.distlib/dylib-cache/simplejson/_speedups.so'
>>>
Does anyone see any problems with this approach to importing C extensions from
zip files?
Regards,
Vinay Sajip
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Safely importing zip files with C extensions
2013/3/27 Vinay Sajip
> When you mount a wheel, its absolute path name is added to
> sys.path, allowing the Python code in it to be imported.
>
Better: just put the wheel path to sys.path
sys.path.append('/tmp/simplejson-3.1.2-cp27-none-linux_x86_64.whl')
and let a sys.path_hook entry do the job.
Such a WheelImporter could even inherit from zipimporter, plus the magic
required for C extensions.
It avoids the mount/nomount methods, only the usual sys.path operations are
necessary from the user.
--
Amaury Forgeot d'Arc
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] astoptimizer: static optimizer working on the AST
Victor Stinner, 26.03.2013 22:56: > * what should be the name of "pyc" files? > * how to handle different configuration of astoptimizer: generate > different "pyc" files? You could use (a part of) a crypto hash of the serialised options as part of the filename. One drawback of a .pyc duplication is that when you ship code to other environments that run with different optimiser flags, those would not pick up the same .pyc files. Although, would it really be a problem if different modules that were compiled with different optimisation settings get mixed on the same machine? Are there any non semantics preserving AST modifications for a module that have an impact on other modules? I.e., is this just a development time concern when trying out different optimisations, or is this a problem for deployed code as well? Stefan ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Safely importing zip files with C extensions
Vinay Sajip, 27.03.2013 20:38:
> >>> w = Wheel('/tmp/simplejson-3.1.2-cp27-none-linux_x86_64.whl')
> >>> w.mount()
> >>> import simplejson._speedups
> >>> dir(simplejson._speedups)
> ['__doc__', '__file__', '__loader__', '__name__', '__package__',
> 'encode_basestring_ascii', 'make_encoder', 'make_scanner', 'scanstring']
> >>> simplejson._speedups.__file__
> '/home/vinay/.distlib/dylib-cache/simplejson/_speedups.so'
I've always hated this setuptools misfeature of copying C extensions from
an installed archive into a user directory, one for each user. At least
during normal installation, they should be properly unpacked into normal
shared library files in the file system.
Whether it then makes sense to special case one-shot trial imports like the
above without installation is a bit of a different question, but I don't
see a compelling reason for adding complexity here. It's not really an
important use case.
Stefan
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Safely importing zip files with C extensions
Amaury Forgeot d'Arc gmail.com> writes:
> Better: just put the wheel path to sys.path
sys.path.append('/tmp/simplejson-3.1.2-cp27-none-linux_x86_64.whl')
> and let a sys.path_hook entry do the job.
That's what the mount() actually does - adds the wheel to a registry that an
import hook uses. You also need a place to check that the wheel being mounted
is compatible with the Python doing the mounting - I'm not sure whether what
the import hook should do if e.g. there is a compatibility problem with the
wheel (e.g. is it clear that it should always raise an ImportError? Or ignore
the wheel - seems wrong? Or do something else?)
Regards,
Vinay Sajip
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Safely importing zip files with C extensions
Jim Fulton is right that weird failures are a characteristic of zipped
eggs, so one of the #1 requests for setuptools is how to prohibit
zipping from ever happening. This is an important reason why wheel is
billed as an installation format -- fewer users with pitchforks. It's
very cool that it works though. Debugging is slightly easier than it
was in the old days because pdb can now read the source code from the
zip.
An unzipped wheel as a directory with the same name as the wheel would
be a more reliable solution that might be interesting to work with. It
would work the same as egg unless you had important files in the
.data/ (currently mostly used for console scripts and include files)
directory. However it was always confusing to have more than one kind
(zipped, unzipped) of egg.
On Wed, Mar 27, 2013 at 4:41 PM, Vinay Sajip wrote:
> Amaury Forgeot d'Arc gmail.com> writes:
>
>
>> Better: just put the wheel path to sys.path
> sys.path.append('/tmp/simplejson-3.1.2-cp27-none-linux_x86_64.whl')
>> and let a sys.path_hook entry do the job.
>
> That's what the mount() actually does - adds the wheel to a registry that an
> import hook uses. You also need a place to check that the wheel being mounted
> is compatible with the Python doing the mounting - I'm not sure whether what
> the import hook should do if e.g. there is a compatibility problem with the
> wheel (e.g. is it clear that it should always raise an ImportError? Or ignore
> the wheel - seems wrong? Or do something else?)
>
> Regards,
>
> Vinay Sajip
>
> ___
> Python-Dev mailing list
> [email protected]
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Safely importing zip files with C extensions
Stefan Behnel behnel.de> writes: > I've always hated this setuptools misfeature of copying C extensions from > an installed archive into a user directory, one for each user. At least > during normal installation, they should be properly unpacked into normal > shared library files in the file system. The user directory location is not a key part of the functionality, it could just as well be a shared location across all users. And this is an option for specific scenarios, not a general substitute for installing the wheel (which unpacks everything into FHS-style locations). A lot of people use virtual envs, which are per-user anyway. I'm not suggesting this is a good idea for system-wide deployments of software. > Whether it then makes sense to special case one-shot trial imports like the > above without installation is a bit of a different question, but I don't > see a compelling reason for adding complexity here. It's not really an > important use case. Well, my post was to elicit some comment about the usefulness of the feature, so fair enough. It doesn't seem especially complex though, unless I've missed something. Regards, Vinay Sajip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Safely importing zip files with C extensions
Daniel Holth gmail.com> writes: > zipping from ever happening. This is an important reason why wheel is > billed as an installation format -- fewer users with pitchforks. It's > very cool that it works though. Debugging is slightly easier than it > was in the old days because pdb can now read the source code from the > zip. Well, it's just an experiment, and I was soliciting comments because I'm not as familiar with the issues as some others are. Distlib is still only at version 0.1.1, and the mount()/unmount() functionality is not set in stone :-) Regards, Vinay Sajip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Safely importing zip files with C extensions
On Wed, Mar 27, 2013 at 1:13 PM, Amaury Forgeot d'Arc wrote:
> 2013/3/27 Vinay Sajip
>
>> When you mount a wheel, its absolute path name is added to
>> sys.path, allowing the Python code in it to be imported.
>>
>
> Better: just put the wheel path to sys.path
> sys.path.append('/tmp/simplejson-3.1.2-cp27-none-linux_x86_64.whl')
> and let a sys.path_hook entry do the job.
>
> Such a WheelImporter could even inherit from zipimporter, plus the magic
> required for C extensions.
>
I implemented just such a path hook zipimporter plus the magic
required for C extensions --- as a challenge to myself to learn more about
the Python import mechanisms.
See https://github.com/bfroehle/pydzipimport.
Cheers,
Brad
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Writing importers and path hooks
On 27 March 2013 21:19, Bradley M. Froehle wrote: > I implemented just such a path hook zipimporter plus the magic required > for C extensions --- as a challenge to myself to learn more about the Python > import mechanisms. > > See https://github.com/bfroehle/pydzipimport. Apologies for hijacking the thread, but it's interesting that you implemented your hook like this. I notice that you didn't use any of the importlib functionality in doing so. Was there a particular reason? I ask because a few days ago, I was writing a very similar importer, as I wanted to try a proof of concept importer based on the new importlib stuff (which is intended to make writing custom importers easier), and I really struggled to get something working. It seems to me that the importlib documentation doesn't help much for people trying to import path hooks. But it might be just me. Does anyone have an example of a simple importlib-based finder/loader? That would be a huge help for me. In return for any pointers, I'll look at putting together a doc patch to clarify how to use importlib to build your own path hooks :-) Thanks, Paul ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Google Summer of Code - Organization Deadline Approaching - March 29
On 03/26/2013 01:51 PM, Brian Curtin wrote: Just an FYI that there are under 3 days to apply to Google Summer of Code for mentoring organizations: http://www.google-melange.com/gsoc/homepage/google/gsoc2013. The student application deadline is later on in May. If you run a project that is interested in applying under the Python umbrella organization, contact Terri Oda at [email protected] I would obviously love it if you got your ideas up before the March 29th deadline so our application will totally shine and we'll be accepted immediately in to the program. But I should note that while the Google deadline is March 29th, I'm happy to accept applications for sub-organizations and new project ideas after that deadline. The students will be descending on April 9th, so if you want to attract the very best, try to get your project ideas up near then if at all possible! Presuming we get accepted, though, I can take extra ideas almost up until the student applications start to come in, but I'd prefer them before April 15th (again, so you can attract the best students). Is anyone here interested in leading CPython through GSOC? Anyone have potential students to get involved, or interested in being a mentor? For those on the fence about mentoring or worried about doing a good job, I have a few experienced GSoC mentors who are currently without projects who have volunteered to help us out this year. I'm happy to pair anyone who's interested with someone who loves working with GSoC students but maybe won't be familiar with the project, so you can learn the mentoring ropes and do code reviews but have a backup mentor there to help both you and the student through the GSoC process. Please let me know ASAP if you'd like to do this so I can introduce you to the person and give them time to learn as much as they can about the projects you're hoping to run before students start arriving. Terri ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Safely importing zip files with C extensions
On Wed, Mar 27, 2013 at 5:19 PM, Bradley M. Froehle wrote: > I implemented just such a path hook zipimporter plus the magic required > for C extensions --- as a challenge to myself to learn more about the Python > import mechanisms. > > See https://github.com/bfroehle/pydzipimport. FYI, there appears to be a bug for Windows with packages: you're using '/__init__' in a couple places that should actually be os.sep+'__init__'. This does seem like a good way to address the issue, for those rare situations where this would be a good idea. The zipped .egg approach was originally intended for user-managed plugin directories for certain types of extensible platforms, where "download a file and stick it in the plugins directory" is a low-effort way to install plugins, without having to build a lot of specialized install capability. As Jim has pointed out, though, this doesn't generalize well to a full-blown packaging system. Technically, you can blame Bob Ippolito for this, since he's the one who talked me into using eggs to install Python libraries in general, not just as a plugin packaging mechanism. ;-) That being said, *unpacked* egg, er, wheels, are still a great way to meet all of the "different apps needing different versions" use cases (without needing one venv per app), and nowadays the existence of automated installer tools means that using one to install a plugin for a low-tech plugin system is not a big deal, as long as that tool supports the simple unpacked wheel scenario. So I wholeheartedly support some kind of mount/unmount or "require"-type mechanism for finding plugins. pkg_resources even has an API for handling simple dynamic plugin dependency resolution scenarios: http://peak.telecommunity.com/DevCenter/PkgResources#locating-plugins It'd be a good idea if distlib provides a similar feature, or at least the APIs upon which apps or frameworks can implement such features. (Historical note for those who weren't around back then: easy_install wasn't even an *idea* until well after eggs were created; the original idea was just that people would build plugins and libraries as eggs and manually drop them in directories, where a plugin support library would discover them and add them to sys.path as needed. And Bob and I also considered a sort of "update site" mechanism ala Eclipse, with a library to let apps fetch plugins. But as soon as eggs existed and PyPI allowed uploads, it was kind of an obvious follow-up to make an installation tool as a kind of "technology demonstration" which promptly became a monster. The full story with all its twists and turns can also be found here: http://mail.python.org/pipermail/python-dev/2006-April/064145.html ) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Can I introspect/reflect to get arguments exec()?
On Tue, Mar 26, 2013 at 11:00 PM, Rocky Bernstein wrote: > Okay. But is the string is still somewhere in the CPython VM stack? (The > result of LOAD_CONST 4 above). Is there a way to pick it up from there? Maybe using C you could peek into the frame's value stack, but that's not exposed to any Python API I know of. But that still doesn't help you, because the value will be removed from the stack before exec() is actually called, which means if you go looking for it in code called from the exec (e.g. the call event itself), you aren't going to see the data. > At the point that we are stopped the exec action hasn't taken place yet. That doesn't help if you're using line-level tracing events. At the beginning of the line, the data's not on the call stack yet, and by the time you enter the frame of the code being exec'd, it'll be off the stack again. Basically, there is no way to do what you're asking for, short of replacing the built-in exec function with your own version. And it still won't help you with stepping through the source of functions that are *compiled* using an exec() or compile(), or any other way of ending up with dynamically-generated code you want to debug. (Unless you use something like DecoratorTools to generate it, that is -- DecoratorTools has some facilities for caching dynamically-generated code so that it works properly with debuggers. But that has to be done by the code doing the generation, not the debugger. If the code generator uses DecoratorTools' caching support, then any debugger that uses the linecache module will Just Work. It might be nice for the stdlib should have something like this, but you could also potentially fake it by replacing the builtin eval, exec, compile, etc. functions w/versions that cache the source.) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Post-PyCon updates to PyParallel
[ python-dev: I've set up a new list for pyparallel discussions: https://lists.snakebite.net/mailman/listinfo/pyparallel. This e-mail will be the last I'll send to python-dev@ regarding the on-going pyparallel work; please drop python-dev@ from the CC and just send to [email protected] -- I'll stay on top of the posts-from-unsubscribed-users moderation for those that want to reply to this e-mail but not subscribe. ] Hi folks, Wanted to give a quick update on the parallel work both during and after PyCon. During the language summit when I presented the slides I uploaded to speakerdeck.com, the majority of questions from other developers revolved around the big issues like data integrity and what happens when parallel objects interact with main-thread objects and vice-versa. So, during the sprints, I explored putting guards in place to throw an exception if we detect that a user has assigned a parallel object to a non-protected main-thread object. (I describe the concept of 'protection' in my follow up posts to python-dev last week: http://mail.python.org/pipermail/python-dev/2013-March/124690.html. Basically, protecting a main-thread object allows code like this to work without crashing: d = async.dict() def foo(): # async.rdtsc() is a helper method # that basically wraps the result of # the assembly RDTSC (read time- # stamp counter) instruction into a # PyLong object. So, it's handy when # I need to test the very functionality # being demonstrated here (creating # an object within a parallel context # and persisting it elsewhere). d['foo'] = async.rdtsc() def bar(): d['bar'] = async.rdtsc() async.submit_work(foo) async.submit_work(bar) ) It was actually pretty easy, far easier than I expected. It was achieved via Px_CHECK_PROTECTION(): https://bitbucket.org/tpn/pyparallel/commits/f3fe082668c6f3f699db990f046291ff66b1b467#LInclude/object.hT1072 Various new tests related to the protection functionality: https://bitbucket.org/tpn/pyparallel/commits/f3fe082668c6f3f699db990f046291ff66b1b467#LLib/async/test/test_primitives.pyT58 The type of changes I had to make to other parts of CPython to perform the protection checks: https://bitbucket.org/tpn/pyparallel/commits/f3fe082668c6f3f699db990f046291ff66b1b467#LObjects/abstract.cT170 That was all working fine... until I started looking at adding support for lists (i.e. appending a parallel thread object to a protected, main-thread list). The problem is that appending to a list will often involve a list resize, which is done via PyMem_REALLOC() and some custom fiddling. That would mean if a parallel thread attempts to append to a list and it needs resizing, all the newly realloc'd memory would be allocated from the parallel context's heap. Now, this heap would stick around as long as the parallel objects have a refcount > 0. However, as soon as the last parallel object's refcount hits 0, the entire context will be scheduled for the cleanup/release/free dance, which will eventually blow away the entire heap and all the memory allocated against that heap... which means all the **ob_item stuff that was reallocated as part of the list resize. Not particularly desirable :-) As I was playing around with ways to potentially pre-allocate lists, it occurred to me that dicts would be affected in the exact same way; I just hadn't run into it yet because my unit tests only ever assigned a few (<5) objects to the protected dicts. Once the threshold gets reached (10?), a "dict resize" would take place, which would involve lots of PyMem_REALLOCs, and we get into the exact same situation mentioned above. So, at that point, I concluded that whole async protection stuff was not a viable long term solution. (In fact, the reason I first added it was simply to have an easy way to test things in unit tests.) The new solution I came up with: new thread-safe, interlocked data types that are *specifically* designed for this exact use case; transferring results from computation in a parallel thread back to a main thread 'container' object. First up is a new list type: xlist() (PyXListObject/PyXList_Type). I've just committed the work-in-progress stuff I've been able to hack out whilst traveling the past few days: https://bitbucket.org/tpn/pyparallel/commits/5b662eba4efe83e94d31bd9db4520a779aea612a It's not finished, and I'm pretty sure it doesn't even compile yet, but the idea is something like this:
Re: [Python-Dev] Post-PyCon updates to PyParallel
On Wed, Mar 27, 2013 at 11:26:51PM -0700, Trent Nelson wrote: > [ python-dev: I've set up a new list for pyparallel discussions: > https://lists.snakebite.net/mailman/listinfo/pyparallel. This > e-mail will be the last I'll send to python-dev@ regarding the > on-going pyparallel work; please drop python-dev@ from the CC > and just send to [email protected] -- I'll stay on > top of the posts-from-unsubscribed-users moderation for those that > want to reply to this e-mail but not subscribe. ] Gah, wrong e-mail address, it's [email protected]. Trent. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
