Re: [Python-Dev] Removing the GIL (Me, not you!)
> But this has been raised before, and was rejected as not worth the > amount of work that would be required to achieve it. In my understanding, there is an important difference between "it was rejected", and "it was not done". Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
>> Sure - but those things don't get modified that often, except for their >> reference count. > > The reference count is the killer, though -- you have > to lock the object even to do that. And it happens > a LOT, to all objects, including immutable ones. Now we are getting into details: you do NOT have to lock an object to modify its reference count. An atomic increment/decrement operation is enough. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] what versions of Python don't have the "addr" field in the socket object?
> I believe this is only in 2.5.1 and later -- can > anyone confirm that? That's correct. Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Making directories and zip files executable
Guido van Rossum wrote: > I could use a refresher on how PJE's patch solves Andy's problem. I'm not sure if you're asking about how you would execute a zip file after the patch has been applied, or about the mechanics of how the patch works. PJE's last post covered the former question, so I'll cover the gist of the latter. The patch works by passing the script argument to the import machinery to see if it is recognised as a valid sys.path entry (i.e. either a directory or a zip file in a default Python installation). If it is, then add that location to the front of sys.path and use the -m switch support to execute the "__main__" module directly. If the filename passed in isn't recognised as a sys.path entry, then it is executed as a script as normal. Cheers, Nick. -- Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia --- http://www.boredomandlaziness.org ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
On 9/12/07, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > Now we are getting into details: you do NOT have to lock > an object to modify its reference count. An atomic > increment/decrement operation is enough. One could measure the performance hit incurred by using atomic operations for refcounting by hacking a few macros -- right? Deferred reference counting (DRC for short) might help... http://www.memorymanagement.org/glossary/d.html#deferred.reference.counting I can explain a little more how this works if anyone's interested. DRC basically eliminates reference counting for locals--that is, pointers from the stack to an object. An object becomes refcounted only when some other object gets a pointer to it. The drawback is that destructors aren't called quite as promptly as in true refcounting. (They're still called in the right order, though--barring cycles, an object's destructor is called before its children's destructors.) What counts as "stack" is up to the implementation; typically it means "the C stack". This could be used to eliminate most refcounting in C code, although listobject.c would keep it. The amount of per-platform assembly code needed is surprisingly small (and won't change, once you've written it--the Tamarin JavaScript VM does this). You could go further and treat the Python f_locals and interpreter stack as "stack". I think this would eliminate all refcounting in the interpreter. Of course, it complicates matters that f_locals is actually an object visible from Python. Just a thought, not a demand, please don't flame me, -j ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
Brett> We should probably document where all of these globals lists are Brett> instead of relying on looking for all file level static Brett> declarations or something. I smell a wiki page. Skip Brett> Or would there be benefit to moving things like this to the Brett> interpreter struct so that threads within a single interpreter Brett> call are locked but interpreters can act much more independently? Would that simplify things all that much? All containers would probably still rely on the GIL. Also, all objects rely on it to do reference count increment/decrement as I recall. Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
Martin> Now we are getting into details: you do NOT have to lock an Martin> object to modify its reference count. An atomic Martin> increment/decrement operation is enough. Implemented in asm I suspect? For common CPUs this could just be part of the normal Python distribution. For uncommon ones this could use a lock until someone gets around to writing the necessary couple lines of assembler. Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Windows package for new SSL package?
I can't figure out how to build a Windows package for ssl-1.1.tar.gz, and probably don't have the tools to do it anyway. I presume that both a Windows machine and Visual Studio (because there's a C extension) is required? Anyone out there who's interested in the challenge? It's at http://www.parc.com/janssen/transient/ssl-1.1.tar.gz. Incidentally, there's no documentation in the package; instead, just use the development documentation at http://docs.python.org/dev/library/ssl.html. Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] SSL-protected server on python.org for testing?
The SSL tests currently use SSL-protected ports on gmail.com and Verisign for testing. That's not what they are for; I think we should shift to using SSL-protected ports on python.org somewhere. Are there any HTTPS servers, or SSL-protected POP or IMAP servers, currently running on python.org already that I could use? The "use" is an SSL handshake with the server, once or twice per test run. Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] frozenset C API?
> > By the way, I think the hostname matching provisions of 2818 (which > > is, after all, only an informational RFC, not a standard) are poorly > > thought out. Many machines have more hostnames than you can shake a > > stick at, and often provide certs with the wrong hostname in them > > (usually because they have no way to determine what the *right* > > hostname is, from inside that machine). > > ...which is why you pretty much need to have a canonical hostname mapped > to each IP you're using on a machine. Basically, you need to map the > hostname you intend to use to an IP, then do reverse-DNS to find out > whether the hostname is in fact the canonical hostname. If not, you're > using the wrong hostname on your cert. Yep. The problem is having a particular service know which certificate it should choose to use, and also to know when the network connectivity has changed. Usually, server ports are bound to wildcard IP addresses, so that they can still be reached even if the network connectivity changes (particularly true for servers running on laptops, or the Python server I'm running on my iPhone). The server has no way of knowing which IP address the client knows it as, and no way of knowing which of its multiple certificates to present, so that the name in the cert will match the name the client thought it was using. Or am I wrong? Is there some interface in the socket API which gives this information? Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SSL-protected server on python.org for testing?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Sep 12, 2007, at 2:05 PM, Bill Janssen wrote: > The SSL tests currently use SSL-protected ports on gmail.com and > Verisign for testing. That's not what they are for; I think we should > shift to using SSL-protected ports on python.org somewhere. Are there > any HTTPS servers, or SSL-protected POP or IMAP servers, currently > running on python.org already that I could use? The "use" is an SSL > handshake with the server, once or twice per test run. svn.python.org? - -Barry -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (Darwin) iQCVAwUBRugvN3EjvBPtnXfVAQJejQP+JdsEJroDOWdN53cDvtdahJ/2AheObhhb UEdOaucxW3i+odPEUmjLncVq70IQJt1T4YQuZ835iT+k6OkIoB+eaTU3OqslB6bv JKMYsb0Jxdl/plqWld/6WBSH+fCGB5x+JrxelKdu2xVdF8i1YHU+FehK2y1k1kZi Bc9hZ7kONN8= =Uamc -END PGP SIGNATURE- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SSL-protected server on python.org for testing?
Yes, port 443 on svn.python.org seems to work for this purpose. Everyone OK with that? If so, I'll change the SSL test code. Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Making directories and zip files executable
On 9/12/07, Nick Coghlan <[EMAIL PROTECTED]> wrote: > Guido van Rossum wrote: > > I could use a refresher on how PJE's patch solves Andy's problem. > > I'm not sure if you're asking about how you would execute a zip file > after the patch has been applied, or about the mechanics of how the > patch works. PJE's last post covered the former question, so I'll cover > the gist of the latter. > > The patch works by passing the script argument to the import machinery > to see if it is recognised as a valid sys.path entry (i.e. either a > directory or a zip file in a default Python installation). Ah, this is the crux! I didn't understand Phillips wording of "an importable path". I still didn't understand your wording "recognised as a valid sys.path entry"; both wordings suggest a link between sys.argv[0] and the current value of sys.path, which isn't the case -- it is whether they are recognized by the "meta import hook"! This only became clear after I re-read the patch with Phillip's and your words in the back of my head. I now like and approve of the patch, and said so on the tracker. --Guido > If it is, > then add that location to the front of sys.path and use the -m switch > support to execute the "__main__" module directly. > > If the filename passed in isn't recognised as a sys.path entry, then it > is executed as a script as normal. > > Cheers, > Nick. > > -- > Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia > --- > http://www.boredomandlaziness.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows package for new SSL package?
> I can't figure out how to build a Windows package for ssl-1.1.tar.gz,
> and probably don't have the tools to do it anyway. I presume that
> both a Windows machine and Visual Studio (because there's a C
> extension) is required?
>
> Anyone out there who's interested in the challenge?
>
> It's at http://www.parc.com/janssen/transient/ssl-1.1.tar.gz.
>
I had a bit of a look at this. I think I managed to get it building:
* find_ssl() is along way from working on Windows. Python itself uses magic
to locate an SSL directory in the main Python directory's parent. On my
system, this is c:\src\openssl-0.9.7e, but obviously that could be almost
anywhere, and with almost any name. See PCBuild\build_ssl.py and
PCBuild\_ssl.mak for the gory details. I'm not sure how you would like to
approach this (insist on an environment variable for the top-level SSL dir
name?), but in the meantime I hacked find_ssl() to:
ssl_incs = [r"\src\openssl-0.9.7e\inc32",]
ssl_libs = [r"\src\openssl-0.9.7e\out32"]
return ssl_incs, ssl_libs, ["libeay32", "ssleay32", "gdi32", "wsock32"]
* The call to find_ssl() appears to discard the 3rd param:
ssl_incs, ssl_libs, libs = find_ssl()
...
ext_modules=[Extension('ssl._ssl2', ['ssl/_ssl2.c'],
include_dirs = ssl_incs + [socket_inc],
library_dirs = ssl_libs,
libraries = ['ssl', 'crypto'],
depends = ['ssl/socketmodule.h'])],
The 'libraries =' line probably means to pass 'libs' rather than the
literal.
* The "depends = ['ssl/socketmodule.h']" fails for me - no header of that
name exists in the ssl directory in your archive.
After those changes I was able to get it built and tested:
"""
Ran 15 tests in 3.157s
OK
"""
Hope this helps,
Mark
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows package for new SSL package?
Thanks, Mark (and David, who replied to me personally). I'll update the setup.py files with your suggestions and do a 1.2 (with more metadata in it, too). Looks like the functionality is working for people, even if the build is still a bit flakey. Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows package for new SSL package?
> * find_ssl() is along way from working on Windows. Python itself uses magic > to locate an SSL directory in the main Python directory's parent. On my > system, this is c:\src\openssl-0.9.7e, but obviously that could be almost > anywhere, and with almost any name. See PCBuild\build_ssl.py and > PCBuild\_ssl.mak for the gory details. I'm not sure how you would like to > approach this (insist on an environment variable for the top-level SSL dir > name?) Can't we look in the registry for this? We have a working Python; perhaps we can just use a Windows-specific registry lookup to find OpenSSL? (I'm just blue-skying here; I have no clue how things work on Windows.) Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows package for new SSL package?
> > * find_ssl() is along way from working on Windows. Python itself > uses magic > > to locate an SSL directory in the main Python directory's parent. On > my > > system, this is c:\src\openssl-0.9.7e, but obviously that could be > almost > > anywhere, and with almost any name. See PCBuild\build_ssl.py and > > PCBuild\_ssl.mak for the gory details. I'm not sure how you would > like to > > approach this (insist on an environment variable for the top-level > SSL dir > > name?) > > Can't we look in the registry for this? We have a working Python; > perhaps we can just use a Windows-specific registry lookup to find > OpenSSL? (I'm just blue-skying here; I have no clue how things work > on Windows.) Not really. Python itself, when building _ssl, doesn't look for a binary install of openssl, but instead a source directory and a working perl interpreter so an openssl can be built suitable for linking with Python. This source directory is just downloaded and unzipped - no registration takes place, and any binaries that may be built are ignored (we just want the .h and .lib files) It might be possible to try and use build_ssl.py to locate the openssl directory, but this will still require that someone building it has Python built from source - I'm fairly sure that someone installing a Python binary will not have build_ssl.py, nor are they likely to have a suitable openssl directory or installation just "hanging around" either. Mark ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows package for new SSL package?
> > Can't we look in the registry for this? We have a working Python; > > perhaps we can just use a Windows-specific registry lookup to find > > OpenSSL? (I'm just blue-skying here; I have no clue how things work > > on Windows.) > > Not really. Python itself, when building _ssl, doesn't look for a binary > install of openssl, but instead a source directory and a working perl > interpreter so an openssl can be built suitable for linking with Python. > This source directory is just downloaded and unzipped - no registration > takes place, and any binaries that may be built are ignored (we just want > the .h and .lib files) In that case, I think your idea of just hard-coding a path is probably the right thing to do. I'll add a note that this is how you need to do it if you are going to try "python setup.py build". Presumably the binary then built with "python setup.py bdist" will install on a Windows machine regardless of where OpenSSL is installed? Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows package for new SSL package?
"Mark Hammond" <[EMAIL PROTECTED]> writes: > It might be possible to try and use build_ssl.py to locate the openssl > directory, but this will still require that someone building it has Python > built from source - I'm fairly sure that someone installing a Python binary > will not have build_ssl.py, nor are they likely to have a suitable openssl > directory or installation just "hanging around" either. Yep - even if a Windows user has an appropriate development environment in general (and can build most standalone extensions with just a binary Python install), as you say the odds are pretty small they'd have an OpenSSL source tree around, with libraries built. At the same time, I suspect that only a small percentage of Windows users will want to rebuild the extension - rather they'll just want a binary installer, something not uncommon to be published for Windows users of many extension modules. So that pushes the problem upstream a bit where having a Python development tree might be more common or familiar. Rather than a lot of complexity to cater to that small percentage, I'd probably just make setup.py need an explicit configuration - editing, or perhaps environment variable - for the location of the root of the OpenSSL source tree. As you say, there's no guaranteed way to find it otherwise, although I suppose it might try checking relative to the Python executable (along the same lines as build_ssl.py) in case it's being built from within the source tree. Adding some comments that following instructions to build Python from source (or at least the standard _ssl module) will yield just such a tree should be a simple enough as a reference for those who need it. The setup.py does also need to understand the different library names (and required system libraries) to build properly under Windows, as you've already highlighted, but that should be relatively easy to vary by platform. -- David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] SSL certs
On Wed, Sep 12, 2007, Bill Janssen wrote: > >>> By the way, I think the hostname matching provisions of 2818 (which >>> is, after all, only an informational RFC, not a standard) are poorly >>> thought out. Many machines have more hostnames than you can shake a >>> stick at, and often provide certs with the wrong hostname in them >>> (usually because they have no way to determine what the *right* >>> hostname is, from inside that machine). >> >> ...which is why you pretty much need to have a canonical hostname mapped >> to each IP you're using on a machine. Basically, you need to map the >> hostname you intend to use to an IP, then do reverse-DNS to find out >> whether the hostname is in fact the canonical hostname. If not, you're >> using the wrong hostname on your cert. > > Yep. The problem is having a particular service know which > certificate it should choose to use, and also to know when the network > connectivity has changed. Usually, server ports are bound to wildcard > IP addresses, so that they can still be reached even if the network > connectivity changes (particularly true for servers running on > laptops, or the Python server I'm running on my iPhone). The server > has no way of knowing which IP address the client knows it as, and no > way of knowing which of its multiple certificates to present, so that > the name in the cert will match the name the client thought it was > using. My understanding is that the client tells the server which hostname it wants to use; the server should then pass down that information. That's how virtual hosting works in the first place. The only difference with SSL is that the hostname must have a unique IP address, so that when the client does a reverse DNS to validate the IP address presented by the SSL certificate, it all comes together correctly. There are, of course, wildcard certs; I don't understand how those work. -- Aahz ([EMAIL PROTECTED]) <*> http://www.pythoncraft.com/ "Many customs in this life persist because they ease friction and promote productivity as a result of universal agreement, and whether they are precisely the optimal choices is much less important." --Henry Spencer http://www.lysator.liu.se/c/ten-commandments.html ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
I was reading GvR's post on this and came up with a theory on how to tackle the problem. I ended up putting it in a blog post. http://www.brainwavelive.com/blog/index.php?/archives/12-Suggestion- for-removing-the-Python-Global-Interpreter-Lock.html What do you think? Prateek On Sep 12, 2007, at 9:08 PM, [EMAIL PROTECTED] wrote: Martin> Now we are getting into details: you do NOT have to lock an Martin> object to modify its reference count. An atomic Martin> increment/decrement operation is enough. Implemented in asm I suspect? For common CPUs this could just be part of the normal Python distribution. For uncommon ones this could use a lock until someone gets around to writing the necessary couple lines of assembler. Skip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ surekap%40gmail.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removing the GIL (Me, not you!)
> What do you think? I think what you are describing is the situation of today, except in a less-performant way. The kernel *already* implements such a "synchronization server", except that all CPUs can act as such. You write "Since we are guaranteeing that synchronized code is running on a single core, it is the equivalent of a lock at the cost of a context switch." This is precisely what a lock costs today: a context switch. Since the Python interpreter is synchronized all of the time, it would completely run on the synchronization server all of the time. As you identify, that single CPU might get overloaded, so your scheme would give no benefits (since Python code could never run in parallel), and only disadvantages (since multiple Python interpreters today can run on multiple CPUs, but could not anymore under your scheme). Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Windows package for new SSL package?
Bill Janssen <[EMAIL PROTECTED]> writes: > In that case, I think your idea of just hard-coding a path is probably > the right thing to do. I'll add a note that this is how you need to do > it if you are going to try "python setup.py build". Presumably the > binary then built with "python setup.py bdist" will install on a Windows > machine regardless of where OpenSSL is installed? Yes (though typically bdist_wininst for the Windows installer), but perhaps not for the reason you think. I think where there's probably a small disconnect here is that, there really isn't an OpenSSL "installed" on the end user's machine. Well, there could be, but Python isn't using it. The OpenSSL library is statically linked as part of the _ssl.pyd module, as it will be with your _ssl2.pyd module. (That's also why there is no OpenSSL to "find" in your setup even with Python installed - at least not any libraries you can use). In other words, both the standard and your extension module on Windows bring along their own OpenSSL. -- David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
