Re: [Python-Dev] Identifier API
Le 14/10/2011 07:44, Georg Brandl a écrit : Am 14.10.2011 00:30, schrieb Victor Stinner: Le jeudi 13 octobre 2011 03:34:00, Victor Stinner a écrit : We would need a new format for Py_BuildValue, e.g. 'a' for ASCII string. Later we can add new functions like _PyDict_GetASCII(). The main difference between my new "const ASCII" string idea and PyIdentifier is the lifetime of objects. Using "const ASCII" strings, the strings are destroyed quickly (to not waste memory), whereas PyIdentifiers ^ are intern strings and so they are only destroyed at Python exit. I don't know if "const ASCII" strings solve a real issue. I implemented my idea. I will do some benchmarks to check if it's useful or not :-) Ok, I did some tests: it is slower with my PyConstASCIIObject. I don't understand why, but it means that the idea is not interesting because the code is not faster. I think you've already given the answer above... I tried with and without interned strings. It doesn't change anything. Victor ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging and binary distributions for Python 3.3
I can't really comment on this. I agree in principle with what you're saying, but I know little about the MSI format so I can't say much more. It feels to me like you're suggesting that the MSI file encapsulate the file layout logic that already has to exist in pysetup, though, which sounds like duplication of effort. Can MSI call out to pysetup to actually install the files and save this duplication? I'm not sure what exactly it is that pysetup does, so I can't say whether there is any duplication. It's possibly to have post-install actions in MSI, so if the files get put into (some) place at first, it would be possible to copy/link them to whatever layout is needed. What I'd like to avoid is that people need to create too many different packages on Windows, for different use cases. It would be better if the author/packager could create one Windows distribution, and have that work in all use cases. As for MSI: it's primary advantages over bdist_wininst is the higher flexibility of integration into systems maintenance infrastructures. UI-less installation, installation through Active Directory, nested installation (as part of some bundle of installers) are all supported by MSI out of the box. IMO, the primary reason to keep bdist_wininst (besides popularity) is that you need to run the packaging on Windows to create an MSI file, whereas bdist_wininst can be created cross-platform (as long as there are no binary extension modules). In addition, bdist_wininst is better wrt. to repeated installations. I'd prefer a setup though where the same package can work in multiple installations without requiring physical copies. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Identifier API
Am 13.10.11 20:38, schrieb Barry Warsaw: On Oct 13, 2011, at 08:08 PM, Martin v. Löwis wrote: Py_CONST_STRING or Py_IDENTIFIER would be fine with me. Given that everything else uses "Id" in their name, Py_IDENTIFIER is probably better? I agree that either is fine, with a slight preference for Py_IDENTIFIER for the same reasons. Ok, so it's Py_IDENTIFIER. Given below, shouldn't that be _Py_IDENTIFIER? It actually is _Py_IDENTIFIER (and was _Py_identifier). Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging and binary distributions for Python 3.3
Thanks for the clarification. I can see why this would be important. But maintaining 3 different interfaces to do essentially the same thing (collect some data from the user, then based on that data put the same set of files in the same places) seems a waste of effort, and a recipe for discrepancies in capabilities. Maybe the wininst and MSI installers should ultimately become simple UIs around a zipfile and an invocation of the packaging APIs? Not that I'm offering to do that work, I'm afraid... I think you are mixing issues: even if they were simple wrappers, they would still be 3 different interfaces (presented to the user). I.e. the user doesn't really care whether there is a zip file inside (as in bdist_wininst) or a cab file (as in bdist_msi), and whether or not the packaging APIs are invoked during installation. So if you want to get rid of interfaces, you really have to drop one of the formats. Making maintenance of the interfaces simpler and more homogenous by having them call into installed Python code is certainly worthwhile. The challenge here is that the installers also work on older Python versions (unless there are extension modules in there), so they could only use the packaging API when installing into 3.3 or newer. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Identifier API
On Oct 14, 2011, at 04:08 PM, Martin v. Löwis wrote: >It actually is _Py_IDENTIFIER (and was _Py_identifier). Yep, I saw your commit to make the change. Thanks! -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging and binary distributions for Python 3.3
On 14 October 2011 15:07, "Martin v. Löwis" wrote: >> I can't really comment on this. I agree in principle with what you're >> saying, but I know little about the MSI format so I can't say much >> more. It feels to me like you're suggesting that the MSI file >> encapsulate the file layout logic that already has to exist in >> pysetup, though, which sounds like duplication of effort. Can MSI call >> out to pysetup to actually install the files and save this >> duplication? > > I'm not sure what exactly it is that pysetup does, so I can't say > whether there is any duplication. It's possibly to have post-install > actions in MSI, so if the files get put into (some) place at first, > it would be possible to copy/link them to whatever layout is needed. OK, that might be a useful approach. > What I'd like to avoid is that people need to create too many different > packages on Windows, for different use cases. It would be better if > the author/packager could create one Windows distribution, and have > that work in all use cases. Absolutely. That is crucial if we're to avoid the sort of fragmentation we've seen in the past (with bdist_wininst, bdist_msi and binary eggs). I don't know if we'll ever get down to one format, but that would be the ideal. > As for MSI: it's primary advantages over bdist_wininst is the > higher flexibility of integration into systems maintenance > infrastructures. UI-less installation, installation through > Active Directory, nested installation (as part of some bundle > of installers) are all supported by MSI out of the box. IMO, > the primary reason to keep bdist_wininst (besides popularity) > is that you need to run the packaging on Windows to create an > MSI file, whereas bdist_wininst can be created cross-platform > (as long as there are no binary extension modules). In addition, > bdist_wininst is better wrt. to repeated installations. I'd > prefer a setup though where the same package can work in > multiple installations without requiring physical copies. One other aspect is that MSI format is essentially opaque (correct me if I'm wrong here). With bdist_msi, if I want to get the compiled binaries out for some reason (maybe to install them in a virtual environment or some type of other custom build) I just unzip the file - the exe header gets ignored. With bdist_msi, I have no idea if there's any way of doing that. Also, there are fewer people with expertise in MSI format. I suspect that even a Unix developer could have a go at modifying the C code in bdist_msi, it's not too MS-specific. I don't know if that's possible for bdist_msi. Speaking personally, the msilib documentation is pretty unreadable, as I don't know anything about the MSI format. Whenever I've tried reading the MS documentation in the past, I've found it pretty impenetrable (a link to a simple tutorial, and some examples of use, in the msilib documentation might help). Paul. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Add a comment explaining this heuristic.
Hi Antoine, > changeset: 701b2e0e6f3f > user:Antoine Pitrou > date:Thu Oct 13 18:07:37 2011 +0200 > summary: > Add a comment explaining this heuristic. > > diff --git a/Objects/stringlib/fastsearch.h b/Objects/stringlib/fastsearch.h > --- a/Objects/stringlib/fastsearch.h > +++ b/Objects/stringlib/fastsearch.h > @@ -115,6 +115,9 @@ > unsigned char needle; > needle = p[0] & 0xff; > #if STRINGLIB_SIZEOF_CHAR > 1 > +/* If looking for a multiple of 256, we'd have two > + many false positives looking for the '\0' byte in UCS2 > + and UCS4 representations. */ I guess this should read “too many”, not “two”. Cheers ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging and binary distributions for Python 3.3
One other aspect is that MSI format is essentially opaque (correct me if I'm wrong here). You are wrong: msiexec /a unpacks an MSI extracts the files from the MSI (documented as "administrative installation", meaning that the result of it can again be installed, as it will also produce a stripped MSI file with just the installation procedure). With bdist_msi, if I want to get the compiled binaries out for some reason (maybe to install them in a virtual environment or some type of other custom build) I just unzip the file - the exe header gets ignored. With bdist_msi, I have no idea if there's any way of doing that. It's little known, but was always well supported. See also http://www.python.org/download/releases/2.4/msi/ Also, there are fewer people with expertise in MSI format. That's certainly true. I suspect that even a Unix developer could have a go at modifying the C code in bdist_msi, it's not too MS-specific. s/bdist_msi/bdist_wininst/ I don't know if that's possible for bdist_msi. No need to modify C code - it's all pure Python :-) However, I agree that's beside the point: you do need to understand MSI fairly well for modifying bdist_msi. I'm skeptical with your assertion that a Unix developer could contribute to bdist_wininst though without a Windows installation - you have to test this stuff or else it will break. Speaking personally, the msilib documentation is pretty unreadable, as I don't know anything about the MSI format. Whenever I've tried reading the MS documentation in the past, I've found it pretty impenetrable (a link to a simple tutorial, and some examples of use, in the msilib documentation might help). If somebody would volunteer to write a tutorial, I could provide input. I'm clearly unqualified to write such a document, both for language barrier reasons, and because I continue to fail guessing what precisely it is that people don't understand. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging and binary distributions for Python 3.3
On 14 October 2011 15:13, "Martin v. Löwis" wrote: >> Thanks for the clarification. I can see why this would be important. >> But maintaining 3 different interfaces to do essentially the same >> thing (collect some data from the user, then based on that data put >> the same set of files in the same places) seems a waste of effort, and >> a recipe for discrepancies in capabilities. >> >> Maybe the wininst and MSI installers should ultimately become simple >> UIs around a zipfile and an invocation of the packaging APIs? Not that >> I'm offering to do that work, I'm afraid... > > I think you are mixing issues: even if they were simple wrappers, they > would still be 3 different interfaces (presented to the user). I.e. > the user doesn't really care whether there is a zip file inside > (as in bdist_wininst) or a cab file (as in bdist_msi), and whether or > not the packaging APIs are invoked during installation. > > So if you want to get rid of interfaces, you really have to drop > one of the formats. > > Making maintenance of the interfaces simpler and more homogenous > by having them call into installed Python code is certainly worthwhile. > The challenge here is that the installers also work on older Python > versions (unless there are extension modules in there), so they > could only use the packaging API when installing into 3.3 or newer. You are right that I'm mixing issues somewhat. I think the two issues are multiple interfaces and multiple formats. - On interfaces, I personally don't mind the existence of multiple choices. Some people like a GUI, others like command lines. Some like platform integration, others like integration with the language environment, or a consistent cross-platform experience. Trying to mandate one interface will always upset someone. So I don't see this as an important goal in itself. (But see below for a proviso [1]). - On formats, I strongly believe that having multiple formats is a problem. But I need to be clear here - an installer (MSI, wininst) is a bundle containing executable code (which drives the interface), plus a chunk of data that is the objects to be installed. (I am oversimplifying here, but bear with me). That's necessary because the package must be a single download, but it means that the delivery format combines code and data. Format to me refers to that data (and how accessible it is). The problem is that the packager has two roles: 1. He does the build, which is an essential service as end users don't necessarily have the means to compile. 2. He chooses the distribution format (and hence the user experience), which is not essential per se. The second role is the problem, as the packager's preferences may not match those of the end user. I am proposing decoupling those 2 roles. In the purest sense, I'd like to see a distribution format that contains *purely* the binaries, with no UI intelligence. That's what the proposed bdist_simple format is about. This format can be consumed by external tools that just read it as data (that's what pysetup, and packaging, does). For a standard user experience on Windows, at least, there needs to be a means of wrapping that data into an installer. MSI is the obvious choice, because it's the MS standard, but there's a problem here in that there's no obvious way to retrieve the raw bdist_simple file from the MSI once you've bundled it. The wininst format is better here, as you can retrieve the original format (just by removing, or ignoring, the EXE header). I have no opinions on how the MSI/wininst installers should present options to the user, nor do I have any strong views on how they should do the installation (my instinct says that they should reuse the packaging core code, but as you say that causes compatibility problems, so may not be feasible). My only strong views are: - *if* there are multiple formats, they should be easily convertible - pysetup should be able to consume one of the formats (trivially if a bdist_simple format exists, less so if MSI becomes the only format...) The second point is because pysetup offers capabilities that MSI/wininst do not - at least right now, and probably always, as I expect the core packaging code to support new features like venvs sooner than Windows-specfic formats. If only because Unix developers can hack on packaging. Conversion between bdist_wininst and the proposed bdist_simple format is easy as they are both fundamentally zipfiles. The MSI format is the odd one out here, as I don't know how to build an MSI from binary data (without using bdist_msi - or to put it another way, the layout of the files that bdist_msi expects to build the installer from isn't documented) and I don't know how to get binaries out of an MSI (without installing). If these two things were possible, we'd have the means to convert between formats at will (and writing a tool that non-packagers could use to convert formats would be possible). I'm focusing on writing code to allow packaging to i
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2011-10-07 - 2011-10-14) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open3077 (+25) closed 21884 (+31) total 24961 (+56) Open issues with patches: 1315 Issues opened (42) == #11085: expose _abcoll as collections.abc http://bugs.python.org/issue11085 reopened by georg.brandl #12967: IDLE RPC Proxy for standard IO streams lacks 'errors' attribut http://bugs.python.org/issue12967 reopened by ned.deily #13124: Add "Running a Build Slave" page to the devguide http://bugs.python.org/issue13124 opened by eric.snow #13125: test_all_project_files() expected failure http://bugs.python.org/issue13125 opened by barry #13126: find() slower than rfind() http://bugs.python.org/issue13126 opened by pitrou #13127: xml.dom.Attr.name is not labeled as read-only http://bugs.python.org/issue13127 opened by dillona #13128: httplib debuglevel on CONNECT doesn't print response headers http://bugs.python.org/issue13128 opened by Matt.Spear #13131: FD leak in urllib2 http://bugs.python.org/issue13131 opened by Valery.Khamenya #13132: distutils sends non-RFC compliant HTTP request http://bugs.python.org/issue13132 opened by mitchellh #13133: FD leaks in ZipFile.read(), ZipFile.extract() and also using e http://bugs.python.org/issue13133 opened by Valery.Khamenya #13139: multiprocessing.map skips finally blocks http://bugs.python.org/issue13139 opened by illicitonion #13140: ThreadingMixIn.daemon_threads is not honored when parent is da http://bugs.python.org/issue13140 opened by flox #13141: get rid of old threading API in the examples http://bugs.python.org/issue13141 opened by flox #13143: os.path.islink documentation is ambiguous http://bugs.python.org/issue13143 opened by Garen #13144: Global Module Index link in the offline documentation is incor http://bugs.python.org/issue13144 opened by graemeglass #13146: Writing a pyc file is not atomic http://bugs.python.org/issue13146 opened by pitrou #13147: Multiprocessing Pool.map_async() does not have an error_callba http://bugs.python.org/issue13147 opened by Jakub.Gedeon #13149: optimization for append-only StringIO http://bugs.python.org/issue13149 opened by pitrou #13150: Most of Python's startup time is sysconfig http://bugs.python.org/issue13150 opened by pitrou #13151: pysetup3 run bdist_wininst fails http://bugs.python.org/issue13151 opened by vinay.sajip #13152: textwrap: support custom tabsize http://bugs.python.org/issue13152 opened by jfeuerstein #13153: IDLE crash with unicode bigger than 0x http://bugs.python.org/issue13153 opened by JBernardo #13156: _PyGILState_Reinit assumes auto thread state will always exist http://bugs.python.org/issue13156 opened by grahamd #13157: Build Python outside the source directory http://bugs.python.org/issue13157 opened by haypo #13160: Rename install_dist to install http://bugs.python.org/issue13160 opened by eric.araujo #13161: problems with help() documentation of __i*__ operators http://bugs.python.org/issue13161 opened by eli.bendersky #13163: `port` and `host` are confused in `_get_socket http://bugs.python.org/issue13163 opened by cool-RR #13164: importing rlcompleter module writes a control sequence in stdo http://bugs.python.org/issue13164 opened by valva #13165: Integrate stringbench in the Tools directory http://bugs.python.org/issue13165 opened by pitrou #13166: Implement packaging.database.Distribution.__str__ http://bugs.python.org/issue13166 opened by eric.araujo #13167: Add get_metadata to packaging http://bugs.python.org/issue13167 opened by eric.araujo #13168: Python 2.6 having trouble finding modules when invoked via a s http://bugs.python.org/issue13168 opened by RandyGalbraith #13169: Regular expressions with 0 to 65536 repetitions raises Overflo http://bugs.python.org/issue13169 opened by techmaurice #13170: distutils2 test failures http://bugs.python.org/issue13170 opened by eric.araujo #13171: Bug in file.read(), can access unknown data. http://bugs.python.org/issue13171 opened by Alexander.Steppke #13172: pysetup run --list-commands fails with a traceback http://bugs.python.org/issue13172 opened by pmoore #13173: Default values for string.Template http://bugs.python.org/issue13173 opened by nitupho #13174: test_os failures on Fedora 15: listxattr() returns ['security. http://bugs.python.org/issue13174 opened by haypo #13175: packaging uses wrong line endings in RECORD files on Windows http://bugs.python.org/issue13175 opened by pmoore #13177: Avoid chained exceptions in lru_cache http://bugs.python.org/issue13177 opened by ezio.melotti #13178: Need tests for Unicode handling in install_distinfo and instal http://bugs.python.org/issue13178 opened by eric.araujo #13179: IDLE uses common tkinter variables across all editor windows http://bugs.python.o
Re: [Python-Dev] Packaging and binary distributions for Python 3.3
- On formats, I strongly believe that having multiple formats is a problem. But I need to be clear here - an installer (MSI, wininst) is a bundle containing executable code (which drives the interface), plus a chunk of data that is the objects to be installed. (I am oversimplifying here, but bear with me). Beyond oversimplifying, I think this is actually wrong: MSI deliberately is *not* an executable format, but just a "dumb" database, to be interpreted by the installation routines that are already on the system. In that sense, it is very similar to pysetup and bdist_simple. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging and binary distributions for Python 3.3
On 14 October 2011 17:46, "Martin v. Löwis" wrote: > >> - On formats, I strongly believe that having multiple formats is a >> problem. But I need to be clear here - an installer (MSI, wininst) is >> a bundle containing executable code (which drives the interface), plus >> a chunk of data that is the objects to be installed. (I am >> oversimplifying here, but bear with me). > > Beyond oversimplifying, I think this is actually wrong: MSI deliberately > is *not* an executable format, but just a "dumb" database, to be interpreted > by the installation routines that are already on the system. > In that sense, it is very similar to pysetup and bdist_simple. Ah. Sorry, I had misunderstood that, then. So in theory, packaging could be taught to extract the distribution files from an MSI file (using msilib, presumably) and install them much like it could with a zip file. That would imply that the only barrier to using MSI as the default format is the fact that the files can only be manipulated on a Windows platform (which is only a problem if Unix users can build binaries for Windows - they would then be able to build but not package them). I wish I felt more comfortable with MSI as a format (as opposed to an opaque clickable installer). I'd be interested to know what others think. Paul. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Identifier API
Instead of an explicit prefix, how about a macro, such as Py_ID(__string__)? That wouldn't be instead, but in addition - you need the variable name, anyway. Not sure whether there is actually a gain in readability - people not familiar with this would assume that it's a function call of some kind, which it would not be. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Identifier API
Martin v. Löwis wrote: That wouldn't be instead, but in addition - you need the variable name, anyway. But the details of exactly how the name is constructed could be kept as an implementation detail. Not sure whether there is actually a gain in readability - people not familiar with this would assume that it's a function call of some kind, which it would not be. To me the benefit would be that the name you write as the argument would be *exactly* the identifier it represents. If you have to manually add a prefix, there's room for a bit of confusion, especially if the prefix itself ends with an underscore. E.g. if the identifier is "__init__" and the prefix is "PyID_", do you write "PyID__init__" (two underscores) or "PyID___init__" (three underscores?) And can you easily spot the difference in your editor? -- Greg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Identifier API
Am 15.10.2011 01:32, schrieb Greg Ewing: > Martin v. Löwis wrote: >> That wouldn't be instead, but in addition - you need the >> variable name, anyway. > > But the details of exactly how the name is constructed > could be kept as an implementation detail. Is there a use case for keeping that detail hidden? >> Not sure whether there is actually >> a gain in readability - people not familiar with this would >> assume that it's a function call of some kind, which it would >> not be. > > To me the benefit would be that the name you write as > the argument would be *exactly* the identifier it > represents. > > If you have to manually add a prefix, there's room for > a bit of confusion, especially if the prefix itself > ends with an underscore. E.g. if the identifier is > "__init__" and the prefix is "PyID_", do you write > "PyID__init__" (two underscores) or "PyID___init__" > (three underscores?) And can you easily spot the > difference in your editor? The compiler can, very easily. Georg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
