Re: [Python-Dev] Long term development external named branches and periodic merges from python
> Could be implemented as a command line command using "revsets"?.
> Propose a new revset to mercurial devels?
It *is* implemented as a command line command using "revsets".
The revset is
max(ancestors(branch("%s")))-outgoing("%s"))
where the first parameter is the branch that contains your changes,
and the second one is the "path" of the repository you want to diff
against.
In English: find the most recent revision in the ancestry of your
branch that is not an outgoing change wrt. the base repository.
ancestors(branch(yours)) gives all revisions preceding your branches'
tip, which will be your own changes, plus all changes from the "default"
branch that have been merged into your branch (including the changes
from where you originally forked the branch).
Subtracting outgoing removes all changes that are not yet in
cpython, leaving only the changes in your ancestry that come
from cpython. max() then finds the most recent such change,
which will be the "default" parent of your last merge, or
the branch point if you haven't merged after branching.
HTH,
Martin
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] STM and python
Given GCC's announcement that Intel's STM will be an extension for C and C++ in GCC 4.7, what does this mean for Python, and the GIL? I've seen efforts made to make STM available as a context, and for use in user code. I've also read about the "old attempts way back" that attempted to use finer grain locking. The understandably failed due to the heavy costs involved in both the locking mechanisms used, and the overhead of a reference counting garbage collection system. However given advances in locking and garbage collection in the last decade, what attempts have been made recently to try these new ideas out? In particular, how unlikely is it that all the thread safe primitives, global contexts, and reference counting functions be made __transaction_atomic, and magical parallelism performance boosts ensue? I'm aware that C89, platforms without STM/GCC, and single threaded performance are concerns. Please ignore these for the sake of discussion about possibilities. http://gcc.gnu.org/wiki/TransactionalMemory http://linux.die.net/man/4/futex ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] STM and python
2011/11/30 Matt Joiner : > Given GCC's announcement that Intel's STM will be an extension for C > and C++ in GCC 4.7, what does this mean for Python, and the GIL? > > I've seen efforts made to make STM available as a context, and for use > in user code. I've also read about the "old attempts way back" that > attempted to use finer grain locking. The understandably failed due to > the heavy costs involved in both the locking mechanisms used, and the > overhead of a reference counting garbage collection system. > > However given advances in locking and garbage collection in the last > decade, what attempts have been made recently to try these new ideas > out? In particular, how unlikely is it that all the thread safe > primitives, global contexts, and reference counting functions be made > __transaction_atomic, and magical parallelism performance boosts > ensue? Have you seen http://morepypy.blogspot.com/2011/08/we-need-software-transactional-memory.html ? -- Regards, Benjamin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 402: Simplified Package Layout and Partitioning
On Sat, Nov 26, 2011 at 11:53 AM, Éric Araujo wrote: > > Le 11/08/2011 20:30, P.J. Eby a écrit : > >> At 04:39 PM 8/11/2011 +0200, Éric Araujo wrote: > >>> I’ll just regret that it's not possible to provide a module docstring > >>> to inform that this is a namespace package used for X and Y. > >> It *is* possible - you'd just have to put it in a "zc.py" file. IOW, > >> this PEP still allows "namespace-defining packages" to exist, as was > >> requested by early commenters on PEP 382. It just doesn't *require* > >> them to exist in order for the namespace contents to be importable. > > That’s quite cool. I guess such a namespace-defining module (zc.py > here) would be importable, right? Yes. > Also, would it cause worse > performance for other zc.* packages than if there were no zc.py? > No. The first import of a subpackage sets up the __path__, and all subsequent imports use it. > >>> A pure virtual package having no source file, I think it should have no >>> __file__ at all. > > Antoine and someone else thought likewise (I can find the link if you > want); do you consider it consensus enough to update the PEP? > Sure. At this point, though, before doing any more work on the PEP I'd like to have some idea of whether there's any chance of it being accepted. At this point, there seems to be a lot of passive, "Usenet nod syndrome" type support for it, but little active support. It doesn't help at all that I'm not really in a position to provide an implementation, and the persons most likely to implement have been leaning somewhat towards 382, or wanting to modify 402 such that it uses .pyp directory extensions so that PEP 395 can be supported... And while 402 is an extension of an idea that Guido proposed a few years ago, he hasn't weighed in lately on whether he still likes that idea, let alone whether he likes where I've taken it. ;-) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] STM and python
> However given advances in locking and garbage collection in the last > decade, what attempts have been made recently to try these new ideas > out? In particular, how unlikely is it that all the thread safe > primitives, global contexts, and reference counting functions be made > __transaction_atomic, and magical parallelism performance boosts > ensue? > I'd say that given that the current libitm implementation uses a single global lock, you're more likely to see a performance loss. TM is useful to synchronize non-trivial operations: an increment or decrement of a reference count is highly trivial (and expensive when performed atomically, as noted), and TM's never going to help if you put each refcount operation in its own transaction: see Armin's http://morepypy.blogspot.com/2011/08/we-need-software-transactional-memory.html for more realistic use cases. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 402: Simplified Package Layout and Partitioning
Hi, Thanks for the replies. > At this point, though, before doing any more work on the PEP I'd > like to have some idea of whether there's any chance of it being accepted. > At this point, there seems to be a lot of passive, "Usenet nod syndrome" > type support for it, but little active support. If this helps, I am +1, and I’m sure other devs will chime in. I think the feature is useful, and I prefer 402’s way to 382’s pyp directories. I do acknowledge that 402 poses problems to PEP 395 which 382 does not, and as I’m not in a position to help, my vote may count less. Cheers ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 402: Simplified Package Layout and Partitioning
> If this helps, I am +1, and I’m sure other devs will chime in. I think > the feature is useful, and I prefer 402’s way to 382’s pyp directories. If that's the obstacle to adopting PEP 382, it would be easy to revert the PEP back to having file markers to indicate package-ness. I insist on having markers of some kind, though (IIUC, this is also what PEP 395 requires). The main problem with file markers is that a) they must not overlap across portions of a package, and b) the actual file name and content is irrelevant. a) means that package authors have to come up with some name, and b) means that the name actually doesn't matter (but the file name extension would). UUIDs would work, as would the name of the portion/distribution. I think the specific choice of name will confuse people into interpreting things in the file name that aren't really intended. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 402: Simplified Package Layout and Partitioning
On Thu, Dec 1, 2011 at 1:28 AM, PJ Eby wrote:
> It doesn't help at all that I'm not really in a position to provide an
> implementation, and the persons most likely to implement have been leaning
> somewhat towards 382, or wanting to modify 402 such that it uses .pyp
> directory extensions so that PEP 395 can be supported...
While I was initially a fan of the possibilities of PEP 402, I
eventually decided that we would be trading an easy problem ("you need
an '__init__.py' marker file or a '.pyp' extension to get Python to
recognise your package directory") for a hard one ("What's your
sys.path look like? What did you mean for it to look like?"). Symlinks
(and the fact we implicitly call realname() during system
initialisation and import) just make things even messier.
*Deliberately* allowing package structures on the filesystem to become
ambiguous is a recipe for future pain (and could potentially undo a
lot of the good work done by PEP 328's elimination of implicit
relative imports).
I acknowledge there is a lot of confusion amongst novices as to how
packages and imports actually work, but my diagnosis of the root cause
of that problem is completely different from that supposed by PEP 402
(as documented in the more recent versions of PEP 395, I've come to
believe it is due to the way we stuff up the default sys.path[0]
initialisation when packages are involved).
So, in the end, I've come to strongly prefer the PEP 382 approach. The
principle of "Explicit is better than implicit" applies to package
detection on the filesystem just as much as it does to any other kind
of API design, and it really isn't that different from the way we
treat actual Python files (i.e. you can *execute* arbitrary files, but
they need to have an appropriate extension if you want to import
them).
Cheers,
Nick.
--
Nick Coghlan | [email protected] | Brisbane, Australia
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] STM and python
I did see this, I'm not convinced it's only relevant to PyPy. On Thu, Dec 1, 2011 at 2:25 AM, Benjamin Peterson wrote: > 2011/11/30 Matt Joiner : >> Given GCC's announcement that Intel's STM will be an extension for C >> and C++ in GCC 4.7, what does this mean for Python, and the GIL? >> >> I've seen efforts made to make STM available as a context, and for use >> in user code. I've also read about the "old attempts way back" that >> attempted to use finer grain locking. The understandably failed due to >> the heavy costs involved in both the locking mechanisms used, and the >> overhead of a reference counting garbage collection system. >> >> However given advances in locking and garbage collection in the last >> decade, what attempts have been made recently to try these new ideas >> out? In particular, how unlikely is it that all the thread safe >> primitives, global contexts, and reference counting functions be made >> __transaction_atomic, and magical parallelism performance boosts >> ensue? > > Have you seen > http://morepypy.blogspot.com/2011/08/we-need-software-transactional-memory.html > ? > > > -- > Regards, > Benjamin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] STM and python
On Thu, 1 Dec 2011 01:31:14 +1100 Matt Joiner wrote: > > However given advances in locking and garbage collection in the last > decade, what attempts have been made recently to try these new ideas > out? In particular, how unlikely is it that all the thread safe > primitives, global contexts, and reference counting functions be made > __transaction_atomic, and magical parallelism performance boosts > ensue? IMHO, it sounds a bit too magical to be true. > I'm aware that C89, platforms without STM/GCC, and single threaded > performance are concerns. Please ignore these for the sake of > discussion about possibilities. > > http://gcc.gnu.org/wiki/TransactionalMemory I find it interesting that the only example of hardware transactional memory mentioned in this page is a Sun CPU project which has been cancelled. Does Intel have anything similar in the works? Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] STM and python
Azul has been using hardware transactional memory on their custom CPUs (and likely STM in their current x86 virtual machine based products) to great effect for their massively parallel Java VM (700+ cpu cores and gobs of ram) for over 4 years. I'll leave it to the reader to do the relevant searching to read more on that. My point is: This is up to any given Python VM implementation to take advantage of or not as it sees fit. Shoe horning it into an existing VM may not make much sense but anyone is welcome to try. -gps ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] STM and python
On Thu, Dec 1, 2011 at 10:58 AM, Gregory P. Smith wrote: > Azul has been using hardware transactional memory on their custom CPUs (and > likely STM in their current x86 virtual machine based products) to great > effect for their massively parallel Java VM (700+ cpu cores and gobs of ram) > for over 4 years. I'll leave it to the reader to do the relevant searching > to read more on that. > > My point is: This is up to any given Python VM implementation to take > advantage of or not as it sees fit. Shoe horning it into an existing VM may > not make much sense but anyone is welcome to try. There's a patch somewhere on the tracker to add an "Armin Rigo hook" to the CPython eval loop so he can play with STM in Python as well (at least, I think it was STM he wanted it for - it might have been something else). Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] STM and python
I saw this, I believe it just exposes an STM primitive to user code. It doesn't make use of STM for Python internals. Explicit STM doesn't seem particularly useful for a language that doesn't expose raw memory in its normal usage. On Thu, Dec 1, 2011 at 4:41 PM, Nick Coghlan wrote: > On Thu, Dec 1, 2011 at 10:58 AM, Gregory P. Smith wrote: >> Azul has been using hardware transactional memory on their custom CPUs (and >> likely STM in their current x86 virtual machine based products) to great >> effect for their massively parallel Java VM (700+ cpu cores and gobs of ram) >> for over 4 years. I'll leave it to the reader to do the relevant searching >> to read more on that. >> >> My point is: This is up to any given Python VM implementation to take >> advantage of or not as it sees fit. Shoe horning it into an existing VM may >> not make much sense but anyone is welcome to try. > > There's a patch somewhere on the tracker to add an "Armin Rigo hook" > to the CPython eval loop so he can play with STM in Python as well (at > least, I think it was STM he wanted it for - it might have been > something else). > > Cheers, > Nick. > > -- > Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Warnings
When updating the documentation, please don't go overboard with warnings. The docs need to be worded affirmatively -- say what a tool does and show how to use it correctly. See http://docs.python.org/documenting/style.html#affirmative-tone The docs for the subprocess module currently have SEVEN warning boxes on one page: http://docs.python.org/library/subprocess.html#module-subprocess The implicit message is that our tools are hazardous and should be avoided. Please show some restraint and aim for clean looking, high-quality technical writing without the FUD. Look at the SQLite3 docs for an example of good writing. The prevention of SQL injection attacks is discussed briefly and effectively without big red boxes littering the page. Raymond ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 402: Simplified Package Layout and Partitioning
On Nov 30, 2011, at 6:39 PM, Nick Coghlan wrote:
> On Thu, Dec 1, 2011 at 1:28 AM, PJ Eby wrote:
>> It doesn't help at all that I'm not really in a position to provide an
>> implementation, and the persons most likely to implement have been leaning
>> somewhat towards 382, or wanting to modify 402 such that it uses .pyp
>> directory extensions so that PEP 395 can be supported...
>
> While I was initially a fan of the possibilities of PEP 402, I
> eventually decided that we would be trading an easy problem ("you need
> an '__init__.py' marker file or a '.pyp' extension to get Python to
> recognise your package directory") for a hard one ("What's your
> sys.path look like? What did you mean for it to look like?"). Symlinks
> (and the fact we implicitly call realname() during system
> initialisation and import) just make things even messier.
> *Deliberately* allowing package structures on the filesystem to become
> ambiguous is a recipe for future pain (and could potentially undo a
> lot of the good work done by PEP 328's elimination of implicit
> relative imports).
>
> I acknowledge there is a lot of confusion amongst novices as to how
> packages and imports actually work, but my diagnosis of the root cause
> of that problem is completely different from that supposed by PEP 402
> (as documented in the more recent versions of PEP 395, I've come to
> believe it is due to the way we stuff up the default sys.path[0]
> initialisation when packages are involved).
>
> So, in the end, I've come to strongly prefer the PEP 382 approach. The
> principle of "Explicit is better than implicit" applies to package
> detection on the filesystem just as much as it does to any other kind
> of API design, and it really isn't that different from the way we
> treat actual Python files (i.e. you can *execute* arbitrary files, but
> they need to have an appropriate extension if you want to import
> them).
I've helped an almost distressing number of newbies overcome their confusion
about sys.path and packages. Systems using Twisted are, almost by definition,
hairy integration problems, and are frequently being created or maintained by
people with little to no previous Python experience.
Given that experience, I completely agree with everything you've written above
(except for the part where you initially liked it). I appreciate the insight
that PEP 402 offers about python's package mechanism (and the difficulties
introduced by namespace packages). Its statement of the problem is good, but
in my opinion its solution points in exactly the wrong direction: packages need
to be _more_ explicit about their package-ness and tools need to be stricter
about how they're laid out. It would be great if sys.path[0] were actually
correct when running a script inside a package, or at least issued a warning
which would explain how to correctly lay out said package. I would love to see
a loud alarm every time a module accidentally got imported by the same name
twice. I wish I knew, once and for all, whether it was 'import Image' or 'from
PIL import Image'.
My hope is that if Python starts to tighten these things up a bit, or at least
communicate better about best practices, editors and IDEs will develop better
automatic discovery features and frameworks will start to normalize their
sys.path setups and stop depending on accidents of current directory and script
location. This will in turn vastly decrease confusion among new python
developers taking on large projects with a bunch of libraries, who mostly don't
care what the rules for where files are supposed to go are, and just want to
put them somewhere that works.
-glyph
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Warnings
On Dec 1, 2011, at 1:10 AM, Raymond Hettinger wrote: > When updating the documentation, please don't go overboard with warnings. > The docs need to be worded affirmatively -- say what a tool does and show how > to use it correctly. > See http://docs.python.org/documenting/style.html#affirmative-tone > > The docs for the subprocess module currently have SEVEN warning boxes on one > page: > http://docs.python.org/library/subprocess.html#module-subprocess > The implicit message is that our tools are hazardous and should be avoided. > > Please show some restraint and aim for clean looking, high-quality technical > writing without the FUD. > > Look at the SQLite3 docs for an example of good writing. The prevention of > SQL injection attacks is discussed briefly and effectively without big red > boxes littering the page. I'm not convinced this is actually a great example of how to outline pitfalls clearly; it doesn't say what an SQL injection attack is, or what the consequences might be. Also, it's not the best example of a positive tone. The narrative is: You probably want to do X. Don't do Y, because it will make you vulnerable to a Q attack. Instead, do Z. Here's an example of Y. Don't do it! Okay, finally, here's an example of Z. It would be better to say "You probably want to do X. Here's how you do X, with Z. Here's an example of Z." Then, later, discuss why some people want to do Y, and why you should avoid that impulse. However, what 'subprocess' is doing clearly isn't an improvement, it's not an effective introduction to secure process execution, just a reference document punctuated with ambiguous anxiety. sqlite3 is at least somewhat specific :). I think both of these documents point to a need for a recommended idiom for discussing security, or at least common antipatterns, within the Python documentation. I like the IETF's "security considerations" section, because it separates things off into a section that can be referred to later, once the developer has had an opportunity to grasp the basics. Any section with security implications can easily say "please refer to the 'security considerations' section for important information on how to avoid common mistakes" without turning into a big security digression on its own. -glyph ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Warnings
On Thu, Dec 1, 2011 at 4:10 PM, Raymond Hettinger wrote: > When updating the documentation, please don't go overboard with warnings. > The docs need to be worded affirmatively -- say what a tool does and show > how to use it correctly. > See http://docs.python.org/documenting/style.html#affirmative-tone > > The docs for the subprocess module currently have SEVEN warning boxes on one > page: > http://docs.python.org/library/subprocess.html#module-subprocess > The implicit message is that our tools are hazardous and should be avoided. I have no problem with eliminating a lot of those specific warnings - I kept them there in the last rewrite (and added a couple of new ones) because avoiding shell injection vulnerabilities is such a driving theme behind the subprocess module design. Since I was already changing a lot of other things, messing with that aspect really wasn't high on my priority list. Now that we have the "frequently used arguments" section, though, the rest of the warnings could fairly readily be downgraded to notes or inline references to that section. > Please show some restraint and aim for clean looking, high-quality technical > writing without the FUD. I do object to you calling genuine attempts to educate programmers about security issues FUD, though. It's not FUD - novice programmers inflict shell injection, script injection and SQL injection vulnerabilities on the world every day. The multiple warnings are there in the subprocess docs because people often only look at the documentation for the specific function they're interested in, not at the broader context of the page it is part of. "Overkill" is a legitimate complaint, but calling attempts to highlight genuinely insecure practices FUD is the kind of attitude that has given the world so many years of persistent vulnerability to buffer overflow attacks :P Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Warnings
On Thu, Dec 1, 2011 at 5:15 PM, Glyph wrote: > I think both of these documents point to a need for a recommended idiom for > discussing security, or at least common antipatterns, within the Python > documentation. I like the IETF's "security considerations" section, because > it separates things off into a section that can be referred to later, once > the developer has had an opportunity to grasp the basics. Any section with > security implications can easily say "please refer to the 'security > considerations' section for important information on how to avoid common > mistakes" without turning into a big security digression on its own. I like that approach - one of the problems with online docs is the fact people don't read them in order, hence the proliferation of warnings for the subprocess module. A clear "Security Considerations" section with appropriate cross links would allow us to be clear and explicit about common problems without littering the docs with red warning boxes for security issues that are inherent in a particular task rather than being a Python-specific problem. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Warnings
On Thu, Dec 1, 2011 at 5:36 PM, Nick Coghlan wrote: > On Thu, Dec 1, 2011 at 5:15 PM, Glyph wrote: >> I think both of these documents point to a need for a recommended idiom for >> discussing security, or at least common antipatterns, within the Python >> documentation. I like the IETF's "security considerations" section, because >> it separates things off into a section that can be referred to later, once >> the developer has had an opportunity to grasp the basics. Any section with >> security implications can easily say "please refer to the 'security >> considerations' section for important information on how to avoid common >> mistakes" without turning into a big security digression on its own. > > I like that approach - one of the problems with online docs is the > fact people don't read them in order, hence the proliferation of > warnings for the subprocess module. A clear "Security Considerations" > section with appropriate cross links would allow us to be clear and > explicit about common problems without littering the docs with red > warning boxes for security issues that are inherent in a particular > task rather than being a Python-specific problem. I created http://bugs.python.org/issue13515 to propose a specific documentation style guide adopt along these lines (expanded a bit to cover other cross-cutting concerns like the pipe buffer blocking I/O problem in subprocess). Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
