Re: [Python-Dev] Support for Linux perf

[Julian Taylor]

On 17.11.2014 23:09, Francis Giraldeau wrote:
> Hi, 
> ...
> The PEP-418 is about performance counters, but there is no mention o
> Anyway, I think we must change CPython to support tools such as perf.
> Any thoughts? 
> 

there are some patches available adding systemtap and dtrace probes,
which should at least help getting function level profiles:

http://bugs.python.org/issue21590

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators

[Nick Coghlan]

On 22 Nov 2014 02:51, "Antoine Pitrou"  wrote:
>
> On Fri, 21 Nov 2014 05:47:58 -0800
> Raymond Hettinger  wrote:
> >
> > Another issue is that it breaks the way I and others have taught for
years that generators are a kind of iterator (an object implementing the
iterator protocol) and that a primary motivation for generators is to
provide a simpler and more direct way of creating iterators.  However,
Chris explained that, "This proposal causes a separation of generators and
iterators, so it's no longer possible to pretend that they're the same
thing."  That is a major and worrisome conceptual shift.
>
> I agree with Raymond on this point.

A particularly relevant variant of the idiom is the approach of writing
"__iter__" directly as a generator, rather than creating a separate custom
iterator class. In that context, the similarities between the __iter__
implementation and the corresponding explicit __next__ implementation is a
beneficial feature.

I'm definitely coming around to the point of view that, even if we wouldn't
design it the way it currently works given a blank slate, the alternative
design doesn't provide sufficient benefit to justify the cost of changing
the behaviour & getting people to retrain their brains.

Cheers,
Nick.

>
> Regards
>
> Antoine.
>
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Nick Coghlan]

On 22 Nov 2014 07:37, "Donald Stufft"  wrote:
> > On Nov 21, 2014, at 3:59 PM, Ned Deily  wrote:
> > Sure, I get that.  But we're not even talking here about the main Python
> > docs since they are part of the CPython repos, only ancillary repos like
> > PEPs and the developer's guide.  The level of activity on those is quite
> > small.  So, thinking about it a bit more, PEPs don't normally have bug
> > tracker issues associated with them so I suppose my concerns about issue
> > tracker aren't a major concern for them.  The dev guide does get issues
> > opened about it and I suppose they could be managed.  But, without
> > tackling the CPython repo workflow (a *much* bigger deal), is the
> > divergence in workflows worth it?  I dunno.

I also think the tutorial and howto guides should be broken out of the main
CPython repo &  made version independent (with internal version specific
notes).

That offers no compelling advantages right now, but becomes far more
beneficial if it comes with a switch to enabling online editing.

> Yea for the smaller repositories I don’t have a whole lot of opinion
> about if the benefit buys us much, especially since one of the goals
> is new-person friendliness but the problem is that it doesn’t translate
> to contributing to CPython itself.

OK, different question. Has anyone here actually even *read* the workflow
PEPs I wrote? They were on the agenda for the language summit, but got
bumped due to lack of time (which I'm still annoyed about, given the
comparatively inconsequential things that chewed up a whole lot of the day).

I've only had a couple of folks from outside the core dev community express
interest in them. Personally, the lack of online editing support annoys me
immensely whenever I need to work on PEPs or the devguide. I also think
it's ridiculous that we have dozens (hundreds?) of folks running community
workshops, and all creating their own custom documentation, rather than us
finding a way to better enable their collaboration on the official tutorial.

The BitBucket proposal in this thread came out of a desire to avoid adding
yet more work to an understaffed group of primarily volunteers maintaining
the infrastructure (the paid admins are more focused on incident response
and general infrastructure support, rather than spinning up new workflow
services).

My preferred answer remains setting up a srlf-hosted forge.python.org, but
I've seen little evidence we have the capacity to deploy & maintain such a
service effectively, given the relative lack of interest shown in the idea
by almost everyone I've spoken to about it. Any progress has only come with
a lot of pushing from me, and I just don't have the personal bandwidth to
sustain that at this point. That's why the related PEPs were deferred, and
the only responses I've received regarding potentially taking them over
have come from folks outside the core development community, which really
doesn't help very much in removing my availability as a bottleneck in the
workflow improvement process.

If nobody wants to maintain a self-hosted forge, or even enable the folks
that have expressed interest in setting it up & maintaining it, then the
right answer is "don't do it" - we should use a commercial service instead.

Regards,
Nick.

>
> ---
> Donald Stufft
> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[R. David Murray]

On Sun, 23 Nov 2014 00:59:42 +1000, Nick Coghlan  wrote:
> OK, different question. Has anyone here actually even *read* the workflow
> PEPs I wrote? They were on the agenda for the language summit, but got
> bumped due to lack of time (which I'm still annoyed about, given the
> comparatively inconsequential things that chewed up a whole lot of the day).

I have (but I'm core).  It's been a while, though, I need to review it.

> I've only had a couple of folks from outside the core dev community express
> interest in them. Personally, the lack of online editing support annoys me
> immensely whenever I need to work on PEPs or the devguide. I also think

Eh, I hate online editing ;).  But I understand its utility.

[...]

> My preferred answer remains setting up a srlf-hosted forge.python.org, but
> I've seen little evidence we have the capacity to deploy & maintain such a
> service effectively, given the relative lack of interest shown in the idea
> by almost everyone I've spoken to about it. Any progress has only come with
> a lot of pushing from me, and I just don't have the personal bandwidth to
> sustain that at this point. That's why the related PEPs were deferred, and
> the only responses I've received regarding potentially taking them over
> have come from folks outside the core development community, which really
> doesn't help very much in removing my availability as a bottleneck in the
> workflow improvement process.
> 
> If nobody wants to maintain a self-hosted forge, or even enable the folks
> that have expressed interest in setting it up & maintaining it, then the
> right answer is "don't do it" - we should use a commercial service instead.

I have an interest in this, but like you lack bandwidth.  I have no
employer that I could petition for 50% open source time, so any time I
spend on it is time I'm not billing...so my available time comes and
goes depending on my client situation, and I'm also one of the few
putting any time in on tracker maintenance (not that I have much lately,
Ezio has done most of it) and I've still got the email docs to finish up
as well as a list of bugs...yeah, bandwidth is a problem.

But maybe I can take over coordination of the volunteers that want to
work on the forge.  We've already got the mailing list set up.

--David
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 22, 2014, at 9:59 AM, Nick Coghlan  wrote:
> 
> 
> On 22 Nov 2014 07:37, "Donald Stufft"  > wrote:
> > > On Nov 21, 2014, at 3:59 PM, Ned Deily  > > > wrote:
> > > Sure, I get that.  But we're not even talking here about the main Python
> > > docs since they are part of the CPython repos, only ancillary repos like
> > > PEPs and the developer's guide.  The level of activity on those is quite
> > > small.  So, thinking about it a bit more, PEPs don't normally have bug
> > > tracker issues associated with them so I suppose my concerns about issue
> > > tracker aren't a major concern for them.  The dev guide does get issues
> > > opened about it and I suppose they could be managed.  But, without
> > > tackling the CPython repo workflow (a *much* bigger deal), is the
> > > divergence in workflows worth it?  I dunno.
> 
> I also think the tutorial and howto guides should be broken out of the main 
> CPython repo &  made version independent (with internal version specific 
> notes).
> 
> That offers no compelling advantages right now, but becomes far more 
> beneficial if it comes with a switch to enabling online editing.
> 
> > Yea for the smaller repositories I don’t have a whole lot of opinion
> > about if the benefit buys us much, especially since one of the goals
> > is new-person friendliness but the problem is that it doesn’t translate
> > to contributing to CPython itself.
> 
> OK, different question. Has anyone here actually even *read* the workflow 
> PEPs I wrote? They were on the agenda for the language summit, but got bumped 
> due to lack of time (which I'm still annoyed about, given the comparatively 
> inconsequential things that chewed up a whole lot of the day).
> 
> I've only had a couple of folks from outside the core dev community express 
> interest in them. Personally, the lack of online editing support annoys me 
> immensely whenever I need to work on PEPs or the devguide. I also think it's 
> ridiculous that we have dozens (hundreds?) of folks running community 
> workshops, and all creating their own custom documentation, rather than us 
> finding a way to better enable their collaboration on the official tutorial.
> 
> The BitBucket proposal in this thread came out of a desire to avoid adding 
> yet more work to an understaffed group of primarily volunteers maintaining 
> the infrastructure (the paid admins are more focused on incident response and 
> general infrastructure support, rather than spinning up new workflow 
> services).
> 
> My preferred answer remains setting up a srlf-hosted forge.python.org 
> , but I've seen little evidence we have the 
> capacity to deploy & maintain such a service effectively, given the relative 
> lack of interest shown in the idea by almost everyone I've spoken to about 
> it. Any progress has only come with a lot of pushing from me, and I just 
> don't have the personal bandwidth to sustain that at this point. That's why 
> the related PEPs were deferred, and the only responses I've received 
> regarding potentially taking them over have come from folks outside the core 
> development community, which really doesn't help very much in removing my 
> availability as a bottleneck in the workflow improvement process.
> 
> If nobody wants to maintain a self-hosted forge, or even enable the folks 
> that have expressed interest in setting it up & maintaining it, then the 
> right answer is "don't do it" - we should use a commercial service instead.
> 
> 

I think I told you before, but if not I’ll say it now :)

From the infrastructure side spinning up a new VM is pretty easy, what we need
is someone to write some salt states in the psf-salt repository that will
deploy an instance of whatever software. I don't have time to figure out the
intracities of the various software and deploy them but I can help with
figuring out our infra layout. The other thing the infrastructure side would
need is some guidance on what size/flavor of Rackspace VM it needs and what
additional services it would need (we have a PostgreSQL database already, does
it need extra disk space? A Queue? Object Store?). There is a more or less
working vagrant setup in the psf-salt[1] repository (a few of the roles don't
work yet without manual configuration, like the hg role) but for what someone
would need for setting up a new service it should all be there.

I'm not sure what else we can do to enable someone who *does* want to work on
it to be able to set it up. However I'm not against using a commerical service
for smaller repositories. It's fine with me, I don't have a big opinion one
way or the other other than I greatly prefer Github over Bitbucket.

[1] https://github.com/python/psf-salt

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python

Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators

[Ron Adam]

On 11/22/2014 08:31 AM, Nick Coghlan wrote:


On 22 Nov 2014 02:51, "Antoine Pitrou" mailto:solip...@pitrou.net>> wrote:
 >
 > On Fri, 21 Nov 2014 05:47:58 -0800
 > Raymond Hettinger mailto:raymond.hettin...@gmail.com>> wrote:
 > >
 > > Another issue is that it breaks the way I and others have taught for
years that generators are a kind of iterator (an object implementing the
iterator protocol) and that a primary motivation for generators is to
provide a simpler and more direct way of creating iterators.  However,
Chris explained that, "This proposal causes a separation of generators and
iterators, so it's no longer possible to pretend that they're the same
thing."  That is a major and worrisome conceptual shift.
 >
 > I agree with Raymond on this point.

A particularly relevant variant of the idiom is the approach of writing
"__iter__" directly as a generator, rather than creating a separate custom
iterator class. In that context, the similarities between the __iter__
implementation and the corresponding explicit __next__ implementation is a
beneficial feature.

I'm definitely coming around to the point of view that, even if we wouldn't
design it the way it currently works given a blank slate, the alternative
design doesn't provide sufficient benefit to justify the cost of changing
the behaviour & getting people to retrain their brains.


This all seems more complex than it should be to me.  The way I tend to 
think about it is simply "for" loops in one form or another, catch 
StopIteration.  So if you iterate an iterator manually rather than using it 
as a "for" iterator, you need to catch StopIteration.


If we write a function to act as an iterator, like __next__, we need to 
raise StopIteration from somewhere in it, or let one bubble out from a 
generator if we are manually iterating it on each call... next(g). It's 
possible we may need to do either or both conditionally.  That could mean 
we need to think about refactoring some part of a program, but it doesn't 
mean Python needs to be fixed.


So the lines that split them isn't quite as clear cut as it may seem they 
should be.  That may just be a misplaced ideal.


Any time a StopIteration is raised.. either manually with "raise", or at 
the end of a generator, it should bubble up until a "for loop" iterating 
over that bit of code, catches it, or a try-except catches it, or fail 
loudly.  I think it does do this in normal generators, so I don't see an 
issue with how StopIteration works.



Which gets us back to generator expressions and comprehensions.

Let me know if I got some part of this wrong...   :-)

Comprehensions are used as a convenient way to create an object.  The 
expression parts of the comprehension define the *body* of a loop, so a 
StopIteration raised in it will bubble out.  As it would in any other case 
where it is raised in the body of a loop.


Generator exprssions on the other hand define the *iterator* to be used in 
a for loop. A StopIteration raised in it is caught by the for loop.


So they both work as they are designed, but they look so similar, it looks 
like one is broken.




It looks to me that there are three options...


OPTION 1:

Make comprehensions act more like generator expressions.

It would mean a while loop in the object creation point is converted to a 
for loop. (or something equivalent.)


Then both a comprehension and a generator expressions can be viewed as 
defining iterators, with the same behaviour, rather than comprehensions 
defining the body of the loop, which has the different but valid behaviour 
of StopIteration escaping.


This would make explaining them a *lot* easier as they become the same 
thing used in a different context, rather than two different things used in 
what appears to be similar contexts.



I think this fits with what Guido wants, but does so in a narrower scope, 
only effecting comprehensions.  StopIteration is less likely to leak out.


But it also allows the use of the stop() hack to raise StopIteration in 
comprehensions and terminate them early.  Currently it doesn't work as it 
does in generator expressions.


If the stop() hack works in both comprehensions and generator expressions 
the same way, then maybe we can view it as less of a hack.



OPTION 2:

Make generator expressions more like comprehensions.

This would mean StopIteration would bubble out of generator expressions as 
the person who posted the original topic on python ideas wanted.  And the 
stop hack would no longer work.


Both generator expressions and comprehensions could be viewed as supplying 
the body in a loop.  This is inconsistent with defining generators that act 
as iterators.  So I'm definitely -1 on this option.



OPTION 3:

Document the differences better.




Cheers,
   Ron






















___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mail

Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators

[Ethan Furman]

On 11/22/2014 06:31 AM, Nick Coghlan wrote:
> 
> A particularly relevant variant of the idiom is the approach of writing
> "__iter__" directly as a generator, rather than creating a separate custom
> iterator class. In that context, the similarities between the __iter__
> implementation and the corresponding explicit __next__ implementation is a
> beneficial feature.

https://docs.python.org/3/reference/datamodel.html?highlight=__iter__#object.__iter__
--
> This method is called when an iterator is required for a container.
> This method should return a new iterator object that can iterate
> over all the objects in the container. For mappings, it should
> iterate over the keys of the container, and should also be made
> available as the method keys().

> Iterator objects also need to implement this method; they are
> required to return themselves. For more information on iterator
> objects, see Iterator Types.

Unless the object is itself at iterator, the __iter__ method is allowed to 
return any iterator object; whether that
iterator is constructed by a separate class entirely, or by using the iter() 
function, or by writing a generator, should
have no bearing on how we write generators themselves.

--
~Ethan~



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators

[Chris Angelico]

On Sun, Nov 23, 2014 at 6:49 AM, Ron Adam  wrote:
>
> OPTION 1:
>
> Make comprehensions act more like generator expressions.
>
> It would mean a while loop in the object creation point is converted to a
> for loop. (or something equivalent.)
>
> Then both a comprehension and a generator expressions can be viewed as
> defining iterators, with the same behaviour, rather than comprehensions
> defining the body of the loop, which has the different but valid behaviour
> of StopIteration escaping.
>
> This would make explaining them a *lot* easier as they become the same thing
> used in a different context, rather than two different things used in what
> appears to be similar contexts.
>
>
> I think this fits with what Guido wants, but does so in a narrower scope,
> only effecting comprehensions.  StopIteration is less likely to leak out.

A list comp is usually compared to a statement-form loop.

def stop(): raise StopIteration
lst = [stop() for x in (1,2,3)]
lst = []
for x in (1,2,3):
lst.append(stop())

At the moment, these are identical (virtually; the pedantic will point
out that 'x' doesn't leak out of the comprehension) - in each case,
the exception raised by the body will bubble up and be printed to the
console.

But a generator expression is different:

lst = list(stop() for x in (1,2,3))

In this form, lst is an empty list, and nothing is printed to the
console. Making comprehensions work more like generator expressions
would, IMO, imply making the same change to all for loops: having a
StopIteration raised by the body of the loop quietly terminate the
loop.

This is backwards. Most of Python is about catching exceptions as
narrowly as possible: you make your "try" blocks small, you use
specific exception types rather than bare "except:" clauses, etc, etc,
etc. You do your best to catch ONLY those exceptions that you truly
understand, unless you're writing a "catch, log, and reraise" or
"catch, log, and go back for more work" generic handler. A 'for' loop
currently is written more-or-less like this:

for var in expr:
body

it = iter(expr)
while True:
try: var = next(it)
except StopIteration: break
body

But this suggestion of yours would make it become:
it = iter(expr)
while True:
try:
var = next(it)
body
except StopIteration: break

I believe this to be the wrong direction to make the change. Instead,
generator expressions should be the ones to change, because that's a
narrowing of exception-catching scope. Currently, every generator
function is implicitly guarded by "try: .. except StopIteration:
pass". This is allowing errors to pass silently, to allow a hack
involving non-local control flow (letting a chained function call
terminate a generator) - violating the Zen of Python.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators

[Raymond Hettinger]

> On Nov 22, 2014, at 6:31 AM, Nick Coghlan  wrote:
> 
> I'm definitely coming around to the point of view that, even if we wouldn't 
> design it the way it currently works given a blank slate, the alternative 
> design doesn't provide sufficient benefit to justify the cost of changing the 
> behaviour & getting people to retrain their brains.

Thanks Nick.  That was well said.

After a couple more days of thinking about PEP 455 and reading many
of the mailing list posts, I am also still flatly opposed to the proposal
and wanted to summarize my thoughts for everyone's consideration.

It looks like the only point in favor of the PEP is makes the internal
semantics of genexps appear to more closely match list comprehensions.

However, it does so not by fixing anything or adding a capability;
rather, it disallows a coding pattern that potentially exposes the
true differences between genexps and list comps.  Even then, the
difference isn't hidden; instead, the proposal just breaks the code
loudly by raising a RuntimeError.

AFAICT, the problem it solves isn't really a problem in practice.
(I do code reviews and teach Python for living, so I get broad
exposure to how python is used in practice).

As collateral damage, the PEP breaks code that is otherwise
well designed, beautiful looking, and functioning correctly.

Even worse, the damage will be long lasting.  In introduces
a new special case in the current clean design of generators.
And, it creates an additional learning burden (we would now
have to teach the special case and how to work around it
with a "try: v=next(it)  except StopIteration: return".

I realize these are sweeping statements, so I elaborate with more
detail and examples below.  If you're not interested in the details,
feel free to skip the rest of the post; you've already gotten the keys
points.


New Special Case


By design, exceptions raised in generators are passed through to the
caller.  This includes IndexErrors, ValueErrors, StopIteration, and
PEP 342's GeneratorExit.  Under the proposed PEP, there general
rule (exceptions are passed through) is broken and there is a new
special case:  StopIteration exceptions are caught and reraised
as RuntimeErrors.  This is a surprising new behavior.


Breaks well established, documented, and tested behaviors
-

From the first day generators were introduced 13 years ago, we have
documented and promised that you can end a terminate a generator by
raising StopIteration or by a return-statment (that is in PEP 255,
in the whatsnew document for 2.2, in the examples we provided for the
myriad of ways to use generators, in standard library code, in the
Martelli's Python Cookbook example, in the documention for itertools,
etc.)


Legitimate Use Cases for Raising StopIteration in a Generator


In a producer/consumer generator chain, the input generator signals
it is done by raising StopIteration and the output generator signals
that it is done by raising StopIteration (this isn't in dispute).

That use case is currently implemented something like this:

def middleware_generator(source_generator):
it = source_generator()
input_value = next(it)
output_value = do_something_interesting(input_value)
yield output_value

Termination of the input stream will then terminate middleware stream.
You can see several real world examples of this code pattern in
Fredrik Lundh's pure python verion of ElementTree
(prepare_descendant, prepare_predicate, and iterfind).

Under the proposal, the new logic would be to:
1) catch input stream StopIteration
2) return from the generator
3) which in turn raises StopIteration yet again.

This doesn't make the code better in any way.  The new code
is wordy, slow, and unnecessarily convoluted:

def middleware_generator(source_generator):
it = source_generator()
try:
input_value = next(it)
except StopIteration:
return # This causes StopIteration to be reraised
output_value = do_something_interesting(input_value)
yield output_value

I don't look forward to teaching people to have to write code like this
and to have to remember yet another special case rule for Python.


Is next() Surprising?
-

The PEP author uses the word "surprise" many times in describing the
motivation for the PEP.  In the context of comparing generator expressions
to list comprehenions, I can see where someone might be surprised that
though similar in appearance, their implementations are quite different
and that some of those differences might not expected.

However, I believe this is where the "surprise" ends.

The behavior of next(it) is to return a value or raise StopIteration.
That is fundamental to what is does (what else could it do?).

This is as basic as list indexing returning a value or raisi

Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators

[Terry Reedy]

On 11/22/2014 2:49 PM, Ron Adam wrote:


On 11/22/2014 08:31 AM, Nick Coghlan wrote:



I'm definitely coming around to the point of view that, even if we
wouldn't
design it the way it currently works given a blank slate, the alternative
design doesn't provide sufficient benefit to justify the cost of changing
the behaviour & getting people to retrain their brains.


Me too.


Which gets us back to generator expressions and comprehensions.


> Comprehensions are used as a convenient way to create an object.  The

expression parts of the comprehension define the *body* of a loop, so a
StopIteration raised in it will bubble out.  As it would in any other
case where it is raised in the body of a loop.

Generator expressions on the other hand define the *iterator* to be used
in a for loop. A StopIteration raised in it is caught by the for loop.

So they both work as they are designed, but they look so similar, it
looks like one is broken.



It looks to me that there are three options...


OPTION 1:

Make comprehensions act more like generator expressions.


I was thinking about this also.


It would mean a while loop in the object creation point is converted to
a for loop. (or something equivalent.)


The code for [e(i) for i in it] already uses a for loop.  The issue is 
when e(i) raise StopIteration.  The solution to accomplish the above is 
to wrap the for loop in try: ... except StopIteration: pass.


def _list_comp_temp():
ret = []
try:
for i in it:
ret.append[e(i)]
except StopIteration:
pass
return ret

I believe this would make list(or set)(genexp) == [genexp] (or {genexp}) 
as desired.  In 2.x, there was a second difference, the leaking of 'i'. 
 3.x eliminated one difference, the leaking of the iteration variable, 
but not the other, the leaking of StopIteration.


The document about execution of comprehensions says "the comprehension 
is executed in a separate scope, so names assigned to in the target list 
don’t “leak”. What we would be adding for comprensions (but not genexps) 
is "and StopIteration raised in evaluating the expression" before 
'"leak"'.  We could then document the equality above.



Then both a comprehension and a generator expressions can be viewed as
defining iterators,


A comprehension is not an iterator.  The above would make a list or set 
comprehension the same as feeding a genexp to list() or set().



I think this fits with what Guido wants, but does so in a narrower
scope, only effecting comprehensions.  StopIteration is less likely to
leak out.


StopIteration would *not* leak out ever.  This is half of what Guido 
wants, the other half being the issue of obscure bugs with genrators in 
general.



But it also allows the use of the stop() hack to raise StopIteration in
comprehensions and terminate them early.


Yep. There is no good way to have next(it) work as intended, including 
in Raymond's example, where it should work, and not let stop() work.  I 
think this falls under our 'consenting adults' principle.



Currently it doesn't work as it does in generator expressions.
If the stop() hack works in both comprehensions and generator
expressions the same way, then maybe we can view it as less of a hack.


--
Terry Jan Reedy


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators

[Ron Adam]

On 11/22/2014 02:16 PM, Chris Angelico wrote:

On Sun, Nov 23, 2014 at 6:49 AM, Ron Adam  wrote:

>
>OPTION 1:
>
>Make comprehensions act more like generator expressions.
>
>It would mean a while loop in the object creation point is converted to a
>for loop. (or something equivalent.)
>
>Then both a comprehension and a generator expressions can be viewed as
>defining iterators, with the same behaviour, rather than comprehensions
>defining the body of the loop, which has the different but valid behaviour
>of StopIteration escaping.
>
>This would make explaining them a*lot*  easier as they become the same thing
>used in a different context, rather than two different things used in what
>appears to be similar contexts.
>
>
>I think this fits with what Guido wants, but does so in a narrower scope,
>only effecting comprehensions.  StopIteration is less likely to leak out.

A list comp is usually compared to a statement-form loop.

def stop(): raise StopIteration
lst = [stop() for x in (1,2,3)]



lst = []
for x in (1,2,3):
 lst.append(stop())

At the moment, these are identical (virtually; the pedantic will point
out that 'x' doesn't leak out of the comprehension) - in each case,
the exception raised by the body will bubble up and be printed to the
console.


This is what I meant by leaking out.  A StopIteration bubbles up.  And your 
examples match my understanding. :-)






But a generator expression is different:


Yes, but they work as I expect them to.




lst = list(stop() for x in (1,2,3))

In this form, lst is an empty list, and nothing is printed to the
console.


I think that is what it should do.  I think of it this way...

>>> def stop(): raise StopIteration
...
>>> def ge():
...for value in (stop() for x in (1,2,3)):
...   yield value
...
>>> list(ge())
[]

Notice the entire body of the comprehension is in the for loop header, and 
no parts of it are in the body except the reference to the already assigned 
value.


The StopIteration is caught by the outer for loop.  Not the for loop in the 
generator expression, or iterator part.




Making comprehensions work more like generator expressions
would, IMO, imply making the same change to all for loops: having a
StopIteration raised by the body of the loop quietly terminate the
loop.


I'm not suggesting making any changes to generator expressions or for loops 
at all.  They would continue to work like they currently do.



Cheers,
   Ron



This is backwards. Most of Python is about catching exceptions as
narrowly as possible: you make your "try" blocks small, you use
specific exception types rather than bare "except:" clauses, etc, etc,
etc. You do your best to catch ONLY those exceptions that you truly
understand, unless you're writing a "catch, log, and reraise" or
"catch, log, and go back for more work" generic handler. A 'for' loop
currently is written more-or-less like this:

for var in expr:
 body

it = iter(expr)
while True:
 try: var = next(it)
 except StopIteration: break
 body

But this suggestion of yours would make it become:
it = iter(expr)
while True:
 try:
 var = next(it)
 body
 except StopIteration: break



I believe this to be the wrong direction to make the change. Instead,
generator expressions should be the ones to change, because that's a
narrowing of exception-catching scope. Currently, every generator
function is implicitly guarded by "try: .. except StopIteration:
pass". This is allowing errors to pass silently, to allow a hack
involving non-local control flow (letting a chained function call
terminate a generator) - violating the Zen of Python.

ChrisA


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators

[Ron Adam]

On 11/22/2014 03:01 PM, Terry Reedy wrote:



Then both a comprehension and a generator expressions can be viewed as
defining iterators,


A comprehension is not an iterator.  The above would make a list or set
comprehension the same as feeding a genexp to list() or set().


Correct, but we could think and reason about them in the same way.

Cheers,
   Ron


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PCBuild project updates

[Steve Dower]

​Hi all

Just attracting some attention to http://bugs.python.org/issue22919 for those 
of us who build Python out of the PCBuild folder. More details/patches there, 
but the tl;dr is

* Still works with VS 2010 (and now VS 2013 and VS 2015 too)
* Build time should be reduced
* Tools\buildbot\*.bat are also updated, so buildbots shouldn't notice any 
change

If you're concerned/interested, I'll be watching the tracker for comments. 
Ideally I'd like at least a couple of people to test build with whatever VS 
version they have, but if nobody can do that then I'll assume that nobody will 
notice anything break sooner than me :)

Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators

[Chris Angelico]

On Sun, Nov 23, 2014 at 8:03 AM, Ron Adam  wrote:
>> Making comprehensions work more like generator expressions
>> would, IMO, imply making the same change to all for loops: having a
>> StopIteration raised by the body of the loop quietly terminate the
>> loop.
>
>
> I'm not suggesting making any changes to generator expressions or for loops
> at all.  They would continue to work like they currently do.

But if you're suggesting making list comps react to StopIteration
raised by their primary expressions, then to maintain the
correspondence between a list comp and its expanded form, for loops
would have to change too. Or should that correspondence be broken, in
that single-expression loop constructs become semantically different
from statement loop constructs?

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators

[Chris Angelico]

On Sun, Nov 23, 2014 at 7:30 AM, Raymond Hettinger
 wrote:
> Legitimate Use Cases for Raising StopIteration in a Generator
> 
>
> In a producer/consumer generator chain, the input generator signals
> it is done by raising StopIteration and the output generator signals
> that it is done by raising StopIteration (this isn't in dispute).
>
> That use case is currently implemented something like this:
>
> def middleware_generator(source_generator):
> it = source_generator()
> input_value = next(it)
> output_value = do_something_interesting(input_value)
> yield output_value

Does your middleware_generator work with just a single element,
yielding either one output value or none? Or is it more likely to be
iterating over the source generator:

def middleware_generator(source_generator):
for input_value in source_generator:
yield do_something_interesting(input_value)

MUCH tidier code, plus it's safe against unexpected StopIterations.
Even if you can't do it this way, all you need is to stick a 'break'
at the end of the loop, if the try/except is so abhorrent. Wouldn't be
the first time I've seen a loop with a hard break at the end of it.

> This doesn't make the code better in any way.  The new code
> is wordy, slow, and unnecessarily convoluted:
>
> def middleware_generator(source_generator):
> it = source_generator()
> try:
> input_value = next(it)
> except StopIteration:
> return # This causes StopIteration to be reraised
> output_value = do_something_interesting(input_value)
> yield output_value

The raising of StopIteration here is an implementation detail; you
might just as well have a comment saying "This causes the function to
set an exception state and return NULL", which is what happens at the
C level.

What happens if do_something_interesting happens to raise
StopIteration? Will you be surprised that this appears identical to
the source generator yielding nothing? That's current behaviour. You
don't have the option of narrowing the try/except scope as you've done
above.

> Is next() Surprising?
> -
>
> If someone in this thread says that they were suprised that next() could
> raise StopIteration, I don't buy it.

Agreed, I don't think that's surprising to anyone.

> Being able to consume a value from an iterator stream is a fundamental
> skill and not hard to learn (when I teach iterators and generators, the
> operation of next() has never been a stumbling block).

In anything other than a generator, you're expected to cope with two
possible results from next(): a returned value or a raised
StopIteration. Suppose you want to read a file with a header - you'd
need to do something like this:

def process_file(f):
f = iter(f)
try: header=next(f)
except StopIteration: cope_somehow_maybe_return
for line in f:
process_line_with_headers(line, header)

Currently, *if and only if* you're writing a generator, you have an
implicit "except StopIteration: return" there. Anywhere else, you need
to catch that exception.

Iterators are an implementation detail of generators. There is no
particular reason for a generator author to be aware of iterator
protocol, any more than this class needs to be:

class X:
def __iter__(self):
return iter([1,2,3,4])

It's perfectly iterable, just as a generator is, and it knows nothing
about StopIteration.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Brett Cannon]

On Sat Nov 22 2014 at 10:00:03 AM Nick Coghlan  wrote:

>
> On 22 Nov 2014 07:37, "Donald Stufft"  wrote:
> > > On Nov 21, 2014, at 3:59 PM, Ned Deily  wrote:
>
> > > Sure, I get that.  But we're not even talking here about the main
> Python
> > > docs since they are part of the CPython repos, only ancillary repos
> like
> > > PEPs and the developer's guide.  The level of activity on those is
> quite
> > > small.  So, thinking about it a bit more, PEPs don't normally have bug
> > > tracker issues associated with them so I suppose my concerns about
> issue
> > > tracker aren't a major concern for them.  The dev guide does get issues
> > > opened about it and I suppose they could be managed.  But, without
> > > tackling the CPython repo workflow (a *much* bigger deal), is the
> > > divergence in workflows worth it?  I dunno.
>
> I also think the tutorial and howto guides should be broken out of the
> main CPython repo &  made version independent (with internal version
> specific notes).
>
> That offers no compelling advantages right now, but becomes far more
> beneficial if it comes with a switch to enabling online editing.
>
> > Yea for the smaller repositories I don’t have a whole lot of opinion
> > about if the benefit buys us much, especially since one of the goals
> > is new-person friendliness but the problem is that it doesn’t translate
> > to contributing to CPython itself.
>
> OK, different question. Has anyone here actually even *read* the workflow
> PEPs I wrote? They were on the agenda for the language summit, but got
> bumped due to lack of time (which I'm still annoyed about, given the
> comparatively inconsequential things that chewed up a whole lot of the day).
>

I did and was looking forward to them coming to fruition.


> I've only had a couple of folks from outside the core dev community
> express interest in them. Personally, the lack of online editing support
> annoys me immensely whenever I need to work on PEPs or the devguide. I also
> think it's ridiculous that we have dozens (hundreds?) of folks running
> community workshops, and all creating their own custom documentation,
> rather than us finding a way to better enable their collaboration on the
> official tutorial.
>
> The BitBucket proposal in this thread came out of a desire to avoid adding
> yet more work to an understaffed group of primarily volunteers maintaining
> the infrastructure (the paid admins are more focused on incident response
> and general infrastructure support, rather than spinning up new workflow
> services).
>
> My preferred answer remains setting up a srlf-hosted forge.python.org,
> but I've seen little evidence we have the capacity to deploy & maintain
> such a service effectively, given the relative lack of interest shown in
> the idea by almost everyone I've spoken to about it. Any progress has only
> come with a lot of pushing from me, and I just don't have the personal
> bandwidth to sustain that at this point. That's why the related PEPs were
> deferred, and the only responses I've received regarding potentially taking
> them over have come from folks outside the core development community,
> which really doesn't help very much in removing my availability as a
> bottleneck in the workflow improvement process.
>
> If nobody wants to maintain a self-hosted forge, or even enable the folks
> that have expressed interest in setting it up & maintaining it, then the
> right answer is "don't do it" - we should use a commercial service instead.
>
There are two goals to any improvement to the development workflow: that
which helps the core devs and that which helps everyone else. For helping
core devs that's getting some CI set up which will test every patch
submitted, single-click patch committal from the issue tracker, etc. For
everyone else it's inline editing and whatever it takes to get patches
accepted faster (I know Nick is pointing out he wants inline editing for
PEPs and docs but I don't view that as critical for core devs who already
have the checkouts available and have the workflow memorized).

>From my perspective, getting our commit workflow improved is the critical
first step before we worry about making it easier to receive patches. If we
can't keep up with an influx of patches that might occur from inline
editing then there is little point in having it; frustrating people that we
can't commit patches as fast as we receive them is not helpful.

Now in terms of how the heck we are ever going to improve our workflow,
that's tricky. As Nick as pointed out we are low on volunteer time. Take
the issue tracker as an example: Ezio Melotti does a large amount of work
and R. David Murray also helps, but that's mostly it (Martin von Löwis has
helped in the past but has been mostly absent as of late). We are not well
covered in the "hit by a bus" scenario.

I understand the viewpoint of not wanting to give up control of our process
to a third party, and I understand not wanting to use closed-source
software

Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators

[Ron Adam]

On 11/22/2014 04:23 PM, Chris Angelico wrote:

On Sun, Nov 23, 2014 at 8:03 AM, Ron Adam  wrote:

>>Making comprehensions work more like generator expressions
>>would, IMO, imply making the same change to all for loops: having a
>>StopIteration raised by the body of the loop quietly terminate the
>>loop.

>
>
>I'm not suggesting making any changes to generator expressions or for loops
>at all.  They would continue to work like they currently do.



But if you're suggesting making list comps react to StopIteration
raised by their primary expressions, then to maintain the
correspondence between a list comp and its expanded form, for loops
would have to change too.
Or should that correspondence be broken, in
that single-expression loop constructs become semantically different
from statement loop constructs?



Se we have these...

 Tuple Comprehension  (...)
 List Comprehension  [...]
 Dict Comprehension  {...}  Colon make's it different from sets.
 Set Comprehension  {...}

I don't think there is any other way to create them. And they don't 
actually expand to any python code that is visible to a programmer.  They 
are self contained literal expressions for creating these few objects.


The expanded form you are referring to is just how we currently explain 
them.  And yes, we will need to alter how we currently explain 
Comprehensions a bit if this is done.



Here is what I think(?) list comps do currently.

 lst = [expr for items in itr if cond]

Is the same as.

 lst = []
 for x in itr:  # StopIteration  caught here.
 if cond:   # StopIteration  bubbles here.
 lst.append(expr)   # StopIteration  bubbles here.



And it would be changed to this...

def comp_expression():
for x in itr:  # StopIteration caught here.
if cond:
   list.append(expr)

# StopIteration from cond and expr caught here.
lst = list(x for x in comp_expression())



So

   [expr for itr if cond]

Becomes a literal form of:

   list(expr for itr if cond)


And I believe that is how they are explained quite often. :-)


(Although It's not currently quite true, is it?)



Cheers,
   Ron

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators

[Chris Angelico]

On Sun, Nov 23, 2014 at 11:05 AM, Ron Adam  wrote:
> Se we have these...
>
>  Tuple Comprehension  (...)
>  List Comprehension  [...]
>  Dict Comprehension  {...}  Colon make's it different from sets.
>  Set Comprehension  {...}
>
> I don't think there is any other way to create them. And they don't actually
> expand to any python code that is visible to a programmer.  They are self
> contained literal expressions for creating these few objects.

Hmmm, there's no such thing as tuple comprehensions. Are you confusing
comprehension syntax (which has a 'for' loop in it) with
tuple/list/etc display, which doesn't?

lst = [1,2,3,4] # not a comprehension
even = [n*2 for n in lst] # comprehension

> Here is what I think(?) list comps do currently.
>
>  lst = [expr for items in itr if cond]
>
> Is the same as.
>
>  lst = []
>  for x in itr:  # StopIteration  caught here.
>  if cond:   # StopIteration  bubbles here.
>  lst.append(expr)   # StopIteration  bubbles here.

Pretty much. It's done in a nested function (so x doesn't leak), but
otherwise, yes.

> And it would be changed to this...
>
> def comp_expression():
> for x in itr:  # StopIteration caught here.
> if cond:
>list.append(expr)
>
> # StopIteration from cond and expr caught here.
> lst = list(x for x in comp_expression())

That wouldn't quite work, but this would:

def ():
ret = []
try:
for x in itr:
if cond:
ret.append(x)
except StopIteration:
pass
return ret
lst = ()

(And yes, the name's syntactically illegal, but if you look at a
traceback, that's what is used.)

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators

[Ron Adam]

On 11/22/2014 06:20 PM, Chris Angelico wrote:

On Sun, Nov 23, 2014 at 11:05 AM, Ron Adam  wrote:

>Se we have these...
>
>  Tuple Comprehension  (...)
>  List Comprehension  [...]
>  Dict Comprehension  {...}  Colon make's it different from sets.
>  Set Comprehension  {...}
>
>I don't think there is any other way to create them. And they don't actually
>expand to any python code that is visible to a programmer.  They are self
>contained literal expressions for creating these few objects.

Hmmm, there's no such thing as tuple comprehensions.


Just didn't think it through quite well enough.  But you are correct, that 
would be a generator expression.


One less case to worry about. :-)



lst = [1,2,3,4] # not a comprehension
even = [n*2 for n in lst] # comprehension


>Here is what I think(?) list comps do currently.
>
>  lst = [expr for items in itr if cond]
>
>Is the same as.
>
>  lst = []
>  for x in itr:  # StopIteration  caught here.
>  if cond:   # StopIteration  bubbles here.
>  lst.append(expr)   # StopIteration  bubbles here.



Pretty much. It's done in a nested function (so x doesn't leak), but
otherwise, yes.



>And it would be changed to this...
>
> def comp_expression():
> for x in itr:  # StopIteration caught here.
> if cond:
>list.append(expr)
>
> # StopIteration from cond and expr caught here.
> lst = list(x for x in comp_expression())



That wouldn't quite work, but this would:


Right, the list.append() should be a yield(expr).



def ():
 ret = []
 try:
 for x in itr:
 if cond:
 ret.append(x)
 except StopIteration:
 pass
 return ret
lst = ()

(And yes, the name's syntactically illegal, but if you look at a
traceback, that's what is used.)


That's fine too.

The real question is how much breakage would such a change make?  That will 
probably need a patch in order to test it out.


Cheers,
   Ron








___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators

[Chris Angelico]

On Sun, Nov 23, 2014 at 11:51 AM, Ron Adam  wrote:
>
>
> On 11/22/2014 06:20 PM, Chris Angelico wrote:
>>
>> Hmmm, there's no such thing as tuple comprehensions.
>
> Just didn't think it through quite well enough.  But you are correct, that
> would be a generator expression.
>
> One less case to worry about. :-)

Ah right, no probs.

>>> >And it would be changed to this...
>>> >
>>> > def comp_expression():
>>> > for x in itr:  # StopIteration caught here.
>>> > if cond:
>>> >list.append(expr)
>>> >
>>> > # StopIteration from cond and expr caught here.
>>> > lst = list(x for x in comp_expression())
>
> Right, the list.append() should be a yield(expr).

In that case, your second generator expression is entirely redundant;
all you want is list(comp_expression()). But the example doesn't say
*why* this version should terminate on a StopIteration raised by expr,
when the statement form would print an exception traceback.

> The real question is how much breakage would such a change make?  That will
> probably need a patch in order to test it out.

There's one attached here:

http://bugs.python.org/issue22906

It doesn't create a __future__ directive, just applies the change globally.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators

[Raymond Hettinger]

> On Nov 22, 2014, at 2:45 PM, Chris Angelico  wrote:
> 
> Does your middleware_generator work with just a single element,
> yielding either one output value or none?


I apologize if I didn't make the point clearly.  The middleware example was 
just simple outline of calling next(), doing some processing, and yielding a
result while letting the StopIteration float through from the next() call.

It was meant to show in summary form a pattern for legitimate uses of next() 
inside a generator.   Some of those uses benefit from letting their 
StopIteration
pass through rather than being caught, returning, and reraising the 
StopIteration.

The worry is that your proposal intentionally breaks that code which is 
currently
bug free, clean, fast, stable, and relying on a part of the API that has been
guaranteed and documented from day one.

Since the middleware() example was ineffective in communicating the need,
here are some real-world examples.

Here's one from Fredrick Lundh's ElementTree code in the standard library
(there are several other examples besides this one in his code are well):

def iterfind(elem, path, namespaces=None):
# compile selector pattern
cache_key = (path, None if namespaces is None
else tuple(sorted(namespaces.items(
if path[-1:] == "/":
path = path + "*" # implicit all (FIXME: keep this?)
try:
selector = _cache[cache_key]
except KeyError:
if len(_cache) > 100:
_cache.clear()
if path[:1] == "/":
raise SyntaxError("cannot use absolute path on element")
next = iter(xpath_tokenizer(path, namespaces)).__next__
token = next()
selector = []
while 1:
try:
selector.append(ops[token[0]](next, token))
except StopIteration:
raise SyntaxError("invalid path")
try:
token = next()
if token[0] == "/":
token = next()
except StopIteration:
break
_cache[cache_key] = selector
# execute selector pattern
result = [elem]
context = _SelectorContext(elem)
for select in selector:
result = select(context, result)
return result

And here is an example from the pure python version of one of the itertools:

def accumulate(iterable, func=operator.add):
'Return running totals'
# accumulate([1,2,3,4,5]) --> 1 3 6 10 15
# accumulate([1,2,3,4,5], operator.mul) --> 1 2 6 24 120
it = iter(iterable)
total = next(it)
yield total
for element in it:
total = func(total, element)
yield total

And here is an example from Django:

def _generator():
it = iter(text.split(' '))
word = next(it)
yield word
pos = len(word) - word.rfind('\n') - 1
for word in it:
if "\n" in word:
lines = word.split('\n')
else:
lines = (word,)
pos += len(lines[0]) + 1
if pos > width:
yield '\n'
pos = len(lines[-1])
else:
yield ' '
if len(lines) > 1:
pos = len(lines[-1])
yield word
return ''.join(_generator())

I could scan for even more examples, but I think you get the gist.
All I'm asking is that you consider that your proposal will do more
harm than good.  It doesn't add any new capability at all.
It just kills some code that currently works.


Raymond
(the author of the generator expressions pep)



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators

[Chris Angelico]

On Sun, Nov 23, 2014 at 12:11 PM, Raymond Hettinger
 wrote:
> The worry is that your proposal intentionally breaks that code which is
> currently
> bug free, clean, fast, stable, and relying on a part of the API that has
> been
> guaranteed and documented from day one.

(I'd just like to mention that this isn't "my proposal", beyond that
I'm doing the summarizing and PEP writing. The proposal itself is
primarily derived from one of Guido's posts on -ideas.)

> Here's one from Fredrick Lundh's ElementTree code in the standard library
> (there are several other examples besides this one in his code are well):
>
> def iterfind(elem, path, namespaces=None):
> # compile selector pattern
> cache_key = (path, None if namespaces is None
> else tuple(sorted(namespaces.items(
> if path[-1:] == "/":
> path = path + "*" # implicit all (FIXME: keep this?)
> try:
> selector = _cache[cache_key]
> except KeyError:
> if len(_cache) > 100:
> _cache.clear()
> if path[:1] == "/":
> raise SyntaxError("cannot use absolute path on element")
> next = iter(xpath_tokenizer(path, namespaces)).__next__
> token = next()
> selector = []
> while 1:
> try:
> selector.append(ops[token[0]](next, token))
> except StopIteration:
> raise SyntaxError("invalid path")
> try:
> token = next()
> if token[0] == "/":
> token = next()
> except StopIteration:
> break
> _cache[cache_key] = selector
> # execute selector pattern
> result = [elem]
> context = _SelectorContext(elem)
> for select in selector:
> result = select(context, result)
> return result

Most of the next() calls are already guarded with try/except; from
what I can see, only the first one isn't. So this wouldn't be much of
a change here.

> And here is an example from the pure python version of one of the itertools:
>
> def accumulate(iterable, func=operator.add):
> 'Return running totals'
> # accumulate([1,2,3,4,5]) --> 1 3 6 10 15
> # accumulate([1,2,3,4,5], operator.mul) --> 1 2 6 24 120
> it = iter(iterable)
> total = next(it)
> yield total
> for element in it:
> total = func(total, element)
> yield total

Again, only the first one needs protection, and all that happens is
that there's clearly a control flow possibility here (that the first
"yield total" might not be reached). Currently, *any* function call
has the potential to be a silent control flow event.

> And here is an example from Django:
>
> def _generator():
> it = iter(text.split(' '))
> word = next(it)
> yield word
> pos = len(word) - word.rfind('\n') - 1
> for word in it:
> if "\n" in word:
> lines = word.split('\n')
> else:
> lines = (word,)
> pos += len(lines[0]) + 1
> if pos > width:
> yield '\n'
> pos = len(lines[-1])
> else:
> yield ' '
> if len(lines) > 1:
> pos = len(lines[-1])
> yield word
> return ''.join(_generator())

When you split a string, you're guaranteed at least one result, ergo
'it' is guaranteed to yield at least one word. So this one wouldn't
need to be changed - it can't possibly raise RuntimeError.

> I could scan for even more examples, but I think you get the gist.
> All I'm asking is that you consider that your proposal will do more
> harm than good.  It doesn't add any new capability at all.
> It just kills some code that currently works.

I have considered it, and I'm not convinced that it will. I see lots
of people saying "code will have to be changed", but that's exactly
the same concern that people raise about switching from the sloppy Py2
merging of text and bytes to the strict Py3 separation - yes, code has
to be changed, but it's definitely much better to have immediate and
obvious failures when something's wrong than to have subtle behavioral
changes.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators

[Ethan Furman]

On 11/22/2014 05:11 PM, Raymond Hettinger wrote:
>> On Nov 22, 2014, at 2:45 PM, Chris Angelico wrote:
>>
>> Does your middleware_generator work with just a single element,
>> yielding either one output value or none?
> 
> I apologize if I didn't make the point clearly.  The middleware example was 
> just simple outline of calling next(), doing some processing, and yielding a
> result while letting the StopIteration float through from the next() call.

[middleware example]

def middleware_generator(source_generator):
it = source_generator()
input_value = next(it)
output_value = do_something_interesting(input_value)
yield output_value

The point that Chris made that you should be refuting is this one:

>> What happens if do_something_interesting happens to raise
>> StopIteration? Will you be surprised that this appears identical to
>> the source generator yielding nothing?

--
~Ethan~



signature.asc
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators

[Ron Adam]

On 11/22/2014 07:06 PM, Chris Angelico wrote:

On Sun, Nov 23, 2014 at 11:51 AM, Ron Adam  wrote:

>
>
>On 11/22/2014 06:20 PM, Chris Angelico wrote:

>>
>>Hmmm, there's no such thing as tuple comprehensions.

>
>Just didn't think it through quite well enough.  But you are correct, that
>would be a generator expression.
>
>One less case to worry about.:-)

Ah right, no probs.



>>> >And it would be changed to this...
>>> >
>>> > def comp_expression():
>>> > for x in itr:  # StopIteration caught here.
>>> > if cond:
>>> >list.append(expr)
>>> >
>>> > # StopIteration from cond and expr caught here.
>>> > lst = list(x for x in comp_expression())

>
>Right, the list.append() should be a yield(expr).



In that case, your second generator expression is entirely redundant;
all you want is list(comp_expression()).

Yes, and that is good.  Simplies it even more.


But the example doesn't say
*why*  this version should terminate on a StopIteration raised by expr,
when the statement form would print an exception traceback.


I presume you are asking why do this?  And not why the example does that?

There has been a desires expressed, more than a few times, to make 
comprehensions more like generator expressions in the past.  It looks to me 
that that desire is still true.  I also think there has been quite a bit of 
confusion in these discussions that could be reduced substantially by 
making Comprehensions work a bit more like generator expressions.


As to why the example does that.. a list constructor iterates over a 
generator expression in a way that follows the iterator protocol.  If you 
make comprehsionsons work as if they are a generator expression fed to a 
constructor... then it too should follow the itorator protocol.  Do you agree?




>The real question is how much breakage would such a change make?  That will
>probably need a patch in order to test it out.

There's one attached here:

http://bugs.python.org/issue22906


Doesn't that patch effect generators and not comprehensions?  If so, it 
wouldn't do what we are talking about here.


Ron


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] 2.7.9 delayed

[Benjamin Peterson]

As you may have noticed, 2.7.9rc1 wasn't tagged today. I've been much
busier than I expected to be, so I haven't had the time to fix up some
2.7.9 lose ends. Hopefully, the delay won't be more than a few days.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] PEP 479: Change StopIteration handling inside generators

[Guido van Rossum]

In order to save everyone's breath, I am *accepting* the proposal of PEP
479. The transition plan is:

- "from __future__ import generator_stop" in 3.5, and a silent deprecation
if StopIteration is allowed to bubble out of a generator (i.e. no warning
is printed unless you explicitly turn it on)

- non-silent deprecation in 3.6

- feature enabled by default in 3.7

The PEP hasn't been updated to include this and it also could use some more
editing -- I'll try to get to that Monday. But the specification of the
proposal is crystal-clear and I have no doubt that this is the right thing
going forward.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Guido van Rossum]

This thread seems to beg for a decision. I think Donald Stufft has it
exactly right: we should move to GitHub, because it is the easiest to use
and most contributors already know it (or are eager to learn thee).
Honestly, the time for core devs (or some other elite corps of dedicated
volunteers) to sysadmin their own machines (virtual or not) is over. We've
never been particularly good at this, and I don't see us getting better or
more efficient.

Moving the CPython code and docs is not a priority, but everything else
(PEPs, HOWTOs etc.) can be moved easily and I am in favor of moving to
GitHub. For PEPs I've noticed that for most PEPs these days (unless the
primary author is a core dev) the author sets up a git repo first anyway,
and the friction of moving between such repos and the "official" repo is a
pain.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 23, 2014, at 12:19 AM, Guido van Rossum  wrote:
> 
> This thread seems to beg for a decision. I think Donald Stufft has it exactly 
> right: we should move to GitHub, because it is the easiest to use and most 
> contributors already know it (or are eager to learn thee). Honestly, the time 
> for core devs (or some other elite corps of dedicated volunteers) to sysadmin 
> their own machines (virtual or not) is over. We've never been particularly 
> good at this, and I don't see us getting better or more efficient.
> 
> Moving the CPython code and docs is not a priority, but everything else 
> (PEPs, HOWTOs etc.) can be moved easily and I am in favor of moving to 
> GitHub. For PEPs I've noticed that for most PEPs these days (unless the 
> primary author is a core dev) the author sets up a git repo first anyway, and 
> the friction of moving between such repos and the "official" repo is a pain.
> 

Heck I am a Core dev technically and I setup a git repo first until it’s moved 
into the hg.python.org .

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Wes Turner]

re: docs hg repos and static HTTP hosting

* I can't remember what the GitHub Pages CDN cache time is
* Does BitBucket support more than one pages repo?
* Does  support Sphinx .. index and :ref: syntax extensions?

* https://github.com/yoloseem/awesome-sphinxdoc
* https://github.com/rtfd/readthedocs.org ("hostthedocs")
* http://conf.writethedocs.org/
* https://docs.python.org/devguide/documenting.html

On Sat, Nov 22, 2014 at 11:19 PM, Guido van Rossum  wrote:

> This thread seems to beg for a decision. I think Donald Stufft has it
> exactly right: we should move to GitHub, because it is the easiest to use
> and most contributors already know it (or are eager to learn thee).
> Honestly, the time for core devs (or some other elite corps of dedicated
> volunteers) to sysadmin their own machines (virtual or not) is over. We've
> never been particularly good at this, and I don't see us getting better or
> more efficient.
>
> Moving the CPython code and docs is not a priority, but everything else
> (PEPs, HOWTOs etc.) can be moved easily and I am in favor of moving to
> GitHub. For PEPs I've noticed that for most PEPs these days (unless the
> primary author is a core dev) the author sets up a git repo first anyway,
> and the friction of moving between such repos and the "official" repo is a
> pain.
>
> --
> --Guido van Rossum (python.org/~guido)
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com
>
>


-- 
Wes Turner
https://westurner.github.io/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Nick Coghlan]

On 23 November 2014 at 15:19, Guido van Rossum  wrote:
> This thread seems to beg for a decision. I think Donald Stufft has it
> exactly right: we should move to GitHub, because it is the easiest to use
> and most contributors already know it (or are eager to learn thee).
> Honestly, the time for core devs (or some other elite corps of dedicated
> volunteers) to sysadmin their own machines (virtual or not) is over. We've
> never been particularly good at this, and I don't see us getting better or
> more efficient.

The learning curve on git is still awful - it offers no compelling
advantages over hg, and GitHub doesn't offer any huge benefits over
BitBucket for Sphinx based documentation (ReadTheDocs works just as
well with either service).

> Moving the CPython code and docs is not a priority, but everything else
> (PEPs, HOWTOs etc.) can be moved easily and I am in favor of moving to
> GitHub. For PEPs I've noticed that for most PEPs these days (unless the
> primary author is a core dev) the author sets up a git repo first anyway,
> and the friction of moving between such repos and the "official" repo is a
> pain.

Note that if folks prefer Git, BitBucket supports both. I would object
strongly to unilaterally forcing existing contributors to switch from
Mercurial to git.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 23, 2014, at 12:59 AM, Nick Coghlan  wrote:
> 
> On 23 November 2014 at 15:19, Guido van Rossum  wrote:
>> This thread seems to beg for a decision. I think Donald Stufft has it
>> exactly right: we should move to GitHub, because it is the easiest to use
>> and most contributors already know it (or are eager to learn thee).
>> Honestly, the time for core devs (or some other elite corps of dedicated
>> volunteers) to sysadmin their own machines (virtual or not) is over. We've
>> never been particularly good at this, and I don't see us getting better or
>> more efficient.
> 
> The learning curve on git is still awful - it offers no compelling
> advantages over hg, and GitHub doesn't offer any huge benefits over
> BitBucket for Sphinx based documentation (ReadTheDocs works just as
> well with either service).

It does have one very big compelling advantage. It’s way more popular.

Besides, the learning curve on hg isn’t any better, it’s just differently
hard.

> 
>> Moving the CPython code and docs is not a priority, but everything else
>> (PEPs, HOWTOs etc.) can be moved easily and I am in favor of moving to
>> GitHub. For PEPs I've noticed that for most PEPs these days (unless the
>> primary author is a core dev) the author sets up a git repo first anyway,
>> and the friction of moving between such repos and the "official" repo is a
>> pain.
> 
> Note that if folks prefer Git, BitBucket supports both. I would object
> strongly to unilaterally forcing existing contributors to switch from
> Mercurial to git.

Going to all the trouble to move to an external repository and choosing the
least popular option out of the two main ones seems like a bad idea in
general.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 23, 2014, at 1:03 AM, Donald Stufft  wrote:
> 
>> 
>> On Nov 23, 2014, at 12:59 AM, Nick Coghlan  wrote:
>> 
>> On 23 November 2014 at 15:19, Guido van Rossum  wrote:
>>> This thread seems to beg for a decision. I think Donald Stufft has it
>>> exactly right: we should move to GitHub, because it is the easiest to use
>>> and most contributors already know it (or are eager to learn thee).
>>> Honestly, the time for core devs (or some other elite corps of dedicated
>>> volunteers) to sysadmin their own machines (virtual or not) is over. We've
>>> never been particularly good at this, and I don't see us getting better or
>>> more efficient.
>> 
>> The learning curve on git is still awful - it offers no compelling
>> advantages over hg, and GitHub doesn't offer any huge benefits over
>> BitBucket for Sphinx based documentation (ReadTheDocs works just as
>> well with either service).
> 
> It does have one very big compelling advantage. It’s way more popular.
> 
> Besides, the learning curve on hg isn’t any better, it’s just differently
> hard.
> 
>> 
>>> Moving the CPython code and docs is not a priority, but everything else
>>> (PEPs, HOWTOs etc.) can be moved easily and I am in favor of moving to
>>> GitHub. For PEPs I've noticed that for most PEPs these days (unless the
>>> primary author is a core dev) the author sets up a git repo first anyway,
>>> and the friction of moving between such repos and the "official" repo is a
>>> pain.
>> 
>> Note that if folks prefer Git, BitBucket supports both. I would object
>> strongly to unilaterally forcing existing contributors to switch from
>> Mercurial to git.
> 
> Going to all the trouble to move to an external repository and choosing the
> least popular option out of the two main ones seems like a bad idea in
> general.
> 

Also important to note, that if people want to use hg on their own, it’s pretty
easy to do that with https://hg-git.github.io/. Last I heard that works pretty
well still. I’ve yet to find one that works the other direction that doesn’t
choke on some repositories.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Chris Angelico]

On Sun, Nov 23, 2014 at 4:59 PM, Nick Coghlan  wrote:
> The learning curve on git is still awful - it offers no compelling
> advantages over hg, and GitHub doesn't offer any huge benefits over
> BitBucket for Sphinx based documentation (ReadTheDocs works just as
> well with either service).

The learning curve for source control as a concept is pretty steep.
Once someone's learned one DVCS, learning another is much easier, and
I don't know that it makes a lot of difference whether you learn git
and then hg, or hg and then git. I learned git first, and mastered hg
basics by keeping a Rosetta Stone chart handy; given that the document
I was reading was intended for the reverse case, I expect it's not too
hard to get the basics of git once you know hg.

Just as git offers few advantages over hg, hg offers few advantages
over git. If git and GitHub are where most people are, I would support
using them for Python. I'm one of those PEP draft authors who starts
with his own repo on GitHub and sends drafts in, and Guido had to
remind me that I can simply test my edits in the peps repo before
posting them (to make sure I haven't mucked up the markup); if the
peps repo were itself hosted on GitHub, or at least in a git repo, I
could have a workflow that directly incorporates that, instead of
being "off to the side" with periodic manual imports.

That said, it does make sense for CPython *itself* to use Mercurial,
simply because it's written in Python. I don't know how strong that
philosophical argument is with people, but I wouldn't argue against
it.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Guido van Rossum]

On Saturday, November 22, 2014, Nick Coghlan  wrote:

> On 23 November 2014 at 15:19, Guido van Rossum  > wrote:
> > This thread seems to beg for a decision. I think Donald Stufft has it
> > exactly right: we should move to GitHub, because it is the easiest to use
> > and most contributors already know it (or are eager to learn thee).
> > Honestly, the time for core devs (or some other elite corps of dedicated
> > volunteers) to sysadmin their own machines (virtual or not) is over.
> We've
> > never been particularly good at this, and I don't see us getting better
> or
> > more efficient.
>
> The learning curve on git is still awful - it offers no compelling
> advantages over hg, and GitHub doesn't offer any huge benefits over
> BitBucket for Sphinx based documentation (ReadTheDocs works just as
> well with either service).


Git may well have a learning curve, but ever since I "got" it I started
preferring it over hg.

Too bad for BitBucket, but most people who started contributing to open
source in the past 5 years already have a GitHub account.


>
> > Moving the CPython code and docs is not a priority, but everything else
> > (PEPs, HOWTOs etc.) can be moved easily and I am in favor of moving to
> > GitHub. For PEPs I've noticed that for most PEPs these days (unless the
> > primary author is a core dev) the author sets up a git repo first anyway,
> > and the friction of moving between such repos and the "official" repo is
> a
> > pain.
>
> Note that if folks prefer Git, BitBucket supports both. I would object
> strongly to unilaterally forcing existing contributors to switch from
> Mercurial to git.
>

What about potential new contributors? And the hg-git bridges that git fans
are always referred to work in the opposite direction too... :-)




-- 
--Guido van Rossum (on iPad)
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Wes Turner]

> The learning curve on git is still awful - it offers no compelling
advantages over hg, and GitHub doesn't offer any huge benefits over
BitBucket for Sphinx based documentation (ReadTheDocs works just as
well with either service).

This was a most helpful resource:

https://github.com/sympy/sympy/wiki/Git-hg-rosetta-stone#rosetta-stone

Wikipedia: https://en.wikipedia.org/wiki/Git_(software)
> Homepage: http://git-scm.com/
> Docs: http://git-scm.com/documentation
> Docs: http://git-scm.com/book/en/
> Docs: http://documentup.com/skwp/git-workflows-book
> Docs: http://learnxinyminutes.com/docs/git/
> Source: git https://github.com/git/git


hg imuutability is certainly a primarily attractive feature;
along with the keyring support.


On Sat, Nov 22, 2014 at 11:59 PM, Nick Coghlan  wrote:

> On 23 November 2014 at 15:19, Guido van Rossum  wrote:
> > This thread seems to beg for a decision. I think Donald Stufft has it
> > exactly right: we should move to GitHub, because it is the easiest to use
> > and most contributors already know it (or are eager to learn thee).
> > Honestly, the time for core devs (or some other elite corps of dedicated
> > volunteers) to sysadmin their own machines (virtual or not) is over.
> We've
> > never been particularly good at this, and I don't see us getting better
> or
> > more efficient.
>
> The learning curve on git is still awful - it offers no compelling
> advantages over hg, and GitHub doesn't offer any huge benefits over
> BitBucket for Sphinx based documentation (ReadTheDocs works just as
> well with either service).
>
> > Moving the CPython code and docs is not a priority, but everything else
> > (PEPs, HOWTOs etc.) can be moved easily and I am in favor of moving to
> > GitHub. For PEPs I've noticed that for most PEPs these days (unless the
> > primary author is a core dev) the author sets up a git repo first anyway,
> > and the friction of moving between such repos and the "official" repo is
> a
> > pain.
>
> Note that if folks prefer Git, BitBucket supports both. I would object
> strongly to unilaterally forcing existing contributors to switch from
> Mercurial to git.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com
>



-- 
Wes Turner
https://westurner.github.io/
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Nick Coghlan]

On 23 November 2014 at 16:03, Donald Stufft  wrote:
>> On Nov 23, 2014, at 12:59 AM, Nick Coghlan  wrote:
>>
>> Note that if folks prefer Git, BitBucket supports both. I would object
>> strongly to unilaterally forcing existing contributors to switch from
>> Mercurial to git.
>
> Going to all the trouble to move to an external repository and choosing the
> least popular option out of the two main ones seems like a bad idea in
> general.

Moving repos from hg.python.org to bitbucket.org is just a matter of
switching some URLs around, and changing some backend systems to cope
with the new location. The end result should be to make life better
for existing contributors *and* new contributors using the web UI, and
be largely transparent to folks using command line tools.

By contrast, proposals to switch from Mercurial to Git impose a
*massive* burden on contributors that don't already know git. That
significant increase in the time investment required will provide *NO*
practical benefit for existing contributors (this is coming from
someone that has used git and Mercurial in parallel for years - trust
me, they're functionally isomorphic), and only make life marginally
easier for potential new contributors (you can log in to BitBucket
with your GitHub ID, and the functional isomorphism means that many
folks already use tools like git-remote-hg  to use the git command
line to interact with the hg.python.org Mercurial repos).

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 23, 2014, at 1:25 AM, Nick Coghlan  wrote:
> 
> On 23 November 2014 at 16:03, Donald Stufft  wrote:
>>> On Nov 23, 2014, at 12:59 AM, Nick Coghlan  wrote:
>>> 
>>> Note that if folks prefer Git, BitBucket supports both. I would object
>>> strongly to unilaterally forcing existing contributors to switch from
>>> Mercurial to git.
>> 
>> Going to all the trouble to move to an external repository and choosing the
>> least popular option out of the two main ones seems like a bad idea in
>> general.
> 
> Moving repos from hg.python.org to bitbucket.org is just a matter of
> switching some URLs around, and changing some backend systems to cope
> with the new location. The end result should be to make life better
> for existing contributors *and* new contributors using the web UI, and
> be largely transparent to folks using command line tools.
> 
> By contrast, proposals to switch from Mercurial to Git impose a
> *massive* burden on contributors that don't already know git. That
> significant increase in the time investment required will provide *NO*
> practical benefit for existing contributors (this is coming from
> someone that has used git and Mercurial in parallel for years - trust
> me, they're functionally isomorphic), and only make life marginally
> easier for potential new contributors (you can log in to BitBucket
> with your GitHub ID, and the functional isomorphism means that many
> folks already use tools like git-remote-hg  to use the git command
> line to interact with the hg.python.org Mercurial repos).

Yea, but then you lose out on the entire ecosystem built around Github.

Like you won’t be able to run travis tests on the docs to make sure that
any Pull Requests don’t silently start breaking the ability to build the
docs.

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Chris Angelico]

On Sun, Nov 23, 2014 at 5:03 PM, Wes Turner  wrote:
> hg imuutability is certainly a primarily attractive feature;
> along with the keyring support.

What exactly do you mean by immutability? Are you talking about how
git allows a "force push" that can destroy data? That can be rejected
in a repo's hook scripts; also, I'm fairly sure I remember reading
somewhere about how to do that with hg, too. It's all bits of data
inside computers; nothing's immutable. Both DVCSes pledge to make it
obvious any time something is altered, not to make it impossible to
alter.

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Nick Coghlan]

On 23 November 2014 at 16:10, Guido van Rossum  wrote:
> On Saturday, November 22, 2014, Nick Coghlan  wrote:
>> The learning curve on git is still awful - it offers no compelling
>> advantages over hg, and GitHub doesn't offer any huge benefits over
>> BitBucket for Sphinx based documentation (ReadTheDocs works just as
>> well with either service).
>
>
> Git may well have a learning curve, but ever since I "got" it I started
> preferring it over hg.

I took the git knowledge I acquired by necessity at Red Hat and
figured out how to apply it to hg. All the same features are there in
hg, they're just switched off by default (mainly because the core
Mercurial devs are adamant that any potentially history destroying
operation in a version control system must be opt-in). In particular,
the evolve extension is an impressive tool that allows history to be
edited in such a way that the edits can be safely shared amongst
repos.

> Too bad for BitBucket, but most people who started contributing to open
> source in the past 5 years already have a GitHub account.

You can log into BitBucket with a GitHub account.

More generally, I'm very, very disappointed to see folks so willing to
abandon fellow community members for the sake of following the crowd.
Perhaps we should all just abandon Python and learn Ruby or JavaScript
because they're better at getting press in Silicon Valley?

>> Note that if folks prefer Git, BitBucket supports both. I would object
>> strongly to unilaterally forcing existing contributors to switch from
>> Mercurial to git.
>
> What about potential new contributors? And the hg-git bridges that git fans
> are always referred to work in the opposite direction too... :-)

We already have lots of potential contributors (if we didn't, review
bandwidth wouldn't be the bottleneck the way it is today), and the
functional differences between GitHub and BitBucket from a barrier to
entry perspective are so low as to be trivial.

For organisations with no vested interest in BitBucket or Mercurial,
sure, they may as well just go with the popular choice.

Unlike most organisations, we have a vested interest in encouraging
the growth of the Python ecosystem and community, and the developers
of both Mercurial and BitBucket are members of that community.

While we have historically done very little to date in reaching out to
the Mercurial developers and seek their assistance in improving the
developer guide, remedying that and providing better Mercurial usage
guidelines is one of my hoped for outcomes in making it easier for
others to contribute to the developer guide.

By contrast, GitHub is a Ruby application, and git is a collection of
C and shell scripts - they may be tools used *by* the Python
community, but they are decidedly not products *of* the Python
community.

Given the choice between two functionally equivalent solutions, one of
which lets us continue using the tools we already use (including all
the associated automation), and is created by members of the same
development community, I think the burden of proof falls on the folks
that want to make the backwards incompatible change that the
additional cost will be worth it.

Moving from self-hosted Mercurial repos to externally hosted Mercurial
repos is a low risk change. It reduces maintenance overhead and lowers
barriers to external contribution, both without alienating existing
contributors by forcing them to change their workflows.

Proposing to *also* switch from Mercurial to git significantly
increases the cost of the change, while providing minimal incremental
benefit.

Regards,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Nick Coghlan]

On 23 November 2014 at 16:27, Donald Stufft  wrote:
>> On Nov 23, 2014, at 1:25 AM, Nick Coghlan  wrote:
>> By contrast, proposals to switch from Mercurial to Git impose a
>> *massive* burden on contributors that don't already know git. That
>> significant increase in the time investment required will provide *NO*
>> practical benefit for existing contributors (this is coming from
>> someone that has used git and Mercurial in parallel for years - trust
>> me, they're functionally isomorphic), and only make life marginally
>> easier for potential new contributors (you can log in to BitBucket
>> with your GitHub ID, and the functional isomorphism means that many
>> folks already use tools like git-remote-hg  to use the git command
>> line to interact with the hg.python.org Mercurial repos).
>
> Yea, but then you lose out on the entire ecosystem built around Github.
>
> Like you won’t be able to run travis tests on the docs to make sure that
> any Pull Requests don’t silently start breaking the ability to build the
> docs.

Travis isn't the only CI system on the internet, and for pure Sphinx
documentation cases, ReadTheDocs runs just as well off BitBucket as it
does off GitHub.

I personally think changing version control systems would be an
incredibly bad idea, and consider it completely out of scope for
discussions of Mercurial repo hosting arrangements. There's a lot that
could be done, with much lower impact, just by changing the way we
manage the existing Mercurial repos. When we can't even work out the
practical details of getting those implemented, suggestions of
"git/GitHub will fix it!" sound like mere wishful thinking.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 23, 2014, at 1:49 AM, Nick Coghlan  wrote:
> 
> On 23 November 2014 at 16:10, Guido van Rossum  wrote:
>> On Saturday, November 22, 2014, Nick Coghlan  wrote:
>>> The learning curve on git is still awful - it offers no compelling
>>> advantages over hg, and GitHub doesn't offer any huge benefits over
>>> BitBucket for Sphinx based documentation (ReadTheDocs works just as
>>> well with either service).
>> 
>> 
>> Git may well have a learning curve, but ever since I "got" it I started
>> preferring it over hg.
> 
> I took the git knowledge I acquired by necessity at Red Hat and
> figured out how to apply it to hg. All the same features are there in
> hg, they're just switched off by default (mainly because the core
> Mercurial devs are adamant that any potentially history destroying
> operation in a version control system must be opt-in). In particular,
> the evolve extension is an impressive tool that allows history to be
> edited in such a way that the edits can be safely shared amongst
> repos.

Often times it’s there multiple times in different incompatible ways so
figuring out which extension to use can be hard. They also have varying
levels of compatibility with other tools. Setuptools uses “bookmarks”
instead of branches sometimes and that confuses bitbucket in some way
that makes bitbucket throw errors at me when I attempt to use it.

I’ve not had much luck personally with the extensions to Hg, they seem
to be second class citizens in terms of workflow and I do think that
them being off by default makes them worse too.

> 
>> Too bad for BitBucket, but most people who started contributing to open
>> source in the past 5 years already have a GitHub account.
> 
> You can log into BitBucket with a GitHub account.
> 
> More generally, I'm very, very disappointed to see folks so willing to
> abandon fellow community members for the sake of following the crowd.
> Perhaps we should all just abandon Python and learn Ruby or JavaScript
> because they're better at getting press in Silicon Valley?

I don’t think this is really fair. It’s not about press, it’s about the
best tool for the job and being pragmatic.

> 
>>> Note that if folks prefer Git, BitBucket supports both. I would object
>>> strongly to unilaterally forcing existing contributors to switch from
>>> Mercurial to git.
>> 
>> What about potential new contributors? And the hg-git bridges that git fans
>> are always referred to work in the opposite direction too... :-)
> 
> We already have lots of potential contributors (if we didn't, review
> bandwidth wouldn't be the bottleneck the way it is today), and the
> functional differences between GitHub and BitBucket from a barrier to
> entry perspective are so low as to be trivial.

That’s not really true. It’s more than just “can I log in”, potential
contributors are more likely to already know how to use Github too and
are more likely to not want to deal with another site. I know personally
if I see a project is on bitbucket my chances of contributing to that
project drop drastically, even if they are using git on bitbucket,
just because I know that I’m going to get frustrated to some degree.

> 
> For organisations with no vested interest in BitBucket or Mercurial,
> sure, they may as well just go with the popular choice.
> 
> Unlike most organisations, we have a vested interest in encouraging
> the growth of the Python ecosystem and community, and the developers
> of both Mercurial and BitBucket are members of that community.
> 
> While we have historically done very little to date in reaching out to
> the Mercurial developers and seek their assistance in improving the
> developer guide, remedying that and providing better Mercurial usage
> guidelines is one of my hoped for outcomes in making it easier for
> others to contribute to the developer guide.
> 
> By contrast, GitHub is a Ruby application, and git is a collection of
> C and shell scripts - they may be tools used *by* the Python
> community, but they are decidedly not products *of* the Python
> community.

Well CPython is a C application so perhaps we should switch to an
interpreter written in Python if the goal is to always pick the
thing that’s written in Python?

> 
> Given the choice between two functionally equivalent solutions, one of
> which lets us continue using the tools we already use (including all
> the associated automation), and is created by members of the same
> development community, I think the burden of proof falls on the folks
> that want to make the backwards incompatible change that the
> additional cost will be worth it.
> 
> Moving from self-hosted Mercurial repos to externally hosted Mercurial
> repos is a low risk change. It reduces maintenance overhead and lowers
> barriers to external contribution, both without alienating existing
> contributors by forcing them to change their workflows.
> 
> Proposing to *also* switch from Mercurial to git significantly
> increases the cost of the change, while provid

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Donald Stufft]

> On Nov 23, 2014, at 2:01 AM, Nick Coghlan  wrote:
> 
> On 23 November 2014 at 16:27, Donald Stufft  wrote:
>>> On Nov 23, 2014, at 1:25 AM, Nick Coghlan  wrote:
>>> By contrast, proposals to switch from Mercurial to Git impose a
>>> *massive* burden on contributors that don't already know git. That
>>> significant increase in the time investment required will provide *NO*
>>> practical benefit for existing contributors (this is coming from
>>> someone that has used git and Mercurial in parallel for years - trust
>>> me, they're functionally isomorphic), and only make life marginally
>>> easier for potential new contributors (you can log in to BitBucket
>>> with your GitHub ID, and the functional isomorphism means that many
>>> folks already use tools like git-remote-hg  to use the git command
>>> line to interact with the hg.python.org Mercurial repos).
>> 
>> Yea, but then you lose out on the entire ecosystem built around Github.
>> 
>> Like you won’t be able to run travis tests on the docs to make sure that
>> any Pull Requests don’t silently start breaking the ability to build the
>> docs.
> 
> Travis isn't the only CI system on the internet, and for pure Sphinx
> documentation cases, ReadTheDocs runs just as well off BitBucket as it
> does off GitHub.

Sure it’s not the only CI system, but as far as I know bitbucket doesn’t
have near the integration possible with CI systems. I make a PR on github
I get it tested and the merge button turns green to let me know that
the tests run. Travis is popular enough that I’ve seen bitbucket projects
hosting a mirror on github *just* for travis.

> 
> I personally think changing version control systems would be an
> incredibly bad idea, and consider it completely out of scope for
> discussions of Mercurial repo hosting arrangements. There's a lot that
> could be done, with much lower impact, just by changing the way we
> manage the existing Mercurial repos. When we can't even work out the
> practical details of getting those implemented, suggestions of
> "git/GitHub will fix it!" sound like mere wishful thinking.
> 
> Cheers,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Move selected documentation repos to PSF BitBucket account?

[Nick Coghlan]

On 23 November 2014 at 17:14, Donald Stufft  wrote:
>> On Nov 23, 2014, at 2:01 AM, Nick Coghlan  wrote:
>> Travis isn't the only CI system on the internet, and for pure Sphinx
>> documentation cases, ReadTheDocs runs just as well off BitBucket as it
>> does off GitHub.
>
> Sure it’s not the only CI system, but as far as I know bitbucket doesn’t
> have near the integration possible with CI systems. I make a PR on github
> I get it tested and the merge button turns green to let me know that
> the tests run. Travis is popular enough that I’ve seen bitbucket projects
> hosting a mirror on github *just* for travis.

In the absence of a proposal to change version control systems
(again), the lack of Mercurial hosting on GitHub makes it rather a
moot point. Given that we can barely muster up any enthusiasm for
rehosting *without* changing version control systems (and the number
of CI systems that integrate with hg.python.org repos other than the
main CPython one is exactly zero), any proposal that involves doing
even *more* work seems doubly doomed.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com