Re: [Python-Dev] nonlocal keyword in 2.x?

[Michael Foord]

Brett Cannon wrote:

On Sun, Nov 1, 2009 at 13:39, Raymond Hettinger  wrote:
  

As I said: Python 2 support is not only about supporting old versions of
Python,
but also supporting users of Python2-only modules. So 2.7 support will
for the most part be a case not of supporting Python versions, but
Python *users*. And contrary to what Antoine said, that *is* a good
reason to backport it.
  

FWIW, I support backporting the nonlocal-keyword in 2.7.
All of the reasons for introducting nonlocal to 3.x also apply to 2.x.
Using the nonlocal keyword in clear and explicit, especially
when compared to the existing workarounds which are not pretty.



We are voting, I'm -0.
  


nonlocal is really useful for tests so I'd love to have this in 2.7, but 
obviously someone has to do the work so I'm only +0.


Michael


-Brett
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
  



--
http://www.ironpythoninaction.com/

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reworking the GIL

[Martin v. Löwis]

> The new GIL does away with this by ditching _Py_Ticker entirely and
> instead using a fixed interval (by default 5 milliseconds, but settable)
> after which we ask the main thread to release the GIL and let another
> thread be scheduled.

I've looked at this part of the implementation, and have a few comments.
a) why is gil_interval a double? Make it an int, counting in
   microseconds.
b) notice that, on Windows, minimum wait resolution may be as large as
   15ms (e.g. on XP, depending on the hardware). Not sure what this
   means for WaitForMultipleObjects; most likely, if you ask for a 5ms
   wait, it waits until the next clock tick. It would be bad if, on
   some systems, a wait of 5ms would mean that it immediately returns.
c) is the gil_drop_request handling thread-safe? If your answer is
   "yes", you should add a comment as to what the assumptions are of
   this code (ISTM that multiple threads may simultaneously attempt
   to set the drop request, overlapping with the holding thread actually
   dropping the GIL). There is also the question whether it is
   thread-safe to write into a "volatile int"; I keep forgetting the
   answer.

Regards,
Martin


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] thanks to everyone cleaning up the tests

[Mark Dickinson]

On Sat, Oct 31, 2009 at 9:47 PM, Brett Cannon  wrote:
> Just wanted to publicly thank everyone who has been causing all the
> checkins to fix and stabilize the test suite (I think it's mostly
> Antoine and Mark, but I could be missing somebody; I'm under a
> deadline so only have marginal higher brain functionality).

Not guilty, your honour.  Blame Antoine and David.  :)

David's new buildslave seems to be making a difference, too.

Mark
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it

[Antoine Pitrou]

Cameron Simpson  zip.com.au> writes:
> 
> Personally, I'm for the iteration spec in a lot of ways.
> 
> Firstly, a .get()/.pick() that always returns the same element feels
> horrible. Is there anyone here who _likes_ it?

I do. Since the caller is asking for an arbitrary element, returning the same
element is legitimate. It's funny how people seem to have a problem with the
word "arbitrary" :)

And I'm -1 on any implicit iteration attaching some state to the object. If you
want to iterate, there's already an obvious way to it.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reworking the GIL

[Sturla Molden]

Martin v. Löwis skrev:

b) notice that, on Windows, minimum wait resolution may be as large as
   15ms (e.g. on XP, depending on the hardware). Not sure what this
   means for WaitForMultipleObjects; most likely, if you ask for a 5ms
   wait, it waits until the next clock tick. It would be bad if, on
   some systems, a wait of 5ms would mean that it immediately returns.
  

Which is why one should use multimedia timers with QPC on Windows.

To get a wait function with much better resolution than Windows' 
default, do this:


1. Set a high resolution with timeBeginPeriod.

2. Loop using a time-out of 0 for WaitForMultipleObjects and put a 
Sleep(0) in the loop not to burn the CPU. Call QPF to get a precise 
timing, and break the loop when the requested time-out has been reached.


3. When you are done, call timeBeginPeriod to turn the multimedia timer off.

This is how you create usleep() in Windows as well: Just loop on QPF and 
Sleep(0) after setting timeBeginPeriod(1).







___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reworking the GIL

[Martin v. Löwis]

Sturla Molden wrote:
> Martin v. Löwis skrev:
>> b) notice that, on Windows, minimum wait resolution may be as large as
>>15ms (e.g. on XP, depending on the hardware). Not sure what this
>>means for WaitForMultipleObjects; most likely, if you ask for a 5ms
>>wait, it waits until the next clock tick. It would be bad if, on
>>some systems, a wait of 5ms would mean that it immediately returns.
>>   
> Which is why one should use multimedia timers with QPC on Windows.

Maybe you should study the code under discussion before making such
a proposal.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reworking the GIL

[Antoine Pitrou]

Martin v. Löwis  v.loewis.de> writes:
> 
> I've looked at this part of the implementation, and have a few comments.
> a) why is gil_interval a double? Make it an int, counting in
>microseconds.

Ok, I'll do that.

> b) notice that, on Windows, minimum wait resolution may be as large as
>15ms (e.g. on XP, depending on the hardware). Not sure what this
>means for WaitForMultipleObjects; most likely, if you ask for a 5ms
>wait, it waits until the next clock tick. It would be bad if, on
>some systems, a wait of 5ms would mean that it immediately returns.

I'll let someone else give an authoritative answer. I did the Windows version
mainly by reading online MSDN docs, and testing it worked fine in an XP VM.

Anyway, here is what the online docs have to say :

« The system clock "ticks" at a constant rate. If the time-out interval is less
than the resolution of the system clock, the wait may time out in less than the
specified length of time. If the time-out interval is greater than one tick but
less than two, the wait can be anywhere between one and two ticks, and so on. »

So it seems that the timeout always happens on a Windows tick boundary, which
means that it can immediately return, but only very rarely so and never more
than once in a row...

> c) is the gil_drop_request handling thread-safe? If your answer is
>"yes", you should add a comment as to what the assumptions are of
>this code (ISTM that multiple threads may simultaneously attempt
>to set the drop request, overlapping with the holding thread actually
>dropping the GIL).

The gil_drop_request is volatile mainly out of precaution, but it's probably not
necessary. It is only written (set/cleared) when holding the gil_mutex; however,
it can be read while not holding that mutex. Exceptionally reading the "wrong"
value would not have any adverse consequences, it would just decrease the
switching quality at the said instant.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it

[Willi Richert]

Hi,

all your points are valid -- for the experienced Python programmer who has 
stumbled across this issue already and solved it in one of several ways. All 
your points, however, do not support the "one obvious way to do it" philosophy 
of Python. It's all about making Python even more clean and beautiful.

wr

Am Montag, 2. November 2009 06:29:00 schrieb Cameron Simpson:
> On 30Oct2009 20:43, Chris Bergstresser  wrote:
> | On Fri, Oct 30, 2009 at 8:29 PM, Steven D'Aprano  
wrote:
> | >> > Iterating over an iterable is
> | >> > what iterators are for.
> | >
> | > set.get(), or set.pick() as Wikipedia calls it, isn't for iterating
> | > over sets. It is for getting an arbitrary element from the set.
> 
> [...]
> 
> | > The purpose is to
> | > return an arbitrary item each time it is called. If two threads get the
> | > same element, that's not a problem; if one thread misses an element
> | > because another thread grabbed it first, that's not a problem either.
> | > If people prefer a random element instead, I have no problem with
> | > that -- personally I think that's overkill, but maybe that's just me.
> |
> |I also think returning a random one is overkill, in the base set.
> | And I'd have the base implementation do the simplest thing possible:
> | return a fixed element (either the first returned when iterated over,
> | or the last stored, or whatever's easiest based on the internals).
> | For me, leaving the choice of *which* element to return on each call
> | is a feature.  It allows subclasses to change the behavior to support
> | other use cases, like a random or round-robin behavior.
> 
> Personally, I'm for the iteration spec in a lot of ways.
> 
> Firstly, a .get()/.pick() that always returns the same element feels
> horrible. Is there anyone here who _likes_ it?
> 
> Secondly, I think the thread-unsafe arguments are weak. Question: is the
> existing set iteration scheme thread safe? i.e. does it return a fresh
> iterator that a thread can happily use concurrently while another thread
> uses its own iterator?  (Absent set modifications). If the answer is yes,
> then I don't buy the thread-unsafe complaints - there are already plenty
> of thread unsafe things much as lots of iterators will break in the face
> of changes to the data structures over which they're iterating. If
> iter(set) gets you a safe iterator (absent set modification and multiple
> users of that iterator) then thread safe methods exist and are easy to
> use. No presently thread-safe program is going to be broken by adding an
> iterating .get() method.
> 
> Thirdly, an iteration-like .get() is dead easy to provide: you just keep a
> _single_, cycling, internal iterator made on demand the same way __iter__
> already works. So why not do just do it? There's no need to construct it
> before anyone calls .get() the first time. Somewhat like:
> 
>   def get(self):
> for x in self:
>   return x
> raise something here
> 
> but not making a new iterator for every caller. Indeed, making a new
> iterater per caller, in addition to being expensive, might easily return
> the same element to every caller.
> 
> Do anyone feel an iteration capable .get() unduely burdens subclasses
> that want to impement different .get()s? Both the suggested potential
> subclass choices (round robin and random) suggest iteration capable
> .get()s (presuming "random" means "shuffle order" rather than potentially
> returning the same element twice).
> 
> Cheers,
> 

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] EC2 buildslaves

[Antoine Pitrou]

 twistedmatrix.com> writes:
> 
> Starting with a mainstream distro doesn't seem like a bad idea.  For 
> example, there isn't currently a 32bit Ubuntu (any version) slave.  That 
> would be a nice gap to fill in, right?

I've setup a buildslave on an EC2 Ubuntu Karmic instance here:
http://www.python.org/dev/buildbot/all/buildslaves/pitrou-ubuntu

However, since it's right now billed on my credit card, I'll shut it down in a
couple of days. I wonder how we can get the PSF to be billed instead of me, if
the PSF accepts to fund such an instance (which, given EC2 prices, is perhaps
not the best use of money?).

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reworking the GIL

[Martin v. Löwis]

>> c) is the gil_drop_request handling thread-safe? If your answer is
>>"yes", you should add a comment as to what the assumptions are of
>>this code (ISTM that multiple threads may simultaneously attempt
>>to set the drop request, overlapping with the holding thread actually
>>dropping the GIL).
> 
> The gil_drop_request is volatile mainly out of precaution, but it's probably 
> not
> necessary. It is only written (set/cleared) when holding the gil_mutex; 
> however,
> it can be read while not holding that mutex. Exceptionally reading the "wrong"
> value would not have any adverse consequences, it would just decrease the
> switching quality at the said instant.

I think it then needs definitely to be volatile - otherwise, the
compiler may chose to cache it in a register (assuming enough registers
are available), instead of re-reading it from memory each time (which is
exactly what volatile then forces it to).

Even if it is read from memory, I still wonder what might happen on
systems that require explicit memory barriers to synchronize across
CPUs. What if CPU 1 keeps reading a 0 value out of its cache, even
though CPU 1 has written an 1 value a long time ago?

IIUC, any (most?) pthread calls would cause synchronization in that
case, which is why applications that also use locks for reading:

http://www.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap04.html#tag_04_10

Of course, on x86, you won't see any issues, because it's cache-coherent
anyway.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reworking the GIL

[Antoine Pitrou]

Martin v. Löwis  v.loewis.de> writes:
> 
[gil_drop_request]
> Even if it is read from memory, I still wonder what might happen on
> systems that require explicit memory barriers to synchronize across
> CPUs. What if CPU 1 keeps reading a 0 value out of its cache, even
> though CPU 1 has written an 1 value a long time ago?
> 
> IIUC, any (most?) pthread calls would cause synchronization in that
> case, which is why applications that also use locks for reading:
> 
> http://www.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap04.html#tag_04_10
> 
> Of course, on x86, you won't see any issues, because it's cache-coherent
> anyway.

I think there are two things here:
- all machines Python runs on should AFAIK be cache-coherent: CPUs synchronize
their views of memory in a rather timely fashion.
- memory ordering: writes made by a CPU can be seen in a different order by
another CPU (i.e. CPU 1 writes A before B, but CPU 2 sees B written before A). I
don't see how this can apply to gil_drop_request, since it's a single variable
(and, moreover, only a single bit of it is significant).

(there's an explanation of memory ordering issues here:
http://www.linuxjournal.com/article/8211)

As a side note, I remember Jeffrey Yasskin trying to specify an ordering model
for Python code
(see http://code.google.com/p/unladen-swallow/wiki/MemoryModel).

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it

[Nick Coghlan]

Antoine Pitrou wrote:
> Cameron Simpson  zip.com.au> writes:
>> Personally, I'm for the iteration spec in a lot of ways.
>>
>> Firstly, a .get()/.pick() that always returns the same element feels
>> horrible. Is there anyone here who _likes_ it?
> 
> I do. Since the caller is asking for an arbitrary element, returning the same
> element is legitimate. It's funny how people seem to have a problem with the
> word "arbitrary" :)

Agreed. Arbitrary is arbitrary - for a stateless method that returns an
arbitrary result, return the same value every time is fine.

> And I'm -1 on any implicit iteration attaching some state to the object. If 
> you
> want to iterate, there's already an obvious way to it.

Also agreed.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] EC2 buildslaves

[Martin v. Löwis]

> I've setup a buildslave on an EC2 Ubuntu Karmic instance here:
> http://www.python.org/dev/buildbot/all/buildslaves/pitrou-ubuntu
> 
> However, since it's right now billed on my credit card, I'll shut it down in a
> couple of days. I wonder how we can get the PSF to be billed instead of me, if
> the PSF accepts to fund such an instance (which, given EC2 prices, is perhaps
> not the best use of money?).

Send a request to psf-bo...@python.org. Such request should include:
- contact information (who'll be running this project)
- project purpose/description
- estimated expenses (in case of doubt, round up rather than
  rounding down)
- a proposal of how payment should proceed. I'm not quite sure whether
  it could be billed on the PSF credit card (you may ask Kurt Kaiser
  in advance); traditionally, it worked best when we received invoices.
There will be a board meeting next Monday, so it might be useful to
have a proposal by then.

As for whether that's good use of the money, I'm skeptical as well.
I don't actually know what EC2 prices *are*, or what the current pricing
for a root server is (plus there had been various offers from people
donating hardware - from people who would be unable to also donate
time).

There was discussion that an EC2 instance can be turned on only when
needed, so we could try to set up something like that (the build master
could then trigger activation of the machine, IIUC). However, it might
be that the machine would have to be up most of the day, as there would
be one build going on always, anyway.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] EC2 buildslaves

[Antoine Pitrou]

Le lundi 02 novembre 2009 à 13:31 +0100, "Martin v. Löwis" a écrit :
> 
> There was discussion that an EC2 instance can be turned on only when
> needed, so we could try to set up something like that (the build master
> could then trigger activation of the machine, IIUC). However, it might
> be that the machine would have to be up most of the day, as there would
> be one build going on always, anyway.

Yes, I think that would be the case. We have frequent commits on each of
the 4 branches under test, with a test suite that takes quite a bit of
time to run in debug mode with -uall.

Moreover, a standard ("small", which also means cheapest) EC2 instance
apparently provides (based on a quick test) between 25% and 50% of the
power of a full CPU core, which makes builds longer. I thought a full
CPU core was provided, but it is not.

An always-on "small" EC2 instance is at least 500$ a year, with a small
storage cost added to that.

Therefore, I think EC2 buildslaves would be interesting under the
condition that they are donated rather than paid for. I don't know
whether anyone has contacts at Amazon.
(but then any donated piece of hardware would be good, not necessary an
EC2 instance)

I assume Jean-Paul made his original suggestion under the premise that
the EC2 instances would only be run when necessary, which is probably
very economical with Twisted's development model (few commits on trunk)
but not with ours.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] EC2 buildslaves

[sstein...@gmail.com]

On Nov 2, 2009, at 6:30 AM, Antoine Pitrou wrote:


 twistedmatrix.com> writes:


Starting with a mainstream distro doesn't seem like a bad idea.  For
example, there isn't currently a 32bit Ubuntu (any version) slave.   
That

would be a nice gap to fill in, right?


I've setup a buildslave on an EC2 Ubuntu Karmic instance here:
http://www.python.org/dev/buildbot/all/buildslaves/pitrou-ubuntu



If you could send me the script that you used to set it up, I could  
give it a shot on RackSpace where it's cheaper (and I have a temporary  
developer account).


S

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] EC2 buildslaves

[Antoine Pitrou]

Le lundi 02 novembre 2009 à 07:42 -0500, sstein...@gmail.com a écrit :
> 
> If you could send me the script that you used to set it up, I could  
> give it a shot on RackSpace where it's cheaper (and I have a temporary  
> developer account).

There's no need for a special script, really.
Install Python, buildbot, subversion (all packaged by your Linux
provider), then a couple of development libraries so that enough
extension modules get built: dev headers for zlib, bz2, openssl and
sqlite3 would be enough IMO.

When you have done that, go to http://wiki.python.org/moin/BuildBot and
follow the instructions at the end. Skip "install buildbot from source"
if you've installed it from your distro's package repo.
Also, you may have to replace "buildbot slave" with "buildbot
create-slave".



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reworking the GIL

[Martin v. Löwis]

> - all machines Python runs on should AFAIK be cache-coherent: CPUs synchronize
> their views of memory in a rather timely fashion.

Ok. I thought that Itanium was an example where this assumption is
actually violated (as many web pages claim such a restriction), however,
it seems that on Itanium, caches are indeed synchronized using MESI.

So claims wrt. lack of cache consistency on Itanium, and the need for
barrier instruction, seem to be caused by the Itanium feature that
allows the processor to fetch memory out-of-order, i.e. an earlier read
may see a later memory state. This is apparently used to satisfy reads
as soon as the cache line is read (so that the cache line can be
discarded earlier). Wrt. to your requirement ("rather timely fashion",
this still seems to be fine).

Still, this all needs to be documented in the code :-)

Regards,
Martin

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] EC2 buildslaves

[Martin v. Löwis]

> An always-on "small" EC2 instance is at least 500$ a year, with a small
> storage cost added to that.

OTOH, that isn't that expensive (compared to the other PSF expenses),
plus people keep donating money, so when we say what we use it for,
there may be a larger return than just the test results.

OTTH, the same could be achieved by buying a hosted server elsewhere.

Regards,
Martin

P.S. Perhaps this *is* the time for Steve to ask for "always-on" machines.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] EC2 buildslaves

[Antoine Pitrou]

> OTOH, that isn't that expensive (compared to the other PSF expenses),
> plus people keep donating money, so when we say what we use it for,
> there may be a larger return than just the test results.
> 
> OTTH, the same could be achieved by buying a hosted server elsewhere.

One advantage of a real hosted server is that we could have a bunch of
our own VMs on it, which is probably not possible (and perhaps not
allowed) on an EC2 instance (not to mention it could really be slow). 

(I'm not volunteering to install and manage VMs, however; I don't think
I'm competent to do that)


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] EC2 buildslaves

[Jesse Noller]

On Mon, Nov 2, 2009 at 8:06 AM, Antoine Pitrou  wrote:
>
>> OTOH, that isn't that expensive (compared to the other PSF expenses),
>> plus people keep donating money, so when we say what we use it for,
>> there may be a larger return than just the test results.
>>
>> OTTH, the same could be achieved by buying a hosted server elsewhere.
>
> One advantage of a real hosted server is that we could have a bunch of
> our own VMs on it, which is probably not possible (and perhaps not
> allowed) on an EC2 instance (not to mention it could really be slow).
>
> (I'm not volunteering to install and manage VMs, however; I don't think
> I'm competent to do that)

Now that the bulk of my pycon work is "completed" - I'll thread swap
to my VM host proposal and the moratorium PEP. So don't worry about
the VM host.

jesse
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reworking the GIL

[Sturla Molden]

Martin v. Löwis skrev:

Maybe you should study the code under discussion before making such
a proposal.

I did, and it does nothing of what I suggested. I am sure I can make the 
Windows GIL
in ceval_gil.h and the mutex in thread_nt.h at lot more precise and 
efficient.


This is the kind of code I was talking about, from ceval_gil.h:

r = WaitForMultipleObjects(2, objects, TRUE, milliseconds);

I would turn on multimedia timer (it is not on by default), and replace 
this

call with a loop, approximately like this:

for (;;) {
  r = WaitForMultipleObjects(2, objects, TRUE, 0);
  /* blah blah blah */   
  QueryPerformanceCounter(&cnt);  
  if (cnt > timeout) break;

  Sleep(0);
}

And the timeout "milliseconds" would now be computed from querying the 
performance

counter, instead of unreliably by the Windows NT kernel.



Sturla










___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reworking the GIL

[Antoine Pitrou]

Sturla Molden  molden.no> writes:
> 
> And the timeout "milliseconds" would now be computed from querying the 
> performance
> counter, instead of unreliably by the Windows NT kernel.

Could you actually test your proposal under Windows and report what kind of
concrete benefits it brings?

Thank you

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reworking the GIL

[Sturla Molden]

Sturla Molden skrev:


I would turn on multimedia timer (it is not on by default), and 
replace this

call with a loop, approximately like this:

for (;;) {
  r = WaitForMultipleObjects(2, objects, TRUE, 0);
  /* blah blah blah */ QueryPerformanceCounter(&cnt);if (cnt > 
timeout) break;

  Sleep(0);
}


And just so you don't ask: There should not just be a Sleep(0) in the 
loop, but a sleep that gets shorter and shorter until a lower threshold 
is reached, where it skips to Sleep(0). That way we avoid hammering om 
WaitForMultipleObjects and QueryPerformanceCounter more than we need. 
And for all that to work better than just giving a timeout to 
WaitForMultipleObjects, we need the multimedia timer turned on.


Sturla



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reworking the GIL

[Sturla Molden]

Sturla Molden skrev:


And just so you don't ask: There should not just be a Sleep(0) in the 
loop, but a sleep that gets shorter and shorter until a lower 
threshold is reached, where it skips to Sleep(0). That way we avoid 
hammering om WaitForMultipleObjects and QueryPerformanceCounter more 
than we need. And for all that to work better than just giving a 
timeout to WaitForMultipleObjects, we need the multimedia timer turned 
on.


The important thing about multimedia timer is that the granularity of 
Sleep() and WaitForMultipleObjects() by default is "10 ms or at most 20 
ms". But if we call


timeBeginPeriod(1);

the MM timer is on and granularity becomes 1 ms or at most 2 ms. But we 
can get even more precise than that by hammering on Sleep(0) for 
timeouts less than 2 ms. We can get typical granularity in the order of 
10 µs, with the occational 100 µs now and then. I know this because I 
was using Windows 2000 to generate TTL signals on the LPT port some 
years ago, and watched them on the oscilloscope.


~ 15 ms granularity is Windows default. But that is brain dead.

By the way Antoine, if you think granularity of 1-2 ms is sufficient, 
i.e. no need for µs precision, then just calling timeBeginPeriod(1) will 
be sufficient.


Sturla








___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reworking the GIL

[Antoine Pitrou]

Sturla Molden  molden.no> writes:
> 
> By the way Antoine, if you think granularity of 1-2 ms is sufficient, 

It certainly is.
But once again, I'm no Windows developer and I don't have a native Windost host
to test on; therefore someone else (you?) has to try.

Also, the MSDN doc (*) says timeBeginPeriod() can have a detrimental effect on
system performance; I don't know how much of it is true.

(*) http://msdn.microsoft.com/en-us/library/dd757624(VS.85).aspx


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reworking the GIL

[Sturla Molden]

Antoine Pitrou skrev:

It certainly is.
But once again, I'm no Windows developer and I don't have a native Windost host
to test on; therefore someone else (you?) has to try.
  
I'd love to try, but I don't have VC++ to build Python, I use GCC on 
Windows.


Anyway, the first thing to try then is to call
 
  timeBeginPeriod(1);


once on startup, and leave the rest of the code as it is. If 2-4 ms is 
sufficient we can use timeBeginPeriod(2), etc. Microsoft is claiming 
Windows performs better with high granularity, which is why it is 10 ms 
by default.



Sturla






___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reworking the GIL

[John Arbash Meinel]

Sturla Molden wrote:
> Antoine Pitrou skrev:
>> It certainly is.
>> But once again, I'm no Windows developer and I don't have a native
>> Windost host
>> to test on; therefore someone else (you?) has to try.
>>   
> I'd love to try, but I don't have VC++ to build Python, I use GCC on
> Windows.
> 
> Anyway, the first thing to try then is to call
>  
>   timeBeginPeriod(1);
> 
> once on startup, and leave the rest of the code as it is. If 2-4 ms is
> sufficient we can use timeBeginPeriod(2), etc. Microsoft is claiming
> Windows performs better with high granularity, which is why it is 10 ms
> by default.
> 
> 
> Sturla

That page claims:
  Windows uses the lowest value (that is, highest resolution) requested
  by any process.

I would posit that the chance of having some random process on your
machine request a high-speed timer is high enough that the overhead for
Python doing the same is probably low.

John
=:->
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reworking the GIL

[Antoine Pitrou]

Sturla Molden  molden.no> writes:
> 
> I'd love to try, but I don't have VC++ to build Python, I use GCC on 
> Windows.

You can use Visual Studio Express, which is free (gratis).



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reworking the GIL

[Martin v. Löwis]

> I did, and it does nothing of what I suggested. I am sure I can make the
> Windows GIL in ceval_gil.h and the mutex in thread_nt.h at lot more precise 
> and
> efficient.

Hmm. I'm skeptical that your code makes it more accurate, and I
completely fail to see that it makes it more efficient (by what
measurement of efficiency?)

Also, why would making it more accurate make it better? IIUC,
accuracy is completely irrelevant here, though efficiency
(low overhead) does matter.

> This is the kind of code I was talking about, from ceval_gil.h:
> 
> r = WaitForMultipleObjects(2, objects, TRUE, milliseconds);
> 
> I would turn on multimedia timer (it is not on by default), and replace
> this
> call with a loop, approximately like this:
> 
> for (;;) {
>   r = WaitForMultipleObjects(2, objects, TRUE, 0);
>   /* blah blah blah */ QueryPerformanceCounter(&cnt);if (cnt >
> timeout) break;
>   Sleep(0);
> }
> 
> And the timeout "milliseconds" would now be computed from querying the
> performance counter, instead of unreliably by the Windows NT kernel.

Hmm. This creates a busy wait loop; if you add larger sleep values,
then it loses accuracy.

Why not just call timeBeginPeriod, and then rely on the higher clock
rate for WaitForMultipleObjects?

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reworking the GIL

[Daniel Stutzbach]

On Mon, Nov 2, 2009 at 11:27 AM, "Martin v. Löwis" wrote:

> Hmm. This creates a busy wait loop; if you add larger sleep values,
> then it loses accuracy.
>

I thought that at first, too, but then I checked the documentation for
Sleep(0):

"A value of zero causes the thread to relinquish the remainder of its time
slice to any other thread of equal priority that is ready to run."

(this is not to say that I think the solution with Sleep is worthwhile,
though...)

--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reworking the GIL

[Jeffrey Yasskin]

On Mon, Nov 2, 2009 at 4:15 AM, Antoine Pitrou  wrote:
> Martin v. Löwis  v.loewis.de> writes:
>>
> [gil_drop_request]
>> Even if it is read from memory, I still wonder what might happen on
>> systems that require explicit memory barriers to synchronize across
>> CPUs. What if CPU 1 keeps reading a 0 value out of its cache, even
>> though CPU 1 has written an 1 value a long time ago?
>>
>> IIUC, any (most?) pthread calls would cause synchronization in that
>> case, which is why applications that also use locks for reading:
>>
>> http://www.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap04.html#tag_04_10
>>
>> Of course, on x86, you won't see any issues, because it's cache-coherent
>> anyway.
>
> I think there are two things here:
> - all machines Python runs on should AFAIK be cache-coherent: CPUs synchronize
> their views of memory in a rather timely fashion.
> - memory ordering: writes made by a CPU can be seen in a different order by
> another CPU (i.e. CPU 1 writes A before B, but CPU 2 sees B written before 
> A). I
> don't see how this can apply to gil_drop_request, since it's a single variable
> (and, moreover, only a single bit of it is significant).
>
> (there's an explanation of memory ordering issues here:
> http://www.linuxjournal.com/article/8211)
>
> As a side note, I remember Jeffrey Yasskin trying to specify an ordering model
> for Python code
> (see http://code.google.com/p/unladen-swallow/wiki/MemoryModel).

Note that that memory model was only for Python code; the C code
implementing it is subject to the C memory model, which is weaker (and
not even fully defined until the next C standard comes out).

To be really safe, we ought to have a couple primitives implementing
"atomic" rather than just "volatile" instructions, but until then a
signal that's just saying "do something" rather than "here's some data
you should look at" should be ok as a volatile int.

I'd like to look at the patch in detail, but I can't guarantee that
I'll get to it in a timely manner. I'd say check it in and let more
threading experts look at it in the tree. We've got some time before a
release for people to fix problems and make further improvements. +1
to Martin's request for detailed documentation though. :)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] nonlocal keyword in 2.x?

[Guido van Rossum]

I'm -0 on backporting nonlocal to 2.7. I could be +0 if we added "from
__future__ import nonlocal_keyword" (or some such phrasing) to enable
it.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] nonlocal keyword in 2.x?

[Raymond Hettinger]

[Guido van Rossum]

I'm -0 on backporting nonlocal to 2.7. I could be +0 if we added "from
__future__ import nonlocal_keyword" (or some such phrasing) to enable
it.


With the "from __future__" option, what keeps you from being
a full +1 on nonlocal?   Is there something that makes it a better
solution for 3.x than 2.x?  Just curious about the pros and cons
from your point of view.


Raymond
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] nonlocal keyword in 2.x?

[Guido van Rossum]

On Mon, Nov 2, 2009 at 10:06 AM, Raymond Hettinger  wrote:
>
> [Guido van Rossum]
>>
>> I'm -0 on backporting nonlocal to 2.7. I could be +0 if we added "from
>> __future__ import nonlocal_keyword" (or some such phrasing) to enable
>> it.
>
> With the "from __future__" option, what keeps you from being
> a full +1 on nonlocal?   Is there something that makes it a better
> solution for 3.x than 2.x?  Just curious about the pros and cons
> from your point of view.

I think the number of projects that can afford to drop support for 2.6
is pretty limited, so I think the utility of the feature is thereby
also limited.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Reworking the GIL

[Sturla Molden]

Martin v. Löwis skrev:

I did, and it does nothing of what I suggested. I am sure I can make the
Windows GIL in ceval_gil.h and the mutex in thread_nt.h at lot more precise and
efficient.



Hmm. I'm skeptical that your code makes it more accurate, and I
completely fail to see that it makes it more efficient (by what
measurement of efficiency?)

Also, why would making it more accurate make it better? IIUC,
accuracy is completely irrelevant here, though efficiency
(low overhead) does matter.

  

This is the kind of code I was talking about, from ceval_gil.h:

r = WaitForMultipleObjects(2, objects, TRUE, milliseconds);

I would turn on multimedia timer (it is not on by default), and replace
this
call with a loop, approximately like this:

for (;;) {
  r = WaitForMultipleObjects(2, objects, TRUE, 0);
  /* blah blah blah */ QueryPerformanceCounter(&cnt);if (cnt >
timeout) break;
  Sleep(0);
}

And the timeout "milliseconds" would now be computed from querying the
performance counter, instead of unreliably by the Windows NT kernel.



Hmm. This creates a busy wait loop; if you add larger sleep values,
then it loses accuracy.

  
Actually an usleep lookes like this, and the call to the wait function 
must go into the for loop. But no, it's not a busy sleep.


static int inited = 0;
static __int64 hz;
static double dhz;
const double sleep_granularity = 2.0E10-3;

void usleep( long us )
{
   __int64 cnt, end;
   double diff;
   if (!inited) {
   timeBeginPeriod(1);
   QueryPerformanceFrequency((LARGE_INTEGER*)&hz);
   dhz = (double)hz;
   inited = 1;
   }  
   QueryPerformanceCounter((LARGE_INTEGER*)&cnt);

   end = cnt + (__int64)(1.0E10-6 * (double)(us) * dhz);
   for (;;) {
   QueryPerformanceCounter((LARGE_INTEGER*)&cnt);
   if (cnt >= end) break;
   diff = (double)(end - cnt)/dhz;
   if (diff > sleep_granularity)
   Sleep((DWORD)(diff - sleep_granularity));
   else
   Sleep(0);  
   }

}



Why not just call timeBeginPeriod, and then rely on the higher clock
rate for WaitForMultipleObjects?

  

That is what I suggested when Antoine said 1-2 ms was enough.



Sturla




___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] language summit topic: issue tracker

[Martin v. Löwis]

> +lots on adding a module field (independent of automatically adding
> maintainers to the nosy list, it would assist in "I just did a major
> cleanup of module X, are there any old bugs I can kill off").

Link (1:1) or Multilink (1:n)? What is the impact on the Component field?

Would you be willing to manage the field (in the sense of managing the
set of values)? If so, please send me a list of values.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] language summit topic: issue tracker

[Martin v. Löwis]

s...@pobox.com wrote:
> Brett> Another summit, another potential time to see if people want to
> Brett> change anything about the issue tracker.
> 
> I have no idea how hard this would be to implement and won't be at the
> language summit to formally present the idea, but it seems to me that some
> integration between the issue tracker and Rietveld would be beneficial. 

See

http://psf.upfronthosting.co.za/roundup/meta/issue247

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] nonlocal keyword in 2.x?

[R. David Murray]

On Mon, 2 Nov 2009 at 10:09, Guido van Rossum wrote:

On Mon, Nov 2, 2009 at 10:06 AM, Raymond Hettinger  wrote:


[Guido van Rossum]


I'm -0 on backporting nonlocal to 2.7. I could be +0 if we added "from
__future__ import nonlocal_keyword" (or some such phrasing) to enable
it.


With the "from __future__" option, what keeps you from being
a full +1 on nonlocal? ? Is there something that makes it a better
solution for 3.x than 2.x? ?Just curious about the pros and cons
from your point of view.


I think the number of projects that can afford to drop support for 2.6
is pretty limited, so I think the utility of the feature is thereby
also limited.


I don't currently have an opinion on this backport proposal, but in
regard to this argument:  if we do not do any 2.x releases after 2.7,
then over time the number of packages that can afford to drop 2.6 support
will grow, yet many will need to retain 2.7 support for much longer.

--David___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] 2.7/3.2 release schedule

[Benjamin Peterson]

I've updated PEP 373 with my proposed release schedule:

- 2.7/3.2 alpha 1 2009-12-05
- 2.7/3.2 alpha 2 2010-01-09
- 2.7/3.2 alpha 3 2010-02-06
- 2.7/3.2 alpha 4 2010-03-06
- 2.7/3.2 beta 1 2010-04-03
- 2.7/3.2 beta 2 2010-05-01
- 2.7/3.2 rc1 2010-05-29
- 2.7/3.2 rc2 2010-06-12
- 2.7/3.2 final 2010-06-26

-- 
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] nonlocal keyword in 2.x?

[Martin v. Löwis]

> I don't currently have an opinion on this backport proposal, but in
> regard to this argument:  if we do not do any 2.x releases after 2.7,
> then over time the number of packages that can afford to drop 2.6 support
> will grow, yet many will need to retain 2.7 support for much longer.

I don't think the argument applies to 2.7 as much as it applied to
earlier releases: 2.7 will have a life time of 18 months perhaps (I
think we still need to decide formally against 2.8, and also decide
when to make the last 2.7 bug fix release). There is some likelihood
that by the time people can agree to drop 2.6 support, 2.7 will be
out of bug fix maintenance already (*). I'm fairly skeptical that people
will be interested in adding new code to specifically clean up their
2.x codebase.

Regards,
Martin

(*) just as it happened with any other release: people are now dropping
support for 2.3, when this got out of security fixes a year ago. By that
measure, people will drop 2.6 support in 2015. I think there are
*really* good chances that many packages will drop 2.6 support along
with dropping 2.x altogether.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] nonlocal keyword in 2.x?

[R. David Murray]

On Mon, 2 Nov 2009 at 22:17, "Martin v. L?wis" wrote:

I don't currently have an opinion on this backport proposal, but in
regard to this argument:  if we do not do any 2.x releases after 2.7,
then over time the number of packages that can afford to drop 2.6 support
will grow, yet many will need to retain 2.7 support for much longer.


I don't think the argument applies to 2.7 as much as it applied to
earlier releases: 2.7 will have a life time of 18 months perhaps (I
think we still need to decide formally against 2.8, and also decide
when to make the last 2.7 bug fix release). There is some likelihood


I was under the impression that if 2.7 was the last release that it
would be maintained (ie: bugfixed) until we decided 3.x uptake was
"sufficient", and that that might be considerably longer than 18 months.
If that is not the case, then what you say is true.

--David___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] nonlocal keyword in 2.x?

[Guido van Rossum]

On Mon, Nov 2, 2009 at 1:27 PM, R. David Murray  wrote:
> On Mon, 2 Nov 2009 at 22:17, "Martin v. Löwis" wrote:
>>>
>>> I don't currently have an opinion on this backport proposal, but in
>>> regard to this argument:  if we do not do any 2.x releases after 2.7,
>>> then over time the number of packages that can afford to drop 2.6 support
>>> will grow, yet many will need to retain 2.7 support for much longer.
>>
>> I don't think the argument applies to 2.7 as much as it applied to
>> earlier releases: 2.7 will have a life time of 18 months perhaps (I
>> think we still need to decide formally against 2.8, and also decide
>> when to make the last 2.7 bug fix release). There is some likelihood
>
> I was under the impression that if 2.7 was the last release that it
> would be maintained (ie: bugfixed) until we decided 3.x uptake was
> "sufficient", and that that might be considerably longer than 18 months.
> If that is not the case, then what you say is true.

Is it even wort doing a 2.7 release? Isn't the effort better spent on
3.2 alone? (Note, these aren't rhetorical questions. It's well
possible that there are good reasons for pushing along with 2.7. Maybe
considering those reasons will also help answering questions about
whether to backport things like nonlocal.)

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] nonlocal keyword in 2.x?

[Martin v. Löwis]

R. David Murray wrote:
> On Mon, 2 Nov 2009 at 22:17, "Martin v. Löwis" wrote:
>>> I don't currently have an opinion on this backport proposal, but in
>>> regard to this argument:  if we do not do any 2.x releases after 2.7,
>>> then over time the number of packages that can afford to drop 2.6
>>> support
>>> will grow, yet many will need to retain 2.7 support for much longer.
>>
>> I don't think the argument applies to 2.7 as much as it applied to
>> earlier releases: 2.7 will have a life time of 18 months perhaps (I
>> think we still need to decide formally against 2.8, and also decide
>> when to make the last 2.7 bug fix release). There is some likelihood
> 
> I was under the impression that if 2.7 was the last release that it
> would be maintained (ie: bugfixed) until we decided 3.x uptake was
> "sufficient", and that that might be considerably longer than 18 months.
> If that is not the case, then what you say is true.

I think that's as-yet undecided. My understanding was that we would
decide, at some point, whether to create a 2.8 release or not, and if
not, that 2.7 would be the final release. To me, this always implied
that there wouldn't be any bug fix releases after 18 months, and no
security releases after five years.

If we would decide to continue doing 2.x releases, the we definitely
wouldn't go beyond 2.9 (because there can't be a 2.10 release,
numerically). At a rate of a release every 18 month, this would mean
we would make bug fix releases for 2.x for no more than 5 years
(from now). But I would really hope that we stop before 2.9.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] nonlocal keyword in 2.x?

[Antoine Pitrou]

Guido van Rossum  python.org> writes:
> 
> Is it even wort doing a 2.7 release? Isn't the effort better spent on
> 3.2 alone? (Note, these aren't rhetorical questions. It's well
> possible that there are good reasons for pushing along with 2.7. Maybe
> considering those reasons will also help answering questions about
> whether to backport things like nonlocal.)

2.7 has an up-to-date backport of the C IO lib (as well as the memoryview
object), which means it is better for people wanting to ease transition to 3.x.

But of course, as Martin said, few people will want to support 2.7 only and not 
2.6.



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] nonlocal keyword in 2.x?

[Martin v. Löwis]

> Is it even wort doing a 2.7 release? Isn't the effort better spent on
> 3.2 alone? (Note, these aren't rhetorical questions. It's well
> possible that there are good reasons for pushing along with 2.7. Maybe
> considering those reasons will also help answering questions about
> whether to backport things like nonlocal.)

If no 2.7 is produced, people will ask for the life time of 2.6. I think
users will appreciate not being forced to 3.x as long as possible, which
essentially means as long as we create 2.x releases. It may be that
users could also "live" with 2.6 bug fix releases only - in this case,
we would have to decide for how long these will have to be created.

There is also the issue of committers who already added stuff to the
trunk after the 2.6 release - some may be unhappy to learn that their
stuff is not going to be released (at least not on that branch).

In the end, the EOL for 2.x will need to be decided by BDFL
pronouncement. Users would continue to ask for 2.x releases until
2020 and beyond if all they have to do is ask. Some committers would
likely also want to continue creating 2.x releases for a few more
years (not sure whether that's because they see themselves also as
users, or because they sympathize with the users, or for other reasons).

In any case, a decision not to release 2.7 should be made before
Benjamin produces the first alpha release.

Regards,
Martin

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] A new way to configure logging

[Olemis Lang]

On Wed, Oct 7, 2009 at 11:26 AM, Olemis Lang  wrote:
> On Wed, Oct 7, 2009 at 10:49 AM, Vinay Sajip  wrote:
>> Olemis Lang  gmail.com> writes:
>>
>>> This kind of problems is similar to the one mentioned in another
>>> thread about modifying config options after executing commands. In
>>> that case I mentioned that the same dict-like interface also holds for
>>> WinReg and so on ...
>>>
>>> So thinking big (yes ! I have a big head ! ) I think that the best
>>> approach in this case is to build an adaptation (simple) layer on top
>>> of ConfigParser, JSON, WinReg, PyCode, YAML, ... and build specific
>>> extensions for these formats . Perhaps the proper interfaces are
>>> already there (e.g. `dict`, `shelve` ) and I'm just blind and looking
>>> somewhere else ;o)
>>
>> Sorry, you've lost me :-)
>>
>
> Never mind . I was just trying to say that using `dict`, an adapter
> could be implemented (if not already there ;o) for multiple formats
> like the ones I mentioned above and the solution would cover many
> config formats ... and also I was saying that I was not sure about
> whether this is correct , I mean for *ANY* config formats,

Intention = answer myself + sharing with you => for a better Pyland

Maybe we could learn something from Augeas [1]_ and use their
experience to include some of those features to `stdlib` . It seems to
be an ambitious attempt to put all the pieces in config-land together;
and AFAICS they handle (a lof of) config formats and have an API for
that, and ... CMIIW anyway

.. [1] Augeas
(http://augeas.net/)

-- 
Regards,

Olemis.

Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Retrieve an arbitrary element from a set withoutremoving it

[Greg Ewing]

Cameron Simpson wrote:


Personally, I'm for the iteration spec in a lot of ways.

Firstly, a .get()/.pick() that always returns the same element feels
horrible. Is there anyone here who _likes_ it?


It doesn't sound very useful to me. However, an iterating
version of it doesn't sound any more useful. What would it
gain you? Why not just iterate over the set? Or make a
copy and repeatedly pop() it?

I completely fail to see a use case for this.

--
Greg
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] 2.7 Release? 2.7 == last of the 2.x line?

[sstein...@gmail.com]

Not that anyone has asked yet, but here's my opinion on two issues  
that have been raised on the python-dev mailing list lately:


	+1 on 2.7 release with as much 3.0 "easy-port goo" as is practicable  
without delaying the product beyond the tentative schedule.  Sooner  
would, of course, be better but I'm sure PEP 373 was produced after  
due consideration of all pertinent factors.


+1 on 2.7 being the last of the 2.x series.  Enough already!

I, personally, haven't even written my first line of 3.x code, nor  
have I had any good reason to.


If I saw the actual end of the line at 2.7, I would actually start  
looking for 3.x versions of my favorite tools and would be much more  
inclined to help push them along ASAP.


Right now, so much that I use on a daily basis doesn't even have a 3.x  
roadmap, much less any sort of working implementation, that I don't  
see switching to 3.x ever unless the 2.x line ends, and soon!


Just my one vote.

S
 
___

Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7 Release? 2.7 == last of the 2.x line?

[James Y Knight]

On Nov 2, 2009, at 6:24 PM, sstein...@gmail.com wrote:


+1 on 2.7 being the last of the 2.x series.  Enough already!


-1. (not that it matters)

I, personally, haven't even written my first line of 3.x code, nor  
have I had any good reason to.


Me neither.

If I saw the actual end of the line at 2.7, I would actually start  
looking for 3.x versions of my favorite tools and would be much more  
inclined to help push them along ASAP.


I'd probably keep using 2.7 to be able to keep using those tools,  
instead.


Right now, so much that I use on a daily basis doesn't even have a  
3.x roadmap, much less any sort of working implementation, that I  
don't see switching to 3.x ever unless the 2.x line ends, and soon!



I don't see switching to 3.x anytime soon either. But what's the rush?

2.x seems to be a fine edition of Python, why not let it keep going to  
2.8 and beyond? Then you wouldn't have to switch to 3.x at all, and  
that'd save you a ton of work. (and save all the people you will have  
to convince to make a 3.x roadmap and do the port a ton of work too!)


It really sounds like you're saying that switching to 3.x isn't worth  
the cost to you, but you want to force people (including yourself) to  
do so anyways, because ...?


James
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7 Release? 2.7 == last of the 2.x line?

[skip]

James> 2.x seems to be a fine edition of Python, why not let it keep
James> going to 2.8 and beyond?

Resources would be my guess.  In much the same way that people move on to
newer releases of Windows, Mac OSX or Linux leaving an ever-dwindling group
available to support old versions, the same will be true of Python.  Over
time more and more of the core developers (and end users) will switch to 3.x
leaving fewer and fewer people with the time or inclination to support 2.x.

I think there are probably enough functional differences between 2.6 and 2.7
that releasing 2.7 makes sense.  The discussion at this point should
probably be what to do when 2.7 is out the door.  It makes sense to me to
merge the py3k branch to trunk coincident with the 2.7/3.2 releases and
creation of the release27-maint and release32-maint branches.  3.3 and
future versions would then be released from trunk and there would be no
further 2.x releases.

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] language summit topic: issue tracker

[skip]

>> ... it seems to me that some integration between the issue tracker
>> and Rietveld would be beneficial.

Martin> See

Martin> http://psf.upfronthosting.co.za/roundup/meta/issue247

Cool.  I still haven't used Rietveld for anything, though I am getting
comfortable with Review Board and like the tool support for code reviews.

Skip
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7 Release? 2.7 == last of the 2.x line?

[sstein...@gmail.com]

On Nov 2, 2009, at 7:26 PM, James Y Knight wrote:


It really sounds like you're saying that switching to 3.x isn't  
worth the cost to you, but you want to force people (including  
yourself) to do so anyways, because ...?


Because that's the future of Python, where the developers who make  
real base language improvements are focusing their attention, and  
because the language would advance faster without this split attention.


A better language, i.e. Python 3.x, will become better faster without  
dragging the 2.x series out any longer.


S



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7 Release? 2.7 == last of the 2.x line?

[Brett Cannon]

On Mon, Nov 2, 2009 at 16:26, James Y Knight  wrote:
>
> On Nov 2, 2009, at 6:24 PM, sstein...@gmail.com wrote:
>
>>        +1 on 2.7 being the last of the 2.x series.  Enough already!
>
> -1. (not that it matters)
>
>> I, personally, haven't even written my first line of 3.x code, nor have I
>> had any good reason to.
>
> Me neither.
>
>> If I saw the actual end of the line at 2.7, I would actually start looking
>> for 3.x versions of my favorite tools and would be much more inclined to
>> help push them along ASAP.
>
> I'd probably keep using 2.7 to be able to keep using those tools, instead.
>
>> Right now, so much that I use on a daily basis doesn't even have a 3.x
>> roadmap, much less any sort of working implementation, that I don't see
>> switching to 3.x ever unless the 2.x line ends, and soon!
>
>
> I don't see switching to 3.x anytime soon either. But what's the rush?
>
> 2.x seems to be a fine edition of Python, why not let it keep going to 2.8
> and beyond? Then you wouldn't have to switch to 3.x at all, and that'd save
> you a ton of work. (and save all the people you will have to convince to
> make a 3.x roadmap and do the port a ton of work too!)
>
> It really sounds like you're saying that switching to 3.x isn't worth the
> cost to you, but you want to force people (including yourself) to do so
> anyways, because ...?

... I think a decent number of us no longer want to maintain the 2.x
series. Honestly, if we go past 2.7 I am simply going to stop
backporting features and bug fixes. It's just too much work keeping so
many branches fixed.

-Brett
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7 Release? 2.7 == last of the 2.x line?

[Barry Warsaw]

On Nov 2, 2009, at 10:48 PM, sstein...@gmail.com wrote:

A better language, i.e. Python 3.x, will become better faster  
without dragging the 2.x series out any longer.


If Python 2.7 becomes the last of the 2.x series, then I personally  
favor back porting as many features from Python 3 as possible.  I  
still think doing so will help people migrate to Python 3 by getting  
their Python 2 code base as close to Python 3 as possible without  
biting the ultimate bullet.  E.g. for me "from __future__ import  
absolute_import, unicode_literals" in Python 2.6 has helped quite a bit.


I also think Guido's call for feature freeze makes a lot more sense  
when 2.7 is the EOL.  Let's give people migrating to Python 3 a nice  
big stable target to hit.  Improving the stdlib also gives people a  
big carrot to move.


I think it's also necessary to give third party library and  
application authors as much help as possible to provide Python 3  
compatible software.  Putting together Python tools involves so many  
dependencies in a fairly deep stack that even one unconverted library  
can cause everything above it to stall on Python 2.


-Barry



PGP.sig
Description: This is a digitally signed message part
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7 Release? 2.7 == last of the 2.x line?

[sstein...@gmail.com]

On Nov 2, 2009, at 11:28 PM, Barry Warsaw wrote:


On Nov 2, 2009, at 10:48 PM, sstein...@gmail.com wrote:

A better language, i.e. Python 3.x, will become better faster  
without dragging the 2.x series out any longer.


If Python 2.7 becomes the last of the 2.x series, then I personally  
favor back porting as many features from Python 3 as possible.  I  
still think doing so will help people migrate to Python 3 by getting  
their Python 2 code base as close to Python 3 as possible without  
biting the ultimate bullet.  E.g. for me "from __future__ import  
absolute_import, unicode_literals" in Python 2.6 has helped quite a  
bit.


I agree as long as:
	A> 2.7 comes out as soon as possible, even if it's missing helpful  
porting features.
	B> 2.7 will get ONLY new features that make it easier to port to 3.x,  
not every feature added to 3.x or all you've done is make "Python 2.7,  
the Python 3 Version." and core developer time will continue to be  
wasted on Python 2.7 instead of moving forward.


I also think Guido's call for feature freeze makes a lot more sense  
when 2.7 is the EOL.  Let's give people migrating to Python 3 a nice  
big stable target to hit.  Improving the stdlib also gives people a  
big carrot to move.


Agreed. And specifically NOT porting every shiny new toy from Python 3  
back to 2.7 makes sure the carrots are only in the 3.x series.


I think it's also necessary to give third party library and  
application authors as much help as possible to provide Python 3  
compatible software.  Putting together Python tools involves so many  
dependencies in a fairly deep stack that even one unconverted  
library can cause everything above it to stall on Python 2.


And that's one of the reasons my explorations into Python 3 have been  
limited to pretty much nothing.


I don't have time to do a bunch of work only to find out that the tool  
I absolutely have to have to finish a project doesn't have a Python 3  
version or has been crippled to make a Python 3 version.


BeautifulSoup, which I use every day, is one such product.  Since the  
crappy old SMGL parser's gone, BeautifulSoup uses the one that's left  
in Python 3 and it makes BeautifulSoup completely useless for my daily  
work.


That's not to say I can't fix that one particular project, but  
customers get cranky when their project is taking longer than expected  
and "Oh, I'm having to convert a lot of things to use Python 3"  
doesn't seem to improve their mood much.


Thanks,

S

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7 Release? 2.7 == last of the 2.x line?

[Guido van Rossum]

On Mon, Nov 2, 2009 at 9:51 PM, sstein...@gmail.com  wrote:
> BeautifulSoup, which I use every day, is one such product.  Since the crappy
> old SMGL parser's gone, BeautifulSoup uses the one that's left in Python 3
> and it makes BeautifulSoup completely useless for my daily work.

This sounds an area where some help might be useful. Perhaps the
quickest solution would simply be to copy the old crappy "sgml" based
html parser into a new version of BeautifulSoup. Though I imagine what
it really needs is a "quirks mode" parser that is compatible with the
HTML dialect accepted by, say, IE6. Maybe a summer of code project?

-- 
--Guido van Rossum (python.org/~guido)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7 Release? 2.7 == last of the 2.x line?

[James Y Knight]

On Nov 3, 2009, at 12:06 AM, Guido van Rossum wrote:

Though I imagine what
it really needs is a "quirks mode" parser that is compatible with the
HTML dialect accepted by, say, IE6. Maybe a summer of code project?


Already exists: html5lib.
http://code.google.com/p/html5lib/

Or if you want a faster (yet I think less exact) HTML parser,  
libxml2's HTML parser, via lxml:

http://codespeak.net/lxml/parsing.html#parsing-html

James
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7 Release? 2.7 == last of the 2.x line?

[Martin v. Löwis]

Barry Warsaw wrote:
> On Nov 2, 2009, at 10:48 PM, sstein...@gmail.com wrote:
> 
>> A better language, i.e. Python 3.x, will become better faster without
>> dragging the 2.x series out any longer.
> 
> If Python 2.7 becomes the last of the 2.x series, then I personally
> favor back porting as many features from Python 3 as possible.

And if *2.6* becomes the last of the 2.x series?

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] 2.7 Release? 2.7 == last of the 2.x line?

[Sturla Molden]

I'd just like to mention that the scientific community is highly 
dependent on NumPy. As long as NumPy is not ported to Py3k, migration is 
out of the question. Porting NumPy is not a trivial issue. It might take 
a complete rewrite of the whole C base using Cython. NumPy's ABI is not 
even PEP 3118 compliant. Changing the ABI for Py3k might break extension 
code written for NumPy using C. And scientists tend to write CPU-bound 
routines in languages like C and Fortran, not Python, so that is a major 
issue as well. If we port NumPy to Py3k, everyone using NumPy will have 
to port their C code to the new ABI. There are lot of people stuck with 
Python 2.x for this reason. It does not just affect individual 
scientists, but also large projects like IBM and CERN's blue brain and 
NASA's space telecope. So please, do not cancel 2.x support before we 
have ported NumPy, Matplotlib and most of their dependant extensions to 
Py3k. The community of scientists and engineers using Python is growing, 
but shutting down 2.x support might bring an end to that.


Sturla Molden


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python-Dev Retrieve an arbitrary element from a set?withoutremoving it

[Cameron Simpson]

On 02Nov2009 10:21, Antoine Pitrou  wrote:
| Cameron Simpson  zip.com.au> writes:
| > 
| > Personally, I'm for the iteration spec in a lot of ways.
| > 
| > Firstly, a .get()/.pick() that always returns the same element feels
| > horrible. Is there anyone here who _likes_ it?
| 
| I do. Since the caller is asking for an arbitrary element, returning the same
| element is legitimate. It's funny how people seem to have a problem with the
| word "arbitrary" :)
| 
| And I'm -1 on any implicit iteration attaching some state to the object. If 
you
| want to iterate, there's already an obvious way to it.

Good point. Colour me convinced by this.

I suggest the doco is really clear about the word arbitrary:-)
-- 
Cameron Simpson  DoD#743
http://www.cskk.ezoshosting.com/cs/

I must really get a thinner pencil. I can't manage this one a bit. It writes
all manner of things I don't intend.
- rhe...@csugrad.cs.vt.edu (Bolo Mk XXXIX)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] thanks to everyone cleaning up the tests

[Neal Norwitz]

On Mon, Nov 2, 2009 at 2:18 AM, Mark Dickinson  wrote:
> On Sat, Oct 31, 2009 at 9:47 PM, Brett Cannon  wrote:
>> Just wanted to publicly thank everyone who has been causing all the
>> checkins to fix and stabilize the test suite (I think it's mostly
>> Antoine and Mark, but I could be missing somebody; I'm under a
>> deadline so only have marginal higher brain functionality).
>
> Not guilty, your honour.  Blame Antoine and David.  :)
>
> David's new buildslave seems to be making a difference, too.

It's really been awesome to see all the work improving the test suite.
 This will make it easier to implement change without fear of breaking
everything.  These changes will reap benefits for a long time to come.

Kudos for picking up this work and to Brett for calling you out. :-)

Thanks,
n
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com