Re: [Python-Dev] File system path encoding on Windows

2016-08-20 Thread Nick Coghlan
On 20 August 2016 at 04:59, Steve Dower  wrote:
> Questions:
> * should we always use Window's Unicode APIs instead of switching between
> bytes/Unicode based on parameter type?

Yes

> * should we allow users to pass bytes and interpret them as utf-8 rather
> than letting Windows do the decoding?

Yes (eventually)

> * should we do it in 3.6, 3.7 or 3.8?

Reading your summary meant this finally clicked with something Victor
has been considering for a while: a "Force UTF-8" switch that told
Python to ignore the locale encoding on Linux, and instead assume
UTF-8 everywhere (command line parameter parsing, environment variable
processing, filesystem encoding, standard streams, etc)

It's essentially the same problem you have on Windows, just with
slightly different symptoms and consequences.

Prompted by that realisation, I'd like to suggest an option that
didn't come up on python-ideas: add such a flag to Python 3.6, and
then actively seek feedback from folks using non-UTF-8 encodings
before making a decision on what to do by default in Python 3.7.

This is a really hard problem for people to reason about abstractly,
but "try running Python with this new flag, and see if anything
breaks" is a much easier question to ask and answer.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] File system path encoding on Windows

2016-08-20 Thread Steve Dower

On 20Aug2016 1231, Nick Coghlan wrote:

I'd like to suggest an option that
didn't come up on python-ideas: add such a flag to Python 3.6, and
then actively seek feedback from folks using non-UTF-8 encodings
before making a decision on what to do by default in Python 3.7.


My biggest concern is that it then falls onto users to know how to start 
Python with that flag. About the only way I'd be okay with this is going 
with option #2 (using mbcs:strict, rather than mbcs:replace) and putting 
a note in the exception to "restart Python with the -X:utf8 option, or 
contact whoever gave you the application if you don't know what this 
means", and hope that the user gets to see the exception rather than it 
being swallowed by the application (and then that the user knows how to 
start it via python.exe, as opposed to a script entry point, a shortcut 
or via a browser or other launcher).


Basically, we make breaking noisier and tell the user to fix it. Some 
users will think that means it is their fault.


On the other hand, having code opt-in or out of the new handling 
requires changing code (which is presumably not going to happen, or we 
wouldn't consider keeping the old behaviour and/or letting the user 
control it), and leads to conflicts between libraries that expect 
different behaviour. If we expect people to use the flag (and I do), we 
need to be really explicit that developers should handle two cases. 
Without the flag, we only need to be explicit about how developers 
should handle the one case.



This is a really hard problem for people to reason about abstractly,
but "try running Python with this new flag, and see if anything
breaks" is a much easier question to ask and answer.


Personally I'm a big fan of doing this with beta releases. There's time 
to switch back to an opt-in model before the final release if things go 
badly. There's already been a lot of interest in the 3.6 prereleases 
(the 3.6.0a3 installer had over 80k downloads in its first fortnight), 
so I think this is viable.


To help, I could add an extra info page to the prerelease installer 
announcing some specific new changes and explicitly asking for feedback 
to bugs.p.o - the ask is already there, but we could beg more in 
prereleases.


Cheers,
Steve
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com


Re: [Python-Dev] File system path encoding on Windows

2016-08-20 Thread Nick Coghlan
On 21 August 2016 at 06:31, Steve Dower  wrote:
> My biggest concern is that it then falls onto users to know how to start
> Python with that flag.

Not necessarily, as this is one of the areas where commercial
redistributors can earn their revenue stream - by deciding that
flipping the default behaviour is the right thing to do for *their*
user base (which is inevitably only a subset of the overall Python
user base).

Making that possible doesn't mean redistributors will actually follow
through, but it's an option worth keeping in mind, as while it does
increase the ecosystem complexity in the near term (since default
behaviour may vary based on how you obtained your Python runtime), in
the longer term it can allow for better informed design decisions at
the reference interpreter level. (For business process wonks, it's
essentially like running through a deliberate divergence/convergence
cycle at the level of the entire language ecosystem:
http://theagilepirate.net/archives/1392 )

> About the only way I'd be okay with this is going
> with option #2 (using mbcs:strict, rather than mbcs:replace) and putting a
> note in the exception to "restart Python with the -X:utf8 option, or contact
> whoever gave you the application if you don't know what this means", and
> hope that the user gets to see the exception rather than it being swallowed
> by the application (and then that the user knows how to start it via
> python.exe, as opposed to a script entry point, a shortcut or via a browser
> or other launcher).
>
> Basically, we make breaking noisier and tell the user to fix it. Some users
> will think that means it is their fault.

I think that makes a good fallback plan if we run into problems during
the beta cycle.

> On the other hand, having code opt-in or out of the new handling requires
> changing code (which is presumably not going to happen, or we wouldn't
> consider keeping the old behaviour and/or letting the user control it), and
> leads to conflicts between libraries that expect different behaviour. If we
> expect people to use the flag (and I do), we need to be really explicit that
> developers should handle two cases. Without the flag, we only need to be
> explicit about how developers should handle the one case.

Library and framework developers will need to handle both cases
anyway, since it's going to be a while before anyone that isn't
bundling their language runtime as part of a fully integrated
application can assume Python 3.6 as their minimum Python version.

>> This is a really hard problem for people to reason about abstractly,
>> but "try running Python with this new flag, and see if anything
>> breaks" is a much easier question to ask and answer.
>
> Personally I'm a big fan of doing this with beta releases. There's time to
> switch back to an opt-in model before the final release if things go badly.
> There's already been a lot of interest in the 3.6 prereleases (the 3.6.0a3
> installer had over 80k downloads in its first fortnight), so I think this is
> viable.

Aye, that sounds like a plausible plan to me.

> To help, I could add an extra info page to the prerelease installer
> announcing some specific new changes and explicitly asking for feedback to
> bugs.p.o - the ask is already there, but we could beg more in prereleases.

I think you'll want to escalate this to a PEP as well - those aren't
just useful as within-development-team design discussion documents,
they also help to give changes more visibility to end users during the
beta process, and to provide an easily referenced "Why did this
change?" explanation for a particular design decision post-release.

That way, "Make it the default during the beta period, potentially
revert back to a '-X:utf-8' option in the final beta if we find
unresolvable-in-the-near-term problems with changing the default" can
be documented as part of the PEP, and folks with concerns about the
change will have something to point people to when trying to convince
them that they really need to test the 3.6 beta releases on their
systems in time to influence that decision.

Cheers,
Nick.

-- 
Nick Coghlan   |   [email protected]   |   Brisbane, Australia
___
Python-Dev mailing list
[email protected]
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com