Re: [Python-ideas] PEP draft: context variables

[Nick Coghlan]

On 15 October 2017 at 15:49, Nathaniel Smith  wrote:

> It's not like this is a new and weird concept in Python either -- e.g.
> when you raise an exception, the relevant 'except' block is determined
> based on where the 'raise' happens (the runtime stack), not where the
> 'raise' was written:
>
> try:
> def foo():
> raise RuntimeError
> except RuntimeError:
> print("this is not going to execute, because Python doesn't work that
> way")
> foo()
>

Exactly - this is a better formulation of what I was trying to get at when
I said that we want the semantics of context variables in synchronous code
to reliably align with the semantics of the synchronous call stack as it
appears in an exception traceback.

Attempting a pithy summary of PEP 550's related semantics for use in
explanations to folks that don't care about all the fine details:

The currently active execution context aligns with the expected flow of
exception handling for any exceptions raised in the code being executed.

And with a bit more detail:

* If the code in question will see the exceptions your code raises, then
your code will also be able to see the context variables that it defined or
set
* By default, this relationship is symmetrical, such that if your code will
see the exceptions that other code raises as a regular Python exception,
then you will also see the context changes that that code makes.
* However, APIs and language features that enable concurrent code execution
within a single operating system level thread (like event loops, coroutines
and generators) may break that symmetry to avoid context variable
management conflicts between concurrently executing code. This is the key
behavioural difference between context variables (which enable this by
design) and thread local variables (which don't).
* Pretty much everything else in the PEP 550 API design is a lower level
performance optimisation detail to make management of this dynamic state
sharing efficient in event-driven code

Even PEP 550's proposal for how yield would work aligns with that "the
currently active execution context is the inverse of how exceptions will
flow" notion: the idea there is that if a context manager's __exit__ method
wouldn't see an exception raised by a piece of code, then that piece of
code also shouldn't be able to see any context variable changes made by
that context manager's __enter__ method (since the changes may not get
reverted correctly on failure in that case).

Exceptions raised in a for loop body *don't* typically get thrown back into
the body of the generator-iterator, so generator-iterators' context
variable changes should be reverted at their yield points.

By contrast, exceptions raised in a with statement body *do* get thrown
back into the body of a generator decorated with contextlib.contextmanager,
so those context variable changes should *not* be reverted at yield points,
and instead left for __exit__ to handle.

Similarly, coroutines are in the exception handling path for the other
coroutines they call (just like regular functions), so those coroutines
should share an execution context rather than each having their own.

All of that leads to it being specifically APIs that already need to do
special things to account for exception handling flows within a single
thread (e.g. asyncio.gather, asyncio.ensure_future,
contextlib.contextmanager) that are likely to have to put some thought into
how they will impact the active execution context.

Code for which the existing language level exception handling semantics
already work just fine should then also be able to rely on the default
execution context management semantics.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add time.time_ns(): system clock with nanosecond resolution

[Stefan Behnel]

Nick Coghlan schrieb am 14.10.2017 um 17:46:
> On 14 October 2017 at 18:21, Antoine Pitrou wrote:
>> On Sat, 14 Oct 2017 10:49:11 +0300
>> Serhiy Storchaka wrote:
>>> I don't like the idea of adding a parallel set of functions.
>>>
>>> In the list of alternatives in PEP 410 there is no an idea about fixed
>>> precision float type with nanoseconds precision. It can be implemented
>>> internally as a 64-bit integer, but provide all methods required for
>>> float-compatible number. It would be simpler and faster than general
>>> Decimal.
>>
>> I agree a parallel set of functions is not ideal, but I think a parallel
>> set of functions is still more appropriate than a new number type
>> specific to the time module.
>>
>> Also, if you change existing functions to return a new type, you risk
>> breaking compatibility even if you are very careful about designing the
>> new type.
>>
> 
> Might it make more sense to have a parallel *module* that works with a
> different base data type rather than parallel functions within the existing
> API?
> 
> That is, if folks wanted to switch to 64-bit nanosecond time, they would
> use:
> 
> * time_ns.time()
> * time_ns.monotonic()
> * time_ns.perf_counter()
> * time_ns.clock_gettime()
> * time_ns.clock_settime()
> 
> The idea here would be akin to the fact we have both math and cmath as
> modules, where the common APIs conceptually implement the same algorithms,
> they just work with a different numeric type (floats vs complex numbers).

I thought of that, too. People are used to rename things on import, so this
would provide a very easy way for them to switch. OTOH, I would guess that
"from time import time" covers more than 90% of the use cases of the time
module and it doesn't really matter if we make people change the first or
the second part of that import statement...

But the real point here is that the data type which the current time module
deals with is really (semantically) different from what is proposed now.
All functionality in the time module assumes to work with "seconds", and
accepts fractions of seconds for better precision. But the common semantic
ground is "seconds". That suggests to me that "nanoseconds" really fits
best into a new module which clearly separates the semantics of the two
data types.

(That being said, I'm a big fan of fractions, so I wonder if a Fraction
with a constant nano denominator wouldn't fit in here...)

Stefan

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add time.time_ns(): system clock with nanosecond resolution

[Victor Stinner]

> (That being said, I'm a big fan of fractions, so I wonder if a Fraction
with a constant nano denominator wouldn't fit in here...)

It was discussed in depth in PEP 410, and the PEP was rejected. Guido voted
for nanoseconds as int, when os.stat_result.st_mtime_ns was added.

Victor
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add time.time_ns(): system clock with nanosecond resolution

[Victor Stinner]

> Might it make more sense to have a parallel *module* that works with a
different base data type rather than parallel functions within the existing
API?

I asked about adding new functions to 4 different modules: os, resource,
signal, time.

For example, I dislike the idea of having os and os_ns modules. We already
have os.stat() which returns time as seconds and nanoseconds (both at the
same time). There is also os.utime() which accepts time as seconds *or*
nanoseconds: os.utime (path, times=seconds) or os.utime(path,
ns=nanoseconds).

If we had a time_ns module, would it only contain 4 clocks or does it have
to duplicate the full API? If yes, it is likely to be a mess to maintain
them. How will user choose between time and time_ns? What if tomorrow
clocks get picosecond resolution? (CPU TSC also has sub-nanosecond
resolution, but OS API uses timespec, 1 ns res.) Add a third module?

I prefer to leave all "time functions" in the "time module".

For example, I don't think that we need to add time.nanosleep() or
time.sleep_ns(), since the precision loss starts after a sleep of 104 days.
Who cares of 1 nanosecond after a sleep of 104 days?
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP draft: context variables

[Paul Moore]

On 15 October 2017 at 05:39, Nick Coghlan  wrote:
> On 15 October 2017 at 05:47, Paul Moore  wrote:
>>
>> On 14 October 2017 at 17:50, Nick Coghlan  wrote:
>> > If you capture the context eagerly, then there are fewer opportunities
>> > to
>> > get materially different values from "data = list(iterable)" and "data =
>> > iter(context_capturing_iterable)".
>> >
>> > While that's a valid intent for folks to want to be able to express, I
>> > personally think it would be more clearly requested via an expression
>> > like
>> > "data = iter_in_context(iterable)" rather than having it be implicit in
>> > the
>> > way generators work (especially since having eager context capture be
>> > generator-only behaviour would create an odd discrepancy between
>> > generators
>> > and other iterators like those in itertools).
>>
>> OK. I understand the point here - but I'm not sure I see the practical
>> use case for iter_in_context. When would something like that be used?
>
>
> Suppose you have some existing code that looks like this:
>
> results = [calculate_result(a, b) for a, b in data]
>
> If calculate_result is context dependent in some way (e.g. a & b might be
> decimal values), then eager evaluation of "calculate_result(a, b)" will use
> the context that's in effect on this line for every result.
>
> Now, suppose you want to change the code to use lazy evaluation, so that you
> don't need to bother calculating any results you don't actually use:
>
> results = (calculate_result(a, b) for a, b in data)
>
> In a PEP 550 world, this refactoring now has a side-effect that goes beyond
> simply delaying the calculation: since "calculate_result(a, b)" is no longer
> executed immediately, it will default to using whatever execution context is
> in effect when it actually does get executed, *not* the one that's in effect
> on this line.
>
> A context capturing helper for iterators would let you decide whether or not
> that's what you actually wanted by instead writing:
>
> results = iter_in_context(calculate_result(a, b) for a, b in data)
>
> Here, "iter_in_context" would indicate explicitly to the reader that
> whenever another item is taken from this iterator, the execution context is
> going to be temporarily reset back to the way it was on this line. And since
> it would be a protocol based iterator-in-iterator-out function, you could
> wrap it around *any* iterator, not just generator-iterator objects.

OK, got it. That sounds to me like a candidate for a stdlib function
(either because it's seen as a common requirement, or because it's
tricky to get right - or both). The PEP doesn't include it, as far as
I can see, though.

But I do agree with MAL, it seems wrong to need a helper for this,
even though it's a logical consequence of the other semantics I
described as intuitive :-(

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP draft: context variables

[Paul Moore]

On 15 October 2017 at 06:43, Nick Coghlan  wrote:
> On 15 October 2017 at 15:05, Guido van Rossum  wrote:
>>
>> I would like to reboot this discussion (again). It feels to me we're
>> getting farther and farther from solving any of the problems we might solve.
>>
>> I think we need to give up on doing anything about generators; the use
>> cases point in too many conflicting directions. So we should keep the
>> semantics there, and if you don't want your numeric or decimal context to
>> leak out of a generator, don't put `yield` inside `with`. (Yury and Stefan
>> have both remarked that this is not a problem in practice, given that there
>> are no bug reports or StackOverflow questions about this topic.)
>
>
> Let me have another go at building up the PEP 550 generator argument from
> first principles.
>
> The behaviour that PEP 550 says *shouldn't* change is the semantic
> equivalence of the following code:
>
> # Iterator form
> class ResultsIterator:
> def __init__(self, data):
> self._itr = iter(data)
> def __next__(self):
> return calculate_result(next(self._itr))
>
> results = _ResultsIterator(data)
>
> # Generator form
> def _results_gen(data):
> for item in data:
> yield calculate_result(item)
>
> results = _results_gen(data)
>
> This *had* been non-controversial until recently, and I still don't
> understand why folks suddenly decided we should bring it into question by
> proposing that generators should start implicitly capturing state at
> creation time just because it's technically possible for them to do so (yes
> we can implicitly change the way all generators work, but no, we can't
> implicitly change the way all *iterators* work).

This is non-controversial to me.

> The behaviour that PEP 550 thinks *should* change is for the following code
> to become roughly semantically equivalent, given the constraint that the
> context manager involved either doesn't manipulate any shared state at all
> (already supported), or else only manipulates context variables (the new
> part that PEP 550 adds):
>
> # Iterator form
> class ResultsIterator:
> def __init__(self, data):
> self._itr = iter(data)
> def __next__(self):
> with adjusted_context():
> return calculate_result(next(self._itr))
>
> results = _ResultsIterator(data)
>
> # Generator form
> def _results_gen(data):
> for item in data:
> with adjusted_context():
> yield calculate_result(item)
>
> results = _results_gen(data)
>
> Today, while these two forms look like they *should* be comparable, they're
> not especially close to being semantically equivalent, as there's no
> mechanism that allows for implicit context reversion at the yield point in
> the generator form.

I'll have to take your word for this, as I can't think of an actual
example that follows the pattern of your abstract description, for
which I can immediately see the difference.

In the absence of being able to understand why the difference matters
in current code, I have no view on whether PEP 550 needs to "fix" this
issue.

> While I think PEP 550 would still be usable without fixing this discrepancy,
> I'd be thoroughly disappointed if the only reason we decided not to do it
> was because we couldn't clearly articulate the difference in reasoning
> between:
>
> * "Generators currently have no way to reasonably express the equivalent of
> having a context-dependent return statement inside a with statement in a
> __next__ method implementation, so let's define one" (aka "context variable
> changes shouldn't leak out of generators, as that will make them *more* like
> explicit iterator __next__ methods"); and
> * "Generator functions should otherwise continue to be unsurprising
> syntactic sugar for objects that implement the regular iterator protocol"
> (aka "generators shouldn't implicitly capture their creation context, as
> that would make them *less* like explicit iterator __init__ methods").

I think that if we can't describe the problem that makes it obvious to
the average Python user, then that implies it's a corner case that's
irrelevant to said average Python user - and so I'd consider fixing it
to be low priority. Specifically, a lot lower priority than providing
a context variable facility - which while still not a *common* need,
at least resonates with the average user in the sense of "I can
imagine writing code that needed context like Decimal does".

(And apologies for presenting an imagined viewpoint as what "the
average user" might think...)

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP draft: context variables

[Stefan Krah]

On Sun, Oct 15, 2017 at 06:53:58AM +0200, M.-A. Lemburg wrote:
> I have a hard time seeing the advantage of having a default
> where the context at the time of execution is dependent on
> where it happens rather than where it's defined.
> 
> IMO, the default should be to use the context where the line
> was defined in the code, since that matches the intuitive
> way of writing and defining code.
> 
> The behavior of also deferring the context to time of
> execution should be the non-standard form to not break
> this intuition, otherwise debugging will be a pain and
> writing fully working code would be really hard in the
> face of changing contexts (e.g. say decimal rounding
> changes in different parts of the code).

It would be a major change, but I also think lexical scoping would work
best for the decimal context.

Isolation of modules and libraries (which IMO is a bigger problem than
the often cited generator issues) would be solved.


It would probably not work best (or even at all) for the async call
chain use case.


Stefan Krah



___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP draft: context variables

[Paul Moore]

On 13 October 2017 at 23:30, Yury Selivanov  wrote:
> At this point of time, there's just one place which describes one well
> defined semantics: PEP 550 latest version.  Paul, if you have
> time/interest, please take a look at it, and say what's confusing
> there.

Hi Yury,
The following is my impressions from a read-through of the initial
part of the PEP. tl; dr - you say "concurrent" too much and it makes
my head hurt :-)

1. The abstract feels like it's talking about async. The phrase
"consistent access to non-local state in the context of out-of-order
execution, such as in Python generators and coroutines" said async to
me, even though it mentioned generators. Probably because any time I
see generators mentioned alongside coroutines (a term I really don't
grasp yet in the context of Python) I immediately assume the reference
is to the weird extensions of generators when send() and yield
expressions are used. It quite genuinely took me two or three attempts
to get past the abstract and actually read the next section, because
the "this is async" idea came across so strongly.

2. The rationale says that "Prior to the advent of asynchronous
programming in Python" threads and TLS were used - and it implies this
was fine. But the section goes on to say "TLS does not work well for
programs which execute concurrently in a single thread". But it uses a
*generator* as the example. I'm sorry, but to me a generator is pure
and simple standard Python, and definitely not "executing concurrently
in a single thread" (see below). So again, the clash between what the
description said and the actual example left me confused (and confused
enough to equate all of this in my mind with "all that async stuff I
don't follow").

3. "This is because implicit Decimal context is stored as a
thread-local, so concurrent iteration of the fractions() generator
would corrupt the state." This makes no sense to me. The example isn't
concurrent. There's only one thread, and no async. So no concurrency.
It's interleaved iteration through two generators, which I understand
is *technically* considered concurrency in the async sense, but
doesn't *feel* like concurrency. At its core, this is the problem I'm
hitting throughout the whole document - the conceptual clash between
examples that don't feel concurrent, and discussions that talk almost
totally in terms of concurrency, means that understanding every
section is a significant mental effort.

4. By the end of the rationale, what I'd got from the document was:
"There's a bug in decimal.context, caused by the fact that it uses
TLS. It's basically a limitation of TLS. To fix it they need a new
mechanism, which this PEP provides." So unless I'm using (or would
expect to use) TLS in my own code, this doesn't affect me. Which
really isn't the point (if I now understand correctly) - the PEP is
actually providing a safe (and hopefully easy to use/understand!)
mechanism for handling a specific class of programming problem,
maintaining dynamic state that follows the execution order of the
code, rather than the lexical structure. (I didn't state that well -
but I hope I got the idea across) Basically, the problem that Lisp
dynamic variables are designed to solve (although I don't think that
describing the feature in terms of Lisp is a good idea either).

4a. I'd much prefer this part of the PEP to be structured as follows:

* There's a class of programming problems that need to allow code
to access "state" in a way that follows the runtime path the code
takes. Prior art in this area include Lisp's dynamic scope, ... (more
examples would be good - IIRC, Perl has this type of variable too).
* Normal local variables can't do this as they are lexically
scoped. Global variables can be used, but they don't work in the
presence of threads.
* TLS work for threads, but hit problems when code execution paths
aren't nested subroutine-style. Examples where this happens are
generators (which suspend execution and yield back to their parent),
and async (which simulates multiple threads by interleaving execution
of generators). [Note - I know this explanation is probably
inaccurate]
* This PEP proposes a general mechanism that will allow
programmers to simply write code that manages state like this, which
will work in all of the above cases.

That's it. Barely any mention of async, no need to focus on the
Decimal bug except as a motivating example of why TLS isn't
sufficient, and so no risk that people think "why not just fix
decimal.context" - so no need to go into detail as to why you can't
"just fix it". And it frames the PEP as providing a new piece of
functionality that *anyone* might find a use for, rather than as a fix
for a corner case of async/TLS interaction.

5. The "Goals" section says "provide a more reliable threading.local()
alternative" which is fine. But the bullet points do exactly the same
as before, using terms that I associate with async to describe the
benefits, and so the

Re: [Python-ideas] PEP draft: context variables

[Amit Green]

Once again, I think Paul Moore gets to the heart of the issue.

Generators are simply confusing & async even more so.

Per my earlier email, the fact that generators look like functions, but are
not functions, is at the root of the confusion.

This next part really gets to the heart of the matter:

On Sun, Oct 15, 2017 at 8:15 AM, Paul Moore  wrote:

> To me, this implies that context variables should follow that
> execution path. If the caller sets a value, the generator sees it. If
> the generator sets a value then yields, the caller will see that. If
> code changes the value between two resumptions of the generator, the
> generator will see those changes. The PEP at this point, though,
> states the behaviour of context variables in a way that I simply don't
> follow - it's using the idea of an "outer context" - which as far as I
> can see, has never been defined at this point
>

 This is the totally natural way to think of generators -- and exactly how
I thought about them when I started -- and how I suspect 99% of beginners
think of them:

   - And exactly what you expect since generators appear to be functions
   (since they use 'def' to create them).

Now, as I understand it, its not what really happens, in fact, they can
have their own context, which the major discussion here is all about:

   1. Do we "bind" the context at the time you create the generator (which
   appears to call the generator, but really doesn't)?; OR
   2. Do we "bind" the context at the time the first .__next__ method is
   called?

And, as far as I can figure out, people are strongly arguing for #2 so it
doesn't break backwards compatibility:

   - And then discussion of using wrappers to request #1 instead of #2

My answer is, you can't do #1 or #2 -- you need to do #3, as the default,
-- what Paul write above -- anything else is "unnatural" and
"non-intuitive".

Now, I fully understand we *actually* want the unnatural behavior of #1 &
#2 in real code (and #3 is not really that useful in real code).

However #3 is the natural understanding of what it does ... so that what I
am arguing needs to be implemented (as the default).

Then when we want either #1 or #2, when we are writing real code, -- there
should be special syntax to indicate that is what we want (and yes, we'll
have to use the special syntax 95%+ of the time since we really want #1 or
#2 95%+ of the time; and don't want #3).

But that is the whole point, we *should* use special syntax to indicate we
are doing something that is non-intuitive.

This special syntax helps beginners understand better & will help them
think about the concepts more clearly (See previous post be me on adding
first class language to defining generators, so its a lot clearer what is
going on with then).

My argument, is a major strength of python, is how the syntax helps teach
you concepts & how easy the language is to pick up.  I've talked to so many
people who have said the same thing about the language when they started.

Generators (and this whole discussion of context variables) is not properly
following that paradigm; and I believe it should.  It would make python
even stronger as a programming language of choice, that not only is easy to
use, but easy to learn from as you start programming.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP draft: context variables

[Paul Moore]

On 15 October 2017 at 13:51, Amit Green  wrote:
> Once again, I think Paul Moore gets to the heart of the issue.
>
> Generators are simply confusing & async even more so.
>
> Per my earlier email, the fact that generators look like functions, but are
> not functions, is at the root of the confusion.

I don't agree. I don't find generators *at all* confusing. They are a
very natural way of expressing things, as has been proven by how
popular they are in the Python community.

I don't *personally* understand async, but I'm currently willing to
reserve judgement until I've been in a situation where it would be
useful, and therefore needed to learn it.

Paul
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] (PEP 555 subtopic) Propagation of context in async code

[Koos Zevenhoven]

Let me respond to my own email. I'm sorry I wrote a long email, but I
figured I'll have to take the time to write this down carefully (and even
in a new thread with a clear title) so that people would know what the
discussion was about.  Probably I could have done better structuring that
email, but I seriously ran out of time.

This is directly related to how "a normal user" writing async code would be
affected by the semantics of this (context arguments/variables). It's also
related to the semantics of contexts combined with normal generator
functions, partly because the situation is somewhat similar, and partly
because we might want the same basic rules to apply in both situations.

(Side note: This also has to do with more obscure cases like multiple
different async frameworks in the same process (or in the same program, or
perhaps the same server, or even larger
​
– whatever the constraints are).

Any of the context propagation and isolation/leakage semantics I have
described (or that I recall anyone else describing) could be implemented in
the PEP 555 approach without any problems. The difference is just an if
statement branch or two in C code.

So, see below for some more discussion between (it would be useful if some
people could reply to this email and say if and why they agree or disagree
with something below -- also non-experts that roughly understand what I'm
talking about):

On Fri, Oct 13, 2017 at 6:49 PM, Koos Zevenhoven  wrote:

> This is a continuation of the PEP 555 discussion in
>
> https://mail.python.org/pipermail/python-ideas/2017-September/046916.html
>
> And this month in
>
> https://mail.python.org/pipermail/python-ideas/2017-October/047279.html
>
> If you are new to the discussion, the best point to start reading this
> might be at my second full paragraph below ("The status quo...").
>
> On Fri, Oct 13, 2017 at 10:25 AM, Nick Coghlan  wrote:
>
>> On 13 October 2017 at 10:56, Guido van Rossum  wrote:
>>
>>> I'm out of energy to debate every point (Steve said it well -- that
>>> decimal/generator example is too contrived), but I found one nit in Nick's
>>> email that I wish to correct.
>>>
>>> On Wed, Oct 11, 2017 at 1:28 AM, Nick Coghlan 
>>> wrote:

 As a less-contrived example, consider context managers implemented as
 generators.

 We want those to run with the execution context that's active when
 they're used in a with statement, not the one that's active when they're
 created (the fact that generator-based context managers can only be used
 once mitigates the risk of creation time context capture causing problems,
 but the implications would still be weird enough to be worth avoiding).

>>>
>>> Here I think we're in agreement about the desired semantics, but IMO all
>>> this requires is some special casing for @contextlib.contextmanager. To me
>>> this is the exception, not the rule -- in most *other* places I would want
>>> the yield to switch away from the caller's context.
>>>
>>>
 For native coroutines, we want them to run with the execution context
 that's active when they're awaited or when they're prepared for submission
 to an event loop, not the one that's active when they're created.

>>>
>>> This caught my eye as wrong. Considering that asyncio's tasks (as well
>>> as curio's and trio's) *are* native coroutines, we want complete isolation
>>> between the context active when `await` is called and the context active
>>> inside the `async def` function.
>>>
>>
>> The rationale for this behaviour *does* arise from a refactoring argument:
>>
>>async def original_async_function():
>> with some_context():
>> do_some_setup()
>> raw_data = await some_operation()
>> data = do_some_postprocessing(raw_data)
>>
>> Refactored:
>>
>>async def async_helper_function():
>> do_some_setup()
>> raw_data = await some_operation()
>> return do_some_postprocessing(raw_data)
>>
>>async def refactored_async_function():
>> with some_context():
>> data = await async_helper_function()
>>
>>
> ​*This* type of refactoring argument I *do* subscribe to.​
>
>
>> However, considering that coroutines are almost always instantiated at
>> the point where they're awaited, I do concede that creation time context
>> capture would likely also work out OK for the coroutine case, which would
>> leave contextlib.contextmanager as the only special case (and it would turn
>> off both creation-time context capture *and* context isolation).
>>
>
> ​The difference between context propagation through coroutine function
> calls and awaits comes up when you need help from "the" event loop, which
> means things like creating new tasks from coroutines. However, we cannot
> even assume that the loop is the only one. So far, it makes no difference
> where you call the coroutine function. It is only when you await it or
> schedule it for execution in a loop when somethi

Re: [Python-ideas] PEP draft: context variables

[Amit Green]

Generators are a wonderful feature of the python language, and one of its
best idea.

They are initially very intuitive to understand & easy to use.

However, once you get beyond that; they are actually quite confusing
because their behavior is not natural.

Thus they have a initial easy learning & acceptance curve; and then as you
go from initial use to more advanced use, there is a sudden "bump" in the
learning curve, which is not as smooth as it could be.

Specifically, the fact that when you call them, they do not actually call
your code  (but instead call a wrapper) is quite confusing.

Example:

import __main__


def read_and_prefix_each_line(path):
with open(path) as f:
data = f.read()

for s in data.splitlines():
yield '!' + s

def print_prefixed_file(path):
reader = read_and_prefix_each_line(path) #LINE
12

print('Here is how %r looks prefixed' % path)

for s in
reader:
#LINE 16
print(s)

print_prefixed_file(__main__.__file__)
print_prefixed_file('nonexistent')

Will produce the following:

Traceback (most recent call last):
  File "x.py", line 20, in 
print_prefixed_file('nonexistent')
  File "x.py", line 16, in print_prefixed_file
for s in reader:
  File "x.py", line 5, in read_and_prefix_each_line
with open(path) as f:
IOError: [Errno 2] No such file or directory: 'nonexistent'

This is quite confusing to a person who has been using generators for a
month, and thinks they understand them.

WHY is the traceback happening at line 16 instead of at line 12, when the
function is called?

It is much more intuitive, and natural [to a beginner], to expect the
failure to open the file "nonexistent" to happen at line 12, not line 16.

So, now, the user, while trying to figure out a bug, has to learn that:

   - NO, calling a generator (which looks like a function) does not
   actually call the body of the function (as the user defined it)
   - Instead it calls some generator wrapper.
   - And finally the first time the __next__ method of the wrapper is
   called, the body of the function (as the user defined it) gets called.

And this learning curve is quite steep.

It is made even harder by the following:

>>> def f(): yield 1
...
>>> f


So the user is now convinced that 'f' really is a function.

Further investigation makes it even more confusing:

>>> f()


At this point, the user starts to suspect that something is kind of unusual
about 'yield' keyword.

Eventually, after a while, the user starts to figure this out:

   >>> def f(): print('f started'); yield 1
...
>>> f()

>>> f().__next__()
f started
1
>>>

And eventually after reading
https://docs.python.org/3/reference/datamodel.html the following sentence:

"The following flag bits are defined for co_flags: bit 0x04 is set if the
function uses the *arguments syntax to accept an arbitrary number of
positional arguments; bit 0x08 is set if the function uses the **keywords
syntax to accept arbitrary keyword arguments; bit 0x20 is set if the
function is a generator."

Finally figures it out:

>>> def f(): yield 1
...
>>> f

>>> f.__code__.co_flags & 0x20
32

My point is ... this learning process is overly confusing to a new user &
not a smooth learning curve.

Here is, just a quick initial proposal on how to fix it:
>>> def f(): yield 1
>>> ...
Syntax Error: the 'yield' keyword can only be used in a generator;
please be sure to use @generator before the definition of the generator

>>> @generator
... def f(): yield 1
...
>>> f
 

Just the fact it says ' wrote:

> On 15 October 2017 at 13:51, Amit Green  wrote:
> > Once again, I think Paul Moore gets to the heart of the issue.
> >
> > Generators are simply confusing & async even more so.
> >
> > Per my earlier email, the fact that generators look like functions, but
> are
> > not functions, is at the root of the confusion.
>
> I don't agree. I don't find generators *at all* confusing. They are a
> very natural way of expressing things, as has been proven by how
> popular they are in the Python community.
>
> I don't *personally* understand async, but I'm currently willing to
> reserve judgement until I've been in a situation where it would be
> useful, and therefore needed to learn it.
>
> Paul
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] (PEP 555 subtopic) Propagation of context in async code

[Amit Green]

On Sun, Oct 15, 2017 at 9:44 AM, Koos Zevenhoven  wrote:

> So, see below for some more discussion between (it would be useful if some
> people could reply to this email and say if and why they agree or disagree
> with something below -- also non-experts that roughly understand what I'm
> talking about):
>

Yes, I understand what you are roughly talking about.

Also, yes, generators are co-routines [though when starting to work with
generators, people don't fully realize this].

But then how to address "my" original problem where the context would
> propagate through awaits, and next/send? From what others have written, it
> seems there are also other situations where that is desired. There are
> several ways to solve the problem as an extension to PEP 555, but below is
> one:
>
>
>
>> ​We need both versions: the one that propagates first_context() into the
>> coroutine, and the one that propagates second_context() into it. Or, using
>> my metaphor from the other thread, we need "both the forest and the trees".
>> ​
>>
>> ​A solution to this would be to have two types of context arguments:
>>
>> 1. (calling) context arguments​
>>
>> and
>>
>> 2. execution context arguments
>>
>>
>>
> So yes, I'm actually serious about this possibility. Now it would be up to
> library and framework authors to pick the right variant of the two. And
> this is definitely something that could be documented very clearly.​
>
>
This is an interesting idea.  I would add you also need:

3.  Shared context, the generator shares the context with it's caller which
means:

   - If the caller changes the context, the generator, see the changed
   context next time it's __next__ function is called
   - If the generator changes the context, the caller sees the changed
   context.
   - [This clearly make changing the context using 'with' totally unusable
   in both the caller & the generator -- unless we add even odder semantics,
   that the generator restores the original context when it exists???]
   - (As per previous email by me, I claim this is the most natural way
   beginners are going to think it works; and needs to be supported; also in
   real code this is not often useful]
   - I'm not sure if this would even work with async or not -- *IF* not, I
   would still have a syntax for the user to attempt this -- and throw a
   Syntax Error when they do, with a good explanation of why this combination
   doesn't work for async.  I believe good explanations are a great way for
   people to learn which features can't be combined together & why.


> If something was not clear, but seems relevant to what I'm trying to
> discuss here, please ask :)
>


I looked for you quote "we need both the forest & the trees", but didn't
find it here.  I quite strongly agree we need both; in fact need also the
third case I highlighted above.

As for what Guido wrote, that we might be trying to solve too many problems
-- probably.  However, these are real issues with context's, not edge
cases.

Thus Guido writing we don't want to allow yield within a 'with' clause (as
it leaks context) .. I would argue two things:

   - There are use cases where we *DO* want this -- rare -- true -- but
   they exist (i.e.: my #3 above)

   - IF, for simplicity, sake, it is decided not to handle this case now;
   then make it a syntax error in the language; i.e.:

  def f():
 with context() as x:
yield 1

   Syntax error: 'yield' may not be used inside a 'with' clause.

This would really help new users not to make a mistake that takes hours to
debug; & help correct their [initial mistaken] thinking on how contexts &
generators interact.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] (PEP 555 subtopic) Propagation of context in async code

[Koos Zevenhoven]

On Sun, Oct 15, 2017 at 5:34 PM, Amit Green  wrote:

> On Sun, Oct 15, 2017 at 9:44 AM, Koos Zevenhoven 
> wrote:
>
>> So, see below for some more discussion between (it would be useful if
>> some people could reply to this email and say if and why they agree or
>> disagree with something below -- also non-experts that roughly understand
>> what I'm talking about):
>>
>
> Yes, I understand what you are roughly talking about.
>
> Also, yes, generators are co-routines [though when starting to work with
> generators, people don't fully realize this].
>
> But then how to address "my" original problem where the context would
>> propagate through awaits, and next/send? From what others have written, it
>> seems there are also other situations where that is desired. There are
>> several ways to solve the problem as an extension to PEP 555, but below is
>> one:
>>
>>
>>
>>> ​We need both versions: the one that propagates first_context() into the
>>> coroutine, and the one that propagates second_context() into it. Or, using
>>> my metaphor from the other thread, we need "both the forest and the trees".
>>> ​
>>>
>>> ​A solution to this would be to have two types of context arguments:
>>>
>>> 1. (calling) context arguments​
>>>
>>> and
>>>
>>> 2. execution context arguments
>>>
>>>
>>>
>> So yes, I'm actually serious about this possibility. Now it would be up
>> to library and framework authors to pick the right variant of the two. And
>> this is definitely something that could be documented very clearly.​
>>
>>
> This is an interesting idea.  I would add you also need:
>
> 3.  Shared context, the generator shares the context with it's caller
> which means:
>
>- If the caller changes the context, the generator, see the changed
>context next time it's __next__ function is called
>- If the generator changes the context, the caller sees the changed
>context.
>- [This clearly make changing the context using 'with' totally
>unusable in both the caller & the generator -- unless we add even odder
>semantics, that the generator restores the original context when it
>exists???]
>- (As per previous email by me, I claim this is the most natural way
>beginners are going to think it works; and needs to be supported; also in
>real code this is not often useful]
>- I'm not sure if this would even work with async or not -- *IF* not,
>I would still have a syntax for the user to attempt this -- and throw a
>Syntax Error when they do, with a good explanation of why this combination
>doesn't work for async.  I believe good explanations are a great way for
>people to learn which features can't be combined together & why.
>
>
Just as a quick note, after skimming through your bullet points: ​All of
this is indeed covered with decorators and ​other explicit mechanisms in
the PEP 555 approach. I don't think we need syntax errors, though.


>
>> If something was not clear, but seems relevant to what I'm trying to
>> discuss here, please ask :)
>>
>
>
> I looked for you quote "we need both the forest & the trees", but didn't
> find it here.  I quite strongly agree we need both; in fact need also the
> third case I highlighted above.
>
>
The ordering of the archive was indeed thoroughly destroyed. Ordering by
date might help. ​But the quote you ask for is here:

https://mail.python.org/pipermail/python-ideas/2017-October/047285.html

​​
–-Koos​



> As for what Guido wrote, that we might be trying to solve too many
> problems -- probably.  However, these are real issues with context's, not
> edge cases.
>
> Thus Guido writing we don't want to allow yield within a 'with' clause (as
> it leaks context) .. I would argue two things:
>
>- There are use cases where we *DO* want this -- rare -- true -- but
>they exist (i.e.: my #3 above)
>
>- IF, for simplicity, sake, it is decided not to handle this case now;
>then make it a syntax error in the language; i.e.:
>
>   def f():
>  with context() as x:
> yield 1
>
>Syntax error: 'yield' may not be used inside a 'with' clause.
>
> This would really help new users not to make a mistake that takes hours to
> debug; & help correct their [initial mistaken] thinking on how contexts &
> generators interact.
>
>

-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add time.time_ns(): system clock with nanosecond resolution

[Guido van Rossum]

I'd like to have time.time_ns() -- this is most parallel to st_mtime_ns.

On Sun, Oct 15, 2017 at 2:59 AM, Victor Stinner 
wrote:

> > Might it make more sense to have a parallel *module* that works with a
> different base data type rather than parallel functions within the existing
> API?
>
> I asked about adding new functions to 4 different modules: os, resource,
> signal, time.
>
> For example, I dislike the idea of having os and os_ns modules. We already
> have os.stat() which returns time as seconds and nanoseconds (both at the
> same time). There is also os.utime() which accepts time as seconds *or*
> nanoseconds: os.utime (path, times=seconds) or os.utime(path,
> ns=nanoseconds).
>
> If we had a time_ns module, would it only contain 4 clocks or does it have
> to duplicate the full API? If yes, it is likely to be a mess to maintain
> them. How will user choose between time and time_ns? What if tomorrow
> clocks get picosecond resolution? (CPU TSC also has sub-nanosecond
> resolution, but OS API uses timespec, 1 ns res.) Add a third module?
>
> I prefer to leave all "time functions" in the "time module".
>
> For example, I don't think that we need to add time.nanosleep() or
> time.sleep_ns(), since the precision loss starts after a sleep of 104 days.
> Who cares of 1 nanosecond after a sleep of 104 days?
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>


-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add time.time_ns(): system clock with nanosecond resolution

[Koos Zevenhoven]

On Sun, Oct 15, 2017 at 6:58 PM, Guido van Rossum  wrote:

> I'd like to have time.time_ns() -- this is most parallel to st_mtime_ns.
>
> ​
Welcome to the list Guido! You sound like a C programmer. For many people,
that was the best language they knew of when they learned to program. But
have you ever tried Python? You should give it a try!


-- Koos

P.S. ​Sorry, couldn't resist :-) I guess having two versions of one
function would not be that bad. I will probably never use the ns version
anyway. But I'd like a more general solution to such problems in the long
run.


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Why not picoseconds?

[Antoine Pitrou]

Since new APIs are expensive and we'd like to be future-proof, why not
move to picoseconds?  That would be safe until clocks reach the THz
barrier, which is quite far away from us.

Regards

Antoine.


On Fri, 13 Oct 2017 16:12:39 +0200
Victor Stinner 
wrote:
> Hi,
> 
> I would like to add new functions to return time as a number of
> nanosecond (Python int), especially time.time_ns().
> 
> It would enhance the time.time() clock resolution. In my experience,
> it decreases the minimum non-zero delta between two clock by 3 times,
> new "ns" clock versus current clock: 84 ns (2.8x better) vs 239 ns on
> Linux, and 318 us (2.8x better) vs 894 us on Windows, measured in
> Python.
> 
> The question of this long email is if it's worth it to add more "_ns"
> time functions than just time.time_ns()?
> 
> I would like to add:
> 
> * time.time_ns()
> * time.monotonic_ns()
> * time.perf_counter_ns()
> * time.clock_gettime_ns()
> * time.clock_settime_ns()
> 
> time(), monotonic() and perf_counter() clocks are the 3 most common
> clocks and users use them to get the best available clock resolution.
> clock_gettime/settime() are the generic UNIX API to access these
> clocks and so should also be enhanced to get nanosecond resolution.
> 
> 
> == Nanosecond resolution ==
> 
> More and more clocks have a frequency in MHz, up to GHz for the "TSC"
> CPU clock, and so the clocks resolution is getting closer to 1
> nanosecond (or even better than 1 ns for the TSC clock!).
> 
> The problem is that Python returns time as a floatting point number
> which is usually a 64-bit binary floatting number (in the IEEE 754
> format). This type starts to loose nanoseconds after 104 days.
> Conversion from nanoseconds (int) to seconds (float) and then back to
> nanoseconds (int) to check if conversions loose precision:
> 
> # no precision loss
> >>> x=2**52+1; int(float(x * 1e-9) * 1e9) - x  
> 0
> # precision loss! (1 nanosecond)
> >>> x=2**53+1; int(float(x * 1e-9) * 1e9) - x  
> -1
> >>> print(datetime.timedelta(seconds=2**53 / 1e9))  
> 104 days, 5:59:59.254741
> 
> While a system administrator can be proud to have an uptime longer
> than 104 days, the problem also exists for the time.time() clock which
> returns the number of seconds since the UNIX epoch (1970-01-01). This
> clock started to loose nanoseconds since mid-May 1970 (47 years ago):
> 
> >>> import datetime
> >>> print(datetime.datetime(1970, 1, 1) + datetime.timedelta(seconds=2**53 / 
> >>> 1e9))  
> 1970-04-15 05:59:59.254741
> 
> 
> == PEP 410 ==
> 
> Five years ago, I proposed a large and complex change in all Python
> functions returning time to support nanosecond resolution using the
> decimal.Decimal type:
> 
>https://www.python.org/dev/peps/pep-0410/
> 
> The PEP was rejected for different reasons:
> 
> * it wasn't clear if hardware clocks really had a resolution of 1
> nanosecond, especially when the clock is read from Python, since
> reading a clock in Python also takes time...
> 
> * Guido van Rossum rejected the idea of adding a new optional
> parameter to change the result type: it's an uncommon programming
> practice (bad design in Python)
> 
> * decimal.Decimal is not widely used, it might be surprised to get such type
> 
> 
> == CPython enhancements of the last 5 years ==
> 
> Since this PEP was rejected:
> 
> * the os.stat_result got 3 fields for timestamps as nanoseconds
> (Python int): st_atime_ns, st_ctime_ns, st_mtime_ns
> 
> * Python 3.3 got 3 new clocks: time.monotonic(), time.perf_counter()
> and time.process_time()
> 
> * I enhanced the private C API of Python handling time (API called
> "pytime") to store all timings as the new _PyTime_t type which is a
> simple 64-bit signed integer. The unit of _PyTime_t is not part of the
> API, it's an implementation detail. The unit is currently 1
> nanosecond.
> 
> 
> This week, I converted one of the last clock to new _PyTime_t format:
> time.perf_counter() now has internally a resolution of 1 nanosecond,
> instead of using the C double type.
> 
> XXX technically https://github.com/python/cpython/pull/3983 is not
> merged yet :-)
> 
> 
> 
> == Clocks resolution in Python ==
> 
> I implemented time.time_ns(), time.monotonic_ns() and
> time.perf_counter_ns() which are similar of the functions without the
> "_ns" suffix, but return time as nanoseconds (Python int).
> 
> I computed the smallest difference between two clock reads (ignoring a
> differences of zero):
> 
> Linux:
> 
> * time_ns(): 84 ns <=== !!!
> * time(): 239 ns <=== !!!
> * perf_counter_ns(): 84 ns
> * perf_counter(): 82 ns
> * monotonic_ns(): 84 ns
> * monotonic(): 81 ns
> 
> Windows:
> 
> * time_ns(): 318000 ns <=== !!!
> * time(): 894070 ns <=== !!!
> * perf_counter_ns(): 100 ns
> * perf_counter(): 100 ns
> * monotonic_ns(): 1500 ns
> * monotonic(): 1500 ns
> 
> The difference on time.time() is significant: 84 ns (2.8x better) vs
> 239 ns on Linux and 318 us (2.8x better) vs 894 us on Windows. The
> difference will be larger next y

Re: [Python-ideas] PEP draft: context variables

[M.-A. Lemburg]

On 15.10.2017 07:13, Nick Coghlan wrote:
> On 15 October 2017 at 14:53, M.-A. Lemburg  wrote:
>> The behavior of also deferring the context to time of
>> execution should be the non-standard form to not break
>> this intuition, otherwise debugging will be a pain and
>> writing fully working code would be really hard in the
>> face of changing contexts (e.g. say decimal rounding
>> changes in different parts of the code).
>>
> 
> No, it really wouldn't, since "the execution context is the context that's
> active when the code is executed" is relatively easy to understand based
> entirely on the way functions, methods, and other forms of delayed
> execution work (including iterators).
> 
> "The execution context is the context that's active when the code is
> executed, *unless* the code is in a generator, in which case, it's the
> context that was active when the generator-iterator was instantiated" is
> harder to follow.

I think you're mixing two concepts here: the context defines a
way code is supposed to be interpreted at runtime. This doesn't
have anything to do with when the code is actually run.

Just think what happens if you write code using a specific
context (let's say rounding to two decimal places), which then
get executed deferred within another context (say rounding
to three decimal places) for part of the generator run and
yet another context (say rounding to whole integers) for the
remainder of the generator.

I can't think of a good use case where such behavior would
be intuitive, expected or even reasonable ;-)

The context should be inherited by the generator when instantiated
and not change after that, so that the context defining the
generator takes precedent over any later context in which the
generator is later run.

Note that the above is not the same as raising an exception and
catching it somewhere else (as Nathaniel brought up). The context
actually changes semantics of code, whereas exceptions only flag
a special state and let other code decide what to do with it
(defined where the exception handling is happening, not where the
raise is caused).

Just for clarification: I haven't followed the thread, just saw
your posting and found the argument you put forward a bit
hard to follow. I may well be missing some context or evaluating
the argument in a different one as the one where it was defined ;-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...   http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...   http://zope.egenix.com/


: Try our mxODBC.Connect Python Database Interface for free ! ::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   http://www.egenix.com/company/contact/
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add time.time_ns(): system clock with nanosecond resolution

[Koos Zevenhoven]

All joking aside, I actually like it that Python also allows one to
interact with lower-level concepts when needed. Maybe there could be even
more of this?

-- Koos

On Sun, Oct 15, 2017 at 8:04 PM, Koos Zevenhoven  wrote:

> On Sun, Oct 15, 2017 at 6:58 PM, Guido van Rossum 
> wrote:
>
>> I'd like to have time.time_ns() -- this is most parallel to st_mtime_ns.
>>
>> ​
> Welcome to the list Guido! You sound like a C programmer. For many people,
> that was the best language they knew of when they learned to program. But
> have you ever tried Python? You should give it a try!
>
>
> -- Koos
>
> P.S. ​Sorry, couldn't resist :-) I guess having two versions of one
> function would not be that bad. I will probably never use the ns version
> anyway. But I'd like a more general solution to such problems in the long
> run.
>
>
> --
> + Koos Zevenhoven + http://twitter.com/k7hoven +
>



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Why not picoseconds?

[Koos Zevenhoven]

On Sun, Oct 15, 2017 at 8:17 PM, Antoine Pitrou  wrote:

>
> Since new APIs are expensive and we'd like to be future-proof, why not
> move to picoseconds?  That would be safe until clocks reach the THz
> barrier, which is quite far away from us.
>
>
​I somewhat like the thought, but would everyone then end up thinking about
what power of 1000 they need to multiply with?

-- Koos​



> Regards
>
> Antoine.
>
>
> On Fri, 13 Oct 2017 16:12:39 +0200
> Victor Stinner 
> wrote:
> > Hi,
> >
> > I would like to add new functions to return time as a number of
> > nanosecond (Python int), especially time.time_ns().
> >
> > It would enhance the time.time() clock resolution. In my experience,
> > it decreases the minimum non-zero delta between two clock by 3 times,
> > new "ns" clock versus current clock: 84 ns (2.8x better) vs 239 ns on
> > Linux, and 318 us (2.8x better) vs 894 us on Windows, measured in
> > Python.
> >
> > The question of this long email is if it's worth it to add more "_ns"
> > time functions than just time.time_ns()?
> >
> > I would like to add:
> >
> > * time.time_ns()
> > * time.monotonic_ns()
> > * time.perf_counter_ns()
> > * time.clock_gettime_ns()
> > * time.clock_settime_ns()
> >
> > time(), monotonic() and perf_counter() clocks are the 3 most common
> > clocks and users use them to get the best available clock resolution.
> > clock_gettime/settime() are the generic UNIX API to access these
> > clocks and so should also be enhanced to get nanosecond resolution.
> >
> >
> > == Nanosecond resolution ==
> >
> > More and more clocks have a frequency in MHz, up to GHz for the "TSC"
> > CPU clock, and so the clocks resolution is getting closer to 1
> > nanosecond (or even better than 1 ns for the TSC clock!).
> >
> > The problem is that Python returns time as a floatting point number
> > which is usually a 64-bit binary floatting number (in the IEEE 754
> > format). This type starts to loose nanoseconds after 104 days.
> > Conversion from nanoseconds (int) to seconds (float) and then back to
> > nanoseconds (int) to check if conversions loose precision:
> >
> > # no precision loss
> > >>> x=2**52+1; int(float(x * 1e-9) * 1e9) - x
> > 0
> > # precision loss! (1 nanosecond)
> > >>> x=2**53+1; int(float(x * 1e-9) * 1e9) - x
> > -1
> > >>> print(datetime.timedelta(seconds=2**53 / 1e9))
> > 104 days, 5:59:59.254741
> >
> > While a system administrator can be proud to have an uptime longer
> > than 104 days, the problem also exists for the time.time() clock which
> > returns the number of seconds since the UNIX epoch (1970-01-01). This
> > clock started to loose nanoseconds since mid-May 1970 (47 years ago):
> >
> > >>> import datetime
> > >>> print(datetime.datetime(1970, 1, 1) + datetime.timedelta(seconds=2**53
> / 1e9))
> > 1970-04-15 05:59:59.254741
> >
> >
> > == PEP 410 ==
> >
> > Five years ago, I proposed a large and complex change in all Python
> > functions returning time to support nanosecond resolution using the
> > decimal.Decimal type:
> >
> >https://www.python.org/dev/peps/pep-0410/
> >
> > The PEP was rejected for different reasons:
> >
> > * it wasn't clear if hardware clocks really had a resolution of 1
> > nanosecond, especially when the clock is read from Python, since
> > reading a clock in Python also takes time...
> >
> > * Guido van Rossum rejected the idea of adding a new optional
> > parameter to change the result type: it's an uncommon programming
> > practice (bad design in Python)
> >
> > * decimal.Decimal is not widely used, it might be surprised to get such
> type
> >
> >
> > == CPython enhancements of the last 5 years ==
> >
> > Since this PEP was rejected:
> >
> > * the os.stat_result got 3 fields for timestamps as nanoseconds
> > (Python int): st_atime_ns, st_ctime_ns, st_mtime_ns
> >
> > * Python 3.3 got 3 new clocks: time.monotonic(), time.perf_counter()
> > and time.process_time()
> >
> > * I enhanced the private C API of Python handling time (API called
> > "pytime") to store all timings as the new _PyTime_t type which is a
> > simple 64-bit signed integer. The unit of _PyTime_t is not part of the
> > API, it's an implementation detail. The unit is currently 1
> > nanosecond.
> >
> >
> > This week, I converted one of the last clock to new _PyTime_t format:
> > time.perf_counter() now has internally a resolution of 1 nanosecond,
> > instead of using the C double type.
> >
> > XXX technically https://github.com/python/cpython/pull/3983 is not
> > merged yet :-)
> >
> >
> >
> > == Clocks resolution in Python ==
> >
> > I implemented time.time_ns(), time.monotonic_ns() and
> > time.perf_counter_ns() which are similar of the functions without the
> > "_ns" suffix, but return time as nanoseconds (Python int).
> >
> > I computed the smallest difference between two clock reads (ignoring a
> > differences of zero):
> >
> > Linux:
> >
> > * time_ns(): 84 ns <=== !!!
> > * time(): 239 ns <=== !!!
> > * perf_counter_ns(): 84 ns
> > * perf_counter(): 82 ns
> > * monotonic_

Re: [Python-ideas] Why not picoseconds?

[MRAB]

On 2017-10-15 19:02, Koos Zevenhoven wrote:
On Sun, Oct 15, 2017 at 8:17 PM, Antoine Pitrou >wrote:



Since new APIs are expensive and we'd like to be future-proof, why not
move to picoseconds?  That would be safe until clocks reach the THz
barrier, which is quite far away from us.


​I somewhat like the thought, but would everyone then end up thinking 
about what power of 1000 they need to multiply with?


A simple solution to that would be to provide the multiplier as a named 
constant.


[snip]
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Why not picoseconds?

[Victor Stinner]

I proposed to use nanoseconds because UNIX has 1 ns resolution in timespec,
the most recent API, and Windows has 100 ns.

Using picoseconds would confuse users who may expect sub-nanosecond
resolution, whereas no OS support them currently.

Moreover, nanoseconds as int already landed in os.stat and os.utime.

Last but not least, I already strugle in pytime.c to prevent integer
overflow with 1 ns resolution. It can quickly become much more complex if
there is no native C int type supporting a range large enough to more 1
picosecond resolution usable. I really like using int64_t for _PyTime_t,
it's well supported, very easy to use (ex: "t = t2 - t1"). 64-bit int
supports year after 2200 for delta since 1970.

Note: I only know Ruby which chose picoseconds.

Victor

Le 15 oct. 2017 19:18, "Antoine Pitrou"  a écrit :

>
> Since new APIs are expensive and we'd like to be future-proof, why not
> move to picoseconds?  That would be safe until clocks reach the THz
> barrier, which is quite far away from us.
>
> Regards
>
> Antoine.
>
>
> On Fri, 13 Oct 2017 16:12:39 +0200
> Victor Stinner 
> wrote:
> > Hi,
> >
> > I would like to add new functions to return time as a number of
> > nanosecond (Python int), especially time.time_ns().
> >
> > It would enhance the time.time() clock resolution. In my experience,
> > it decreases the minimum non-zero delta between two clock by 3 times,
> > new "ns" clock versus current clock: 84 ns (2.8x better) vs 239 ns on
> > Linux, and 318 us (2.8x better) vs 894 us on Windows, measured in
> > Python.
> >
> > The question of this long email is if it's worth it to add more "_ns"
> > time functions than just time.time_ns()?
> >
> > I would like to add:
> >
> > * time.time_ns()
> > * time.monotonic_ns()
> > * time.perf_counter_ns()
> > * time.clock_gettime_ns()
> > * time.clock_settime_ns()
> >
> > time(), monotonic() and perf_counter() clocks are the 3 most common
> > clocks and users use them to get the best available clock resolution.
> > clock_gettime/settime() are the generic UNIX API to access these
> > clocks and so should also be enhanced to get nanosecond resolution.
> >
> >
> > == Nanosecond resolution ==
> >
> > More and more clocks have a frequency in MHz, up to GHz for the "TSC"
> > CPU clock, and so the clocks resolution is getting closer to 1
> > nanosecond (or even better than 1 ns for the TSC clock!).
> >
> > The problem is that Python returns time as a floatting point number
> > which is usually a 64-bit binary floatting number (in the IEEE 754
> > format). This type starts to loose nanoseconds after 104 days.
> > Conversion from nanoseconds (int) to seconds (float) and then back to
> > nanoseconds (int) to check if conversions loose precision:
> >
> > # no precision loss
> > >>> x=2**52+1; int(float(x * 1e-9) * 1e9) - x
> > 0
> > # precision loss! (1 nanosecond)
> > >>> x=2**53+1; int(float(x * 1e-9) * 1e9) - x
> > -1
> > >>> print(datetime.timedelta(seconds=2**53 / 1e9))
> > 104 days, 5:59:59.254741
> >
> > While a system administrator can be proud to have an uptime longer
> > than 104 days, the problem also exists for the time.time() clock which
> > returns the number of seconds since the UNIX epoch (1970-01-01). This
> > clock started to loose nanoseconds since mid-May 1970 (47 years ago):
> >
> > >>> import datetime
> > >>> print(datetime.datetime(1970, 1, 1) + datetime.timedelta(seconds=2**53
> / 1e9))
> > 1970-04-15 05:59:59.254741
> >
> >
> > == PEP 410 ==
> >
> > Five years ago, I proposed a large and complex change in all Python
> > functions returning time to support nanosecond resolution using the
> > decimal.Decimal type:
> >
> >https://www.python.org/dev/peps/pep-0410/
> >
> > The PEP was rejected for different reasons:
> >
> > * it wasn't clear if hardware clocks really had a resolution of 1
> > nanosecond, especially when the clock is read from Python, since
> > reading a clock in Python also takes time...
> >
> > * Guido van Rossum rejected the idea of adding a new optional
> > parameter to change the result type: it's an uncommon programming
> > practice (bad design in Python)
> >
> > * decimal.Decimal is not widely used, it might be surprised to get such
> type
> >
> >
> > == CPython enhancements of the last 5 years ==
> >
> > Since this PEP was rejected:
> >
> > * the os.stat_result got 3 fields for timestamps as nanoseconds
> > (Python int): st_atime_ns, st_ctime_ns, st_mtime_ns
> >
> > * Python 3.3 got 3 new clocks: time.monotonic(), time.perf_counter()
> > and time.process_time()
> >
> > * I enhanced the private C API of Python handling time (API called
> > "pytime") to store all timings as the new _PyTime_t type which is a
> > simple 64-bit signed integer. The unit of _PyTime_t is not part of the
> > API, it's an implementation detail. The unit is currently 1
> > nanosecond.
> >
> >
> > This week, I converted one of the last clock to new _PyTime_t format:
> > time.perf_counter() now has internally a resolution of 1 nanosecond,
> > instead o

Re: [Python-ideas] Why not picoseconds?

[Stephan Houben]

Hi all,

I propose multiples of the Planck time, 5.39 × 10 −44 s.
Unlikely computers can be more accurate than that anytime soon.

On a more serious note, I am not sure what problem was solved by moving from
double to a fixed-precision format.

I do know that it now introduced the issue of finding
the correct base unit of the fixed-precision format.

Stephan




2017-10-15 20:28 GMT+02:00 Victor Stinner :

> I proposed to use nanoseconds because UNIX has 1 ns resolution in
> timespec, the most recent API, and Windows has 100 ns.
>
> Using picoseconds would confuse users who may expect sub-nanosecond
> resolution, whereas no OS support them currently.
>
> Moreover, nanoseconds as int already landed in os.stat and os.utime.
>
> Last but not least, I already strugle in pytime.c to prevent integer
> overflow with 1 ns resolution. It can quickly become much more complex if
> there is no native C int type supporting a range large enough to more 1
> picosecond resolution usable. I really like using int64_t for _PyTime_t,
> it's well supported, very easy to use (ex: "t = t2 - t1"). 64-bit int
> supports year after 2200 for delta since 1970.
>
> Note: I only know Ruby which chose picoseconds.
>
> Victor
>
> Le 15 oct. 2017 19:18, "Antoine Pitrou"  a écrit :
>
>>
>> Since new APIs are expensive and we'd like to be future-proof, why not
>> move to picoseconds?  That would be safe until clocks reach the THz
>> barrier, which is quite far away from us.
>>
>> Regards
>>
>> Antoine.
>>
>>
>> On Fri, 13 Oct 2017 16:12:39 +0200
>> Victor Stinner 
>> wrote:
>> > Hi,
>> >
>> > I would like to add new functions to return time as a number of
>> > nanosecond (Python int), especially time.time_ns().
>> >
>> > It would enhance the time.time() clock resolution. In my experience,
>> > it decreases the minimum non-zero delta between two clock by 3 times,
>> > new "ns" clock versus current clock: 84 ns (2.8x better) vs 239 ns on
>> > Linux, and 318 us (2.8x better) vs 894 us on Windows, measured in
>> > Python.
>> >
>> > The question of this long email is if it's worth it to add more "_ns"
>> > time functions than just time.time_ns()?
>> >
>> > I would like to add:
>> >
>> > * time.time_ns()
>> > * time.monotonic_ns()
>> > * time.perf_counter_ns()
>> > * time.clock_gettime_ns()
>> > * time.clock_settime_ns()
>> >
>> > time(), monotonic() and perf_counter() clocks are the 3 most common
>> > clocks and users use them to get the best available clock resolution.
>> > clock_gettime/settime() are the generic UNIX API to access these
>> > clocks and so should also be enhanced to get nanosecond resolution.
>> >
>> >
>> > == Nanosecond resolution ==
>> >
>> > More and more clocks have a frequency in MHz, up to GHz for the "TSC"
>> > CPU clock, and so the clocks resolution is getting closer to 1
>> > nanosecond (or even better than 1 ns for the TSC clock!).
>> >
>> > The problem is that Python returns time as a floatting point number
>> > which is usually a 64-bit binary floatting number (in the IEEE 754
>> > format). This type starts to loose nanoseconds after 104 days.
>> > Conversion from nanoseconds (int) to seconds (float) and then back to
>> > nanoseconds (int) to check if conversions loose precision:
>> >
>> > # no precision loss
>> > >>> x=2**52+1; int(float(x * 1e-9) * 1e9) - x
>> > 0
>> > # precision loss! (1 nanosecond)
>> > >>> x=2**53+1; int(float(x * 1e-9) * 1e9) - x
>> > -1
>> > >>> print(datetime.timedelta(seconds=2**53 / 1e9))
>> > 104 days, 5:59:59.254741
>> >
>> > While a system administrator can be proud to have an uptime longer
>> > than 104 days, the problem also exists for the time.time() clock which
>> > returns the number of seconds since the UNIX epoch (1970-01-01). This
>> > clock started to loose nanoseconds since mid-May 1970 (47 years ago):
>> >
>> > >>> import datetime
>> > >>> print(datetime.datetime(1970, 1, 1) + datetime.timedelta(seconds=2**53
>> / 1e9))
>> > 1970-04-15 05:59:59.254741
>> >
>> >
>> > == PEP 410 ==
>> >
>> > Five years ago, I proposed a large and complex change in all Python
>> > functions returning time to support nanosecond resolution using the
>> > decimal.Decimal type:
>> >
>> >https://www.python.org/dev/peps/pep-0410/
>> >
>> > The PEP was rejected for different reasons:
>> >
>> > * it wasn't clear if hardware clocks really had a resolution of 1
>> > nanosecond, especially when the clock is read from Python, since
>> > reading a clock in Python also takes time...
>> >
>> > * Guido van Rossum rejected the idea of adding a new optional
>> > parameter to change the result type: it's an uncommon programming
>> > practice (bad design in Python)
>> >
>> > * decimal.Decimal is not widely used, it might be surprised to get such
>> type
>> >
>> >
>> > == CPython enhancements of the last 5 years ==
>> >
>> > Since this PEP was rejected:
>> >
>> > * the os.stat_result got 3 fields for timestamps as nanoseconds
>> > (Python int): st_atime_ns, st_ctime_ns, st_mtime_ns
>> >
>> > * Python 3.3 got 3 

Re: [Python-ideas] Why not picoseconds?

[Eric V. Smith]

On 10/15/2017 3:13 PM, Stephan Houben wrote:

Hi all,

I propose multiples of the Planck time, 5.39 × 10 ^−44 s.
Unlikely computers can be more accurate than that anytime soon.

On a more serious note, I am not sure what problem was solved by 
moving from

double to a fixed-precision format.

I do know that it now introduced the issue of finding
the correct base unit of the fixed-precision format.


From Victor's original message, describing the current functions using 
64-bit binary floating point numbers (aka double). They lose precision:


"The problem is that Python returns time as a floatting point number
which is usually a 64-bit binary floatting number (in the IEEE 754
format). This type starts to loose nanoseconds after 104 days."

Eric.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add time.time_ns(): system clock with nanosecond resolution

[Todd]

On Sat, Oct 14, 2017 at 11:46 AM, Nick Coghlan  wrote:

>
> On 14 October 2017 at 18:21, Antoine Pitrou  wrote:
>
>> On Sat, 14 Oct 2017 10:49:11 +0300
>> Serhiy Storchaka 
>> wrote:
>> > I don't like the idea of adding a parallel set of functions.
>> >
>> > In the list of alternatives in PEP 410 there is no an idea about fixed
>> > precision float type with nanoseconds precision. It can be implemented
>> > internally as a 64-bit integer, but provide all methods required for
>> > float-compatible number. It would be simpler and faster than general
>> > Decimal.
>>
>> I agree a parallel set of functions is not ideal, but I think a parallel
>> set of functions is still more appropriate than a new number type
>> specific to the time module.
>>
>> Also, if you change existing functions to return a new type, you risk
>> breaking compatibility even if you are very careful about designing the
>> new type.
>>
>
> Might it make more sense to have a parallel *module* that works with a
> different base data type rather than parallel functions within the existing
> API?
>
> That is, if folks wanted to switch to 64-bit nanosecond time, they would
> use:
>
> * time_ns.time()
> * time_ns.monotonic()
> * time_ns.perf_counter()
> * time_ns.clock_gettime()
> * time_ns.clock_settime()
>
> The idea here would be akin to the fact we have both math and cmath as
> modules, where the common APIs conceptually implement the same algorithms,
> they just work with a different numeric type (floats vs complex numbers).
>
> Cheers,
> Nick.
>

What if we had a class, say time.time_base.  The user could specify the
base units (such as "s", "ns", 1e-7, etc) and the data type ('float',
'int', 'decimal', etc.) when the class is initialized.   It would then
present as methods the entire time API using that precision and data type.
Then the existing functions could internally wrap an instance of the class
where the base units are "1" and the data type is "float".  That way the
user could pick the representation most appropriate for their use-case,
rather than python needing who knows how many different time formats for .

The other advantage is that third-party module could potentially subclass
this with additional options, such as an astronomy module providing an
option to choose between sidereal time vs. solar time, without having to
duplicate the entire API.
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP draft: context variables

[Guido van Rossum]

All this arguing based on "equivalence" between different code fragments is
nuts. The equivalences were never meant to be exact, and people don't
typically understand code using generators using these equivalencies.

The key problem we're trying to address is creating a "context" abstraction
that can be controlled by async task frameworks without making the *access*
API specific to the framework. Because coroutines and generators are
similar under the covers, Yury demonstrated the issue with generators
instead of coroutines (which are unfamiliar to many people). And then
somehow we got hung up about fixing the problem in the example.

I want to back out of this entirely, because the problem (while it can be
demonstrated) is entirely theoretical, and the proposed solution is made
too complicated by attempting to solve the problem for generators as well
as for tasks.

--Guido

On Sat, Oct 14, 2017 at 10:43 PM, Nick Coghlan  wrote:

> On 15 October 2017 at 15:05, Guido van Rossum  wrote:
>
>> I would like to reboot this discussion (again). It feels to me we're
>> getting farther and farther from solving any of the problems we might solve.
>>
>> I think we need to give up on doing anything about generators; the use
>> cases point in too many conflicting directions. So we should keep the
>> semantics there, and if you don't want your numeric or decimal context to
>> leak out of a generator, don't put `yield` inside `with`. (Yury and Stefan
>> have both remarked that this is not a problem in practice, given that there
>> are no bug reports or StackOverflow questions about this topic.)
>>
>
> Let me have another go at building up the PEP 550 generator argument from
> first principles.
>
> The behaviour that PEP 550 says *shouldn't* change is the semantic
> equivalence of the following code:
>
> # Iterator form
> class ResultsIterator:
> def __init__(self, data):
> self._itr = iter(data)
> def __next__(self):
> return calculate_result(next(self._itr))
>
> results = _ResultsIterator(data)
>
> # Generator form
> def _results_gen(data):
> for item in data:
> yield calculate_result(item)
>
> results = _results_gen(data)
>
> This *had* been non-controversial until recently, and I still don't
> understand why folks suddenly decided we should bring it into question by
> proposing that generators should start implicitly capturing state at
> creation time just because it's technically possible for them to do so (yes
> we can implicitly change the way all generators work, but no, we can't
> implicitly change the way all *iterators* work).
>
> The behaviour that PEP 550 thinks *should* change is for the following
> code to become roughly semantically equivalent, given the constraint that
> the context manager involved either doesn't manipulate any shared state at
> all (already supported), or else only manipulates context variables (the
> new part that PEP 550 adds):
>
> # Iterator form
> class ResultsIterator:
> def __init__(self, data):
> self._itr = iter(data)
> def __next__(self):
> with adjusted_context():
> return calculate_result(next(self._itr))
>
> results = _ResultsIterator(data)
>
> # Generator form
> def _results_gen(data):
> for item in data:
> with adjusted_context():
> yield calculate_result(item)
>
> results = _results_gen(data)
>
> Today, while these two forms look like they *should* be comparable,
> they're not especially close to being semantically equivalent, as there's
> no mechanism that allows for implicit context reversion at the yield point
> in the generator form.
>
> While I think PEP 550 would still be usable without fixing this
> discrepancy, I'd be thoroughly disappointed if the only reason we decided
> not to do it was because we couldn't clearly articulate the difference in
> reasoning between:
>
> * "Generators currently have no way to reasonably express the equivalent
> of having a context-dependent return statement inside a with statement in a
> __next__ method implementation, so let's define one" (aka "context variable
> changes shouldn't leak out of generators, as that will make them *more*
> like explicit iterator __next__ methods"); and
> * "Generator functions should otherwise continue to be unsurprising
> syntactic sugar for objects that implement the regular iterator protocol"
> (aka "generators shouldn't implicitly capture their creation context, as
> that would make them *less* like explicit iterator __init__ methods").
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add time.time_ns(): system clock with nanosecond resolution

[Steven D'Aprano]

On Sun, Oct 15, 2017 at 08:04:25PM +0300, Koos Zevenhoven wrote:
> On Sun, Oct 15, 2017 at 6:58 PM, Guido van Rossum  wrote:
> 
> > I'd like to have time.time_ns() -- this is most parallel to st_mtime_ns.
> >
> > ​
> Welcome to the list Guido! You sound like a C programmer. For many people,
> that was the best language they knew of when they learned to program. But
> have you ever tried Python? You should give it a try!

You seem to be making a joke, but I have *no idea* what the joke is.

Are you making fun of Guido's choice of preferred name? "time_ns" 
instead of "time_nanoseconds" perhaps?

Other than that, I cannot imagine what the joke is about. Sorry for 
being so slow.


-- 
Steve
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add time.time_ns(): system clock with nanosecond resolution

[Guido van Rossum]

Sorry, that's an in-joke. Koos is expressing his disappointment in the
rejection of PEP 555 in a way that's only obvious if you're Dutch.

On Sun, Oct 15, 2017 at 3:16 PM, Steven D'Aprano 
wrote:

> On Sun, Oct 15, 2017 at 08:04:25PM +0300, Koos Zevenhoven wrote:
> > On Sun, Oct 15, 2017 at 6:58 PM, Guido van Rossum 
> wrote:
> >
> > > I'd like to have time.time_ns() -- this is most parallel to
> st_mtime_ns.
> > >
> > > ​
> > Welcome to the list Guido! You sound like a C programmer. For many
> people,
> > that was the best language they knew of when they learned to program. But
> > have you ever tried Python? You should give it a try!
>
> You seem to be making a joke, but I have *no idea* what the joke is.
>
> Are you making fun of Guido's choice of preferred name? "time_ns"
> instead of "time_nanoseconds" perhaps?
>
> Other than that, I cannot imagine what the joke is about. Sorry for
> being so slow.
>
>
> --
> Steve
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>



-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Add time.time_ns(): system clock with nanosecond resolution

[Juancarlo Añez]

It could be: time.time(ns=True)

On Sun, Oct 15, 2017 at 7:44 PM, Guido van Rossum  wrote:

> Sorry, that's an in-joke. Koos is expressing his disappointment in the
> rejection of PEP 555 in a way that's only obvious if you're Dutch.
>
> On Sun, Oct 15, 2017 at 3:16 PM, Steven D'Aprano 
> wrote:
>
>> On Sun, Oct 15, 2017 at 08:04:25PM +0300, Koos Zevenhoven wrote:
>> > On Sun, Oct 15, 2017 at 6:58 PM, Guido van Rossum 
>> wrote:
>> >
>> > > I'd like to have time.time_ns() -- this is most parallel to
>> st_mtime_ns.
>> > >
>> > > ​
>> > Welcome to the list Guido! You sound like a C programmer. For many
>> people,
>> > that was the best language they knew of when they learned to program.
>> But
>> > have you ever tried Python? You should give it a try!
>>
>> You seem to be making a joke, but I have *no idea* what the joke is.
>>
>> Are you making fun of Guido's choice of preferred name? "time_ns"
>> instead of "time_nanoseconds" perhaps?
>>
>> Other than that, I cannot imagine what the joke is about. Sorry for
>> being so slow.
>>
>>
>> --
>> Steve
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>> https://mail.python.org/mailman/listinfo/python-ideas
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>


-- 
Juancarlo *Añez*
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Why not picoseconds?

[Nick Coghlan]

On 16 October 2017 at 04:28, Victor Stinner 
wrote:

> I proposed to use nanoseconds because UNIX has 1 ns resolution in
> timespec, the most recent API, and Windows has 100 ns.
>
> Using picoseconds would confuse users who may expect sub-nanosecond
> resolution, whereas no OS support them currently.
>
> Moreover, nanoseconds as int already landed in os.stat and os.utime.
>

And this precedent also makes sense to me as the rationale for using an
"_ns" API suffix within the existing module rather than introducing a new
module.


> Last but not least, I already strugle in pytime.c to prevent integer
> overflow with 1 ns resolution. It can quickly become much more complex if
> there is no native C int type supporting a range large enough to more 1
> picosecond resolution usable. I really like using int64_t for _PyTime_t,
> it's well supported, very easy to use (ex: "t = t2 - t1"). 64-bit int
> supports year after 2200 for delta since 1970.
>

Hopefully by the time we decide it's worth worrying about picoseconds in
"regular" code, compiler support for decimal128 will be sufficiently
ubiquitous that we'll be able to rely on that as our 3rd generation time
representation (where the first gen is seconds as a 64 bit binary float and
the second gen is nanoseconds as a 64 bit integer).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] PEP draft: context variables

[Nick Coghlan]

On 15 October 2017 at 20:45, Paul Moore  wrote:

> On 15 October 2017 at 06:43, Nick Coghlan  wrote:
>


> > # Generator form
> > def _results_gen(data):
> > for item in data:
> > with adjusted_context():
> > yield calculate_result(item)
> >
> > results = _results_gen(data)
> >
> > Today, while these two forms look like they *should* be comparable,
> they're
> > not especially close to being semantically equivalent, as there's no
> > mechanism that allows for implicit context reversion at the yield point
> in
> > the generator form.
>
> I'll have to take your word for this, as I can't think of an actual
> example that follows the pattern of your abstract description, for
> which I can immediately see the difference.
>

Interestingly, thinking about the problem in terms of exception handling
flow reminded me of the fact that having a generator-iterator yield while
inside a with statement or try/except block is already considered an
anti-pattern in many situations, precisely because it means that any
exceptions that get thrown in (including GeneratorExit) will be intercepted
when that may not be what the author really intended.

Accordingly, the canonical
guaranteed-to-be-consistent-with-the-previous-behaviour iterator ->
generator transformation already involves the use of a temporary variable
to move the yield outside any exception handling constructs and ensure that
the exception handling only covers the code that it was originally written
to cover:

def _results_gen(data):
for item in data:
with adjusted_context():
result_for_item = calculate_result(item)
yield result_for_item

results = _results_gen(data)

The exception handling expectations with coroutines are different, since an
"await cr" expression explicitly indicates that any exceptions "cr" fails
to handle *will* propagate back through to where the await appears, just as
"f()" indicates that unhandled exceptions in "f" will be seen by the
current frame.

And even if as a new API context variables were to be defined in a
yield-tolerant way, a lot of existing context managers still wouldn't be
"yield safe", since they may be manipulating thread local or process global
state, rather than context variables or a particular object instance.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Why not picoseconds?

[Guido van Rossum]

On Sun, Oct 15, 2017 at 8:40 PM, Nick Coghlan  wrote:

> Hopefully by the time we decide it's worth worrying about picoseconds in
> "regular" code, compiler support for decimal128 will be sufficiently
> ubiquitous that we'll be able to rely on that as our 3rd generation time
> representation (where the first gen is seconds as a 64 bit binary float and
> the second gen is nanoseconds as a 64 bit integer).
>

I hope we'll never see time_ns() and friends as the second generation --
it's a hack that hopefully we can retire in those glorious days of hardware
decimal128 support.

-- 
--Guido van Rossum (python.org/~guido)
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Why not picoseconds?

[Stephan Houben]

2017-10-15 22:08 GMT+02:00 Eric V. Smith :
>From Victor's original message, describing the current functions using
64-bit binary floating point numbers (aka double). They lose precision:

"The problem is that Python returns time as a floatting point number
> which is usually a 64-bit binary floatting number (in the IEEE 754
> format). This type starts to loose nanoseconds after 104 days."
>
>
Do we realize that at this level of accuracy, relativistic time dilatation
due
to continental drift starts to matter?

Stephan



> Eric.
>
>
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/