Re: [Python-Dev] datetime nanosecond support
On 07/24/2012 08:47 PM, Guido van Rossum wrote:
>
> So what functionality specifically do you require? You speak in
> generalities but I need specifics.
The ability of thinking datetime.datetime as a flexible class that can
give you the representation you need when you need. To be more specific
think about this case:
User selects year, month, day, hour, minute, millisecond, nanosecond of
a network event from a browser the javascript code does a ajax call with
time of this format (variant of iso8601):
-MM-DDTHH:MM:SS.mmnnn (where nnn is the nanosecond representation).
The python server takes that string, converts to a datetime, does all
the math with its data and gives the output back using labeling data
with int(nano_datetime.strftime('MMSSmmnnn')) so I've a sequence
number that javascript can sort and handle easily.
It's this flexibility of conversion I'm talking about.
>
>> As you may think using that approach in a web application is very
>> limiting since there is no strftime() in this custom class.
>
> Apparently you didn't need it? :-) Web frameworks usually have their
> own date/time formatting anyway.
Which is usually derived from python's datetime, like in web2py (
http://web2py.com/books/default/chapter/29/6#Record-representation ) in
which timestamps are real python datetime objects and It's ORM
responsability to find the right representation of that data at database
level.
This lead, as you know, to one of the main advantages of any ORM:
abstract from the database layer and the SQL syntax.
The same applies for another well known framework, Django ( your
personal favorite :) ) in which DateTimeField (
https://docs.djangoproject.com/en/dev/ref/models/fields/#django.db.models.DateTimeField
) is a date and time
represented in Python by a datetime.datetime instance.
We didn't need to build a webapp yet. I've been hired for it :) So I'll
do very soon. Unluckly if datetime does not support nanoseconds, I
cannot blame any ORM for not supporting it natively.
>
>>> I didn't read the entire bug, but it mentioned something about storing
>>> datetimes in databases. Do databases support nanosecond precision?
>>>
>>
>> Yeah. According to
>> http://wiki.ispirer.com/sqlways/postgresql/data-types/timestamp at least
>> Oracle support timestamps with nanoseconds accuracy, SQL server supports
>> 100 nanosecond accuracy.
>> Since I use Postgresql personally the best way to accomplish it (also
>> suggested by the #postgresql on freenode) is to store the timestamp with
>> nanosecond (like 1343158283.880338907242) as bigint and let the ORM (so
>> a python ORM) do all the conversion job.
>> An yet again, having nanosecond support in datetime would make things
>> much more easy.
>
> How so, given that the database you use doesn't support it?
Wasn't the job of an ORM to abstract from actual database (either
relational or not relational) such that people who use the ORM do not
care about how data is represented behind it?
If yes It's job of the ORM to figure out what's the best representation
of a data on the given relational or non relational database.
>
> TBH, I think that adding nanosecond precision to the datetime type is
> not unthinkable. You'll have to come up with some clever backward
> compatibility in the API though, and that will probably be a bit ugly
> (you'd have a microsecond parameter with a range of 0-100 and a
> nanosecond parameter with a range of 0-1000). Also the space it takes
> in memory would probably increase (there's no room for an extra 10
> bits in the carefully arranged 8-byte internal representation).
Sure, that are all open issues but as soon as you are in favour of
adding nanosecond support we can start addressing them. I'm sure there
would be other people here that would like to participate to those
issues too.
>
> But let me be clear -- are you willing to help implement any of this?
> You can't just order a feature, you know...
>
Of course, as I wrote in my second message in the issue (
http://bugs.python.org/issue15443#msg166333 ) I'm ready and excited to
contribute to the python core if I can.
Best Regards,
--
Vincenzo Ampolo
http://vincenzo-ampolo.net
http://goshawknest.wordpress.com
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] datetime nanosecond support
Am 25.07.2012 03:46, schrieb Guido van Rossum: > First you will have to show how you'd have to code this *without* > nanosecond precision in datetime and how tedious that is. (I expect > that representing the timestamp as a long integer expressing a posix > timestamp times a billion would be very reasonable.) I'd vote for two separate numbers, the first similar to JDN (Julian Day Number [1]), the second for nanoseconds per day. 3600 * 100 fit nicely into an unsigned 32bit int. This approach has the neat benefit that we'd get rid of the timestamp_t limitations and year 2038 bug at once. IIRC datetime used to break for dates before 1970 on some system because timestamp_t was unsigned. Python could finally support dates BC! JDN is widely used by astronomers and historians to supports a wide range of years as well as convert between calendar systems. Its day 0 is January 1, 4713 BC in proleptic Julian calendar. The conversion between Julian and Gregorian calendar makes JDN hard to use. Rata Die (Januar 1, 1 AD at midnight in proleptic Gregorian calender) sounds like a good idea. People in need for a high precision timer should also consinder TAI [2] instead of UTC as TAI doesn't have leap seconds. DJB's daemontools specifies a tai64n log format [3] that is similar to your idea. Christian [1] http://en.wikipedia.org/wiki/Julian_Day_Number [2] http://en.wikipedia.org/wiki/International_Atomic_Time [3] http://cr.yp.to/daemontools/tai64n.html ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] datetime nanosecond support
On Wed, 25 Jul 2012 11:24:14 +0200 Christian Heimes wrote: > Am 25.07.2012 03:46, schrieb Guido van Rossum: > > First you will have to show how you'd have to code this *without* > > nanosecond precision in datetime and how tedious that is. (I expect > > that representing the timestamp as a long integer expressing a posix > > timestamp times a billion would be very reasonable.) > > I'd vote for two separate numbers, the first similar to JDN (Julian Day > Number [1]), the second for nanoseconds per day. 3600 * 100 fit > nicely into an unsigned 32bit int. But 24 * 3600 * 1e9 doesn't. Perhaps I didn't understand your proposal. Regards Antoine. -- Software development and contracting: http://pro.pitrou.net ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] datetime nanosecond support
On Wed, Jul 25, 2012 at 7:24 PM, Christian Heimes wrote: > Am 25.07.2012 03:46, schrieb Guido van Rossum: > I'd vote for two separate numbers, the first similar to JDN (Julian Day > Number [1]), the second for nanoseconds per day. 3600 * 100 fit > nicely into an unsigned 32bit int. This approach has the neat benefit > that we'd get rid of the timestamp_t limitations and year 2038 bug at > once. IIRC datetime used to break for dates before 1970 on some system > because timestamp_t was unsigned. Python could finally support dates BC! Alternatively, use Decimal as the internal representation (backed by cdecimal if additional speed is needed). However, rather than getting buried in the weeds right here: 1. For the reasons presented, I think it's worth attempting to define a common API that is based on datetime, but is tailored towards high precision time operations (at least using a different internal representation, perhaps supporting TAI). 2. I don't think the stdlib is the right place to define the initial version of this. It seems most sensible to first fork the pure Python version of datetime, figure out the details to get that working as a new distribution on PyPI, and then fork the C implementation to make the PyPI version faster. Assuming it can be completed in time, the revised API could then be brought back as a PEP (alternatively, depending on the details of the proposal, the use case may be deemed sufficiently rare that it is just kept as a specialist module on PyPI). Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] datetime nanosecond support
Am 25.07.2012 13:48, schrieb Antoine Pitrou: >> I'd vote for two separate numbers, the first similar to JDN (Julian Day >> Number [1]), the second for nanoseconds per day. 3600 * 100 fit >> nicely into an unsigned 32bit int. > > But 24 * 3600 * 1e9 doesn't. Perhaps I didn't understand your proposal. What the h... was I thinking? I confused nano with micro and forgot the hours, how embarrassing. :( days 32bit signed integer numbers of days since Jan 1, 1 AD in proleptic Gregorian calendar (aka modern civil calendar). That's Rata Die minus one day since it defines Jan 1, 1 AD as day 1. This allows days between year 5.8 Mio in the past and 5.8 Mio in the future ((1<<31) // 365.242 ~ 5879618). nanoseconds --- 64bit signed or unsigned integer more than enough for nanosecond granularity (47bits), we could easily push it to pico seconds resolution (57bits) in the future. Christian ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] datetime nanosecond support
Am 25.07.2012 14:11, schrieb Nick Coghlan: > 1. For the reasons presented, I think it's worth attempting to define > a common API that is based on datetime, but is tailored towards high > precision time operations (at least using a different internal > representation, perhaps supporting TAI). This is a great opportunity to implement two requests at once. Some people want high precision datetime objects while others would like to see a datetime implementation that works with dates BC. >>> from datetime import datetime, timedelta >>> epoch = datetime.datetime(1970, 1, 1) >>> epoc - timedelta(days=1969*365.242) datetime.datetime(1, 1, 1, 12, 2, 52, 78) >>> epoch - timedelta(days=1970*365.242) Traceback (most recent call last): File "", line 1, in OverflowError: date value out of range > 2. I don't think the stdlib is the right place to define the initial > version of this. +1 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] datetime nanosecond support
Christian Heimes wrote: > > days > > 32bit signed integer > > numbers of days since Jan 1, 1 AD in proleptic Gregorian calendar (aka > modern civil calendar). That's Rata Die minus one day since it defines > Jan 1, 1 AD as day 1. This allows days between year 5.8 Mio in the past > and 5.8 Mio in the future ((1<<31) // 365.242 ~ 5879618). > > nanoseconds > --- > 64bit signed or unsigned integer > > more than enough for nanosecond granularity (47bits), we could easily > push it to pico seconds resolution (57bits) in the future. An alternate strategy might be to use tai64/tai64n/tai64na, which can represent any time over the course of a few hundred billion years to second/nanosecond/attosecond, respectively. They're well-defined, and there's a fair bit of software that can use or manipulate dates in these formats. tai64 is defined similar to propleptic Gregorian in that it uses an idealized 24*60*60 second day, etc. Charles -- --- Charles Cazabon GPL'ed software available at: http://pyropus.ca/software/ --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] datetime nanosecond support
On Wed, Jul 25, 2012 at 7:11 AM, Christian Heimes wrote: > Am 25.07.2012 14:11, schrieb Nick Coghlan: >> 1. For the reasons presented, I think it's worth attempting to define >> a common API that is based on datetime, but is tailored towards high >> precision time operations (at least using a different internal >> representation, perhaps supporting TAI). > > This is a great opportunity to implement two requests at once. Some > people want high precision datetime objects while others would like to > see a datetime implementation that works with dates BC. Beware, people requesting dates BC rarely know what they are asking for. (E.g. Jesus wasn't born on 12/25/0001.) The calendrical ambiguities are such that representing dates that far in the past is better left to a specialized class. Read the original discussions about the datetime type; it loses meaning for dates long ago even if it can represent them, but the choice was made to ignore these and to offer a uniform abstraction for 1 <= year <= . TBH I'm more worried about years >= 1. :-) from datetime import datetime, timedelta epoch = datetime.datetime(1970, 1, 1) epoc - timedelta(days=1969*365.242) > datetime.datetime(1, 1, 1, 12, 2, 52, 78) epoch - timedelta(days=1970*365.242) > Traceback (most recent call last): > File "", line 1, in > OverflowError: date value out of range > > >> 2. I don't think the stdlib is the right place to define the initial >> version of this. > > +1 +1 -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] datetime nanosecond support
1. For the reasons presented, I think it's worth attempting to define a common API that is based on datetime, but is tailored towards high precision time operations (at least using a different internal representation, perhaps supporting TAI). IIUC, the concern is about the DB-API, which does sort-of mandate that the datetime module is used to represent SQL time types. So not basing this off datetime isn't really an option. Providing a subtype should work, though. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] feature freeze questions
I have a few questions about feature freeze: (1) Is increasing test coverage acceptable during feature freeze, even if it does not involve fixing a bug? (2) When adding new tests (e.g. in the course of fixing a bug or increasing test coverage), are we allowed to refactor other tests so that supporting test code can be shared? Or should the tests be added in a less DRY fashion and refactored only after the branch goes back to pre-alpha? (3) What types of documentation changes are allowed during feature freeze? For example, are we only allowed to fix incorrect information, or is it acceptable to improve or add to the information about existing functionality? Thanks, --Chris ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] feature freeze questions
2012/7/25 Chris Jerdonek : > I have a few questions about feature freeze: > > (1) Is increasing test coverage acceptable during feature freeze, even > if it does not involve fixing a bug? > > (2) When adding new tests (e.g. in the course of fixing a bug or > increasing test coverage), are we allowed to refactor other tests so > that supporting test code can be shared? Or should the tests be added > in a less DRY fashion and refactored only after the branch goes back > to pre-alpha? You can do basically anything you want to tests just as long as you don't make them less stable. > > (3) What types of documentation changes are allowed during feature > freeze? For example, are we only allowed to fix incorrect > information, or is it acceptable to improve or add to the information > about existing functionality? All documentation changes are accepted. -- Regards, Benjamin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython (3.2): Issue #15445: Updated logging configuration documentation to highlight
On Thu, Jul 26, 2012 at 4:20 AM, vinay.sajip wrote: > + .. note:: Because portions of the configuration are passed through > + :func:`eval`, use of this function may open its users to a security > risk. > + While the function only binds to a socket on ``localhost``, and so does > + not accept connections from remote machines, there are scenarios where > + untrusted code could be run under the account of the process which > calls > + :func:`listen`. Specifically, if the process calling :func:`listen` > runs > + on a multi-user machine where users cannot trust each other, then a > + malicious user could arrange to run essentially arbitrary code in a > + victim user's process, simply by connecting to the victim's > + :func:`listen` socket and sending a configuration which runs whatever > + code the attacker wants to have executed in the victim's process. This > is > + especially easy to do if the default port is used, but not hard even > if a > + different port is used). Looking at PEP 391, it appears it should be possible to replace the current use of eval() with a combination of the much safer ast.literal_eval() and the str.format attribute/item access micro-language. Worth exploring for 3.4 (http://bugs.python.org/issue15452), as it would be better to actually try to close this attack vector rather than just documenting that it exists. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
