bug#21911: TAI-to-UTC conversion leaps at wrong time

[Mark H Weaver]

Zefram  writes:

> Probing the TAI-to-UTC conversion offered by srfi-19's time-tai->date,
> in the minutes around the leap second in 2012:
>
> scheme@(guile-user)> (use-modules (srfi srfi-19))
> scheme@(guile-user)> (for-each (lambda (d) (write (list d
> (date->string (time-tai->date (add-duration (julian-day->time-tai
> 2456109) (make-time time-duration 0 d)) 0) "~4"))) (newline)) (list
> 43000 43160 43164 43165 43166 43167 43199 43200 43201 43202))
> (43000 "2012-06-30T23:56:40Z")
> (43160 "2012-06-30T23:59:20Z")
> (43164 "2012-06-30T23:59:24Z")
> (43165 "2012-06-30T23:59:25Z")
> (43166 "2012-06-30T23:59:25Z")
> (43167 "2012-06-30T23:59:26Z")
> (43199 "2012-06-30T23:59:58Z")
> (43200 "2012-06-30T23:59:59Z")
> (43201 "2012-06-30T23:59:60Z")
> (43202 "2012-07-01T00:00:01Z")

This is fixed by commit 5106377a3460e1e35daf14ea6edbe80426347155 on the
stable-2.2 branch.  I'm closing this bug now, but feel free to reopen if
appropriate.

Thanks!
  Mark

bug#22034: time-utc->date shows bogus zone-dependent leap second

[Mark H Weaver]

Zefram  writes:

> time-utc->date seems to think that a leap second occurs at a different
> time in each time zone:
>
> scheme@(guile-user)> (use-modules (srfi srfi-19))
> scheme@(guile-user)> (define (tdate d) (write (list (date->string d "~4") 
> (date->string (time-utc->date (date->time-utc d) 3600) "~4"))) (newline))
> scheme@(guile-user)> (tdate (make-date 0 59 59 22 30 6 2012 0))
> ("2012-06-30T22:59:59Z" "2012-06-30T23:59:59+0100")
> scheme@(guile-user)> (tdate (make-date 0 0 0 23 30 6 2012 0))
> ("2012-06-30T23:00:00Z" "2012-06-30T23:59:60+0100")
> scheme@(guile-user)> (tdate (make-date 0 1 0 23 30 6 2012 0))
> ("2012-06-30T23:00:01Z" "2012-07-01T00:00:01+0100")
>
> These are three consecutive seconds that occur an hour before a genuine
> leap second (at 23:59:60Z).  Observe that time-utc->date, applied to the
> middle second, describes it as a leap second happening at 23:59:60+01:00,
> which is bogus.  [...]

This is fixed by commit 5106377a3460e1e35daf14ea6edbe80426347155 on the
stable-2.2 branch.

> Matching up with this, the actual leap second is never correctly described
> with a non-zero zone offset.  It should be, for example, 00:59:60+01:00.

If I understand correctly, 'time-utc->date' should never return a date
object with 60 in the seconds field, because those extra seconds have no
representation in time-utc.  They only have representations in time-tai
and time-monotonic.

Anyway, thanks very much for these reports!  I'm closing this bug now,
but feel free to reopen if you think there are still issues to resolve.

  Mark

bug#21902: doc incorrectly describes Julian Date

[Mark H Weaver]

Zefram  writes:

> The manual says, in the section "SRFI-19 Introduction",
>
> #Also, for those not familiar with the terminology, a "Julian Day" is
> # a real number which is a count of days and fraction of a day, in UTC,
> # starting from -4713-01-01T12:00:00Z, ie. midday Monday 1 Jan 4713 B.C.
>
> There are two errors in the first statement of the epoch for Julian Date,
> in ISO 8601 format.  The JD epoch is noon on 1 January 4713 BC *in the
> proleptic Julian calendar*.  The ISO 8601 format is properly never used on
> the Julian calendar: ISO 8601 specifies the use of the Gregorian calendar,
> including proleptically where necessary (as it most certainly is here).
> On the proleptic Gregorian calendar, the JD epoch is noon on 24 November
> 4714 BC, and so the ISO 8601 expression should have some "-11-24".
>
> The second error is in how the year is expressed in ISO 8601.  The initial
> "-" does not mean the BC era, it means that the year number is negative.
> ISO 8601 specifies that the AD era is always used, with year numbers
> going negative where necessary; this arrangement is commonly known as
> "astronomical year numbering".  So "" means 1 BC, "-0001" means 2
> BC, and "-4713" means 4714 BC.  So the "-4713" is not correct for the
> attempted expression of the Julian calendar date, but happens to be
> correct for the Gregorian calendar date.
>
> Putting it together, a correct ISO 8601 expression for the Julian Date
> epoch is "-4713-11-24T12:00:00Z".
>
> The word-based statement of the JD epoch is correct as far as it goes,
> but would benefit considerably by the addition of a clause stating that
> it is in the proleptic Julian calendar.  (Generally, a clarification
> of which calendar is being used is helpful with the statement of any
> date prior to the UK's switch of calendar in 1752.)  The description of
> Modified Julian Date is essentially correct.
>
> However, there's a third problem: misuse of the term "UTC" for historical
> times.  The description of Julian Date says it's counted "in UTC",
> and the statement of the MJD epoch describes its 1858 time as being
> specified in UTC.  UTC is defined entirely by its relationship to TAI,
> which is defined by the operation of atomic clocks.  TAI is therefore
> only defined for the period since the operation of the first caesium
> atomic clock in the middle of 1955.  The UTC<->TAI relationship isn't
> actually defined even that far back: UTC begins at the beginning of
> 1961 (and that was not in the modern form with leap seconds).  It is
> therefore incorrect to apply the term "UTC" to any time prior to 1961.
> These two references to UTC should instead be to "UT", the wider class
> of closely-matching time scales of which UTC is one representative.
> Also, in the first sentence of this doc section, the phrase "universal
> time (UTC)" should be either "universal time (UT)" or (more likely)
> "coordinated universal time (UTC)".

I changed the text, based partly on your proposed patch and partly based
on similar recent fixes in the upstream SRFI-19 document, in commit
5106377a3460e1e35daf14ea6edbe80426347155 on the stable-2.2 branch.

I'm closing this bug now, but feel free to reopen if there are still
problems.

 Thanks!
   Mark

bug#22033: time-utc format is lossy

[Mark H Weaver]

tags 22033 + notabug
close 22033
thanks

Hi Zefram,

Zefram  writes:

> I wrote:
>>   These two seconds are perfectly
>>distinct parts of the UTC time scale, and the time-utc format ought to
>>preserve their distinction.
>
> This is a problematic goal.  At the time I wrote the bug report I didn't
> have a satisfactory idea of how to achieve it, but I think I've come up
> with one now.
>
> The essential problem is that the SRFI-19 time structure expects to
> encapsulate a scalar value -- as it says, a count of seconds since
> some epoch -- but there is no natural scalar representation of a UTC
> time.  Because of the irregularity imposed by its leaps, the natural
> representation of a UTC time is a two-part structure, consisting of an
> integer identifying the day and a fractional count of seconds elapsed
> within the day.  Because UTC days contain differing numbers of seconds,
> this is a variable-radix system.

More precisely, UTC days contain differing numbers of TAI seconds.
However, they contain equal numbers of UTC seconds.

I don't see how we can fix this given the definition of UTC.  UTC, when
represented as a number of seconds since some epoch, simply cannot
represent leap seconds that cause UTC to jump backwards, as all leap
seconds so far have done.  This is an inherent problem with UTC, and is
one of the reasons that TAI is more appropriate than UTC for many
applications.

Your objections here are valid, and cut to the heart of the
long-standing debate over whether leap seconds are a good idea, a debate
which continues today.  If you're curious to read more on this,
 is a good starting point.

You might also be interested to know that your idea to encode leap
seconds within the 'nanoseconds' field was also proposed by Markus Kuhn
and mentioned by Olin Shivers on the SRFI-19 mailing list during the
early discussion of SRFI-19:

  https://srfi-email.schemers.org/srfi-19/msg/2772123

It's an interesting idea, but I don't think it's something that we can
unilaterally change in an existing, long-finalized SRFI.  It would need
to be part of a new SRFI, I think.

So, I'm closing this as not-a-bug, although I acknowledge that the issue
you raised is valid.  Feel free to reopen and continue the discussion if
you disagree.

In any case, thanks very much for your many interesting and detailed bug
reports, and I apologize for the long delay in addressing them.

Regards,
  Mark

bug#21912: TAI<->UTC conversion botches the unknown

[Mark H Weaver]

Zefram  writes:

> Guile ought to be aware of how far its leap table extends, and signal
> an error when asked to perform a TAI<->UTC conversion that falls outside
> its scope.

I sympathize with your preference to raise an error rather than return
invalid results, but unfortunately I don't think this proposal is really
practical, given the SRFI-19 API.  The main problem is that, although
it's not specified in the document, the SRFI-19 reference implementation
uses UTC for all julian days, modified julian days, and dates.  For
example, 'time-tai->date' and 'time-tai->julian-day' include an implicit
conversion to UTC.

As a result, if we were to adopt your proposal, it would never be
possible to print a date more than 6 months in the future, and for users
of stable distributions such as Debian, it is not be possible to print
the current date.

We need a way to print future dates.  In theory, we could print future
dates in TAI without the leap second table, using the convention of
printing TAI dates as in , but
that's not what SRFI-19 does, and it has no API to support that way of
printing TAI dates.

Also, it's debatable whether it's desirable to print future dates in TAI
this way, since in the far future the times will drift far away from
mean solar time, and the calendar will eventually drift from the usual
alignment of the solar year to the months on the calendar.

So, I think this is a valid issue, but I don't see how it can be
addressed within SRFI-19.  I would need to be addressed in a new SRFI,
and even then it's not clear to me what's the right way forward.

What do you think?

Regards,
  Mark

bug#21901: bit shift wrong on maximal right shift

[Mark H Weaver]

Zefram  writes:

> With Guile 2.0.11:
>
> scheme@(guile-user)> (ash 123 (ash -1 63))
> $1 = 123

Fixed in commit 1990aa916382d0afcebd5315a6d6f555949ff654 on the
stable-2.2 branch.  Closing this bug now.

Thanks very much for finding this subtle bug.

  Mark

bug#21903: date->string duff ISO 8601 negative years

[Mark H Weaver]

Zefram  writes:

> The date->string function from (srfi srfi-19), used on ISO 8601 formats
> "~1", "~4" and "~5", for years preceding AD 1, has an off-by-one error:
>
> scheme@(guile-user)> (use-modules (srfi srfi-19))
> scheme@(guile-user)> (date->string (julian-day->date 0 0) "~4")
> $1 = "-4714-11-24T12:00:00Z"
>
> The date in question, the JD epoch, is 24 November 4714 BC (in the
> proleptic Gregorian calendar).  In ISO 8601 format, that year is properly
> represented as "-4713", not "-4714", because ISO 8601 uses the AD era
> exclusively.  4714 BC = AD -4713.

I agree that this is definitely a bug, but I'm nervous about deviating
from the SRFI-19 reference implementation, and therefore probably from
most other implementations of SRFI-19, in this way.

I think that this bug should be reported to the SRFI-19 mailing list.

   https://srfi.schemers.org/srfi-19/

There have been several other bugs reported and fixed in upstream
SRFI-19 over the years, including some as recently as June 2017, so I'm
hopeful that they will take this bug seriously and issue a fix.

Would you like to report it to them?

   Mark

bug#21904: date->string duff ISO 8601 format for non-4-digit years

[Mark H Weaver]

Zefram  writes:

> scheme@(guile-user)> (date->string (julian-day->date 1722000 0) "~1")
> $4 = "2-07-29"
> scheme@(guile-user)> (date->string (julian-day->date 173 0) "~1")
> $5 = "24-06-23"
> scheme@(guile-user)> (date->string (julian-day->date 200 0) "~1")
> $6 = "763-09-18"

This particular subset of bugs, for years 0-, was fixed in the
upstream SRFI-19 reference implementation, and so I included the same
fix in commit 5106377a3460e1e35daf14ea6edbe80426347155.  That fix pads
the year to have at least 4 characters with the requested padding
character (0 by default).  However, it does not handle adding the sign
where mandated by ISO 8601.

As with your related bug , I think this bug
should be reported to upstream SRFI-19, and hopefully they will take it
seriously.  I'm reluctant to have Guile deviate from most (all?) other
SRFI-19 implementations in this respect.

There's also the issue that 'string->date' would need to be fixed to
successfully parse the years as printed by 'date->string'.

Would you like to report these issues to upstream SRFI-19?

 Regards,
   Mark

bug#21904: date->string duff ISO 8601 format for non-4-digit years

[Mark H Weaver]

Zefram  writes:

> For year numbers 1 and above, it is necessary to use more than four
> digits for the year, and that's permitted, but ISO 8601 requires that
> more than four digits are preceded by a sign.  For positive year numbers
> the sign must be "+".  So one should write "+22666" rather than "22666",
> for example.

I skimmed a draft of ISO 8601 that I was able to find gratis online:

  
https://web.archive.org/web/20171019211402/https://www.loc.gov/standards/datetime/ISO_DIS%208601-1.pdf
  
https://web.archive.org/web/2017102043/https://www.loc.gov/standards/datetime/ISO_DIS%208601-2.pdf

and also the ISO 8601 Wikipedia page:

  https://en.wikipedia.org/wiki/ISO_8601#Years

and I'm left with a different interpretation about what the standard
permits.  As the Wikipedia page says:

  To represent years before  or after , the standard also
  permits the expansion of the year representation but only by prior
  agreement between the sender and the receiver.[19] An expanded year
  representation [±Y] must have an agreed-upon number of extra year
  digits beyond the four-digit minimum, and it must be prefixed with a +
  or − sign[20] [...]

Note the words "but only by prior agreement between the sender and the
receiver", and "must have an agreed-upon number of extra year digits".

You seem to have reached the conclusion that the sender can choose the
number of digits dynamically, leaving the receiver to auto-detect the
number of digits, but that seems to contradict to requirements given
above.

My interpretation is that although ISO 8601 permits the use of expanded
year formats, it seems to require that in a given format, the year must
have a fixed number of digits, and it must _always_ include a sign.  In
other words, the receiver should know ahead of time, by prior agreement,
how many digits to expect, and there should _always_ be a sign, even if
the year happens to be in the range 0-.

In order to support years outside the range 0- and in accordance
with ISO 8601, I think that 'date->string' and 'string->date' would need
to be extended to allow the caller to specify how many digits to use in
the expanded 'year' format, presumably by adding a new format escape.
If the specified number of digits is greater than 4, then a sign would
*always* be printed.  'string->date' would know how many digits to
expect, and whether to expect a sign.

Ideally, such an extension of 'date->string' and 'string->date' would be
adopted by upstream SRFI-19.  However, if that's unsuccessful, I'd be
open to unilaterally adding such an extension.  There's precedent for
this in Guile, e.g. see our (srfi srfi-9 gnu) extensions to SRFI-9.

Another question is whether or not we should raise an exception when
attempting to print a year that cannot be represented in the requested
year format.

What do you think?

  Mark

bug#21903: date->string duff ISO 8601 negative years

[Mark H Weaver]

Mark H Weaver  writes:

> Zefram  writes:
>
>> The date->string function from (srfi srfi-19), used on ISO 8601 formats
>> "~1", "~4" and "~5", for years preceding AD 1, has an off-by-one error:
>>
>> scheme@(guile-user)> (use-modules (srfi srfi-19))
>> scheme@(guile-user)> (date->string (julian-day->date 0 0) "~4")
>> $1 = "-4714-11-24T12:00:00Z"
>>
>> The date in question, the JD epoch, is 24 November 4714 BC (in the
>> proleptic Gregorian calendar).  In ISO 8601 format, that year is properly
>> represented as "-4713", not "-4714", because ISO 8601 uses the AD era
>> exclusively.  4714 BC = AD -4713.
>
> I agree that this is definitely a bug, but I'm nervous about deviating
> from the SRFI-19 reference implementation, and therefore probably from
> most other implementations of SRFI-19, in this way.

Also see my comments here:

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21904#17

which mention that ISO 8601 apparently requires that the sender and
receiver agree ahead of time whether an extended format will be used, in
which case a sign is *always* required, even when printing years in the
range 0-.

   Mark

bug#21906: julian-day->date negative input breakage

[Mark H Weaver]

Zefram  writes:

> scheme@(guile-user)> (use-modules (srfi srfi-19))
> scheme@(guile-user)> (julian-day->date 0 0)
> $1 = # year: -4714 zone-offset: 0>
> scheme@(guile-user)> (julian-day->date -1 0)
> $2 = # year: -4714 zone-offset: 0>
> scheme@(guile-user)> (julian-day->date -10 0)
> $3 = # year: -4987 zone-offset: 0>
> scheme@(guile-user)> (julian-day->date -1000 0)
> $4 = # year: -32092 zone-offset: 0>

This is fixed by commit a58c7abd72648f77e4ede5f62a2c4e7969bb7f95 on the
stable-2.2 branch.  I'm closing this bug, but please reopen if
appropriate.

 Thanks!
   Mark

bug#21904: date->string duff ISO 8601 format for non-4-digit years

[Mark H Weaver]

Mark H Weaver  writes:
> Another question is whether or not we should raise an exception when
> attempting to print a year that cannot be represented in the requested
> year format.

I thought about it some more, and I'm now inclined to think that the
approach in your patches is reasonable, or at least it's the least bad
thing we can do when asked to print a year that doesn't fit within the
standard format, given the existing SRFI-19 API.

I also just noticed that the SRFI-19's reference implementation's
formatting of negative years is very badly broken (e.g. it prints "00-2"
when the year field is -2) and Guile had the same behavior after I
applied the fix from upstream to pad the year to 4 digits.

So, for now, I went ahead and implemented the behavior that you
recommended, with one difference: where you hardcode the padding
character to #\0 when formatting years, I use the padding character
specified by the user, following the SRFI-19 reference implementation.

What do you think?

  Mark

bug#21903: date->string duff ISO 8601 negative years

[Mark H Weaver]

Mark H Weaver  writes:

> Mark H Weaver  writes:
>
>> Zefram  writes:
>>
>>> The date->string function from (srfi srfi-19), used on ISO 8601 formats
>>> "~1", "~4" and "~5", for years preceding AD 1, has an off-by-one error:
>>>
>>> scheme@(guile-user)> (use-modules (srfi srfi-19))
>>> scheme@(guile-user)> (date->string (julian-day->date 0 0) "~4")
>>> $1 = "-4714-11-24T12:00:00Z"
>>>
>>> The date in question, the JD epoch, is 24 November 4714 BC (in the
>>> proleptic Gregorian calendar).  In ISO 8601 format, that year is properly
>>> represented as "-4713", not "-4714", because ISO 8601 uses the AD era
>>> exclusively.  4714 BC = AD -4713.
>>
>> I agree that this is definitely a bug, but I'm nervous about deviating
>> from the SRFI-19 reference implementation, and therefore probably from
>> most other implementations of SRFI-19, in this way.
>
> Also see my comments here:
>
>   https://debbugs.gnu.org/cgi/bugreport.cgi?bug=21904#17
>
> which mention that ISO 8601 apparently requires that the sender and
> receiver agree ahead of time whether an extended format will be used, in
> which case a sign is *always* required, even when printing years in the
> range 0-.

Since writing this, I've discovered that the SRFI-19 reference
implementation's formatting of negative years is very badly broken.  For
example, when the year is -2, it prints "00-2".  Guile's behavior was
similarly broken for a short time, after I applied upstream fixes.

Since the current behavior of SRFI-19 and Guile is so clearly broken in
the case of negative years, I'm no longer concerned with maintaining
compatibility with SRFI-19 here.  I also feel more urgency to apply a
fix.

So, I went ahead and implemented your recommended behavior, in commit
a58c7abd72648f77e4ede5f62a2c4e7969bb7f95 on the stable-2.2 branch.

I'm closing this bug now, but please reopen if appropriate.

 Thanks,
   Mark