Option 1 sounds good to me. Let's take to a vote.
On Tue, Apr 2, 2019 at 8:53 PM Micah Kornfield <emkornfi...@gmail.com>
wrote:
> Based on the discussion so far, my attempt at concrete Schema proposals
> below. Jacques I think summarizes what we've discussed, apologies if
> I've misunderstood. Wes would Option 1 work to support the Pandas Time
> Delta use-case? I'm leaning towards Option 1 if it satisfies everyone (but
> happy to implement whatever we come to a consensus on).
>
> ** Option 1: New Type: **
> /// An absolute length of time unrelated to any calendar artifacts. For
> the purposes
> /// of Arrow Implementations, adding this value to a Timestamp ("t1")
> naively (i.e. simply summing
> /// the two number) is acceptable even though in some cases the resulting
> Timestamp (t2) would
> /// not account for leap-seconds during the elapsed time between "t1" and
> "t2". Similarly, representing
> /// the difference between two Unix timestamp is acceptable, but would
> yield a value that is possibly a few seconds
> /// off from the true elapsed time.
> ///
> /// The resolution defaults to
> /// millisecond, but can be any of the other supported TimeUnit values as
> /// with Timestamp and Time types. This type is always represented as
> /// an 8-byte integer.
> table DurationInterval {
> unit: TimeUnit = MILLISECOND;
> }
>
> ** Option 2: New TimeDelta enum on Interval Unit (strong definition around
> leap-seconds): **
>
> enum IntervalUnit: short { YEAR_MONTH, DAY_TIME, TIME_DELTA}
> // A "calendar" interval which models types that don't necessarily
> // have a precise duration without the context of a base timestamp (e.g.
> // days can differ in length during day light savings time transitions).
> In the case
> // of TimeDelta it is possible no precise definition is possible if the
> base timestamp occurs
> // at an instant when a leap second was added (but would only differ by at
> most 1 second).
> // YEAR_MONTH - Indicates the number of elapsed whole months, stored as
> // 4-byte integers.
> // DAY_TIME - Indicates the number of elapsed days and milliseconds,
> // stored as 2 contiguous 32-bit integers (8-bytes in total). Support
> // of this IntervalUnit is not required for full arrow compatibility.
> // TIME_DELTA - Indicates absolute time difference between Unix Timstamps
> (i.e. excluding leap seconds). This value is always represented as an
> 8-byte integer.
> table Interval {
> unit: IntervalUnit;
> resolution: TimeUnit // Only relevant for TIME_DELTA
> }
>
> On Tue, Apr 2, 2019 at 10:03 AM Wes McKinney <wesmck...@gmail.com> wrote:
>
> > Since there were some mentions of leap seconds:
> >
> > I think the intent of the timedelta/duration type should be to express
> > the difference between UNIX timestamps (from second to nanosecond
> > resolution), which don't include leap seconds. We use the
> > timedelta64[ns] type in pandas for example, which is a
> > nanosecond-resolution difference of UNIX timestamps.
> >
> > On Tue, Apr 2, 2019 at 10:05 AM Jacques Nadeau <jacq...@apache.org>
> wrote:
> > >
> > > >
> > > > I could go either way, it has some benefits for forward
> compatibility I
> > > > suppose, but on the other hand YAGNI, if you feel strongly, I'm ok
> > > > including it. However, the more optional fields we have for a
> specific
> > > > enum value, makes me lean more towards a new type instead of just an
> > enum.
> > > >
> > > I'm okay with skipping for now. Appreciate the focus on only what we
> > > actually need.
> > >
> > >
> > >
> > > > Could you elaborate on defining standard arithmetic conversions
> between
> > > > time-delta/duration in seconds and other time unit (days, months,
> > years) as
> > > > part of the standard/format, I'm still not sure I understand what the
> > > > use-case is here.
> > > >
> > >
> > > Here goes nothing...
> > >
> > > Seems like there are two options for durations:
> > > 1) they aren't related to any other type
> > > 2) they have a relationship to timestamps and dates.
> > >
> > > If 1, then the only thing I could understand is real world duration how
> > > seconds are defined (and fractions thereof). E.g. [1] :D. In this
> > > situation, there is no way to express any unit of time of higher
> > > granularity than a second (e.g. days) since it is up to application
> > > implementer to define the relationship. This severely limits the
> > > expressiveness of the concept. (I can't ever use something
> TimeUnit.DAYS)
> > > and stops the ability to cover the existing interval YEAR_MONTH type I
> > > believe (since it has a resolution of months).
> > >
> > > If 2, then we must define the canonical value of ts + duration,
> otherwise
> > > duration are somewhat meaningless, thus the proposed translation chart
> > > (which causes its own oddities depending on the resolution of the time
> > type
> > > you are adding to).
> > >
> > > That being said, having started to remember previous discussions on
> this,
> > > I'm most inclined to simply pick #1 and ignore the need for anything
> > more.
> > > The curiousness of interval math in database systems underscores the
> fact
> > > that it apparently doesn't matter that much. In most cases, today + 3
> > > months is close enough to today + 90 days for government work.
> > >
> > > Let's +2 a patch and get it merged quickly so we never have to think
> > about
> > > this again :)
> > >
> > > [1] "the duration of 9,192,631,770 periods
> > > <https://en.wikipedia.org/wiki/Frequency> of the radiation
> > corresponding to
> > > the transition between the two hyperfine levels
> > > <https://en.wikipedia.org/wiki/Hyperfine_structure> of the ground
> state
> > of
> > > the caesium-133 <https://en.wikipedia.org/wiki/Caesium-133> atom" (at
> a
> > > temperature of 0 K <https://en.wikipedia.org/wiki/Absolute_zero>)
> > >
> > > >
> >
>
