Hi Ji, This sounds like a universal requirement, as 64-bit is not sufficient to hold the precision for nano-second.
For the extension type, we have two choices: 1. Extending struct(int64, int32), which represents the design of SoA (Struct of Arrays). 2. Extending fixed width binary(12), which represents the design of AoS (Array of Structs) Given the universal requirement, I'd prefer a new type. Best, Liya Fan On Wed, Aug 5, 2020 at 11:18 AM Ji Liu <tianc...@apache.org> wrote: > Hi all, > > Now in Arrow Timestamp type, it support different TimeUnit(seconds, > milliseconds, microseconds, nanoseconds) with int64 type for storage. In > most cases this is enough, but if the timestamp value range of external > system exceeds int64_t::max, then it's impossible to directly convert to > Arrow Timestamp, consider the following user case: > > A timestamp in other system with int64 + int32(stores milliseconds and > nanoseconds) can represent data from 0000-00-00 to 9999-12-31 > 23:59:59.999999999, if we want to convert type like this, how should we do? > One probably create an extension type with struct(int64, int32) for > storage. > > Besides ExtensionType, are we considering extending our Timestamp for wider > range or maybe a new type for cases above? > > > Thanks, > Ji Liu >
