I like this proposal. > We'll get agreement across Spark, Hive, and Impala.
Shall we include Parquet and ORC? If they don't support it, it's hard for general query engines like Spark to support it. On Wed, Dec 12, 2018 at 3:36 AM Li Jin <ice.xell...@gmail.com> wrote: > Of course. I added some comments in the doc. > > On Tue, Dec 11, 2018 at 12:01 PM Imran Rashid <im...@therashids.com> > wrote: > >> Hi Li, >> >> thanks for the comments! I admit I had not thought very much about >> python support, its a good point. But I'd actually like to clarify one >> thing about the doc -- though it discusses java types, the point is >> actually about having support for these logical types at the SQL level. >> The doc uses java names instead of SQL names just because there is so much >> confusion around the SQL names, as they haven't been implemented >> consistently. Once there is support for the additional logical types, then >> we'd absolutely want to get the same support in python. >> >> Its great to hear there are existing python types we can map each >> behavior to. Could you add a comment on the doc on each of the types, >> mentioning the equivalent in python? >> >> thanks, >> Imran >> >> On Fri, Dec 7, 2018 at 1:33 PM Li Jin <ice.xell...@gmail.com> wrote: >> >>> Imran, >>> >>> Thanks for sharing this. When working on interop between Spark and >>> Pandas/Arrow in the past, we also faced some issues due to the different >>> definitions of timestamp in Spark and Pandas/Arrow, because Spark timestamp >>> has Instant semantics and Pandas/Arrow timestamp has either LocalDateTime >>> or OffsetDateTime semantics. (Detailed discussion is in the PR: >>> https://github.com/apache/spark/pull/18664#issuecomment-316554156.) >>> >>> For one I am excited to see this effort going but also would love to see >>> interop of Python to be included/considered in the picture. I don't think >>> it adds much to what has already been proposed already because Python >>> timestamps are basically LocalDateTime or OffsetDateTime. >>> >>> Li >>> >>> >>> >>> On Thu, Dec 6, 2018 at 11:03 AM Imran Rashid >>> <iras...@cloudera.com.invalid> wrote: >>> >>>> Hi, >>>> >>>> I'd like to discuss the future of timestamp support in Spark, in >>>> particular with respect of handling timezones in different SQL types. In >>>> a nutshell: >>>> >>>> * There are at least 3 different ways of handling the timestamp type >>>> across timezone changes >>>> * We'd like Spark to clearly distinguish the 3 types (it currently >>>> implements 1 of them), in a way that is backwards compatible, and also >>>> compliant with the SQL standard. >>>> * We'll get agreement across Spark, Hive, and Impala. >>>> >>>> Zoltan Ivanfi (Parquet PMC, also my coworker) has written up a detailed >>>> doc, describing the problem in more detail, the state of various SQL >>>> engines, and how we can get to a better state without breaking any current >>>> use cases. The proposal is good for Spark by itself. We're also going to >>>> the Hive & Impala communities with this proposal, as its better for >>>> everyone if everything is compatible. >>>> >>>> Note that this isn't proposing a specific implementation in Spark as >>>> yet, just a description of the overall problem and our end goal. We're >>>> going to each community to get agreement on the overall direction. Then >>>> each community can figure out specifics as they see fit. (I don't think >>>> there are any technical hurdles with this approach eg. to decide whether >>>> this would be even possible in Spark.) >>>> >>>> Here's a link to the doc Zoltan has put together. It is a bit long, >>>> but it explains how such a seemingly simple concept has become such a mess >>>> and how we can get to a better state. >>>> >>>> >>>> https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit#heading=h.dq3b1mwkrfky >>>> >>>> Please review the proposal and let us know your opinions, concerns and >>>> suggestions. >>>> >>>> thanks, >>>> Imran >>>> >>>
