I've posted a design doc on SPARK-12297, which builds on what Zoltan posted here earlier. It addresses the parquet issues and also considers current inconsistencies in timestamp behavior for spark across data formats and versions. I believe this incorporates all of the prior concerns and feedback. I should post a PR for it soon.
https://docs.google.com/document/d/1mcbkVo-PSsFh6iOOYx6Rk_34aY25H_zT1E2f7KmLMOU/edit# thanks, Imran On Wed, Aug 16, 2017 at 12:33 PM, Zoltan Ivanfi <z...@cloudera.com> wrote: > Dear Spark Community, > > Based on earlier feedback from the Spark community, we would like to > suggest a short-term fix for the timestamp interoperability problem[1] > between different SQL-on-Hadoop engines. I created a design document[2] and > would like to ask you to review it and let me know of any concerns and/or > suggestions you may have. > > [1] https://issues.apache.org/jira/browse/SPARK-12297 > [2] https://docs.google.com/document/d/1XmyVjr3eOJiNFjVeSnmjIU60Hq- > XiZB03pgi3r1razM/edit > > Thanks, > > Zoltan >
