Option B is good too, have date as timestamp and format later. Thanks, -Durgesh
> On Jan 22, 2016, at 9:50 AM, Spencer, Alex (Santander) > <[email protected]> wrote: > > Hi Andy, > > Sorry this is in Scala but you may be able to do something similar? I use > Joda's DateTime class. I ran into a lot of difficulties with the serializer, > but if you are an admin on the box you'll have less issues by adding in some > Kryo serializers. > > import org.joda.time > > val dateFormat = format.DateTimeFormat.forPattern("yyyy-MM-dd"); > val tranDate = dateFormat.parseDateTime(someDateString) > > > Alex >  > -----Original Message----- > From: Andrew Holway [mailto:[email protected]] > Sent: 21 January 2016 19:25 > To: [email protected] > Subject: Date / time stuff with spark. > > Hello, > > I am importing this data from HDFS into a data frame with > sqlContext.read.json(). > > {“a": 42, “a": 56, "Id": "621368e2f829f230", “smunkId": > "CKm26sDMucoCFReRGwodbHAAgw", “popsicleRange": "17610", "time": > "2016-01-20T23:59:53+00:00â€} > > I want to do some date/time operations on this json data but I cannot find > clear documentation on how to > > A) specify the “time†field as a date/time in the schema. > B) the format the date should be in to be correctly in the raw data for an > easy import. > > Cheers, > > Andrew > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] For additional > commands, e-mail: [email protected] > > Emails aren't always secure, and they may be intercepted or changed after > they've been sent. Santander doesn't accept liability if this happens. If you > think someone may have interfered with this email, please get in touch with > the > sender another way. This message doesn't create or change any contract. > Santander doesn't accept responsibility for damage caused by any viruses > contained in this email or its attachments. Emails may be monitored. If you've > received this email by mistake, please let the sender know at once that it's > gone to the wrong person and then destroy it without copying, using, or > telling > anyone about its contents. > Santander UK plc Reg. No. 2294747 and Abbey National Treasury Services plc > Reg. > No. 2338548 Registered Offices: 2 Triton Square, Regent's Place, London NW1 > 3AN. > Registered in England. www.santander.co.uk. Authorised by the Prudential > Regulation Authority and regulated by the Financial Conduct Authority and the > Prudential Regulation Authority. FCA Reg. No. 106054 and 146003 respectively. > Santander Sharedealing is a trading name of Abbey Stockbrokers Limited Reg. > No. > 02666793. Registered Office: Kingfisher House, Radford Way, Billericay, Essex > CM12 0GZ. Authorised and regulated by the Financial Conduct Authority. FCA > Reg. > No. 154210. You can check this on the Financial Services Register by visiting > the FCA’s website www.fca.org.uk/register or by contacting the FCA on 0800 > 111 > 6768. Santander UK plc is also licensed by the Financial Supervision > Commission > of the Isle of Man for its branch in the Isle of Man. Deposits held with the > Isle of Man branch are covered by the Isle of Man Depositors’ Compensation > Scheme as set out in the Isle of Man Depositors’ Compensation Scheme > Regulations > 2010. In the Isle of Man, Santander UK plc’s principal place of business is > at > 19/21 Prospect Hill, Douglas, Isle of Man, IM1 1ET. Santander and the flame > logo > are registered trademarks. > Santander Asset Finance plc. Reg. No. 1533123. Registered Office: 2 Triton > Square, Regent’s Place, London NW1 3AN. Registered in England. Santander > Corporate & Commercial is a brand name used by Santander UK plc, Abbey > National > Treasury Services plc and Santander Asset Finance plc. > Ref:[PDB#1-4A] > ТÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÒÐÐ¥FòVç7V'67&–&RÂRÖ֖âW6W"×Vç7V'67&–&T7&²æ6†Ræ÷&pФf÷"FF—F–öæÂ6öÖÖæG2ÂRÖ֖âW6W"Ö†VÇ7&²æ6†Ræ÷&pÐ > Ð --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
