It does not work now, could you file a jira for it?
On Wed, Apr 15, 2015 at 9:29 AM, Suraj Shetiya wrote:
> Thank you :)
>
> That worked. I had another query regarding date being used as filter.
>
> With the new df which has the column cast as date I am unable to apply a
> filter that compares th
Thank you :)
That worked. I had another query regarding date being used as filter.
With the new df which has the column cast as date I am unable to apply a
filter that compares the dates.
The query I am using is :
df.filter(df.Datecol > datetime.date(2015,1,1)).show()
I do not want to use date a
Hey Suraj,
You should use "date" for DataType:
df.withColumn(df.DateCol.cast("date"))
Davies
On Sat, Apr 11, 2015 at 10:57 PM, Suraj Shetiya wrote:
> Humble reminder
>
> On Sat, Apr 11, 2015 at 12:16 PM, Suraj Shetiya
> wrote:
>>
>> Hi,
>>
>> Below is one line from the json file.
>> I have hi
Humble reminder
On Sat, Apr 11, 2015 at 12:16 PM, Suraj Shetiya
wrote:
> Hi,
>
> Below is one line from the json file.
> I have highlighted the field that represents the date.
>
> "YEAR":2015,"QUARTER":1,"MONTH":1,"DAY_OF_MONTH":31,"DAY_OF_WEEK":6,
> *"FL_DATE":"2015-01-31"*,"UNIQUE_CARRIER":"NK
Hi,
Below is one line from the json file.
I have highlighted the field that represents the date.
"YEAR":2015,"QUARTER":1,"MONTH":1,"DAY_OF_MONTH":31,"DAY_OF_WEEK":6,
*"FL_DATE":"2015-01-31"*,"UNIQUE_CARRIER":"NK","AI
RLINE_ID":20416,"CARRIER":"NK","TAIL_NUM":"N614NK","FL_
NUM":126,"ORIGIN_AIRPOR
What's the format you have in json file?
On Fri, Apr 10, 2015 at 6:57 PM, Suraj Shetiya wrote:
> Hi,
>
> In pyspark when if I read a json file using sqlcontext I find that the date
> field is not infered as date instead it is converted to string. And when I
> try to convert it to date using df.wi
