Mendelson, Assaf
Cc: user
Subject: Re: DataFrame select non-existing column
The problem is that I do not know which data frames has the pass.mobile column.
I just list a HDFS directory which contain the parquet files and some files has
the column and some don't. I really don't want
structure and work on that (then you can do withColumn("mobile",...)
> instead of "pass.mobile") but this would change the schema.
>
>
> -Original Message-
> From: Kristoffer Sjögren [mailto:sto...@gmail.com]
> Sent: Saturday, November 19, 2016 4:57 PM
> To: Mendel
om]
Sent: Saturday, November 19, 2016 4:57 PM
To: Mendelson, Assaf
Cc: user
Subject: Re: DataFrame select non-existing column
Thanks. Here's my code example [1] and the printSchema() output [2].
This code still fails with the following message: "No such struct field mobile
in auction,
ng like:
>
> df.withColumn("newColName",pyspark.sql.functions.lit(None))
>
> Assaf.
> -Original Message-
> From: Kristoffer Sjögren [mailto:sto...@gmail.com]
> Sent: Friday, November 18, 2016 9:19 PM
> To: Mendelson, Assaf
> Cc: user
> Subject: Re
DataFrame select non-existing column
Thanks for your answer. I have been searching the API for doing that but I
could not find how to do it?
Could you give me a code snippet?
On Fri, Nov 18, 2016 at 8:03 PM, Mendelson, Assaf
wrote:
> You can always add the columns to old dataframes giving them
Depending on your use case, 'df.withColumn("my_existing_or_new_col",
lit(0l))' could work?
On Fri, Nov 18, 2016 at 11:18 AM, Kristoffer Sjögren
wrote:
> Thanks for your answer. I have been searching the API for doing that
> but I could not find how to do it?
>
> Could you give me a code snippet?
Thanks for your answer. I have been searching the API for doing that
but I could not find how to do it?
Could you give me a code snippet?
On Fri, Nov 18, 2016 at 8:03 PM, Mendelson, Assaf
wrote:
> You can always add the columns to old dataframes giving them null (or some
> literal) as a preproc
You can always add the columns to old dataframes giving them null (or some
literal) as a preprocessing.
-Original Message-
From: Kristoffer Sjögren [mailto:sto...@gmail.com]
Sent: Friday, November 18, 2016 4:32 PM
To: user
Subject: DataFrame select non-existing column
Hi
We have evolve
