not work for my purpose. It actually
does nothing.
Kazuaki Ishizaki
From: Jacek Laskowski
To: Kazuaki Ishizaki/Japan/IBM@IBMJP
Cc: user
Date: 2016/08/15 04:56
Subject: Re: Change nullable property in Dataset schema
On Wed, Aug 10, 2016 at 12:04 AM, Kazuaki Ishizaki
wrote:
To: Kazuaki Ishizaki/Japan/IBM@IBMJP
Cc: "user@spark.apache.org"
Date: 2016/08/16 04:35
Subject: Re: Change nullable property in Dataset schema
why do you want the array to have nullable = false? what is the benefit?
On Wed, Aug 3, 2016 at 10:45 AM, Kazuaki Ishiza
ty in Dataframe schema, I
> found the following approaches.
> http://stackoverflow.com/questions/33193958/change-
> nullable-property-of-column-in-spark-dataframe
> https://github.com/apache/spark/pull/13873(Not merged yet)
>
> However, I cannot find how to change nullable property
On Wed, Aug 10, 2016 at 12:04 AM, Kazuaki Ishizaki wrote:
> import testImplicits._
> test("test") {
> val ds1 = sparkContext.parallelize(Seq(Array(1, 1), Array(2, 2),
> Array(3, 3)), 1).toDS
You should just Seq(...).toDS
> val ds2 = ds1.map(e => e)
Why are you e => e (since it's id
azuaki Ishizaki
From: Kazuaki Ishizaki/Japan/IBM@IBMJP
To: user@spark.apache.org
Date: 2016/08/03 23:46
Subject: Change nullable property in Dataset schema
Dear all,
Would it be possible to let me know how to change nullable property in
Dataset?
When I looked for how to change nullabl
https://github.com/apache/spark/pull/13873 (Not merged yet)
However, I cannot find how to change nullable property in Dataset schema.
Even when I wrote the following program, nullable property for "value:
array" in ds2.schema is not changed.
If my understanding is correct, current Spark 2
