Re: Spark Writing to parquet directory : java.io.IOException: Disk quota exceeded

2017-11-22 Thread Chetan Khatri
Anybody reply on this ? On Tue, Nov 21, 2017 at 3:36 PM, Chetan Khatri wrote: > > Hello Spark Users, > > I am getting below error, when i am trying to write dataset to parquet > location. I have enough disk space available. Last time i was facing same > kind of error which were resolved by incre

Re: Spark Writing to parquet directory : java.io.IOException: Disk quota exceeded

2017-11-22 Thread Vadim Semenov
The error message seems self-explanatory, try to figure out what's the disk quota you have for your user. On Wed, Nov 22, 2017 at 8:23 AM, Chetan Khatri wrote: > Anybody reply on this ? > > On Tue, Nov 21, 2017 at 3:36 PM, Chetan Khatri < > chetan.opensou...@gmail.com> wrote: > >> >> Hello Spark

Spark.ml roadmap 2.3.0 and beyond

2017-11-22 Thread Stephen Boesch
The roadmaps for prior releases e.g. 1.6 2.0 2.1 2.2 were available: 2.2.0 https://issues.apache.org/jira/browse/SPARK-18813 2.1.0 https://issues.apache.org/jira/browse/SPARK-15581 .. It seems those roadmaps were not available per se' for 2.3.0 and later? Is there a different mechanism for that

SparkSQL not support CharType

2017-11-22 Thread 163
Hi, when I use Dataframe with table schema, It goes wrong: val test_schema = StructType(Array( StructField("id", IntegerType, false), StructField("flag", CharType(1), false), StructField("time", DateType, false))); val df = spark.read.format("com.databricks.spark.csv") .schema(test_s