Hello Sir/Madam,
I am running one spark application having 3 slaves and one master.
I am wring the my information using the parquet format.
but when i am trying to read it got some error.
Please help me to resolve this problem.
code ;
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._
case class Table(Address: String, Couple_time: Int, WT_ID: Int, WT_Name:
String)
val df2 = sc.textFile("/root/Desktop/database.txt").map(_.split(",")).map(p
=> Table(p(0),p(1).trim.toInt, p(2).trim.toInt, p(3)))toDF
df2.write.parquet("Desktop/database2.parquet")
After that on master computer there is on folder name database2 have the
_success file
and on my slaves
has the following tree
database2.parquet
└── _temporary
└── 0
├── task_201603071435_0000_m_000001
│ └── part-r-00002.gz.parquet
├── task_201603071435_0000_m_000004
│ └── part-r-00005.gz.parquet
└── _temporary
But when i am trying to read this file using following command i get the
error
val df1 = sqlContext.read.parquet("Desktop/database2.parquet")
error
ava.lang.AssertionError: assertion failed: No schema defined, and no
Parquet data file or summary file found under file:/root/database2.parquet.
at scala.Predef$.assert(Predef.scala:179)
at
org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache.org$apache$spark$sql$parquet$ParquetRelation2$MetadataCache$$readSchema(newParquet.scala:443)
at
org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$15.apply(newParquet.scala:385)
at
org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache$$anonfun$15.apply(newParquet.scala:385)
at scala.Option.orElse(Option.scala:257)
at
org.apache.spark.sql.parquet.ParquetRelation2$MetadataCache.refresh(newParquet.scala:385)
at org.apache.spark.sql.parquet.ParquetRelation2.org
$apache$spark$sql$parquet$ParquetRelation2$$metadataCache$lzycompute(newParquet.scala:154)
at org.apache.spark.sql.parquet.ParquetRelation2.org
$apache$spark$sql$parquet$ParquetRelation2$$metadataCache(newParquet.scala:152)
at
org.apache.spark.sql.parquet.ParquetRelation2$$anonfun$dataSchema$1.apply(newParquet.scala:193)
at
org.apache.spark.sql.parquet.ParquetRelation2$$anonfun$dataSchema$1.apply(newParquet.scala:193)
at scala.Option.getOrElse(Option.scala:120)
at
org.apache.spark.sql.parquet.ParquetRelation2.dataSchema(newParquet.scala:193)
at org.apache.spark.sql.sources.HadoopFsRel
Thanks.