Hey Michael, Thanks for the clarification. I was actually assuming the query would fail. Ok, so this means I will have to do the validation in an RDD transformation feeding into the SchemaRDD.
On Wed, Dec 10, 2014 at 6:27 PM, Michael Armbrust <mich...@databricks.com> wrote: > As the scala doc for applySchema says, "It is important to make sure that > the structure of every [[Row]] of the provided RDD matches the provided > schema. Otherwise, there will be runtime exceptions." We don't check as > doing runtime reflection on all of the data would be very expensive. You > will only get errors if you try to manipulate the data, but otherwise it > will pass it though. > > I have written some debugging code (developer API, not guaranteed to be > stable) though that you can use. > > import org.apache.spark.sql.execution.debug._ > schemaRDD.typeCheck() > > On Wed, Dec 10, 2014 at 6:19 PM, Alessandro Baretta <alexbare...@gmail.com > > wrote: > >> Hello, >> >> I defined a SchemaRDD by applying a hand-crafted StructType to an RDD. >> Some >> of the Rows in the RDD are malformed--that is, they do not conform to the >> schema defined by the StructType. When running a select statement on this >> SchemaRDD I would expect SparkSQL to either reject the malformed rows or >> fail. Instead, it returns whatever data it finds, even if malformed. Is >> this the desired behavior? Is there no method in SparkSQL to check for >> validity with respect to the schema? >> >> Thanks. >> >> Alex >> > >
