Hello Jon, Are you able to connect to existing Hive and read tables created in hive ? Regards, deepak
On Thu, Mar 26, 2015 at 4:16 PM, Jon Chase <jon.ch...@gmail.com> wrote: > I've filed this as https://issues.apache.org/jira/browse/SPARK-6554 > > On Thu, Mar 26, 2015 at 6:29 AM, Jon Chase <jon.ch...@gmail.com> wrote: > >> Spark 1.3.0, Parquet >> >> I'm having trouble referencing partition columns in my queries. >> >> In the following example, 'probeTypeId' is a partition column. For >> example, the directory structure looks like this: >> >> /mydata >> /probeTypeId=1 >> ...files... >> /probeTypeId=2 >> ...files... >> >> I see the column when I reference load a DF using the /mydata directory >> and call df.printSchema(): >> >> ... >> |-- probeTypeId: integer (nullable = true) >> ... >> >> Parquet is also aware of the column: >> optional int32 probeTypeId; >> >> And this works fine: >> >> sqlContext.sql("select probeTypeId from df limit 1"); >> >> ...as does df.show() - it shows the correct values for the partition >> column. >> >> >> However, when I try to use a partition column in a where clause, I get an >> exception stating that the column was not found in the schema: >> >> sqlContext.sql("select probeTypeId from df where probeTypeId = 1 limit >> 1"); >> >> >> >> ... >> ... >> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 >> in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage >> 0.0 (TID 0, localhost): java.lang.IllegalArgumentException: Column >> [probeTypeId] was not found in schema! >> at parquet.Preconditions.checkArgument(Preconditions.java:47) >> at >> parquet.filter2.predicate.SchemaCompatibilityValidator.getColumnDescriptor(SchemaCompatibilityValidator.java:172) >> at >> parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:160) >> at >> parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:142) >> at >> parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:76) >> at >> parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:41) >> at parquet.filter2.predicate.Operators$Eq.accept(Operators.java:162) >> at >> parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:46) >> at parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:41) >> at parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:22) >> at >> parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:108) >> at >> parquet.filter2.compat.RowGroupFilter.filterRowGroups(RowGroupFilter.java:28) >> at >> parquet.hadoop.ParquetRecordReader.initializeInternalReader(ParquetRecordReader.java:158) >> ... >> ... >> >> >> >> What am I doing wrong? >> >> >> >> >> >> >> >> Here's the full stack trace: >> >> using local[*] for master >> 06:05:55,675 |-INFO in >> ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute >> not set >> 06:05:55,683 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - >> About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender] >> 06:05:55,694 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - >> Naming appender as [STDOUT] >> 06:05:55,721 |-INFO in >> ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default >> type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] >> property >> 06:05:55,768 |-INFO in >> ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of >> ROOT logger to INFO >> 06:05:55,768 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction >> - Attaching appender named [STDOUT] to Logger[ROOT] >> 06:05:55,769 |-INFO in >> ch.qos.logback.classic.joran.action.ConfigurationAction - End of >> configuration. >> 06:05:55,770 |-INFO in >> ch.qos.logback.classic.joran.JoranConfigurator@6aaceffd - Registering >> current configuration as safe fallback point >> >> INFO org.apache.spark.SparkContext Running Spark version 1.3.0 >> WARN o.a.hadoop.util.NativeCodeLoader Unable to load native-hadoop >> library for your platform... using builtin-java classes where applicable >> INFO org.apache.spark.SecurityManager Changing view acls to: jon >> INFO org.apache.spark.SecurityManager Changing modify acls to: jon >> INFO org.apache.spark.SecurityManager SecurityManager: authentication >> disabled; ui acls disabled; users with view permissions: Set(jon); users >> with modify permissions: Set(jon) >> INFO akka.event.slf4j.Slf4jLogger Slf4jLogger started >> INFO Remoting Starting remoting >> INFO Remoting Remoting started; listening on addresses :[akka.tcp:// >> sparkDriver@192.168.1.134:62493] >> INFO org.apache.spark.util.Utils Successfully started service >> 'sparkDriver' on port 62493. >> INFO org.apache.spark.SparkEnv Registering MapOutputTracker >> INFO org.apache.spark.SparkEnv Registering BlockManagerMaster >> INFO o.a.spark.storage.DiskBlockManager Created local directory at >> /var/folders/x7/9hdp8kw9569864088tsl4jmm0000gn/T/spark-150e23b2-ff19-4a51-8cfc-25fb8e1b3f2b/blockmgr-6eea286c-7473-4bda-8886-7250156b68f4 >> INFO org.apache.spark.storage.MemoryStore MemoryStore started with >> capacity 1966.1 MB >> INFO org.apache.spark.HttpFileServer HTTP File server directory is >> /var/folders/x7/9hdp8kw9569864088tsl4jmm0000gn/T/spark-cf4687bd-1563-4ddf-b697-21c96fd95561/httpd-6343b9c9-bb66-43ac-ac43-6da80c7a1f95 >> INFO org.apache.spark.HttpServer Starting HTTP Server >> INFO o.spark-project.jetty.server.Server jetty-8.y.z-SNAPSHOT >> INFO o.s.jetty.server.AbstractConnector Started >> SocketConnector@0.0.0.0:62494 >> INFO org.apache.spark.util.Utils Successfully started service 'HTTP file >> server' on port 62494. >> INFO org.apache.spark.SparkEnv Registering OutputCommitCoordinator >> INFO o.spark-project.jetty.server.Server jetty-8.y.z-SNAPSHOT >> INFO o.s.jetty.server.AbstractConnector Started >> SelectChannelConnector@0.0.0.0:4040 >> INFO org.apache.spark.util.Utils Successfully started service 'SparkUI' >> on port 4040. >> INFO org.apache.spark.ui.SparkUI Started SparkUI at >> http://192.168.1.134:4040 >> INFO org.apache.spark.executor.Executor Starting executor ID <driver> on >> host localhost >> INFO org.apache.spark.util.AkkaUtils Connecting to HeartbeatReceiver: >> akka.tcp://sparkDriver@192.168.1.134:62493/user/HeartbeatReceiver >> INFO o.a.s.n.n.NettyBlockTransferService Server created on 62495 >> INFO o.a.spark.storage.BlockManagerMaster Trying to register BlockManager >> INFO o.a.s.s.BlockManagerMasterActor Registering block manager >> localhost:62495 with 1966.1 MB RAM, BlockManagerId(<driver>, localhost, >> 62495) >> INFO o.a.spark.storage.BlockManagerMaster Registered BlockManager >> INFO o.a.h.conf.Configuration.deprecation mapred.max.split.size is >> deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize >> INFO o.a.h.conf.Configuration.deprecation >> mapred.reduce.tasks.speculative.execution is deprecated. Instead, use >> mapreduce.reduce.speculative >> INFO o.a.h.conf.Configuration.deprecation >> mapred.committer.job.setup.cleanup.needed is deprecated. Instead, use >> mapreduce.job.committer.setup.cleanup.needed >> INFO o.a.h.conf.Configuration.deprecation mapred.min.split.size.per.rack >> is deprecated. Instead, use >> mapreduce.input.fileinputformat.split.minsize.per.rack >> INFO o.a.h.conf.Configuration.deprecation mapred.min.split.size is >> deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize >> INFO o.a.h.conf.Configuration.deprecation mapred.min.split.size.per.node >> is deprecated. Instead, use >> mapreduce.input.fileinputformat.split.minsize.per.node >> INFO o.a.h.conf.Configuration.deprecation mapred.reduce.tasks is >> deprecated. Instead, use mapreduce.job.reduces >> INFO o.a.h.conf.Configuration.deprecation mapred.input.dir.recursive is >> deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive >> INFO o.a.h.hive.metastore.HiveMetaStore 0: Opening raw store with >> implemenation class:org.apache.hadoop.hive.metastore.ObjectStore >> INFO o.a.h.hive.metastore.ObjectStore ObjectStore, initialize called >> INFO DataNucleus.Persistence Property >> hive.metastore.integral.jdo.pushdown unknown - will be ignored >> INFO DataNucleus.Persistence Property datanucleus.cache.level2 unknown - >> will be ignored >> INFO o.a.h.hive.metastore.ObjectStore Setting MetaStore object pin >> classes with >> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" >> INFO o.a.h.h.metastore.MetaStoreDirectSql MySQL check failed, assuming >> we are not on mysql: Lexical error at line 1, column 5. Encountered: "@" >> (64), after : "". >> INFO DataNucleus.Datastore The class >> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as >> "embedded-only" so does not have its own datastore table. >> INFO DataNucleus.Datastore The class >> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as >> "embedded-only" so does not have its own datastore table. >> INFO DataNucleus.Datastore The class >> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as >> "embedded-only" so does not have its own datastore table. >> INFO DataNucleus.Datastore The class >> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as >> "embedded-only" so does not have its own datastore table. >> INFO DataNucleus.Query Reading in results for query >> "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used >> is closing >> INFO o.a.h.hive.metastore.ObjectStore Initialized ObjectStore >> INFO o.a.h.hive.metastore.HiveMetaStore Added admin role in metastore >> INFO o.a.h.hive.metastore.HiveMetaStore Added public role in metastore >> INFO o.a.h.hive.metastore.HiveMetaStore No user is added in admin role, >> since config is empty >> INFO o.a.h.hive.ql.session.SessionState No Tez session required at this >> point. hive.execution.engine=mr. >> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". >> SLF4J: Defaulting to no-operation (NOP) logger implementation >> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for >> further details. >> root >> |-- clientMarketId: integer (nullable = true) >> |-- clientCountryId: integer (nullable = true) >> |-- clientRegionId: integer (nullable = true) >> |-- clientStateId: integer (nullable = true) >> |-- clientAsnId: integer (nullable = true) >> |-- reporterZoneId: integer (nullable = true) >> |-- reporterCustomerId: integer (nullable = true) >> |-- responseCode: integer (nullable = true) >> |-- measurementValue: integer (nullable = true) >> |-- year: integer (nullable = true) >> |-- month: integer (nullable = true) >> |-- day: integer (nullable = true) >> |-- providerOwnerZoneId: integer (nullable = true) >> |-- providerOwnerCustomerId: integer (nullable = true) >> |-- providerId: integer (nullable = true) >> |-- probeTypeId: integer (nullable = true) >> >> ====================================================== >> INFO hive.ql.parse.ParseDriver Parsing command: select probeTypeId from >> df where probeTypeId = 1 limit 1 >> INFO hive.ql.parse.ParseDriver Parse Completed >> ==== results for select probeTypeId from df where probeTypeId = 1 limit 1 >> ====================================================== >> INFO o.a.s.sql.parquet.ParquetRelation2 Reading 33.33333333333333% of >> partitions >> INFO org.apache.spark.storage.MemoryStore ensureFreeSpace(191336) called >> with curMem=0, maxMem=2061647216 >> INFO org.apache.spark.storage.MemoryStore Block broadcast_0 stored as >> values in memory (estimated size 186.9 KB, free 1966.0 MB) >> INFO org.apache.spark.storage.MemoryStore ensureFreeSpace(27530) called >> with curMem=191336, maxMem=2061647216 >> INFO org.apache.spark.storage.MemoryStore Block broadcast_0_piece0 >> stored as bytes in memory (estimated size 26.9 KB, free 1965.9 MB) >> INFO o.a.spark.storage.BlockManagerInfo Added broadcast_0_piece0 in >> memory on localhost:62495 (size: 26.9 KB, free: 1966.1 MB) >> INFO o.a.spark.storage.BlockManagerMaster Updated info of block >> broadcast_0_piece0 >> INFO org.apache.spark.SparkContext Created broadcast 0 from NewHadoopRDD >> at newParquet.scala:447 >> INFO o.a.h.conf.Configuration.deprecation mapred.max.split.size is >> deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize >> INFO o.a.h.conf.Configuration.deprecation mapred.min.split.size is >> deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize >> INFO o.a.s.s.p.ParquetRelation2$$anon$1$$anon$2 Using Task Side Metadata >> Split Strategy >> INFO org.apache.spark.SparkContext Starting job: runJob at >> SparkPlan.scala:121 >> INFO o.a.spark.scheduler.DAGScheduler Got job 0 (runJob at >> SparkPlan.scala:121) with 1 output partitions (allowLocal=false) >> INFO o.a.spark.scheduler.DAGScheduler Final stage: Stage 0(runJob at >> SparkPlan.scala:121) >> INFO o.a.spark.scheduler.DAGScheduler Parents of final stage: List() >> INFO o.a.spark.scheduler.DAGScheduler Missing parents: List() >> INFO o.a.spark.scheduler.DAGScheduler Submitting Stage 0 >> (MapPartitionsRDD[3] at map at SparkPlan.scala:96), which has no missing >> parents >> INFO org.apache.spark.storage.MemoryStore ensureFreeSpace(5512) called >> with curMem=218866, maxMem=2061647216 >> INFO org.apache.spark.storage.MemoryStore Block broadcast_1 stored as >> values in memory (estimated size 5.4 KB, free 1965.9 MB) >> INFO org.apache.spark.storage.MemoryStore ensureFreeSpace(3754) called >> with curMem=224378, maxMem=2061647216 >> INFO org.apache.spark.storage.MemoryStore Block broadcast_1_piece0 >> stored as bytes in memory (estimated size 3.7 KB, free 1965.9 MB) >> INFO o.a.spark.storage.BlockManagerInfo Added broadcast_1_piece0 in >> memory on localhost:62495 (size: 3.7 KB, free: 1966.1 MB) >> INFO o.a.spark.storage.BlockManagerMaster Updated info of block >> broadcast_1_piece0 >> INFO org.apache.spark.SparkContext Created broadcast 1 from broadcast at >> DAGScheduler.scala:839 >> INFO o.a.spark.scheduler.DAGScheduler Submitting 1 missing tasks from >> Stage 0 (MapPartitionsRDD[3] at map at SparkPlan.scala:96) >> INFO o.a.s.scheduler.TaskSchedulerImpl Adding task set 0.0 with 1 tasks >> INFO o.a.spark.scheduler.TaskSetManager Starting task 0.0 in stage 0.0 >> (TID 0, localhost, PROCESS_LOCAL, 1687 bytes) >> INFO org.apache.spark.executor.Executor Running task 0.0 in stage 0.0 >> (TID 0) >> INFO o.a.s.s.p.ParquetRelation2$$anon$1 Input split: >> ParquetInputSplit{part: >> file:/Users/jon/Downloads/sparksql/1partitionsminusgeo/year=2015/month=1/day=14/providerOwnerZoneId=0/providerOwnerCustomerId=0/providerId=287/probeTypeId=1/part-r-00001.parquet >> start: 0 end: 8851183 length: 8851183 hosts: [] requestedSchema: message >> root { >> optional int32 probeTypeId; >> } >> readSupportMetadata: >> {org.apache.spark.sql.parquet.row.requested_schema={"type":"struct","fields":[{"name":"probeTypeId","type":"integer","nullable":true,"metadata":{}}]}, >> org.apache.spark.sql.parquet.row.metadata={"type":"struct","fields":[{"name":"clientMarketId","type":"integer","nullable":true,"metadata":{}},{"name":"clientCountryId","type":"integer","nullable":true,"metadata":{}},{"name":"clientRegionId","type":"integer","nullable":true,"metadata":{}},{"name":"clientStateId","type":"integer","nullable":true,"metadata":{}},{"name":"clientAsnId","type":"integer","nullable":true,"metadata":{}},{"name":"reporterZoneId","type":"integer","nullable":true,"metadata":{}},{"name":"reporterCustomerId","type":"integer","nullable":true,"metadata":{}},{"name":"responseCode","type":"integer","nullable":true,"metadata":{}},{"name":"measurementValue","type":"integer","nullable":true,"metadata":{}},{"name":"year","type":"integer","nullable":true,"metadata":{}},{"name":"month","type":"integer","nullable":true,"metadata":{}},{"name":"day","type":"integer","nullable":true,"metadata":{}},{"name":"providerOwnerZoneId","type":"integer","nullable":true,"metadata":{}},{"name":"providerOwnerCustomerId","type":"integer","nullable":true,"metadata":{}},{"name":"providerId","type":"integer","nullable":true,"metadata":{}},{"name":"probeTypeId","type":"integer","nullable":true,"metadata":{}}]}}} >> ERROR org.apache.spark.executor.Executor Exception in task 0.0 in stage >> 0.0 (TID 0) >> java.lang.IllegalArgumentException: Column [probeTypeId] was not found in >> schema! >> at parquet.Preconditions.checkArgument(Preconditions.java:47) >> ~[parquet-common-1.6.0rc3.jar:na] >> at >> parquet.filter2.predicate.SchemaCompatibilityValidator.getColumnDescriptor(SchemaCompatibilityValidator.java:172) >> ~[parquet-column-1.6.0rc3.jar:na] >> at >> parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:160) >> ~[parquet-column-1.6.0rc3.jar:na] >> at >> parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:142) >> ~[parquet-column-1.6.0rc3.jar:na] >> at >> parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:76) >> ~[parquet-column-1.6.0rc3.jar:na] >> at >> parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:41) >> ~[parquet-column-1.6.0rc3.jar:na] >> at parquet.filter2.predicate.Operators$Eq.accept(Operators.java:162) >> ~[parquet-column-1.6.0rc3.jar:na] >> at >> parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:46) >> ~[parquet-column-1.6.0rc3.jar:na] >> at parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:41) >> ~[parquet-hadoop-1.6.0rc3.jar:na] >> at parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:22) >> ~[parquet-hadoop-1.6.0rc3.jar:na] >> at >> parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:108) >> ~[parquet-column-1.6.0rc3.jar:na] >> at >> parquet.filter2.compat.RowGroupFilter.filterRowGroups(RowGroupFilter.java:28) >> ~[parquet-hadoop-1.6.0rc3.jar:na] >> at >> parquet.hadoop.ParquetRecordReader.initializeInternalReader(ParquetRecordReader.java:158) >> ~[parquet-hadoop-1.6.0rc3.jar:na] >> at >> parquet.hadoop.ParquetRecordReader.initialize(ParquetRecordReader.java:138) >> ~[parquet-hadoop-1.6.0rc3.jar:na] >> at >> org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:133) >> ~[spark-core_2.10-1.3.0.jar:1.3.0] >> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:104) >> ~[spark-core_2.10-1.3.0.jar:1.3.0] >> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66) >> ~[spark-core_2.10-1.3.0.jar:1.3.0] >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >> ~[spark-core_2.10-1.3.0.jar:1.3.0] >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >> ~[spark-core_2.10-1.3.0.jar:1.3.0] >> at >> org.apache.spark.rdd.NewHadoopRDD$NewHadoopMapPartitionsWithSplitRDD.compute(NewHadoopRDD.scala:244) >> ~[spark-core_2.10-1.3.0.jar:1.3.0] >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >> ~[spark-core_2.10-1.3.0.jar:1.3.0] >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >> ~[spark-core_2.10-1.3.0.jar:1.3.0] >> at >> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) >> ~[spark-core_2.10-1.3.0.jar:1.3.0] >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >> ~[spark-core_2.10-1.3.0.jar:1.3.0] >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >> ~[spark-core_2.10-1.3.0.jar:1.3.0] >> at >> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) >> ~[spark-core_2.10-1.3.0.jar:1.3.0] >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >> ~[spark-core_2.10-1.3.0.jar:1.3.0] >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >> ~[spark-core_2.10-1.3.0.jar:1.3.0] >> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) >> ~[spark-core_2.10-1.3.0.jar:1.3.0] >> at org.apache.spark.scheduler.Task.run(Task.scala:64) >> ~[spark-core_2.10-1.3.0.jar:1.3.0] >> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203) >> ~[spark-core_2.10-1.3.0.jar:1.3.0] >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> [na:1.8.0_31] >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> [na:1.8.0_31] >> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_31] >> WARN o.a.spark.scheduler.TaskSetManager Lost task 0.0 in stage 0.0 (TID >> 0, localhost): java.lang.IllegalArgumentException: Column [probeTypeId] was >> not found in schema! >> at parquet.Preconditions.checkArgument(Preconditions.java:47) >> at >> parquet.filter2.predicate.SchemaCompatibilityValidator.getColumnDescriptor(SchemaCompatibilityValidator.java:172) >> at >> parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:160) >> at >> parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:142) >> at >> parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:76) >> at >> parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:41) >> at parquet.filter2.predicate.Operators$Eq.accept(Operators.java:162) >> at >> parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:46) >> at parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:41) >> at parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:22) >> at >> parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:108) >> at >> parquet.filter2.compat.RowGroupFilter.filterRowGroups(RowGroupFilter.java:28) >> at >> parquet.hadoop.ParquetRecordReader.initializeInternalReader(ParquetRecordReader.java:158) >> at >> parquet.hadoop.ParquetRecordReader.initialize(ParquetRecordReader.java:138) >> at >> org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:133) >> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:104) >> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66) >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >> at >> org.apache.spark.rdd.NewHadoopRDD$NewHadoopMapPartitionsWithSplitRDD.compute(NewHadoopRDD.scala:244) >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >> at >> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >> at >> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) >> at org.apache.spark.scheduler.Task.run(Task.scala:64) >> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> >> ERROR o.a.spark.scheduler.TaskSetManager Task 0 in stage 0.0 failed 1 >> times; aborting job >> INFO o.a.s.scheduler.TaskSchedulerImpl Removed TaskSet 0.0, whose tasks >> have all completed, from pool >> INFO o.a.s.scheduler.TaskSchedulerImpl Cancelling stage 0 >> INFO o.a.spark.scheduler.DAGScheduler Job 0 failed: runJob at >> SparkPlan.scala:121, took 0.132538 s >> INFO o.s.j.server.handler.ContextHandler stopped >> o.s.j.s.ServletContextHandler{/metrics/json,null} >> INFO o.s.j.server.handler.ContextHandler stopped >> o.s.j.s.ServletContextHandler{/stages/stage/kill,null} >> INFO o.s.j.server.handler.ContextHandler stopped >> o.s.j.s.ServletContextHandler{/,null} >> INFO o.s.j.server.handler.ContextHandler stopped >> o.s.j.s.ServletContextHandler{/static,null} >> INFO o.s.j.server.handler.ContextHandler stopped >> o.s.j.s.ServletContextHandler{/executors/threadDump/json,null} >> INFO o.s.j.server.handler.ContextHandler stopped >> o.s.j.s.ServletContextHandler{/executors/threadDump,null} >> INFO o.s.j.server.handler.ContextHandler stopped >> o.s.j.s.ServletContextHandler{/executors/json,null} >> INFO o.s.j.server.handler.ContextHandler stopped >> o.s.j.s.ServletContextHandler{/executors,null} >> INFO o.s.j.server.handler.ContextHandler stopped >> o.s.j.s.ServletContextHandler{/environment/json,null} >> INFO o.s.j.server.handler.ContextHandler stopped >> o.s.j.s.ServletContextHandler{/environment,null} >> INFO o.s.j.server.handler.ContextHandler stopped >> o.s.j.s.ServletContextHandler{/storage/rdd/json,null} >> INFO o.s.j.server.handler.ContextHandler stopped >> o.s.j.s.ServletContextHandler{/storage/rdd,null} >> INFO o.s.j.server.handler.ContextHandler stopped >> o.s.j.s.ServletContextHandler{/storage/json,null} >> INFO o.s.j.server.handler.ContextHandler stopped >> o.s.j.s.ServletContextHandler{/storage,null} >> INFO o.s.j.server.handler.ContextHandler stopped >> o.s.j.s.ServletContextHandler{/stages/pool/json,null} >> INFO o.s.j.server.handler.ContextHandler stopped >> o.s.j.s.ServletContextHandler{/stages/pool,null} >> INFO o.s.j.server.handler.ContextHandler stopped >> o.s.j.s.ServletContextHandler{/stages/stage/json,null} >> INFO o.s.j.server.handler.ContextHandler stopped >> o.s.j.s.ServletContextHandler{/stages/stage,null} >> INFO o.s.j.server.handler.ContextHandler stopped >> o.s.j.s.ServletContextHandler{/stages/json,null} >> INFO o.s.j.server.handler.ContextHandler stopped >> o.s.j.s.ServletContextHandler{/stages,null} >> INFO o.s.j.server.handler.ContextHandler stopped >> o.s.j.s.ServletContextHandler{/jobs/job/json,null} >> INFO o.s.j.server.handler.ContextHandler stopped >> o.s.j.s.ServletContextHandler{/jobs/job,null} >> INFO o.s.j.server.handler.ContextHandler stopped >> o.s.j.s.ServletContextHandler{/jobs/json,null} >> INFO o.s.j.server.handler.ContextHandler stopped >> o.s.j.s.ServletContextHandler{/jobs,null} >> INFO org.apache.spark.ui.SparkUI Stopped Spark web UI at >> http://192.168.1.134:4040 >> INFO o.a.spark.scheduler.DAGScheduler Stopping DAGScheduler >> INFO o.a.s.MapOutputTrackerMasterActor MapOutputTrackerActor stopped! >> INFO org.apache.spark.storage.MemoryStore MemoryStore cleared >> INFO o.apache.spark.storage.BlockManager BlockManager stopped >> INFO o.a.spark.storage.BlockManagerMaster BlockManagerMaster stopped >> INFO o.a.s.s.OutputCommitCoordinator$OutputCommitCoordinatorActor >> OutputCommitCoordinator stopped! >> INFO org.apache.spark.SparkContext Successfully stopped SparkContext >> >> org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 >> in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage >> 0.0 (TID 0, localhost): java.lang.IllegalArgumentException: Column >> [probeTypeId] was not found in schema! >> at parquet.Preconditions.checkArgument(Preconditions.java:47) >> at >> parquet.filter2.predicate.SchemaCompatibilityValidator.getColumnDescriptor(SchemaCompatibilityValidator.java:172) >> at >> parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumn(SchemaCompatibilityValidator.java:160) >> at >> parquet.filter2.predicate.SchemaCompatibilityValidator.validateColumnFilterPredicate(SchemaCompatibilityValidator.java:142) >> at >> parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:76) >> at >> parquet.filter2.predicate.SchemaCompatibilityValidator.visit(SchemaCompatibilityValidator.java:41) >> at parquet.filter2.predicate.Operators$Eq.accept(Operators.java:162) >> at >> parquet.filter2.predicate.SchemaCompatibilityValidator.validate(SchemaCompatibilityValidator.java:46) >> at parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:41) >> at parquet.filter2.compat.RowGroupFilter.visit(RowGroupFilter.java:22) >> at >> parquet.filter2.compat.FilterCompat$FilterPredicateCompat.accept(FilterCompat.java:108) >> at >> parquet.filter2.compat.RowGroupFilter.filterRowGroups(RowGroupFilter.java:28) >> at >> parquet.hadoop.ParquetRecordReader.initializeInternalReader(ParquetRecordReader.java:158) >> at >> parquet.hadoop.ParquetRecordReader.initialize(ParquetRecordReader.java:138) >> at >> org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:133) >> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:104) >> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66) >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >> at >> org.apache.spark.rdd.NewHadoopRDD$NewHadoopMapPartitionsWithSplitRDD.compute(NewHadoopRDD.scala:244) >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >> at >> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >> at >> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) >> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) >> at org.apache.spark.scheduler.Task.run(Task.scala:64) >> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> at java.lang.Thread.run(Thread.java:745) >> >> Driver stacktrace: >> at org.apache.spark.scheduler.DAGScheduler.org >> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1203) >> at >> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192) >> at >> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1191) >> at >> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) >> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) >> at >> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1191) >> at >> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693) >> at >> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693) >> at scala.Option.foreach(Option.scala:236) >> at >> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693) >> at >> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393) >> at >> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354) >> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) >> >> Mar 26, 2015 6:06:02 AM INFO: parquet.filter2.compat.FilterCompat: >> Filtering using predicate: eq(probeTypeId, 1) >> Mar 26, 2015 6:06:02 AM WARNING: parquet.hadoop.ParquetRecordReader: Can >> not initialize counter due to context is not a instance of >> TaskInputOutputContext, but is >> org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl >> Mar 26, 2015 6:06:02 AM INFO: parquet.filter2.compat.FilterCompat: >> Filtering using predicate: eq(probeTypeId, 1) >> >> Process finished with exit code 255 >> >> > -- Deepak
