Hi Evelina, Do you use Spark HBase Connector ( hbase-spark ) as part of the unit-test setup?
regards, Guru On Wed, Oct 30, 2024 at 5:35 PM Evelina Dumitrescu <evelina.dumitrescu....@gmail.com> wrote: > > Hello, > > TLDR; The question is asked also here: > https://stackoverflow.com/questions/79139516/incompatible-configuration-used-between-spark-and-hbasetestingutility > > We are using the MiniDFSCluster and MiniHbaseCluster from HBaseTestingUtility > to run unit tests for our Spark jobs. > The Spark configuration that we use is : > > conf.set("spark.sql.catalogImplementation", "hive") > .set("spark.sql.warehouse.dir", getWarehousePath) > .set("javax.jdo.option.ConnectionURL", > s"jdbc:derby:;databaseName=$getMetastorePath;create=true") > .set("shark.test.data.path", dataFilePath) > .set("hive.exec.dynamic.partition.mode", "nonstrict") > .set("spark.kryo.registrator", "CustomKryoRegistrar") > .set("spark.serializer", > "org.apache.spark.serializer.KryoSerializer") > > .registerKryoClasses(Array(classOf[org.apache.hadoop.hbase.client.Result])) > > For the MiniDFSCluster and MiniHbaseCluster we use the default > HbaseTestingUtility configuration. > The release versions that we use are: > - hbase-testing-util Cloudera CDP 2.4.6.7.2.16.0-287 > - Spark 2.11 > > > > In our unit tests, when we try to run a Spark job that reads Hive data, we > get the following exception: > > > ``` > org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in > stage 14.0 failed 1 times, most recent failure: Lost task 0.0 in stage 14.0 > (TID 14, localhost, executor driver): java.lang.Unsuppo > rtedOperationException: Byte-buffer read unsupported by > org.apache.hadoop.fs.BufferedFSInputStream > > at > org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:158) > > at > org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:154) > > at > org.apache.parquet.hadoop.util.H2SeekableInputStream$H2Reader.read(H2SeekableInputStream.java:81) > > at > org.apache.parquet.hadoop.util.H2SeekableInputStream.readFully(H2SeekableInputStream.java:90) > > at > org.apache.parquet.hadoop.util.H2SeekableInputStream.readFully(H2SeekableInputStream.java:75) > > at > org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:546) > > at > org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:516) > > at > org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:510) > > at > org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:459) > > at > org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anonfun$buildReaderWithPartitionValues$1.footerFileMetaData$lzycompute$1(ParquetFileFormat.scala:371) > > at > org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anonfun$buildReaderWithPartitionValues$1.footerFileMetaData$1(ParquetFileFormat.scala:370) > > at > org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anonfun$buildReaderWithPartitionValues$1.apply(ParquetFileFormat.scala:374) > > at > org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anonfun$buildReaderWithPartitionValues$1.apply(ParquetFileFormat.scala:352) > > at > org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:124) > > at > org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:177) > > at > org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101) > > at > org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.scan_nextBatch_0$(Unknown > Source) > > at > org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown > Source) > > at > org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) > > at > org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:645) > > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:270) > > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:262) > > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858) > > at > org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:858) > > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) > > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346) > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:310) > > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) > > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346) > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:310) > > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) > > at org.apache.spark.scheduler.Task.run(Task.scala:123) > > at > org.apache.spark.executor.Executor$TaskRunner$$anonfun$12.apply(Executor.scala:456) > > at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1334) > > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:462) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:750) > > > Driver stacktrace: > at > org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1935) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1923) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1922) > at > scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) > at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) > at > org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1922) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:953) > at > org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:953) > at scala.Option.foreach(Option.scala:257) > at > org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:953) > ... > Cause: java.lang.UnsupportedOperationException: Byte-buffer read > unsupported by org.apache.hadoop.fs.BufferedFSInputStream > at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:158) > at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:154) > at > org.apache.parquet.hadoop.util.H2SeekableInputStream$H2Reader.read(H2SeekableInputStream.java:81) > at > org.apache.parquet.hadoop.util.H2SeekableInputStream.readFully(H2SeekableInputStream.java:90) > at > org.apache.parquet.hadoop.util.H2SeekableInputStream.readFully(H2SeekableInputStream.java:75) > at > org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:546) > at > org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:516) > at > org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:510) > at > org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:459) > at > org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anonfun$buildReaderWithPartitionValues$1.footerFileMetaData$lzycompute$1(ParquetFileFormat.scala:371) > ``` > > > Is there an incompatible configuration used between Spark, MiniDFSCluster and > MiniHbaseCluster ? --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
