[I] Spark Test fails `vectorized reader: missing all struct fields` [datafusion-comet]

[via GitHub]

comphead opened a new issue, #1843:
URL: https://github.com/apache/datafusion-comet/issues/1843

   ### Describe the bug
   
   when reading with the specified schema
   ```
     test("vectorized reader: missing all struct fields") {
       Seq(true, false).foreach { offheapEnabled =>
         withSQLConf(
           SQLConf.USE_V1_SOURCE_LIST.key -> "parquet",
           CometConf.COMET_EXEC_ENABLED.key -> "true",
           CometConf.COMET_ENABLED.key -> "true",
           CometConf.COMET_EXPLAIN_FALLBACK_ENABLED.key -> "false",
           CometConf.COMET_NATIVE_SCAN_IMPL.key -> "native_datafusion",
           SQLConf.PARQUET_VECTORIZED_READER_NESTED_COLUMN_ENABLED.key -> 
"true",
           SQLConf.COLUMN_VECTOR_OFFHEAP_ENABLED.key -> 
offheapEnabled.toString) {
           val data = Seq(
             Tuple1((1, "a")),
             Tuple1((2, null)),
             Tuple1(null)
           )
   
           val readSchema = new StructType().add("_1",
             new StructType()
               .add("_3", IntegerType, nullable = true)
               .add("_4", LongType, nullable = true),
             nullable = true)
   
           withParquetFile(data) { file =>
             checkAnswer(spark.read.schema(readSchema).parquet(file),
               Row(null) :: Row(null) :: Row(null) :: Nil
             )
           }
         }
       }
     }
   ```
   
   The test fails on correctness issue
   
   ```
   !== Correct Answer - 3 ==   == Spark Answer - 3 ==
   !struct<>                   struct<_1:struct<_3:int,_4:bigint>>
   ![null]                     [[null,null]]
   ![null]                     [[null,null]]
    [null]                     [null]
   ```
   
   
   ### Steps to reproduce
   
   _No response_
   
   ### Expected behavior
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org