mbutrovich commented on code in PR #7691:
URL: https://github.com/apache/datafusion-comet/pull/7691#discussion_r4645809519


##########
spark/src/test/resources/sql-tests/expressions/hash/hash_fixed_width_nulls.sql:
##########
@@ -0,0 +1,47 @@
+-- Licensed to the Apache Software Foundation (ASF) under one
+-- or more contributor license agreements.  See the NOTICE file
+-- distributed with this work for additional information
+-- regarding copyright ownership.  The ASF licenses this file
+-- to you under the Apache License, Version 2.0 (the
+-- "License"); you may not use this file except in compliance
+-- with the License.  You may obtain a copy of the License at
+--
+--   http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing,
+-- software distributed under the License is distributed on an
+-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+-- KIND, either express or implied.  See the License for the
+-- specific language governing permissions and limitations
+-- under the License.
+
+-- Plain encoding keeps the columns off the dictionary path, so they reach the 
64-row chunk loop
+-- in `hash_fixed_width_with_nulls`.
+-- Config: parquet.enable.dictionary=false
+
+-- One file of 1300 rows, so each block of 256 rows is four aligned 64-row 
chunks and the last 20
+-- rows are a partial chunk. Each block has its own validity pattern. Blocks 1 
and 2 have 16 and
+-- 15 valid rows per chunk, one at and one below `MIN_VALID_FOR_DENSE`, so 
they choose different
+-- branches. Update them if that constant changes.
+statement
+CREATE TABLE test_hash_fixed_width_nulls USING parquet AS
+SELECT /*+ COALESCE(1) */ id, IF(is_null, NULL, CAST(id * 7919 - 5000000 AS 
INT)) AS i,
+  IF(is_null, NULL, id * 1000000007 - 1000000000000) AS l,
+  IF(is_null, NULL, CASE id % 97 WHEN 0 THEN double('-0.0') WHEN 1 THEN 
double('NaN')
+    ELSE id / 7.0 - 50 END) AS d
+FROM (
+  SELECT id, CASE CAST(id DIV 256 AS INT)

Review Comment:
   > Adding a `CAST(d AS FLOAT)` column, a `smallint` column that includes its 
bounds, and a couple of `double` rows with `Infinity` and `-Infinity` would 
exercise those instantiations against Spark.
   
   Thanks, `hash_fixed_width_nulls.sql` now has `float`, `tinyint` and 
`smallint` with their bounds, `date`, and `timestamp` columns, and `Infinity` 
and `-Infinity` rows in `d` and `f`. It also hashes `-d` and `-f`, which sets 
the sign bit of the NaN rows, and hashes the new columns together with both 
hashes, so each column after the first reaches the chunk loop with existing 
hashes. Year-month and day-time intervals reach the loop as well, but a CTAS 
can't write them to Parquet.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to