gabotechs opened a new issue, #19849:
URL: https://github.com/apache/datafusion/issues/19849
### Describe the bug
Depending on the number of partitions used for running TPC-DS q14, there is
a different number of leading null columns in the response, for example,
running with 1 partition returns this result:
<details><summary>Running with 1 partition</summary>
```
+---------+------------+------------+---------------+--------------------+------------------+
| channel | i_brand_id | i_class_id | i_category_id | sum_sales |
sum_number_sales |
+---------+------------+------------+---------------+--------------------+------------------+
| | | | | 674173362.5100008 |
155629 |
| catalog | | | | 237410857.46999985 |
46322 |
| catalog | 1001001.0 | | | 1697729.02 |
347 |
| catalog | 1001001.0 | 1.0 | | 855204.24 |
167 |
| catalog | 1001001.0 | 1.0 | 1.0 | 115019.61000000002 |
20 |
| catalog | 1001001.0 | 1.0 | 2.0 | 146344.47000000003 |
27 |
| catalog | 1001001.0 | 1.0 | 3.0 | 22597.190000000002 | 3
|
| catalog | 1001001.0 | 1.0 | 4.0 | 107555.43 |
23 |
| catalog | 1001001.0 | 1.0 | 5.0 | 122521.31 |
25 |
| catalog | 1001001.0 | 1.0 | 6.0 | 16883.97 | 3
|
| catalog | 1001001.0 | 1.0 | 7.0 | 46329.78000000001 | 9
|
| catalog | 1001001.0 | 1.0 | 8.0 | 77861.84999999999 |
13 |
| catalog | 1001001.0 | 1.0 | 9.0 | 99985.34999999999 |
21 |
| catalog | 1001001.0 | 1.0 | 10.0 | 100105.28 |
23 |
| catalog | 1001001.0 | 2.0 | | 125167.21999999999 |
24 |
| catalog | 1001001.0 | 2.0 | 2.0 | 43967.969999999994 | 7
|
| catalog | 1001001.0 | 2.0 | 3.0 | 68565.37999999999 |
14 |
| catalog | 1001001.0 | 2.0 | 5.0 | 12633.87 | 3
|
| catalog | 1001001.0 | 3.0 | | 198685.08000000002 |
43 |
| catalog | 1001001.0 | 3.0 | 1.0 | 11100.789999999999 | 5
|
| catalog | 1001001.0 | 3.0 | 2.0 | 60551.64 |
14 |
| catalog | 1001001.0 | 3.0 | 4.0 | 28455.23 | 4
|
| catalog | 1001001.0 | 3.0 | 6.0 | 36821.60999999999 | 7
|
| catalog | 1001001.0 | 3.0 | 7.0 | 17250.82 | 6
|
| catalog | 1001001.0 | 3.0 | 8.0 | 14426.92 | 4
|
| catalog | 1001001.0 | 3.0 | 9.0 | 30078.07 | 3
|
| catalog | 1001001.0 | 4.0 | | 109585.97 |
31 |
| catalog | 1001001.0 | 4.0 | 2.0 | 45473.850000000006 |
13 |
| catalog | 1001001.0 | 4.0 | 3.0 | 16558.92 | 8
|
| catalog | 1001001.0 | 4.0 | 4.0 | 47553.2 |
10 |
| catalog | 1001001.0 | 5.0 | | 59790.60999999999 |
17 |
| catalog | 1001001.0 | 5.0 | 9.0 | 30112.109999999993 |
12 |
| catalog | 1001001.0 | 5.0 | 10.0 | 29678.5 | 5
|
| catalog | 1001001.0 | 6.0 | | 10261.82 | 3
|
| catalog | 1001001.0 | 6.0 | 9.0 | 10261.82 | 3
|
| catalog | 1001001.0 | 7.0 | | 18244.94 | 3
|
| catalog | 1001001.0 | 7.0 | 7.0 | 18244.94 | 3
|
| catalog | 1001001.0 | 8.0 | | 55768.46 |
13 |
| catalog | 1001001.0 | 8.0 | 7.0 | 28872.489999999998 | 7
|
| catalog | 1001001.0 | 8.0 | 10.0 | 26895.97 | 6
|
| catalog | 1001001.0 | 9.0 | | 30944.190000000002 | 5
|
| catalog | 1001001.0 | 9.0 | 6.0 | 30944.190000000002 | 5
|
| catalog | 1001001.0 | 11.0 | | 82810.87 |
12 |
| catalog | 1001001.0 | 11.0 | 9.0 | 82810.87 |
12 |
| catalog | 1001001.0 | 12.0 | | 38427.52 | 9
|
| catalog | 1001001.0 | 12.0 | 10.0 | 38427.52 | 9
|
| catalog | 1001001.0 | 15.0 | | 112838.1 |
20 |
| catalog | 1001001.0 | 15.0 | 9.0 | 53508.79000000001 | 7
|
| catalog | 1001001.0 | 15.0 | 10.0 | 59329.31 |
13 |
| catalog | 1001002.0 | | | 3527831.329999999 |
706 |
| catalog | 1001002.0 | 1.0 | | 2673969.8899999997 |
530 |
| catalog | 1001002.0 | 1.0 | 1.0 | 2673969.8899999997 |
530 |
| catalog | 1001002.0 | 2.0 | | 140831.90999999997 |
29 |
| catalog | 1001002.0 | 2.0 | 1.0 | 140831.90999999997 |
29 |
| catalog | 1001002.0 | 3.0 | | 320175.87000000005 |
67 |
| catalog | 1001002.0 | 3.0 | 1.0 | 320175.87000000005 |
67 |
| catalog | 1001002.0 | 4.0 | | 133287.96 |
21 |
| catalog | 1001002.0 | 4.0 | 1.0 | 133287.96 |
21 |
| catalog | 1001002.0 | 5.0 | | 16606.9 | 9
|
| catalog | 1001002.0 | 5.0 | 1.0 | 16606.9 | 9
|
| catalog | 1001002.0 | 6.0 | | 15133.01 | 4
|
| catalog | 1001002.0 | 6.0 | 1.0 | 15133.01 | 4
|
| catalog | 1001002.0 | 7.0 | | 24471.260000000002 |
10 |
| catalog | 1001002.0 | 7.0 | 1.0 | 24471.260000000002 |
10 |
| catalog | 1001002.0 | 8.0 | | 63773.05 |
12 |
| catalog | 1001002.0 | 8.0 | 1.0 | 63773.05 |
12 |
| catalog | 1001002.0 | 9.0 | | 9167.189999999999 | 3
|
| catalog | 1001002.0 | 9.0 | 1.0 | 9167.189999999999 | 3
|
| catalog | 1001002.0 | 12.0 | | 29108.42 | 7
|
| catalog | 1001002.0 | 12.0 | 1.0 | 29108.42 | 7
|
| catalog | 1001002.0 | 15.0 | | 31143.450000000004 | 6
|
| catalog | 1001002.0 | 15.0 | 1.0 | 31143.450000000004 | 6
|
| catalog | 1001002.0 | 16.0 | | 70162.42 | 8
|
| catalog | 1001002.0 | 16.0 | 1.0 | 70162.42 | 8
|
| catalog | 1002001.0 | | | 2114110.72 |
380 |
| catalog | 1002001.0 | 1.0 | | 348693.97 |
55 |
| catalog | 1002001.0 | 1.0 | 1.0 | 76392.13 |
14 |
| catalog | 1002001.0 | 1.0 | 2.0 | 118394.33 |
21 |
| catalog | 1002001.0 | 1.0 | 4.0 | 29395.79 | 5
|
| catalog | 1002001.0 | 1.0 | 5.0 | 35541.97 | 4
|
| catalog | 1002001.0 | 1.0 | 6.0 | 26104.36 | 3
|
| catalog | 1002001.0 | 1.0 | 9.0 | 18793.969999999998 | 4
|
| catalog | 1002001.0 | 1.0 | 10.0 | 44071.42 | 4
|
| catalog | 1002001.0 | 2.0 | | 1233961.7000000002 |
225 |
| catalog | 1002001.0 | 2.0 | 1.0 | 239511.02000000002 |
51 |
| catalog | 1002001.0 | 2.0 | 2.0 | 147993.14 |
26 |
| catalog | 1002001.0 | 2.0 | 3.0 | 100086.93 |
17 |
| catalog | 1002001.0 | 2.0 | 4.0 | 53524.420000000006 |
13 |
| catalog | 1002001.0 | 2.0 | 5.0 | 48494.05999999999 |
10 |
| catalog | 1002001.0 | 2.0 | 6.0 | 142857.04 |
20 |
| catalog | 1002001.0 | 2.0 | 7.0 | 116557.98 |
16 |
| catalog | 1002001.0 | 2.0 | 8.0 | 92743.93000000001 |
24 |
| catalog | 1002001.0 | 2.0 | 9.0 | 203943.99000000008 |
38 |
| catalog | 1002001.0 | 2.0 | 10.0 | 88249.19 |
10 |
| catalog | 1002001.0 | 3.0 | | 91054.32 |
17 |
| catalog | 1002001.0 | 3.0 | 2.0 | 25171.130000000005 | 6
|
| catalog | 1002001.0 | 3.0 | 7.0 | 27766.699999999997 | 3
|
| catalog | 1002001.0 | 3.0 | 8.0 | 38116.49 | 8
|
| catalog | 1002001.0 | 4.0 | | 182427.69 |
32 |
| catalog | 1002001.0 | 4.0 | 1.0 | 66896.68000000001 |
15 |
+---------+------------+------------+---------------+--------------------+------------------+
```
</details>
<details><summary>Running with 16 partition</summary>
```
+---------+------------+------------+---------------+--------------------+------------------+
| channel | i_brand_id | i_class_id | i_category_id | sum_sales |
sum_number_sales |
+---------+------------+------------+---------------+--------------------+------------------+
| | | | | 47788579.87000008 |
11068 |
| | | | | 46294358.790000014 |
10609 |
| | | | | 40499040.270000026 |
9321 |
| | | | | 37952602.74999999 |
8889 |
| | | | | 50256292.019999996 |
11540 |
| | | | | 27943616.98000001 |
6397 |
| | | | | 43114338.76999998 |
10000 |
| | | | | 56239021.04000009 |
13003 |
| | | | | 25682800.66000003 |
6012 |
| | | | | 38529122.80999997 |
8922 |
| | | | | 59222982.1600001 |
13528 |
| | | | | 48322926.86000004 |
11228 |
| | | | | 39166012.09999997 |
9010 |
| | | | | 32661391.26000001 |
7453 |
| | | | | 43315152.10000006 |
10008 |
| | | | | 37185124.07 |
8641 |
| catalog | | | | 16671923.720000004 |
3228 |
| catalog | | | | 16630833.009999989 |
3143 |
| catalog | | | | 14038550.020000005 |
2798 |
| catalog | | | | 13135427.839999994 |
2638 |
| catalog | | | | 17604907.440000013 |
3399 |
| catalog | | | | 10119873.489999998 |
1959 |
| catalog | | | | 14698922.719999984 |
2919 |
| catalog | | | | 19534422.180000022 |
3931 |
| catalog | | | | 9075046.950000005 |
1756 |
| catalog | | | | 13829338.200000003 |
2662 |
| catalog | | | | 21769645.880000025 |
4087 |
| catalog | | | | 16890254.58999999 |
3343 |
| catalog | | | | 13897305.679999996 |
2680 |
| catalog | | | | 11719010.150000002 |
2217 |
| catalog | | | | 14773719.71000001 |
2947 |
| catalog | | | | 13021675.889999993 |
2615 |
| catalog | 1001001.0 | | | 188446.33 |
41 |
| catalog | 1001001.0 | | | 53508.79000000001 | 7
|
| catalog | 1001001.0 | | | 100105.28 |
23 |
| catalog | 1001001.0 | | | 114412.26999999999 |
25 |
| catalog | 1001001.0 | | | 77231.7 |
15 |
| catalog | 1001001.0 | | | 174489.14999999997 |
42 |
| catalog | 1001001.0 | | | 206490.30000000002 |
38 |
| catalog | 1001001.0 | | | 45473.850000000006 |
13 |
| catalog | 1001001.0 | | | 146344.47000000003 |
27 |
| catalog | 1001001.0 | | | 152599.38 |
28 |
| catalog | 1001001.0 | | | 206412.37 |
36 |
| catalog | 1001001.0 | | | 119368.20999999999 |
23 |
| catalog | 1001001.0 | | | 45014.149999999994 |
12 |
| catalog | 1001001.0 | | | 50948.8 |
14 |
| catalog | 1001001.0 | | | 16883.97 | 3
|
| catalog | 1001001.0 | 1.0 | | 100105.28 |
23 |
| catalog | 1001001.0 | 1.0 | | 99985.34999999999 |
21 |
| catalog | 1001001.0 | 1.0 | | 107555.43 |
23 |
| catalog | 1001001.0 | 1.0 | | 161349.39 |
29 |
| catalog | 1001001.0 | 1.0 | | 146344.47000000003 |
27 |
| catalog | 1001001.0 | 1.0 | | 122521.31 |
25 |
| catalog | 1001001.0 | 1.0 | | 77861.84999999999 |
13 |
| catalog | 1001001.0 | 1.0 | | 22597.190000000002 | 3
|
| catalog | 1001001.0 | 1.0 | | 16883.97 | 3
|
| catalog | 1001001.0 | 1.0 | 1.0 | 115019.61000000002 |
20 |
| catalog | 1001001.0 | 1.0 | 2.0 | 146344.47000000003 |
27 |
| catalog | 1001001.0 | 1.0 | 3.0 | 22597.190000000002 | 3
|
| catalog | 1001001.0 | 1.0 | 4.0 | 107555.43 |
23 |
| catalog | 1001001.0 | 1.0 | 5.0 | 122521.31 |
25 |
| catalog | 1001001.0 | 1.0 | 6.0 | 16883.97 | 3
|
| catalog | 1001001.0 | 1.0 | 7.0 | 46329.78000000001 | 9
|
| catalog | 1001001.0 | 1.0 | 8.0 | 77861.84999999999 |
13 |
| catalog | 1001001.0 | 1.0 | 9.0 | 99985.34999999999 |
21 |
| catalog | 1001001.0 | 1.0 | 10.0 | 100105.28 |
23 |
| catalog | 1001001.0 | 2.0 | | 68565.37999999999 |
14 |
| catalog | 1001001.0 | 2.0 | | 43967.969999999994 | 7
|
| catalog | 1001001.0 | 2.0 | | 12633.87 | 3
|
| catalog | 1001001.0 | 2.0 | 2.0 | 43967.969999999994 | 7
|
| catalog | 1001001.0 | 2.0 | 3.0 | 68565.37999999999 |
14 |
| catalog | 1001001.0 | 2.0 | 5.0 | 12633.87 | 3
|
| catalog | 1001001.0 | 3.0 | | 60551.64 |
14 |
| catalog | 1001001.0 | 3.0 | | 14426.92 | 4
|
| catalog | 1001001.0 | 3.0 | | 36821.60999999999 | 7
|
| catalog | 1001001.0 | 3.0 | | 30078.07 | 3
|
| catalog | 1001001.0 | 3.0 | | 28455.23 | 4
|
| catalog | 1001001.0 | 3.0 | | 28351.61 |
11 |
| catalog | 1001001.0 | 3.0 | 1.0 | 11100.789999999999 | 5
|
| catalog | 1001001.0 | 3.0 | 2.0 | 60551.64 |
14 |
| catalog | 1001001.0 | 3.0 | 4.0 | 28455.23 | 4
|
| catalog | 1001001.0 | 3.0 | 6.0 | 36821.60999999999 | 7
|
| catalog | 1001001.0 | 3.0 | 7.0 | 17250.82 | 6
|
| catalog | 1001001.0 | 3.0 | 8.0 | 14426.92 | 4
|
| catalog | 1001001.0 | 3.0 | 9.0 | 30078.07 | 3
|
| catalog | 1001001.0 | 4.0 | | 47553.2 |
10 |
| catalog | 1001001.0 | 4.0 | | 45473.850000000006 |
13 |
| catalog | 1001001.0 | 4.0 | | 16558.92 | 8
|
| catalog | 1001001.0 | 4.0 | 2.0 | 45473.850000000006 |
13 |
| catalog | 1001001.0 | 4.0 | 3.0 | 16558.92 | 8
|
| catalog | 1001001.0 | 4.0 | 4.0 | 47553.2 |
10 |
| catalog | 1001001.0 | 5.0 | | 29678.5 | 5
|
| catalog | 1001001.0 | 5.0 | | 30112.109999999993 |
12 |
| catalog | 1001001.0 | 5.0 | 9.0 | 30112.109999999993 |
12 |
| catalog | 1001001.0 | 5.0 | 10.0 | 29678.5 | 5
|
| catalog | 1001001.0 | 6.0 | | 10261.82 | 3
|
| catalog | 1001001.0 | 6.0 | 9.0 | 10261.82 | 3
|
| catalog | 1001001.0 | 7.0 | | 18244.94 | 3
|
| catalog | 1001001.0 | 7.0 | 7.0 | 18244.94 | 3
|
| catalog | 1001001.0 | 8.0 | | 26895.97 | 6
|
+---------+------------+------------+---------------+--------------------+------------------+
```
</details>
I don't think results should be changing depending on the partitions used.
### To Reproduce
This issue can be reproduced with TPC-DS q14 version from DuckDB here:
https://github.com/duckdb/duckdb/blob/main/extension/tpcds/dsdgen/queries/14.sql.
The current one present in this repository is different and does not show the
issue.
Staps for reproducing:
1. Locally, replace the contents of
https://github.com/apache/datafusion/blob/main/datafusion/core/tests/tpc-ds/14.sql
with
https://github.com/duckdb/duckdb/blob/main/extension/tpcds/dsdgen/queries/14.sql
2. Make sure the TPC-DS dataset is available in the benchmark:
```cd benchmarks && ./bench.sh data tpcds```
3. Run the benchmark for TPC-DS q14 in debug mode with just 1 partition to
see results while being on the `benchmarks/` dir:
```cargo run --profile release-nonlto --bin dfbench -- tpcds
--iterations 1 --debug --path data/tpcds_sf1 --query_path
"../datafusion/core/tests/tpc-ds" --query 14 --partitions 1```
4. Run the benchmark for TPC-DS q14 in debug mode with more partitions to
see results while being on the `benchmarks/` dir:
```cargo run --profile release-nonlto --bin dfbench -- tpcds
--iterations 1 --debug --path data/tpcds_sf1 --query_path
"../datafusion/core/tests/tpc-ds" --query 14 --partitions 16```
### Expected behavior
Results should be the same regardless of the amount of partitions
### Additional context
Initially reported here
https://github.com/pydantic/datafusion-distributed/pull/2#discussion_r2693942638
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
