alamb commented on issue #18070: URL: https://github.com/apache/datafusion/issues/18070#issuecomment-3418219330
> Could you check the size of the inputs for the NLJoin? If this is NLJoin related it could be similar to [#17547](https://github.com/apache/datafusion/issues/17547) and [#17488](https://github.com/apache/datafusion/issues/17488) . I ran `EXPLAIN ANAYZE` The relevant part of the plan I think is like this (tiny left input - 1 row, giant right input 22M rows) ``` NLJoin ProjectionExec (1 row) CoalesceBatchesExec (21917655 rows) ``` The full plan is here: <details><summary>Full Output</summary> <p> ```sql +-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -----+ | plan_type | plan | +-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -----+ | Plan with Metrics | ProjectionExec: expr=[count(Int64(1))@0 as count(*)], metrics=[output_rows=1, elapsed_compute=250ns] | | | AggregateExec: mode=Final, gby=[], aggr=[count(Int64(1))], metrics=[output_rows=1, elapsed_compute=431.127µs] | | | CoalescePartitionsExec, metrics=[output_rows=16, elapsed_compute=12.875µs] | | | AggregateExec: mode=Partial, gby=[], aggr=[count(Int64(1))], metrics=[output_rows=16, elapsed_compute=20.151647ms] | | | ProjectionExec: expr=[], metrics=[output_rows=21018273, elapsed_compute=837.882µs] | | | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=21018273, elapsed_compute=17.333442058s] | | | FilterExec: array_has_any(fsq_category_ids@0, category_ids@1), metrics=[output_rows=21018273, elapsed_compute=173.866690988s] | | | ProjectionExec: expr=[fsq_category_ids@1 as fsq_category_ids, category_ids@0 as category_ids], metrics=[output_rows=21917655, elapsed_compute=2.68371ms] | | | NestedLoopJoinExec: join_type=Right, metrics=[output_rows=21917655, elapsed_compute=35.600322924s, build_input_batches=1, build_input_rows=1, input_batches=2356, input_rows=21917655, output_batches=4696, build_mem_used=55068, build_time=7.882335ms, join_time=35.590911587s] | | | ProjectionExec: expr=[array_agg(categories_raw.category_id)@0 as category_ids], metrics=[output_rows=1, elapsed_compute=43.375µs] | | | GlobalLimitExec: skip=0, fetch=500, metrics=[output_rows=1, elapsed_compute=42ns] | | | AggregateExec: mode=Single, gby=[], aggr=[array_agg(categories_raw.category_id)], metrics=[output_rows=1, elapsed_compute=79.125µs] | | | DataSourceExec: file_groups={1 group: [[release/dt=2025-09-09/categories/parquet/categories.zstd.parquet]]}, projection=[category_id], file_type=parquet, metrics=[output_rows=1245, elapsed_compute=1ns, batches_splitted=0, bytes_scanned=6295, file_open_errors=0, file_scan_errors=0, files_ranges_pruned_statistics=0, num_predicate_creation_errors=0, page_index_rows_matched=0, page_index_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, row_groups_matched_bloom_filter=0, row_groups_matched_statistics=0, row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=0, bloom_filter_eval_time=2ns, metadata_load_time=89.590751ms, page_index_eval_time=2ns, row_pushdown_eval_time=2ns, statistics_eval_time=2ns, time_elapsed_opening=90.102958ms, time_elapsed_processing=7.179959ms, time_elapsed_scanning_total=51.312167ms, time_elapsed_scanning_until_data=51.186792ms] | | | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=21917655, elapsed_compute=80.199596ms] | | | FilterExec: CAST(date_refreshed@0 AS Date32) >= 2024-10-18, projection=[fsq_category_ids@1], metrics=[output_rows=21917655, elapsed_compute=1.918010644s] | | | DataSourceExec: file_groups={16 groups: [[release/dt=2025-09-09/places/parquet/places-00000.zstd.parquet:0..115950054, release/dt=2025-09-09/places/parquet/places-00001.zstd.parquet:0..119625610, release/dt=2025-09-09/places/parquet/places-00002.zstd.parquet:0..111979679, release/dt=2025-09-09/places/parquet/places-00003.zstd.parquet:0..99590199, release/dt=2025-09-09/places/parquet/places-00004.zstd.parquet:0..114422801, ...], [release/dt=2025-09-09/places/parquet/places-00006.zstd.parquet:15283909..123129325, release/dt=2025-09-09/places/parquet/places-00007.zstd.parquet:0..109503606, release/dt=2025-09-09/places/parquet/places-00008.zstd.parquet:0..116880554, release/dt=2025-09-09/places/parquet/places-00009.zstd.parquet:0..107480580, release/dt=2025-09-09/places/parquet/places-00010.zstd.parquet:0..112300670, ...], [release/dt=2025-09-09/places/parquet/places-00012.zstd.parquet:37168422..107444227, release/dt=2025-09-09/places/parque t/places-00013.zstd.parquet:0..110757380, release/dt=2025-09-09/places/parquet/places-00014.zstd.parquet:0..122112592, release/dt=2025-09-09/places/parquet/places-00015.zstd.parquet:0..119978222, release/dt=2025-09-09/places/parquet/places-00016.zstd.parquet:0..128256136, ...], [release/dt=2025-09-09/places/parquet/places-00018.zstd.parquet:44848422..120106873, release/dt=2025-09-09/places/parquet/places-00019.zstd.parquet:0..118004710, release/dt=2025-09-09/places/parquet/places-00020.zstd.parquet:0..118719840, release/dt=2025-09-09/places/parquet/places-00021.zstd.parquet:0..118914980, release/dt=2025-09-09/places/parquet/places-00022.zstd.parquet:0..120055444, ...], [release/dt=2025-09-09/places/parquet/places-00024.zstd.parquet:22110584..112529288, release/dt=2025-09-09/places/parquet/places-00025.zstd.parquet:0..117357063, release/dt=2025-09-09/places/parquet/places-00026.zstd.parquet:0..101323034, release/dt=2025-09-09/places/parquet/places-00027.zstd.parquet:0..113101190, rel ease/dt=2025-09-09/places/parquet/places-00028.zstd.parquet:0..113015606, ...], ...]}, projection=[date_refreshed, fsq_category_ids], file_type=parquet, predicate=CAST(date_refreshed@0 AS Date32) >= 2024-10-18, metrics=[output_rows=105292526, elapsed_compute=16ns, batches_splitted=0, bytes_scanned=327606810, file_open_errors=0, file_scan_errors=0, files_ranges_pruned_statistics=0, num_predicate_creation_errors=0, page_index_rows_matched=0, page_index_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, row_groups_matched_bloom_filter=0, row_groups_matched_statistics=0, row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=0, bloom_filter_eval_time=230ns, metadata_load_time=41.780586406s, page_index_eval_time=15.576µs, row_pushdown_eval_time=230ns, statistics_eval_time=230ns, time_elapsed_opening=2.202947921s, time_elapsed_processing=3.475536283s, time_elapsed_scanning_total=695.001470448s, time_elapsed_scanning_until_data=64.328045 286s] | | | | +-------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -----+ 1 row(s) fetched. Elapsed 58.123 seconds. ``` </p> </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
