alamb commented on code in PR #22032:
URL: https://github.com/apache/datafusion/pull/22032#discussion_r3306016820
##########
benchmarks/src/benchmark_runner/cli.rs:
##########
@@ -17,65 +17,973 @@
//! CLI construction and argument conversion for `benchmark_runner`.
//!
-//! This module owns the clap command tree for the initial runner surface:
-//! top-level help and suite listing.
+//! This module owns the clap command tree for `list`, `info`, and `query`.
+//! Suite options are loaded from `.suite` files and registered as dynamic clap
+//! arguments so help output can describe suite-specific flags. Short aliases
+//! with more than one character, such as `-sf`, are normalized before clap
+//! parses the command because clap only supports one-character short flags
+//! directly.
-use clap::builder::styling::{AnsiColor, Styles};
-use clap::{ArgMatches, Command};
+use crate::benchmark_runner::native::NativeBenchmarks;
+use crate::benchmark_runner::output::write_help_section;
+use crate::benchmark_runner::style::{HELP_STYLES, header};
+use crate::benchmark_runner::suite::{SuiteConfig, SuiteOption, SuiteRegistry};
+use crate::util::CommonOpt;
+use clap::{Arg, ArgAction, ArgMatches, Args, Command};
use datafusion_common::{Result, exec_datafusion_err};
+use std::collections::{BTreeMap, BTreeSet};
+use std::ffi::{OsStr, OsString};
+use std::fmt::Write as _;
-const HELP_STYLES: Styles = Styles::styled()
- .header(AnsiColor::Green.on_default().bold())
- .usage(AnsiColor::Green.on_default().bold())
- .literal(AnsiColor::Cyan.on_default().bold())
- .placeholder(AnsiColor::Cyan.on_default());
+const BENCHMARK_TARGET_HEADING: &str = "SQL Benchmark Target";
Review Comment:
I foundsome of these arguments confusing like why is `DATAFUSION_ARG_IDS`
and `RUN_ENV_ARG_IDS` almost, but not quite the same? And what are they used
for?
Maybe we could clean them up or add some comments
##########
benchmarks/src/benchmark_runner/cli.rs:
##########
@@ -17,65 +17,973 @@
//! CLI construction and argument conversion for `benchmark_runner`.
//!
-//! This module owns the clap command tree for the initial runner surface:
-//! top-level help and suite listing.
+//! This module owns the clap command tree for `list`, `info`, and `query`.
+//! Suite options are loaded from `.suite` files and registered as dynamic clap
+//! arguments so help output can describe suite-specific flags. Short aliases
+//! with more than one character, such as `-sf`, are normalized before clap
+//! parses the command because clap only supports one-character short flags
+//! directly.
-use clap::builder::styling::{AnsiColor, Styles};
-use clap::{ArgMatches, Command};
+use crate::benchmark_runner::native::NativeBenchmarks;
+use crate::benchmark_runner::output::write_help_section;
+use crate::benchmark_runner::style::{HELP_STYLES, header};
+use crate::benchmark_runner::suite::{SuiteConfig, SuiteOption, SuiteRegistry};
+use crate::util::CommonOpt;
+use clap::{Arg, ArgAction, ArgMatches, Args, Command};
use datafusion_common::{Result, exec_datafusion_err};
+use std::collections::{BTreeMap, BTreeSet};
+use std::ffi::{OsStr, OsString};
+use std::fmt::Write as _;
-const HELP_STYLES: Styles = Styles::styled()
- .header(AnsiColor::Green.on_default().bold())
- .usage(AnsiColor::Green.on_default().bold())
- .literal(AnsiColor::Cyan.on_default().bold())
- .placeholder(AnsiColor::Cyan.on_default());
+const BENCHMARK_TARGET_HEADING: &str = "SQL Benchmark Target";
+const RUNNER_OPTIONS_HEADING: &str = "Runner Options";
+const DATAFUSION_OPTIONS_HEADING: &str = "DataFusion Options";
+const SUITE_OPTIONS_HEADING: &str = "Suite Options";
+const DATAFUSION_ARG_IDS: &[&str] = &[
+ "partitions",
+ "batch_size",
+ "mem_pool_type",
+ "memory_limit",
+ "sort_spill_reservation_bytes",
+ "debug",
+ "simulate_latency",
+];
+const RUN_ENV_ARG_IDS: &[&str] = &[
+ "iterations",
+ "partitions",
+ "batch_size",
+ "mem_pool_type",
+ "memory_limit",
+ "sort_spill_reservation_bytes",
+ "debug",
+ "simulate_latency",
+];
+const HIDDEN_RUN_ARG_IDS: &[&str] = &["debug", "sort_spill_reservation_bytes"];
Review Comment:
It would also really help here to have a comment explaining why they are
hidden
##########
benchmarks/sql_benchmarks/tpch/tpch.suite:
##########
@@ -1,16 +1,48 @@
-name = "tpch"
description = "TPC-H SQL benchmarks"
+# Query patterns control how numeric QUERY_ID values map to .benchmark files
+# during discovery and command resolution. Use exactly one query-id token:
+# - {QUERY_ID_PADDED}: two-digit ids, such as q01.benchmark
+# - {QUERY_ID}: unpadded ids, such as query-1.benchmark
+# If omitted, this defaults to q{QUERY_ID_PADDED}.benchmark.
+query_pattern = "q{QUERY_ID_PADDED}.benchmark"
+
+# Path replacements define path-like variables used while parsing benchmark
+# files. Relative paths are resolved from this suite file's directory and then
+# passed to SqlBenchmark's replacement mapping, so benchmark SQL can refer to
+# values such as ${DATA_DIR}. For timed runs, the runner's --path/-p option
+# overrides DATA_DIR.
+[path_replacements]
+DATA_DIR = "../../data"
+
[[options]]
name = "format"
short = "f"
+env = "TPCH_FILE_TYPE"
default = "parquet"
values = ["parquet", "csv", "mem"]
help = "Selects the TPC-H data format."
[[options]]
name = "scale-factor"
short = "sf"
+env = "BENCH_SIZE"
default = "1"
values = ["1", "10"]
help = "Selects the TPC-H scale factor."
+
+[[examples]]
Review Comment:
As a minor aside, these commands don't work yet
```shell
andrewlamb@Andrews-MacBook-Pro-3:~/Software/datafusion$ ^C
andrewlamb@Andrews-MacBook-Pro-3:~/Software/datafusion$ cargo run --bin
benchmark_runner -- run tpch 15 -f csv
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.30s
Running `target/debug/benchmark_runner run tpch 15 -f csv`
External error: error: unrecognized subcommand 'run'
Usage: benchmark_runner [COMMAND]
For more information, try '--help'.
```
I think you said they are coming in the next PR so that is good
##########
benchmarks/src/benchmark_runner/cli.rs:
##########
@@ -17,65 +17,973 @@
//! CLI construction and argument conversion for `benchmark_runner`.
//!
-//! This module owns the clap command tree for the initial runner surface:
-//! top-level help and suite listing.
+//! This module owns the clap command tree for `list`, `info`, and `query`.
Review Comment:
I found this file hard to understand why it was being all fancy with Clap
I think it is because the help is dynamically created based on the test
suite definitions (aka the .suite files). It would probably help if you could
give some short context in a comment (basically explaining that this file
dynamically builds up the clap arguments so that it can dynamically discover
the suites that are defined in .suite files)
I actually had claude make a PR with a proposed simplifcation, see
- https://github.com/Omega359/arrow-datafusion/pull/8
##########
benchmarks/src/benchmark_runner/native.rs:
##########
@@ -0,0 +1,362 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Native benchmark metadata for `benchmark_runner`.
+//!
+//! Native benchmarks are the Rust `RunOpt`-based benchmarks historically
+//! exposed through `dfbench.rs`. SQL suites take precedence, so native entries
+//! whose selector matches a discovered `.suite` are hidden from this runner.
+
+use crate::benchmark_runner::output::write_options_section;
+use crate::benchmark_runner::style::{HELP_STYLES, header};
+use crate::benchmark_runner::suite::SuiteRegistry;
+use crate::{
+ cancellation, clickbench, h2o, hj, imdb, nlj, smj, sort_pushdown,
sort_tpch, tpcds,
+};
+use clap::{Arg, ArgAction, Args, ColorChoice, Command};
+use datafusion::error::Result;
+use std::collections::BTreeSet;
+use std::fmt::Write as _;
+
+/// Metadata and clap wiring for one native Rust benchmark.
+#[derive(Debug, Clone)]
+pub(crate) struct NativeBenchmark {
+ kind: NativeBenchmarkKind,
+}
+
+/// Builds commands and formatted help for one native benchmark.
+impl NativeBenchmark {
+ /// Creates metadata for one native benchmark kind.
+ fn new(kind: NativeBenchmarkKind) -> Self {
+ Self { kind }
+ }
+
+ /// Returns the command-line selector for this benchmark.
+ pub(crate) fn name(&self) -> &'static str {
+ self.kind.name()
+ }
+
+ /// Returns the description shown by `list` and help output.
+ pub(crate) fn description(&self) -> &'static str {
+ self.kind.description()
+ }
+
+ /// Builds the clap command used to render native benchmark options.
+ pub(crate) fn command(&self) -> Command {
+ configure_help(
+ self.kind.augment_args(
+ Command::new(self.name())
+ .about(self.description())
+ .styles(HELP_STYLES)
+ .color(ColorChoice::Always),
+ ),
+ )
+ }
+
+ /// Builds the hidden clap subcommand used to recognize native info
selectors.
+ pub(crate) fn info_command(&self) -> Command {
+ Command::new(self.name())
+ .about(self.description())
+ .styles(HELP_STYLES)
+ .color(ColorChoice::Always)
+ }
+
+ /// Renders compact native help text for the `info` command.
+ pub(crate) fn info_text(&self) -> Result<String> {
+ let mut command = self
+ .command()
+ .bin_name(format!("benchmark_runner info {}", self.name()));
+ format_native_help(&mut command)
+ }
+}
+
+/// Formats native benchmark help in the compact runner style.
+fn format_native_help(command: &mut Command) -> Result<String> {
+ let mut output = String::new();
+
+ if let Some(about) = command.get_about() {
+ writeln!(output, "{about}\n")?;
+ }
+
+ writeln!(
+ output,
+ "{}",
+ header(command.render_usage().to_string().trim_end())
+ )?;
+ writeln!(output)?;
+ write_options_section(&mut output, command)?;
+
+ Ok(output)
+}
+
+/// Applies benchmark-runner help behavior to a native `RunOpt` command.
+fn configure_help(command: Command) -> Command {
+ hide_env_annotations(command).disable_help_flag(true).arg(
+ Arg::new("help")
+ .short('h')
+ .long("help")
+ .action(ArgAction::HelpShort)
+ .help("Print help"),
+ )
+}
+
+/// Hides environment variable annotations inherited from `RunOpt` arguments.
+fn hide_env_annotations(mut command: Command) -> Command {
+ let arg_ids = command
+ .get_arguments()
+ .map(|arg| arg.get_id().to_string())
+ .collect::<Vec<_>>();
+
+ for id in arg_ids {
+ command = command.mut_arg(id, |arg| arg.hide_env(true));
+ }
+
+ command
+}
+
+/// Visible native benchmark registry after excluding names shadowed by suites.
+#[derive(Debug, Clone)]
+pub(crate) struct NativeBenchmarks {
+ visible: Vec<NativeBenchmark>,
+}
+
+/// Discovers and looks up native benchmarks exposed by the runner.
+impl NativeBenchmarks {
+ /// Builds the visible native benchmark list for the current suite
registry.
+ pub(crate) fn new(suites: &SuiteRegistry) -> Self {
+ let suite_names = suites
+ .suites()
+ .iter()
+ .map(|suite| suite.name.as_str())
+ .collect::<BTreeSet<_>>();
+ let visible = native_benchmarks()
+ .into_iter()
+ .filter(|benchmark| !suite_names.contains(benchmark.name()))
+ .collect();
+
+ Self { visible }
+ }
+
+ /// Returns visible native benchmarks in display order.
+ pub(crate) fn visible(&self) -> &[NativeBenchmark] {
+ &self.visible
+ }
+
+ /// Returns whether a native benchmark selector is visible.
+ pub(crate) fn contains(&self, name: &str) -> bool {
+ self.get(name).is_some()
+ }
+
+ /// Finds one visible native benchmark by selector.
+ pub(crate) fn get(&self, name: &str) -> Option<&NativeBenchmark> {
+ self.visible
+ .iter()
+ .find(|benchmark| benchmark.name() == name)
+ }
+}
+
+/// Enumerates the native `RunOpt` benchmarks that can be exposed.
+#[derive(Debug, Clone, Copy)]
Review Comment:
I am a little worried about yet another list of benchmarks to keep up to
date, but if we will eventually remove dfbench I think a single list is
reasonable
##########
benchmarks/src/benchmark_runner/cli.rs:
##########
@@ -17,65 +17,973 @@
//! CLI construction and argument conversion for `benchmark_runner`.
//!
-//! This module owns the clap command tree for the initial runner surface:
-//! top-level help and suite listing.
+//! This module owns the clap command tree for `list`, `info`, and `query`.
+//! Suite options are loaded from `.suite` files and registered as dynamic clap
+//! arguments so help output can describe suite-specific flags. Short aliases
+//! with more than one character, such as `-sf`, are normalized before clap
+//! parses the command because clap only supports one-character short flags
+//! directly.
-use clap::builder::styling::{AnsiColor, Styles};
-use clap::{ArgMatches, Command};
+use crate::benchmark_runner::native::NativeBenchmarks;
+use crate::benchmark_runner::output::write_help_section;
+use crate::benchmark_runner::style::{HELP_STYLES, header};
+use crate::benchmark_runner::suite::{SuiteConfig, SuiteOption, SuiteRegistry};
+use crate::util::CommonOpt;
+use clap::{Arg, ArgAction, ArgMatches, Args, Command};
use datafusion_common::{Result, exec_datafusion_err};
+use std::collections::{BTreeMap, BTreeSet};
+use std::ffi::{OsStr, OsString};
+use std::fmt::Write as _;
-const HELP_STYLES: Styles = Styles::styled()
- .header(AnsiColor::Green.on_default().bold())
- .usage(AnsiColor::Green.on_default().bold())
- .literal(AnsiColor::Cyan.on_default().bold())
- .placeholder(AnsiColor::Cyan.on_default());
+const BENCHMARK_TARGET_HEADING: &str = "SQL Benchmark Target";
Review Comment:
I had claude clean them up and it seems to have gone well
- https://github.com/Omega359/arrow-datafusion/pull/9
##########
benchmarks/src/benchmark_runner/native.rs:
##########
@@ -0,0 +1,362 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Native benchmark metadata for `benchmark_runner`.
+//!
+//! Native benchmarks are the Rust `RunOpt`-based benchmarks historically
+//! exposed through `dfbench.rs`. SQL suites take precedence, so native entries
+//! whose selector matches a discovered `.suite` are hidden from this runner.
+
+use crate::benchmark_runner::output::write_options_section;
+use crate::benchmark_runner::style::{HELP_STYLES, header};
+use crate::benchmark_runner::suite::SuiteRegistry;
+use crate::{
+ cancellation, clickbench, h2o, hj, imdb, nlj, smj, sort_pushdown,
sort_tpch, tpcds,
+};
+use clap::{Arg, ArgAction, Args, ColorChoice, Command};
+use datafusion::error::Result;
+use std::collections::BTreeSet;
+use std::fmt::Write as _;
+
+/// Metadata and clap wiring for one native Rust benchmark.
+#[derive(Debug, Clone)]
+pub(crate) struct NativeBenchmark {
+ kind: NativeBenchmarkKind,
+}
+
+/// Builds commands and formatted help for one native benchmark.
+impl NativeBenchmark {
+ /// Creates metadata for one native benchmark kind.
+ fn new(kind: NativeBenchmarkKind) -> Self {
+ Self { kind }
+ }
+
+ /// Returns the command-line selector for this benchmark.
+ pub(crate) fn name(&self) -> &'static str {
+ self.kind.name()
+ }
+
+ /// Returns the description shown by `list` and help output.
+ pub(crate) fn description(&self) -> &'static str {
+ self.kind.description()
+ }
+
+ /// Builds the clap command used to render native benchmark options.
+ pub(crate) fn command(&self) -> Command {
+ configure_help(
+ self.kind.augment_args(
+ Command::new(self.name())
+ .about(self.description())
+ .styles(HELP_STYLES)
+ .color(ColorChoice::Always),
+ ),
+ )
+ }
+
+ /// Builds the hidden clap subcommand used to recognize native info
selectors.
+ pub(crate) fn info_command(&self) -> Command {
+ Command::new(self.name())
+ .about(self.description())
+ .styles(HELP_STYLES)
+ .color(ColorChoice::Always)
+ }
+
+ /// Renders compact native help text for the `info` command.
+ pub(crate) fn info_text(&self) -> Result<String> {
+ let mut command = self
+ .command()
+ .bin_name(format!("benchmark_runner info {}", self.name()));
+ format_native_help(&mut command)
+ }
+}
+
+/// Formats native benchmark help in the compact runner style.
+fn format_native_help(command: &mut Command) -> Result<String> {
+ let mut output = String::new();
+
+ if let Some(about) = command.get_about() {
+ writeln!(output, "{about}\n")?;
+ }
+
+ writeln!(
+ output,
+ "{}",
+ header(command.render_usage().to_string().trim_end())
+ )?;
+ writeln!(output)?;
+ write_options_section(&mut output, command)?;
+
+ Ok(output)
+}
+
+/// Applies benchmark-runner help behavior to a native `RunOpt` command.
+fn configure_help(command: Command) -> Command {
+ hide_env_annotations(command).disable_help_flag(true).arg(
+ Arg::new("help")
+ .short('h')
+ .long("help")
+ .action(ArgAction::HelpShort)
+ .help("Print help"),
+ )
+}
+
+/// Hides environment variable annotations inherited from `RunOpt` arguments.
+fn hide_env_annotations(mut command: Command) -> Command {
+ let arg_ids = command
+ .get_arguments()
+ .map(|arg| arg.get_id().to_string())
+ .collect::<Vec<_>>();
+
+ for id in arg_ids {
+ command = command.mut_arg(id, |arg| arg.hide_env(true));
+ }
+
+ command
+}
+
+/// Visible native benchmark registry after excluding names shadowed by suites.
Review Comment:
What is the plan for native benchmarks that are shadowed by suites? Will
they be removed once we switch entirely over to benchmrk_runner?
##########
benchmarks/src/benchmark_runner/native.rs:
##########
@@ -0,0 +1,362 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Native benchmark metadata for `benchmark_runner`.
+//!
+//! Native benchmarks are the Rust `RunOpt`-based benchmarks historically
Review Comment:
Anther important property is they can't be setup/run with only SQL
Though I do wonder if we could (as a follow on) potentially port them to
being SQL based (perhaps with some user defined functions)
##########
benchmarks/src/benchmark_runner/selector.rs:
##########
@@ -0,0 +1,356 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Benchmark target resolution.
+//!
+//! Selectors name a suite, such as `tpch`. Optional query IDs are normalized
+//! so values like `1`, `01`, and `001` resolve through the suite's query file
+//! pattern.
+
+use crate::benchmark_runner::suite::{
+ SuiteConfig, SuiteExample, SuiteOption, SuiteRegistry,
+};
+use datafusion_common::{DataFusionError, Result};
+use std::collections::{BTreeMap, HashMap};
+use std::path::PathBuf;
+
+/// Concrete benchmark file selected by suite and optional query id.
+#[derive(Debug, Clone)]
+pub struct ResolvedBenchmark {
+ /// Benchmark file selected for inspection or execution.
+ pub path: PathBuf,
+ /// Numeric query id when it can be inferred from a `qNN.benchmark` file.
+ #[cfg_attr(
+ not(test),
+ expect(dead_code, reason = "used by the stacked run command branch")
+ )]
+ pub query_id: Option<usize>,
+ /// Human-readable benchmark label used in run output and JSON.
+ #[cfg_attr(
+ not(test),
+ expect(dead_code, reason = "used by the stacked run command branch")
+ )]
+ pub label: String,
+}
+
+/// Suite metadata needed after selector resolution.
+///
+/// This intentionally carries only display and suite-option metadata. It does
+/// not include the full suite config, filesystem paths, or discovery caches.
+#[derive(Debug, Clone)]
+pub struct ResolvedSuite {
+ /// Suite selector used on the command line.
+ pub name: String,
+ /// Human-readable suite description.
+ pub description: String,
+ /// Suite-specific CLI options shown by `info`.
+ pub options: Vec<SuiteOption>,
+ /// Example commands shown by `info`.
+ pub examples: Vec<SuiteExample>,
+}
+
+/// Copies display metadata from a suite config into a resolved suite value.
+impl From<&SuiteConfig> for ResolvedSuite {
+ fn from(suite: &SuiteConfig) -> Self {
+ Self {
+ name: suite.name.clone(),
+ description: suite.description.clone(),
+ options: suite.options.clone(),
+ examples: suite.examples.clone(),
+ }
+ }
+}
+
+/// Fully resolved SQL benchmark target ready for inspection or execution.
+#[derive(Debug, Clone)]
+pub struct ResolvedBenchmarkTarget {
+ /// Containing suite metadata, when the target came from or is inside a
+ /// discovered suite.
+ pub suite: Option<ResolvedSuite>,
+ /// Final suite option values after applying defaults and CLI overrides.
+ #[cfg_attr(
+ not(test),
+ expect(dead_code, reason = "used by the stacked run command branch")
+ )]
+ pub option_values: BTreeMap<String, String>,
+ /// SQL replacement values passed to `SqlBenchmark`.
+ pub replacement_values: HashMap<String, String>,
+ /// Concrete benchmark files selected for inspection or execution.
+ pub benchmarks: Vec<ResolvedBenchmark>,
+}
+
+/// Resolves suite selectors, query ids, and suite option values.
+impl ResolvedBenchmarkTarget {
+ /// Resolves a user selector into the concrete benchmark files to inspect
or run.
+ ///
+ /// Suite selectors apply validated suite options and optional query-id
+ /// filtering.
+ pub fn resolve(
+ registry: &SuiteRegistry,
+ selector: &str,
+ query_id: Option<&str>,
+ suite_options: &BTreeMap<String, String>,
+ ) -> Result<Self> {
+ let suite = registry.get(selector).ok_or_else(|| {
+ DataFusionError::Configuration(format!(
+ "unknown benchmark suite '{selector}'"
+ ))
+ })?;
+ let option_values = suite.resolve_option_values(suite_options)?;
+ let replacement_values = suite.option_replacements(&option_values);
+ let queries = suite.discover_queries()?;
+ let benchmarks = if let Some(query_id) = query_id {
+ let id = parse_query_id(query_id)?;
+ let query = queries
+ .into_iter()
+ .find(|query| query.id == id)
+ .ok_or_else(|| {
+ DataFusionError::Configuration(format!(
+ "suite '{}' has no QUERY_ID {}",
+ suite.name, id
+ ))
+ })?;
+ vec![ResolvedBenchmark {
+ path: query.path,
+ query_id: Some(id),
+ label: format!("{}/q{id:02}", suite.name),
+ }]
+ } else {
+ queries
+ .into_iter()
+ .map(|query| ResolvedBenchmark {
+ label: format!("{}/q{:02}", suite.name, query.id),
+ path: query.path,
+ query_id: Some(query.id),
+ })
+ .collect()
+ };
+
+ if benchmarks.is_empty() {
+ return Err(DataFusionError::Configuration(format!(
+ "suite '{}' has no benchmark files",
+ suite.name
+ )));
+ }
+
+ Ok(Self {
+ suite: Some(ResolvedSuite::from(suite)),
+ option_values,
+ replacement_values,
+ benchmarks,
+ })
+ }
+
+ /// Returns the single resolved benchmark required by parse-only inspection
Review Comment:
what is a parse only inspection command? Maybe we could give an example
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
