andygrove opened a new pull request, #1942:
URL: https://github.com/apache/datafusion-ballista/pull/1942
# Which issue does this PR close?
<!-- Draft — happy to file a tracking issue if the maintainers prefer one.
-->
N/A (draft)
# Rationale for this change
The `tpch` benchmark's `benchmark ballista` command can currently only read
data from the local filesystem: `--path` is parsed as a `PathBuf` (which
corrupts URL schemes such as `s3://bucket` into `s3:/bucket`), table
registration probes the local filesystem with `Path::exists()`, and the
benchmark's client `SessionState` is built without object-store support.
This makes it impossible to benchmark Ballista against data held in
S3-compatible object storage (e.g. MinIO or AWS S3), which is the natural way
to run larger scale factors on a distributed cluster where the data does not
fit on a single node's local disk. The `ballista-scheduler` and
`ballista-executor` binaries already register S3 support via
`session_*_with_s3_support`; only the benchmark client was missing it.
# What changes are included in this PR?
Scoped to `benchmark ballista` in `benchmarks/src/bin/tpch.rs`:
- `BallistaBenchmarkOpt::path` changes from `PathBuf` to `String` so URL
schemes survive argument parsing.
- The client session is now built from `session_config_with_s3_support()` and
`session_state_with_s3_support()` (the same helpers the scheduler/executor
binaries use), so the client can plan against an object store. Because this
registers the `S3Options` config extension, the existing `-c key=value`
override flag now also accepts `s3.*` keys (`s3.endpoint`,
`s3.access_key_id`, `s3.secret_access_key`, `s3.region`, `s3.allow_http`).
Standard `AWS_*` environment variables continue to work as well.
- `find_path()` registers the per-table directory directly for object-store
URLs (paths containing `://`), bypassing the local `Path::exists()` check.
- `benchmarks/Cargo.toml` enables `ballista-core/build-binary`, which gates
the
`object_store` module (and pulls in the `aws-config`/`object_store` deps).
Local-filesystem behavior is unchanged: non-URL paths still go through the
existing `find_path()` resolution.
Example:
```
AWS_ENDPOINT=http://localhost:9000 AWS_ALLOW_HTTP=true \
AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_REGION=us-east-1 \
tpch benchmark ballista --host <scheduler> --port 50050 \
--path s3://tpch/sf1000 --format parquet --iterations 3
```
# Are there any user-facing changes?
Yes — the `tpch benchmark ballista --path` option now additionally accepts
object-store URLs (e.g. `s3://bucket/prefix`), and `-c s3.*` config overrides
are now honored. This is additive and backward compatible for local paths. No
public library API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
