alamb commented on code in PR #18004:
URL: https://github.com/apache/datafusion/pull/18004#discussion_r2421019598
##########
datafusion-cli/examples/cli-session-context.rs:
##########
@@ -84,11 +91,17 @@ impl CliSessionContext for MyUnionerContext {
pub async fn main() {
let my_ctx = MyUnionerContext::default();
+ let profile_mode = InstrumentedObjectStoreMode::default();
Review Comment:
A minor code structure thing (not needed) but I think it would be nicer to
encapsulate more of this construction into the `InstrumentedObjectStoreRegistry`
Maybe something like putting the default registries in `::new()`
``rust
let instrumented_registry = Arc::new(InstrumentedObjectStoreRegistry::new());
```
And then maybe you can add builder style APIs if you need, like
```rust
let instrumented_registry = InstrumentedObjectStoreRegistry::new()
.with_inner(inner_registry)
.with_profile_mode(mode)
```
##########
datafusion-cli/src/object_storage/instrumented.rs:
##########
@@ -166,7 +171,28 @@ impl InstrumentedObjectStoreRegistry {
) -> Self {
Self {
inner: registry,
- instrument_mode: default_mode,
+ instrument_mode: AtomicU8::new(default_mode as u8),
+ stores: RwLock::new(Vec::new()),
+ }
+ }
+
+ /// Provides access to all of the [`InstrumentedObjectStore`]s managed by
this
+ /// [`InstrumentedObjectStoreRegistry`]
+ pub fn stores(&self) -> Vec<Arc<InstrumentedObjectStore>> {
+ self.stores.read().clone()
+ }
+
+ /// Returns the current [`InstrumentedObjectStoreMode`] for this
+ /// [`InstrumentedObjectStoreRegistry`]
+ pub fn mode(&self) -> InstrumentedObjectStoreMode {
Review Comment:
Could you please name the methods consistently? `mode` and
`set_instrument_mode` are not consistent
I think this would be consistent with the other method and the name of the
field
```suggestion
pub fn instrument_mode(&self) -> InstrumentedObjectStoreMode {
```
##########
datafusion-cli/src/print_options.rs:
##########
@@ -174,10 +177,78 @@ impl PrintOptions {
query_start_time,
);
+ self.write_output(&mut writer, formatted_exec_details)
+ }
+
+ fn write_output<W: io::Write>(
+ &self,
+ writer: &mut W,
+ formatted_exec_details: String,
+ ) -> Result<()> {
if !self.quiet {
writeln!(writer, "{formatted_exec_details}")?;
+
+ if self.instrumented_registry.mode() !=
InstrumentedObjectStoreMode::Disabled
+ {
+ writeln!(writer, "{OBJECT_STORE_PROFILING_HEADER}")?;
+ for store in self.instrumented_registry.stores() {
+ writeln!(writer, "{store}")?;
+ }
+ }
Review Comment:
I wonder if (as a follow on PR) we should also move the timing into this
structure too
I think we mentioned before it is somewhat strange that something named
`PrintOptions` has an object store registry, but I see why it is like this
##########
datafusion-cli/src/command.rs:
##########
@@ -244,3 +277,52 @@ impl OutputFormat {
}
}
}
+
+#[cfg(test)]
+mod tests {
+ use datafusion::{
+ execution::object_store::DefaultObjectStoreRegistry,
prelude::SessionContext,
+ };
+
+ use crate::{
+ object_storage::instrumented::{
+ InstrumentedObjectStoreMode, InstrumentedObjectStoreRegistry,
+ },
+ print_options::MaxRows,
+ };
+
+ use super::*;
+
+ #[tokio::test]
+ async fn command_execute_profile_mode() {
+ let ctx = SessionContext::new();
+
+ let profile_mode = InstrumentedObjectStoreMode::default();
+ let instrumented_registry =
Arc::new(InstrumentedObjectStoreRegistry::new(
+ Arc::new(DefaultObjectStoreRegistry::new()),
+ profile_mode,
+ ));
+ let mut print_options = PrintOptions {
+ format: PrintFormat::Automatic,
+ quiet: false,
+ maxrows: MaxRows::Unlimited,
+ color: true,
+ instrumented_registry: Arc::clone(&instrumented_registry),
+ };
+
+ let mut cmd: Command = "object_store_profiling"
+ .parse()
+ .expect("expected parse to succeed");
+ assert!(cmd.execute(&ctx, &mut print_options).await.is_ok());
Review Comment:
I think it would be good here to also assert here that
`print_options.instrumented_registry.profile_mode` is updated as well. Perhaps
something like
```rust
assert_eq!(
print_options.instrumented_registry.profile_mode(),
InstrumentedObjectStoreMode::default()
);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
