milenkovicm opened a new pull request, #1096:
URL: https://github.com/apache/datafusion-ballista/pull/1096
# Which issue does this PR close?
Closes #1092.
# Rationale for this change
This change provides ergonomic way to configure SessionContextExt, similar
to DataFusion context creation from defined SessionStore.
# What changes are included in this PR?
This change provides two new methods in `SessionContextExt` `async fn
standalone_with_state(state: SessionState)` and `async fn
remote_with_state(url: &str,state: SessionState)`
Which accepts pre configured `SessionState` as parameter. `SessionState`
should be configured in the same way like when `SessionContext` is configured
in DataFusion.
```rust
let state = SessionStateBuilder::new().with_default_features().build();
let ctx: SessionContext = SessionContext::remote_with_state(&url,
state).await?;
```
This change also exposes a `BallistaSessionConfigExt` which provides method
to configure ballista specific settings like, `BallistaConfiguration`, codecs
or even `QueryPlanner`.
```rust
use ballista_client::extension::BallistaSessionConfigExt;
let session_config = SessionConfig::new_with_ballista()
.with_information_schema(true)
.set_str(BALLISTA_JOB_NAME, "Super Cool Ballista App");
let state = SessionStateBuilder::new()
.with_default_features()
.with_config(session_config)
.build();
let ctx: SessionContext = SessionContext::remote_with_state(&url,
state).await?;
```
`LogicalExtensionCodec` and `PhysicalExtensionCodec` can be changed as well:
```rust
use ballista_client::extension::BallistaSessionConfigExt;
let logical_codec = Arc::new(BadLogicalCodec::default());
let physical_codec = Arc::new(MockPhysicalCodec::default());
let session_config = SessionConfig::new_with_ballista()
.with_information_schema(true)
.with_ballista_physical_extension_codec(physical_codec.clone())
.with_ballista_logical_extension_codec(logical_codec.clone())
;
let state = SessionStateBuilder::new()
.with_default_features()
.with_config(session_config)
.build();
let ctx: SessionContext =
SessionContext::standalone_with_state(state).await?;
```
In this case logical and physical codec will be also be propagated to
standalone.
Lastly, BallistaQueryPlanner can be replaced:
```rust
let session_config = SessionConfig::new_with_ballista()
.with_information_schema(true)
.set_str(BALLISTA_PLANNER_OVERRIDE, "false");
let state = SessionStateBuilder::new()
.with_default_features()
.with_config(session_config)
.with_query_planner(Arc::new(BadPlanner::default()))
.build();
let ctx: SessionContext =
SessionContext::standalone_with_state(state).await?;
```
At the moment there is a hacky way telling ballista not to override provided
planner with `.set_str(BALLISTA_PLANNER_OVERRIDE, "false");` as
its not possible to detect if the planner is changed.
# Are there any user-facing changes?
Introduction of two new methods and one extension to new functionality, no
braking change
Notes:
- Object store will not be set automatically anymore, will look into
`ballista_core::object_store_registry::with_object_store_registry` deprecation
in follow up commits when we expose configuration on scheduler and executor api
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
