robtandy opened a new issue, #72:
URL: https://github.com/apache/datafusion-ray/issues/72
Right now, the `DFRayContext` accepts options that govern its runtime
behavior like
```python
ctx = DFRayContext(
batch_size=batch_size,
partitions_per_worker=partitions_per_worker,
prefetch_buffer_size=prefetch_buffer_size,
worker_pool_min=worker_pool_min,
```
While we cannot accept a `SessionConfig` from datafusion-python due to lack
of ABI stability in rust, we should still try to adhere to the DataFusion
Python API as much as possible.
So, we allow setting datafusion configuration options via `DFRayContext.set`.
Setting options in two places is confusion. I think it will be more clear
if we set the options in the __init__ via `DFRayContext.set` instead.
We can call them
- `datafusion.ray.execution.batch_size`
- `datafusion.ray.execution.partitions_per_processor` (use updated name
processor instead of worker here)
- `datafusion.ray.execution.prefetch_buffer_size`
- `datafusion.ray.execution.processor_pool_min` (use updated name processor)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
