freedomlovesfrank opened a new issue, #2443:
URL: https://github.com/apache/datafusion-ballista/issues/2443
### The value of integrating
The Ballista executor can run on mimalloc —
`ballista/executor/src/bin/main.rs` sets it behind `#[cfg(feature =
"mimalloc")]`, with `mimalloc = { version = "0.1" }`. So a default executor
build runs on the platform allocator, and turning the feature on adds a C
library and its build script to a cluster component you deploy many copies of.
rusty_alloc is relevant here specifically because it is not a different
design: it is the mimalloc v2.4.5 algorithm rebuilt in safe Rust, implementing
about 150 of mimalloc's roughly 157 `mi_*` entry points and validated against
the C original as a differential oracle. So the comparison is like-for-like
rather than a claim that some other allocator suits you better.
That matters for an executor. A Ballista cluster runs scheduler and executor
processes that stay up and take partitions of a distributed query — a workload
that allocates record batches on one thread and frees them on another as work
moves through the plan. On that cross-thread free path the rusty_alloc README
measures 0.83x mimalloc's instructions (`xthread` — every free performed by a
non-owning thread), and it has 8,851 downloads on crates.io.
Not asking you to drop the mimalloc feature — this would be a second one
beside it.
### How to integrate
```toml
[features]
rusty-alloc = ["dep:rusty_alloc-api"] # opt in, exactly like `mimalloc`
[dependencies]
rusty_alloc-api = { version = "2.1", optional = true }
```
```rust
#[cfg(all(feature = "rusty-alloc", not(feature = "mimalloc")))]
#[global_allocator]
static GLOBAL: rusty_alloc_api::RustyAlloc = rusty_alloc_api::RustyAlloc;
```
Nothing else changes: same scheduler, same plan serialization, same SQL
results. The executor Docker image would build with no C compiler at all, which
is the part worth checking even if the numbers came out level.
### Remade With Rust & MATA
We are rebuilding the C libraries Rust projects still depend on, in safe
Rust — allocators, codecs, compression, parsers, identity and storage — so a
build can drop its C pieces without losing performance. The org is at
https://github.com/Remade-With-Rust if anything else fits. If this is not for
you, please close the issue.
- x.com/farmer_timmm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
