andygrove commented on PR #1059:
URL:
https://github.com/apache/datafusion-ballista/pull/1059#issuecomment-2374024160
Thanks @palaska. This looks great. I wasn't able to run any queries though
with these changes.
I built with `cargo build --release`.
I started the scheduler:
```
$ ./target/release/ballista-scheduler
2024-09-25T12:57:29.760309Z INFO main ThreadId(01)
ballista_scheduler::cluster: Initializing Sled database in temp directory
2024-09-25T12:57:29.769023Z INFO main ThreadId(01)
ballista_scheduler::scheduler_process: Ballista v0.12.0 Scheduler listening on
0.0.0.0:50050
2024-09-25T12:57:29.769038Z INFO main ThreadId(01)
ballista_scheduler::scheduler_process: Starting Scheduler grpc server with task
scheduling policy of PullStaged
2024-09-25T12:57:29.769137Z INFO main ThreadId(01)
ballista_scheduler::cluster::kv: Initializing heartbeat listener
2024-09-25T12:57:29.769155Z INFO main ThreadId(01)
ballista_scheduler::scheduler_server::query_stage_scheduler: Starting
QueryStageScheduler
2024-09-25T12:57:29.769175Z INFO tokio-runtime-worker ThreadId(31)
ballista_core::event_loop: Starting the event loop query_stage
```
I started the executor:
```
$ ./target/release/ballista-executor -c 8
2024-09-25T12:57:34.575905Z INFO main ThreadId(01)
ballista_executor::executor_process: Running with config:
2024-09-25T12:57:34.575919Z INFO main ThreadId(01)
ballista_executor::executor_process: work_dir: /tmp/.tmpnbFKWN
2024-09-25T12:57:34.575921Z INFO main ThreadId(01)
ballista_executor::executor_process: concurrent_tasks: 8
2024-09-25T12:57:34.576719Z INFO tokio-runtime-worker ThreadId(32)
ballista_executor::executor_process: Ballista v0.12.0 Rust Executor Flight
Server listening on 0.0.0.0:50051
2024-09-25T12:57:34.576759Z INFO tokio-runtime-worker ThreadId(30)
ballista_executor::execution_loop: Starting poll work loop with scheduler
```
The executor immediately started producing errors:
```
2024-09-25T12:57:34.577270Z WARN tokio-runtime-worker ThreadId(30)
ballista_executor::execution_loop: Executor poll work loop failed. If this
continues to happen the Scheduler might be marked as dead. Error: status:
Cancelled, message: "h2 protocol error: http2 error", details: [], metadata:
MetadataMap { headers: {} }
2024-09-25T12:57:34.678711Z WARN tokio-runtime-worker ThreadId(02)
ballista_executor::execution_loop: Executor poll work loop failed. If this
continues to happen the Scheduler might be marked as dead. Error: status:
Cancelled, message: "h2 protocol error: http2 error", details: [], metadata:
MetadataMap { headers: {} }
```
I then went back to the scheduler process and saw errors there as well:
```
thread 'tokio-runtime-worker' panicked at
/home/andy/git/apache/datafusion-ballista/ballista/scheduler/src/scheduler_server/grpc.rs:95:56:
called `Option::unwrap()` on a `None` value
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'tokio-runtime-worker' panicked at
/home/andy/git/apache/datafusion-ballista/ballista/scheduler/src/scheduler_server/grpc.rs:95:56:
called `Option::unwrap()` on a `None` value
```
Here is the backtrace:
```
thread 'tokio-runtime-worker' panicked at
/home/andy/git/apache/datafusion-ballista/ballista/scheduler/src/scheduler_server/grpc.rs:95:56:
called `Option::unwrap()` on a `None` value
stack backtrace:
0: rust_begin_unwind
1: core::panicking::panic_fmt
2: core::panicking::panic
3: core::option::unwrap_failed
4: ballista_scheduler::scheduler_server::grpc::<impl
ballista_core::serde::generated::ballista::scheduler_grpc_server::SchedulerGrpc
for
ballista_scheduler::scheduler_server::SchedulerServer<T,U>>::poll_work::{{closure}}
5:
<<ballista_core::serde::generated::ballista::scheduler_grpc_server::SchedulerGrpcServer<T>
as tower_service::Service<http::request::Request<B>>>::call::PollWorkSvc<T> as
tonic::server::service::UnaryService<ballista_core::serde::generated::ballista::PollWorkParams>>::call::{{closure}}
6:
<ballista_core::serde::generated::ballista::scheduler_grpc_server::SchedulerGrpcServer<T>
as tower_service::Service<http::request::Request<B>>>::call::{{closure}}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
