andygrove opened a new issue, #37: URL: https://github.com/apache/datafusion-iceberg/issues/37
### Describe the bug `IcebergSchemaProvider::register_table` and `deregister_table` are synchronous `SchemaProvider` methods that DataFusion calls while executing SQL. They run the catalog call inside `tokio::task::spawn_blocking` with `Handle::current().block_on`, then wait for the `JoinHandle` with `futures::executor::block_on` on the calling thread ([schema.rs#L176-L207](https://github.com/apache/datafusion-iceberg/blob/a2bc9427d0659591b5f8122b90fec710fe2f5de6/crates/datafusion/src/schema.rs#L176-L207), [schema.rs#L221-L243](https://github.com/apache/datafusion-iceberg/blob/a2bc9427d0659591b5f8122b90fec710fe2f5de6/crates/datafusion/src/schema.rs#L221-L243)). That blocks the thread that called into DataFusion. On a current-thread runtime, that thread is the only one that drives tokio's I/O and timer drivers. `Handle::block_on` on the blocking-pool thread doesn't drive them, so a catalog call that awaits network I/O or a timer never completes. That covers REST, Glue, HMS and similar catalogs. ### To Reproduce 1. Wrap a `MemoryCatalog` in a `Catalog` implementation whose methods await `tokio::time::sleep(Duration::from_millis(2))` before delegating, to stand in for a network catalog. 2. On a runtime built with `tokio::runtime::Builder::new_current_thread().enable_all()`, register it through `IcebergCatalogProvider` and run `CREATE TABLE ice.ns.created (id INT NOT NULL)`. No result after 10 seconds: the statement hangs. Calling `register_table` from a thread with no tokio runtime panics: ``` thread '<unnamed>' panicked at crates/datafusion/src/schema.rs:177:22: there is no reactor running, must be called from the context of a Tokio 1.x runtime ``` On a multi-thread runtime the statement completed in this test, but it keeps the calling thread blocked for the whole catalog round trip. ### Expected behavior `CREATE TABLE` and `DROP TABLE` complete or return an error on either runtime flavor. The synchronous methods don't panic when called outside a runtime. ### Additional context #24 (section 3) discusses moving remote create/drop behind an async boundary; this issue is the concrete failure in the current implementation. `#[tokio::test]` uses a current-thread runtime by default. The existing tests pass only because `MemoryCatalog` on a local filesystem never needs the I/O or timer driver. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
