janbraunsdorff opened a new issue, #1313: URL: https://github.com/apache/datafusion-ballista/issues/1313
Hello, During the migration from Spark to Datafusion / Ballista, the error __recursion limit reached__ during parsing the logical plan on the scheduler. For refference I used [this](https://github.com/milenkovicm/ballista_delta) as an example to work with Delta on s3. **Error Message** ``` [2025-09-07T15:45:45Z ERROR ballista_scheduler::scheduler_server::grpc] Could not parse logical plan protobuf: Internal error: failed to decode logical plan: DecodeError { description: "recursion limit reached", stack: [...] } ``` **Dependencies** ```toml ballista = {version = "48"} ballista-core = { version = "48", default-features = true} ballista-executor = { version = "48", default-features = true} ballista-scheduler = { version = "48", default-features = true, features = ["graphviz-support", "rest-api", "prometheus-metrics"]} datafusion = { version = "48" } datafusion-proto = { version = "48" } deltalake = { version = "0.27", features = ["datafusion", "s3"] } ``` I saw there is a configuraion named `datafusion.sql_parser.recursion_limit` and added them to the config like below. And also overwrite the setting via `.option_mut().sql_parser.recusion_limit = 500` and set the env variable to `DATAFUSION_SQL_PARSER_RECURSION_LIMIT=500` ```rs pub fn custom_session_config() -> SessionConfig { SessionConfig::from_env().unwrap() .with_option_extension(BallistaConfig::default()) .with_information_schema(true) .with_target_partitions(16) .ballista_restricted_configuration() .with_information_schema(true) .set_usize("datafusion.sql_parser.recursion_limit", 500) } ``` ```rs // in the main funciton let config: SchedulerConfig = SchedulerConfig { override_logical_codec: Some(Arc::new(BallistaDeltaLogicalCodec::default())), override_physical_codec: Some(Arc::new(BallistaDeltaPhysicalCodec::default())), override_config_producer: Some(Arc::new(custom_session_config)), override_session_builder: Some(Arc::new(custom_session_state)), bind_host: "0.0.0.0".to_string(), ..Default::default() }; ``` How can the limit increased? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
