janbraunsdorff commented on issue #1313:
URL:
https://github.com/apache/datafusion-ballista/issues/1313#issuecomment-3264005526
@milenkovicm , this was my fist thought, too. the error in my first post was
from the scheduler. Tthe error message in Client looks like this:
```
thread 'main' panicked at ballista/delta-client/src/bin/main.rs:690:31:
called `Result::unwrap()` on an `Err` value:
ArrowError(ExternalError(Execution("Fail to execute query due to
ExecuteQueryFailureResult { failure: Some(PlanParsingFailure(\"Could not parse
logical plan protobuf: Internal error: failed to decode logical plan:
DecodeError { description: \\\"recursion limit reached\\\", stack:
[(\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"), (\\\"ProjectionNode\\\",
\\\"input\\\"), (\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"),
(\\\"JoinNode\\\", \\\"left\\\"), (\\\"LogicalPlanNode\\\",
\\\"logical_plan_type\\\"), (\\\"ProjectionNode\\\", \\\"input\\\"),
(\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"), (\\\"JoinNode\\\",
\\\"left\\\"), (\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"),
(\\\"ProjectionNode\\\", \\\"input\\\"), (\\\"LogicalPlanNode\\\",
\\\"logical_plan_type\\\"), (\\\"JoinNode\\\", \\\"left\\\"),
(\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"), (\\\"ProjectionNode\\\",
\\\"input\\\"), (\\\"LogicalPlanNode\\\"
, \\\"logical_plan_type\\\"), (\\\"JoinNode\\\", \\\"left\\\"),
(\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"), (\\\"ProjectionNode\\\",
\\\"input\\\"), (\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"),
(\\\"ProjectionNode\\\", \\\"input\\\"), (\\\"LogicalPlanNode\\\",
\\\"logical_plan_type\\\"), (\\\"JoinNode\\\", \\\"left\\\"),
(\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"), (\\\"ProjectionNode\\\",
\\\"input\\\"), (\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"),
(\\\"JoinNode\\\", \\\"left\\\"), (\\\"LogicalPlanNode\\\",
\\\"logical_plan_type\\\"), (\\\"ProjectionNode\\\", \\\"input\\\"),
(\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"), (\\\"ProjectionNode\\\",
\\\"input\\\"), (\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"),
(\\\"ProjectionNode\\\", \\\"input\\\"), (\\\"LogicalPlanNode\\\",
\\\"logical_plan_type\\\"), (\\\"JoinNode\\\", \\\"left\\\"),
(\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"), (\\\"ProjectionNode\\\",
\\\"input\\\"), (\\\
"LogicalPlanNode\\\", \\\"logical_plan_type\\\"), (\\\"JoinNode\\\",
\\\"left\\\"), (\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"),
(\\\"ProjectionNode\\\", \\\"input\\\"), (\\\"LogicalPlanNode\\\",
\\\"logical_plan_type\\\"), (\\\"JoinNode\\\", \\\"left\\\"),
(\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"), (\\\"ProjectionNode\\\",
\\\"input\\\"), (\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"),
(\\\"SelectionNode\\\", \\\"input\\\"), (\\\"LogicalPlanNode\\\",
\\\"logical_plan_type\\\"), (\\\"ProjectionNode\\\", \\\"input\\\"),
(\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"), (\\\"JoinNode\\\",
\\\"left\\\"), (\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"),
(\\\"ProjectionNode\\\", \\\"input\\\"), (\\\"LogicalPlanNode\\\",
\\\"logical_plan_type\\\"), (\\\"JoinNode\\\", \\\"left\\\"),
(\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"), (\\\"ProjectionNode\\\",
\\\"input\\\"), (\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"),
(\\\"JoinNode\\\", \\\"lef
t\\\"), (\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"),
(\\\"ProjectionNode\\\", \\\"input\\\"), (\\\"LogicalPlanNode\\\",
\\\"logical_plan_type\\\"), (\\\"ProjectionNode\\\", \\\"input\\\"),
(\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"), (\\\"JoinNode\\\",
\\\"left\\\"), (\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"),
(\\\"ProjectionNode\\\", \\\"input\\\"), (\\\"LogicalPlanNode\\\",
\\\"logical_plan_type\\\"), (\\\"JoinNode\\\", \\\"left\\\"),
(\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"), (\\\"ProjectionNode\\\",
\\\"input\\\"), (\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"),
(\\\"JoinNode\\\", \\\"left\\\"), (\\\"LogicalPlanNode\\\",
\\\"logical_plan_type\\\"), (\\\"ProjectionNode\\\", \\\"input\\\"),
(\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"), (\\\"ProjectionNode\\\",
\\\"input\\\"), (\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"),
(\\\"ProjectionNode\\\", \\\"input\\\"), (\\\"LogicalPlanNode\\\",
\\\"logical_plan_type\\\"), (\\\"P
rojectionNode\\\", \\\"input\\\"), (\\\"LogicalPlanNode\\\",
\\\"logical_plan_type\\\"), (\\\"JoinNode\\\", \\\"left\\\"),
(\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"), (\\\"ProjectionNode\\\",
\\\"input\\\"), (\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"),
(\\\"ProjectionNode\\\", \\\"input\\\"), (\\\"LogicalPlanNode\\\",
\\\"logical_plan_type\\\"), (\\\"ProjectionNode\\\", \\\"input\\\"),
(\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"), (\\\"ProjectionNode\\\",
\\\"input\\\"), (\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"),
(\\\"ProjectionNode\\\", \\\"input\\\"), (\\\"LogicalPlanNode\\\",
\\\"logical_plan_type\\\"), (\\\"ProjectionNode\\\", \\\"input\\\"),
(\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"), (\\\"ProjectionNode\\\",
\\\"input\\\"), (\\\"LogicalPlanNode\\\", \\\"logical_plan_type\\\"),
(\\\"AggregateNode\\\", \\\"input\\\"), (\\\"LogicalPlanNode\\\",
\\\"logical_plan_type\\\"), (\\\"ProjectionNode\\\", \\\"input\\\"),
(\\\"LogicalPlanNode
\\\", \\\"logical_plan_type\\\")] }.\\nThis was likely caused by a bug in
DataFusion's code and we would welcome that you file an bug report in our issue
tracker\")) }")), None)
```
my client config looks like this. On client side, ich can print the schema.
but the collect fails.
```rs
let config = SessionConfig::from_env().unwrap()
.with_option_extension(BallistaConfig::default())
.with_information_schema(true)
.with_target_partitions(16)
.ballista_restricted_configuration()
.with_ballista_logical_extension_codec(Arc::new(BallistaDeltaLogicalCodec::default()))
.with_ballista_physical_extension_codec(Arc::new(BallistaDeltaPhysicalCodec::default()))
.with_ballista_grpc_client_max_message_size(2 * 1024 * 1024 * 1024)
.with_ballista_job_name("demo")
.with_collect_statistics(true)
.set_u64("datafusion.sql_parser.recursion_limit", 500);
let state = session_state_with_s3_support(config)?;
let ctx = SessionContext::remote_with_state(&scheduler_host, state).await?;
let df_final: DataFrame = .....
println!("{}", df_final.schema()); // <--- Works
df_online.collect().await.unwrap(); // <--- fails
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
