wenjin272 commented on issue #1016:
URL: https://github.com/apache/flink-agents/issues/1016#issuecomment-5630650357
I agree that this issue is primarily about the recovery identity model,
rather than the batch execution API. However, pltbkd’s question about how
`durable_id` should be propagated into `DurableCall` made me revisit the Python
batch API introduced in #926. I did not look closely enough at the Python API
shape during that review, so I missed this concern at the time.
Instead of adding another `durable_id` field to `DurableCall`, or requiring
users to wrap its `func` with `with_durable_id`, could we avoid exposing
`DurableCall` as a required user-facing abstraction?
`durable_execute_async` already represents a single durable invocation and
accepts `durable_id`. Ideally, batch execution could compose those existing
calls:
```python
outcomes = await ctx.durable_gather(
ctx.durable_execute_async(foo, x, durable_id="foo-x"),
ctx.durable_execute_async(bar, y, durable_id="bar-y"),
)
```
If standard asyncio primitives can be supported, this could even be
`asyncio.gather(...)`, similar to LangGraph’s Functional API. Otherwise,
`ctx.durable_gather(...)` could provide the same Python-native shape while
using the existing durable batch implementation internally.
With this design, each call carries its own recovery identity when it is
created, while `durable_gather` is only responsible for coordinating
concurrency, preserving result order, and recovering the calls as one batch.
Therefore, we would not need a separate `durable_id` representation or
propagation mechanism for `DurableCall`; `DurableCall` could remain an internal
implementation detail.
There is still some implementation work involved because the current
`AsyncExecutionResult` is lazy and explicitly does not support
`asyncio.gather`. The runtime would need to collect the call handles and plan
or reserve the complete batch before starting execution. However, that
complexity can remain internal instead of being reflected in the public Python
API.
This would also allow the current discussion to stay focused on the actual
identity semantics: an explicit `durable_id` should be the authoritative
recovery identifier, while the automatically derived identity can remain the
backward-compatible fallback when no `durable_id` is provided. WDYT @pltbkd
@da-daken ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
