SameerMesiah97 opened a new pull request, #63035: URL: https://github.com/apache/airflow/pull/63035
**Description** This change adds support for callback workloads (`ExecuteCallback`) to the `AwsLambdaExecutor`. Previously, the executor only supported task workloads (`ExecuteTask`). This update extends the executor to accept, queue, and execute callback workloads alongside task workloads. The executor now maintains a `queued_callbacks` structure and updates `queue_workload()` to register `ExecuteCallback` workloads using the callback identifier (`callback.id`) as the workload key. The workload processing flow has been updated to support callbacks throughout the executor lifecycle. `_process_workloads()` now handles both `ExecuteTask` and `ExecuteCallback` workloads, dispatching them through the same execution pathway. `execute_async()` has been extended to serialize both workload types and forward them to the Lambda runtime using `python -m airflow.sdk.execution_time.execute_workload`. Additionally, `attempt_task_runs()` and related task tracking logic have been updated to support string-based workload identifiers for callbacks while maintaining JSON-serialized `TaskInstanceKey` identifiers for task workloads. **Rationale** The ExecutorCallback framework introduced in Airflow 3.2 allows executors to execute synchronous callbacks on worker infrastructure. Executors must therefore be able to accept and dispatch `ExecuteCallback` workloads in addition to task workloads. `AwsLambdaExecutor` delegates execution to the Task SDK runtime by invoking `airflow.sdk.execution_time.execute_workload` inside the Lambda environment. As reflected in the direction of PR #62645, the expectation is that handling of both task and callback execution will ultimately occur within the Task SDK runtime rather than inside individual executors. Forwarding callback workloads to the same runtime entrypoint used for tasks aligns the Lambda executor with this model. **Tests** Added unit tests verifying that: * Callback workloads are queued, processed, and dispatched to Lambda using the Task SDK runtime command. * Callback workloads correctly propagate queue overrides specified in `callback.data["queue"]`. * Callback workloads are correctly adopted using string-based `external_executor_id` values. **Documentation** Docstrings have been updated in `AwsLambdaExecutor` to describe support for both task and callback workloads. **Backwards Compatibility** This change preserves existing behavior for task workloads. Task workloads continue to use JSON-serialized `TaskInstanceKey` identifiers for executor tracking and scheduler adoption. Callback workloads use their callback identifier string as the workload key. Related: #62887 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
