villebro opened a new issue, #29839: URL: https://github.com/apache/superset/issues/29839
## [SIP] Proposal for Global Async Task Framework ### Motivation *Note*: This replaces [[SIP-141] Global Async Queries 2.0](#29515) which aimed at completing [[SIP-39] Global Async Query Support](#9190). ### Proposed Change Superset currently has varied and and often times opaque solutions for executing async operations (all require Celery): - SQL Lab supports async query execution, and utilizes long polling for checking for results - Thumbnails and Alerts & Reports are executed as background Celery tasks - Chart queries can be executed async by enabling the GAQ feature flag, and supports both WebSocket and long polling - Scheduled cron jobs (added via `CeleryConfig` in `superset_config.py`) Currently none of the above support deduplication or cancellation of queued or running tasks, or even viewing which tasks are queued/executing. In addition, beyond sharing Celery as the execution backend, there is limited sharing of code, like utils or ORM models, which has lead to significant code duplication. This both increases the risk of regressions, limits reusability of good functionality and adds significant maintenance burden. For this reason this SIP recommends adding a new Global Async Task Framework (GATF), which will introduce the following: - A new ORM model with a menu which makes it possible to view and cancel queued or executing tasks. Admins will have access to all tasks, while regular users will only be able to view tasks they have spawned. This model will be ephemeral by nature, i.e. the task entries will be removed once they are completed. - Add locking for all tasks to ensure deduplication. This applies particularly to async chart queries and thumbnails, which currently can cause significant resource waste. - Deprecate long polling in both chart and SQL Lab queries - going forward only WebSockets would be supported. ### New or Changed Public Interfaces A new ORM model would be introduced for async tasks with a string based identifier. When a new task is created, an entry is added to the table if it's not already there. For instance, for thumbnails, we would use the digest as the identifier. And for chart queries, we would use the cache key and so on. If the entry is already there, we consider the task already locked, and don't schedule a new one. In addition, there would be the following fields (non exhaustive list, mainly here to give a general picture of the model): - created, updated etc (the basic stuff we have in other similar models) - type (alerts & reports, sql lab query, chart query, thumbnail) - status (queued, executing) - payload (any necessary ) - last heartbeat (the executor thread would post a heartbeat here evey x seconds, ensuring that tasks that fail due to worker malfunction can be cleaned up) We propose making WebSockets the sole mechanism for broadcasting task completion. This means that we will remove long polling support from async chart queries, and replace long polling in SQL Lab with WebSockets. ### New dependencies None - however, we would likely make the WebSocket server mandatory for both SQL Lab and async chart queries. ### Migration Plan and Compatibility **Phase 1 - GAQ** In the first phase, we clean up GAQ by simplifying the architecture (see details about redundant form data cache keys etc from SIP-141) and remove long polling support. We also introduce the new ORM model, along with a UI, and migrate the GAQ metadata from Redis to the new ORM model. At the end of phase one, we will have removed long polling from GAQ, and will have both chart query deduplication and cancellation support. #### Phase 2 - The rest In the second phase, we migrate the remaining async tasks to GATF. At the end of this phase, thumbnails will be deduplicated, and it will be possible to cancel any queued or running async task via the UI. ### Rejected Alternatives SIP-39 and SIP-141 were rejected in favor of making a more general purpose Async Task Framework. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
