villebro commented on issue #29839:
URL: https://github.com/apache/superset/issues/29839#issuecomment-2269979472
Summary from meeting today (@mistercrunch, @michael-s-molina and myself in
attendance):
- @villebro proposed migrating Thumbnails to the new framework first, as it
would be quite straight forward to migrate, and is known to cause significant
resource waste due to missing locking.
- @mistercrunch proposed investigating if Celery could be replaced with
something better maintained, like Dask. Some reasons for considering this
include:
- Lack of active maintenance of Celery
- Lack of advanced features that are available in Dask
- Opaque design of Celery, esp. the Kombu transport layer
- @villebro noted that it may be difficult to find a replacement for Celery
that works well with Flask. However, we will investigate if this would indeed
be possible, e.g. by extracting the needed Flask properties during task
creation and placing them in the payload, or proxying/wrapping them so that
Flask APIs are only used when available (e.g. `g`, `current_app`, `request`).
- @michael-s-molina proposed implementing the APIs for the new framework on
Celery, and later replacing it with something else. @villebro noted that the
choice of scheduler may well affect the API design, as the public APIs of the
scheduler will likely affect the final API.
- @michael-s-molina wanted to emphasize, that if we do introduce Dask (or
another replacement for Celery), we need to commit to a deprecation and removal
plan that concludes in the full removal of Celery in a future major version of
Superset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
