Hi, Yeah celery is the way to go. There's https://github.com/sontek/pyramid_celery to check out. Some things to keep in mind is, you'll need a shared DB, good-old ZODB with filestorage is not enough, because more processes will need to have access. Do not store valuable data in the celery queue, consider it ephemeral, go through a real DB.
On Tuesday, January 5, 2021 at 10:31:15 PM UTC+1 tfl...@gmail.com wrote: > Hi Jonathan, > > Thank you for this description of Celery! > I'll try to have a look at it if I can get a little time, it seems to be a > good replacement (probably more reliable!) of my own developments... > > Best regards, > Thierry > -- > https://www.ulthar.net -- http://pyams.readthedocs.io > > Le mar. 5 janv. 2021 à 22:18, 'Jonathan Vanasco' via pylons-discuss < > pylons-...@googlegroups.com> a écrit : > >> Thierry, >> >> That is what I mostly use Celery for (it's also used for generating >> Reports, ACME SSL Certificates and a few other things). A Pyramid Request >> will defer a job(s) to Celery via the "Transport Backend" and receive a >> task/messaging ID. Subsequent requests will poll the Celery "Result >> Backend" for the status of that job, based on the ID. This way we show >> the "Still processing!" message to users. >> >> Celery is run as a worker with multiple processes. The Celery task >> manager grabs a task off the queue, then does the resizing, uploads to S3, >> and notifies the Result Backend when complete. I use Redis for Result and >> Transport. I think there was once a ZeroMQ integration; there is >> definitely RabbitMQ integration, it's one of the more popular options. >> >> On Monday, January 4, 2021 at 12:25:44 PM UTC-5 tfl...@gmail.com wrote: >> >>> Hi Jonathan, >>> >>> I didn't have a look at Celery yet, maybe it could work... >>> My goal is to be able to start long-running tasks (medias files >>> conversions for example) whose definition is stored in database and managed >>> by end users from my main Pyramid application; for this, I create a >>> dedicated sub-process which is "managed" using ZeroMQ messages (I use this >>> because it's also used for other non-Python/Pyramid applications). >>> Tasks are then started using dedicated threads. >>> >>> Everything is OK until now... My only requirement now is to be able to >>> get access to main Pyramid's registry from my sub-process and it's threads, >>> and I don't really understand why I get a pointer to >>> ZCA global registry... :( >>> >>> Best regards, >>> Thierry >>> -- >>> https://www.ulthar.net -- http://pyams.readthedocs.io >>> >>> Le lun. 4 janv. 2021 à 17:51, 'Jonathan Vanasco' via pylons-discuss < >>> pylons-...@googlegroups.com> a écrit : >>> >>>> Have you considered using Celery for this? I started offloading >>>> everything related to subprocesses and message queues to it a while back, >>>> and have been much happier. >>>> >>>> On Monday, January 4, 2021 at 11:20:46 AM UTC-5 tfl...@gmail.com wrote: >>>> >>>>> Hi, >>>>> >>>>> I need to create custom sub-processes from my main Pyramid >>>>> application; these processes are used to handle "commands" received from >>>>> ZeroMQ messages. >>>>> If I use the components registry to register adapters and utilities >>>>> which are also required in these processes, is there a way to start these >>>>> processes so that they can "inherit" from the main application registry ? >>>>> >>>>> I'm actually using Pyramid 1.10.5, and I generally do a >>>>> "config.hook_zca()" on application startup before doing >>>>> the "config.setup_registry()" and "config.scan()". >>>>> I tried to do the same operations in my sub-process "run" method, but >>>>> I always get the "global" registry instead of my "webapp" registry which >>>>> was configured in my main application... >>>>> >>>>> Best regards, >>>>> Thierry >>>>> -- >>>>> https://www.ulthar.net -- http://pyams.readthedocs.io >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "pylons-discuss" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to pylons-discus...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/pylons-discuss/9a3e8a04-1a29-4f18-885e-dc3ac1c759b3n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "pylons-discuss" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to pylons-discus...@googlegroups.com. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/pylons-discuss/5ac83ff1-747f-4df9-994c-ffa48f339013n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/pylons-discuss/5ac83ff1-747f-4df9-994c-ffa48f339013n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discuss+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/0a0d6985-c5b5-48b6-88f9-cce5d837c583n%40googlegroups.com.
