(sorry, I've deleted my previous message because I have more detailed information).
Thank you for pointing that out, I had seen that issue but I missed out that I had to use Scheduler(..., with_spawn=True) so I did that change and, apparently, scheduler is stable (hasn't crashed so far). However, since that change, all the tasks that are from a different app that the one from where the scheduler was run, fail with this traceback: Traceback (most recent call last): File "/home/limon/medios/gluon/scheduler.py", line 494, in executor functions = current._scheduler.tasks AttributeError: '_thread._local' object has no attribute '_scheduler' Let me explain a bit more: I have one web2py instance with several applications, each one with its own database. One of these apps is the main one and the scheduler connects to its database. ...web2py/applications/main/... ...web2py/applications/app1/... ...web2py/applications/app2/... I run three scheduler workers for the main app with this command: /opt/virtualenvs/py39/bin/python /home/limon/web2py/web2py.py -K main,main,main In applications/main/models/scheduler.py I instantiate the Scheduler like this: scheduler = Scheduler(db, max_empty_runs=0, heartbeat=5, use_spawn=True) And then, from several parts of my application I queue tasks. Some of these tasks are defined in the "main" application, but some others are defined in "app1" or "app2". Well, since the change to Scheduler(..., use_spawn=True) all the tasks within "main" application run ok, but all the other ones fail with the traceback I showed before. Notice I run the three scheduler workers for "main" application, I'm not sure if that has something to do with the issue. But I can confirm all this setup was working smoothly before use_spawn=True. What could be happening? Any help will be much appreciated. I'll keep investigating and post here if I find something. Thanks! El miércoles, 21 de junio de 2023 a la(s) 07:08:37 UTC-3, Massimiliano escribió: > There was an issue but should be fixed now. > > https://github.com/web2py/web2py/issues/1999 > > > Il giorno lun 19 giu 2023 alle ore 20:57 Lisandro <rostagno...@gmail.com> > ha scritto: > >> I've recently upgraded to web2py Version >> 2.24.1-stable+timestamp.2023.03.23.05.07.17 >> It's running on python 3.9.14, Rocky Linux RHEL9, using PostgreSQL 15.2 >> for database. >> >> Since I did the upgrade, the scheduler fails from time to time with this >> traceback: >> >> ERROR:web2py.scheduler.main#1531711: error storing result >> Traceback (most recent call last): >> File "/var/www/medios/gluon/scheduler.py", line 1077, in >> wrapped_report_task >> self.report_task(task, task_report) >> File "/var/www/medios/gluon/scheduler.py", line 1101, in report_task >> db(sr.id == task.run_id).update( >> File "/var/www/medios/gluon/packages/dal/pydal/objects.py", line 2789, >> in update >> ret = db._adapter.update(table, self.query, row.op_values()) >> File "/var/www/medios/gluon/packages/dal/pydal/adapters/base.py", line >> 586, in update >> raise e >> File "/var/www/medios/gluon/packages/dal/pydal/adapters/base.py", line >> 581, in update >> self.execute(sql) >> File "/var/www/medios/gluon/packages/dal/pydal/adapters/__init__.py", >> line 69, in wrap >> return f(*args, **kwargs) >> File "/var/www/medios/gluon/packages/dal/pydal/adapters/base.py", line >> 468, in execute >> rv = self.cursor.execute(command, *args[1:], **kwargs) >> psycopg2.OperationalError: server closed the connection unexpectedly >> This probably means the server terminated abnormally >> before or while processing the request. >> Traceback (most recent call last): >> File "/var/www/medios/gluon/scheduler.py", line 1077, in >> wrapped_report_task >> self.report_task(task, task_report) >> File "/var/www/medios/gluon/scheduler.py", line 1101, in report_task >> db(sr.id == task.run_id).update( >> File "/var/www/medios/gluon/packages/dal/pydal/objects.py", line 2789, >> in update >> ret = db._adapter.update(table, self.query, row.op_values()) >> File "/var/www/medios/gluon/packages/dal/pydal/adapters/base.py", line >> 586, in update >> raise e >> File "/var/www/medios/gluon/packages/dal/pydal/adapters/base.py", line >> 581, in update >> self.execute(sql) >> File "/var/www/medios/gluon/packages/dal/pydal/adapters/__init__.py", >> line 69, in wrap >> return f(*args, **kwargs) >> File "/var/www/medios/gluon/packages/dal/pydal/adapters/base.py", line >> 468, in execute >> rv = self.cursor.execute(command, *args[1:], **kwargs) >> psycopg2.OperationalError: server closed the connection unexpectedly >> This probably means the server terminated abnormally >> before or while processing the request. >> During handling of the above exception, another exception occurred: >> Traceback (most recent call last): >> File "/var/www/medios/gluon/shell.py", line 321, in run >> exec(python_code, _env) >> File "<string>", line 1, in <module> >> File "/var/www/medios/gluon/scheduler.py", line 949, in loop >> self.wrapped_report_task(task, self.execute(task)) >> File "/var/www/medios/gluon/scheduler.py", line 1082, in >> wrapped_report_task >> db.rollback() >> File "/var/www/medios/gluon/packages/dal/pydal/base.py", line 825, in >> rollback >> self._adapter.rollback() >> File "/var/www/medios/gluon/packages/dal/pydal/adapters/__init__.py", >> line 57, in wrap >> return f(*args, **kwargs) >> File "/var/www/medios/gluon/packages/dal/pydal/adapters/base.py", line >> 1012, in rollback >> return self.connection.rollback() >> psycopg2.InterfaceError: connection already closed >> >> >> I've checked PostgreSQL logs but there is no error or apparent problem at >> the time scheduler fails. The database instance has several databases and >> no error log is reported, everything runs smoothly. It's just the scheduler >> that reports that error (and after it, it doesn't run anymore). Where else >> should I look? >> >> Any help will be much appreciated. >> Warm regards, >> Lisandro >> >> -- >> Resources: >> - http://web2py.com >> - http://web2py.com/book (Documentation) >> - http://github.com/web2py/web2py (Source code) >> - https://code.google.com/p/web2py/issues/list (Report Issues) >> --- >> You received this message because you are subscribed to the Google Groups >> "web2py-users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to web2py+un...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/web2py/9ffdc6f1-8448-4784-a344-7f176545e9d9n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/web2py/9ffdc6f1-8448-4784-a344-7f176545e9d9n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Massimiliano > -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/web2py/07c8843d-7a78-48f9-bc97-aad5d343e672n%40googlegroups.com.
