Ok thanks again for the input. Corey
On Wed, Jul 25, 2018 at 2:15 PM, Joshua Harlow <harlo...@fastmail.com> wrote: > So the only diff is that GreenThreadPoolExecutor was customized to work > for eventlet (with a similar/same api as ThreadPoolExecutor); as for > performance I would expect (under eventlet) that GreenThreadPoolExecutor > would have better performance because it can use the native eventlet green > objects (which it does try to use) instead of having to go threw the layers > that ThreadPoolExecutor would have to use to achieve the same (and in this > case as you found out it looks like those layers are not patched correctly > in the newest ThreadPoolExecutor). > > Otherwise yes, under eventlet imho swap out the executor (assuming you can > do this) and under threading swap in threadpool executor (ideally if done > correctly the same stuff should 'just work'). > > Corey Bryant wrote: > >> Josh, >> >> Thanks for the input. GreenThreadPoolExecutor does not have the deadlock >> issue, so that is promising (at least with futurist 1.6.0). >> >> Does ThreadPoolExecutor have better performance than >> GreenThreadPoolExecutor? Curious if we could just swap out >> ThreadPoolExecutor for GreenThreadPoolExecutor. >> >> Thanks, >> Corey >> >> On Wed, Jul 25, 2018 at 12:54 PM, Joshua Harlow <harlo...@fastmail.com >> <mailto:harlo...@fastmail.com>> wrote: >> >> Have you tried the following instead of threadpoolexecutor (which >> honestly should work as well, even under eventlet + eventlet >> patching). >> >> https://docs.openstack.org/futurist/latest/reference/index. >> html#futurist.GreenThreadPoolExecutor >> <https://docs.openstack.org/futurist/latest/reference/index. >> html#futurist.GreenThreadPoolExecutor> >> >> If you have the ability to specify which executor your code is >> using, and you are running under eventlet I'd give preference to the >> green thread pool executor under that situation (and if not running >> under eventlet then prefer the threadpool executor variant). >> >> As for @tomoto question; honestly openstack was created before >> asyncio was a thing so that was a reason and assuming eventlet >> patching is actually working then all the existing stdlib stuff >> should keep on working under eventlet (including >> concurrent.futures); otherwise eventlet.monkey_patch isn't working >> and that's breaking the eventlet api. If their contract is that only >> certain things work when monkey patched, that's fair, but that needs >> to be documented somewhere (honestly it's time imho to get the hell >> off eventlet everywhere but that likely requires rewrites of a lot >> of things, oops...). >> >> -Josh >> >> Corey Bryant wrote: >> >> Hi All, >> >> I'm trying to add Py3 packaging support for Ubuntu Rocky and >> while there >> are a lot of issues involved with supporting Py3.7, this is one >> of the >> big ones that I could use a hand with. >> >> With py3.7, there's a deadlock when eventlet monkeypatch of stdlib >> thread modules is combined with use of ThreadPoolExecutor. I >> know this >> affects at least designate. The same or similar also affects heat >> (though I've not dug into the code the traceback after canceling >> tests >> matches that seen with designate). And it may affect other >> projects that >> I haven't touched yet. >> >> How to recreate [1]: >> * designate: Add a tox.ini py37 target and run >> designate.tests.test_workers.test_processing.TestProcessingE >> xecutor.test_execute_multiple_tasks >> * heat: Add a tox.ini py37 target and run tests >> * general: Run bpo34173-recreate.py >> <https://bugs.python.org/file47709/bpo34173-recreate.py >> <https://bugs.python.org/file47709/bpo34173-recreate.py>> from >> issue >> 34173 (see below). >> [1] ubuntu cosmic has py3.7 >> >> In issue 508 (see below) @tomoto asks "Eventlet and asyncio >> solve same >> problem. Why would you want concurrent.futures and eventlet in >> same >> application?" >> >> I told @tomoto that I'd seek input to that question from >> upstream. I >> know there've been efforts to move away from eventlet but I just >> don't >> have the knowledge to provide a good answer to him. >> >> Here are the bugs/issues I currently have open for this: >> https://github.com/eventlet/eventlet/issues/508 >> <https://github.com/eventlet/eventlet/issues/508> >> <https://github.com/eventlet/eventlet/issues/508 >> <https://github.com/eventlet/eventlet/issues/508>> >> https://bugs.launchpad.net/designate/+bug/1782647 >> <https://bugs.launchpad.net/designate/+bug/1782647> >> <https://bugs.launchpad.net/designate/+bug/1782647 >> <https://bugs.launchpad.net/designate/+bug/1782647>> >> https://bugs.python.org/issue34173 >> <https://bugs.python.org/issue34173> >> <https://bugs.python.org/issue34173 >> <https://bugs.python.org/issue34173>> >> >> Any help with this would be greatly appreciated! >> >> Thanks, >> Corey >> >> ____________________________________________________________ >> ______________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> <http://openstack-dev-requ...@lists.openstack.org?subject:un >> subscribe> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev> >> >> >> ____________________________________________________________ >> ______________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev> >> >> >> ____________________________________________________________ >> ______________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib >> e >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
