[issue29569] threading.Timer class: Continue periodical execution till action returns True
slytomcat added the comment: done 2017-06-06 14:28 GMT+03:00 Cheryl Sabella : > > Cheryl Sabella added the comment: > > This bug report has been closed, but the PR is still open. Please close > the PR. Thank you. > > -- > nosy: +csabella > > ___ > Python tracker > <http://bugs.python.org/issue29569> > ___ > -- ___ Python tracker <http://bugs.python.org/issue29569> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29569] threading.Timer class: Continue periodical execution till action returns True
Changes by slytomcat : -- pull_requests: +81 ___ Python tracker <http://bugs.python.org/issue29569> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29569] threading.Timer class: Continue periodical execution till action returns True
New submission from slytomcat: I think that functionality of threading.Timer class can be easily extended to generate the sequence of runs with specified period. The idea comes from the GLib.timeout_add function. Run method of threading.Timer should look like: def run(self): """Continue execution after wait till function returns True""" while(not self.finished.wait(self.interval)): if not self.function(*self.args, **self.kwargs): break self.finished.set() -- components: Library (Lib) messages: 287858 nosy: slytomcat priority: normal severity: normal status: open title: threading.Timer class: Continue periodical execution till action returns True type: enhancement versions: Python 3.7 ___ Python tracker <http://bugs.python.org/issue29569> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29603] More informative Queue class: new method that returns number of unfinished tasks
New submission from slytomcat: Class queue.Queue control the number of unfinished tasks via method task_done(). But it is only possible to get the information about all task done (via join() method). I'm sure that exposing the number of unfinished tasks (unfinished_tasks class variable) can be very useful in many situations when you need more control over the process. But it is not good idea to provide write access to this internal variable (as it controls internal queue class status). Better way - provide RO access via class method like qsize or empty. It can be look like this: def unfinished(self): return self.unfinished_tasks One example of this method usage: there is not optimal function _adjust_thread_count in concurrent.futures.ThreadPoolExecutor with following comment: # TODO(bquinlan): Should avoid creating new threads if there are more # idle threads than items in the work queue. It can be easily done with following condition: if self._work_queue.unfinished() <= len(self._threads): return -- components: Library (Lib) messages: 288189 nosy: rhettinger, slytomcat priority: normal severity: normal status: open title: More informative Queue class: new method that returns number of unfinished tasks versions: Python 3.7 ___ Python tracker <http://bugs.python.org/issue29603> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29569] threading.Timer class: Continue periodical execution till action returns True
Changes by slytomcat : -- pull_requests: +157 ___ Python tracker <http://bugs.python.org/issue29569> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29603] More informative Queue class: new method that returns number of unfinished tasks
Changes by slytomcat : -- pull_requests: +158 ___ Python tracker <http://bugs.python.org/issue29603> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29603] More informative Queue class: new method that returns number of unfinished tasks
Changes by slytomcat : -- type: -> enhancement ___ Python tracker <http://bugs.python.org/issue29603> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29569] threading.Timer class: Continue periodical execution till action returns True
Changes by slytomcat : -- pull_requests: -81 ___ Python tracker <http://bugs.python.org/issue29569> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29603] More informative Queue class: new method that returns number of unfinished tasks
slytomcat added the comment: Raymond, Serhiy, thanks for your opinions. I agree that this method like empty, full, and qsize returns information that may be out-of-date in time of its usage. But like those empty, full, and qsize it provides information that helps to make some decisions. What I disagree that this information can be correctly estimated based on qsize and number of parallel consumers as there is no information what exactly consumers do at the moment. Such estimation will be absolutely useless for a decision making. In the example (see https://github.com/slytomcat/cpython/commit/ea313986d86c083e81624fbc8a7ab6a75784e15d) we need to estimate the number of unfinished tasks and comparing it to the number of active threads we can decide: should we create a new thread or not. There is no reason to wait until all tasks done (via join) we need to make decision in working conditions. Actually I've implemented this solution (Queue.unfinished + modified concurrent.futures.ThreadPoolExecutor) in my project (see https://github.com/slytomcat/yandex-disk-client) and it works perfectly: the number of working threads is increasing only on massive flow of tasks, while weak flow of tasks keeps low number of working threads. And even when Queue.unfinished gives wrong (outdated) information, it can lead to creation one new thread that is not really required at the moment. That is not a big mistake, I suppose (comparing to creation all N threads during placing first N tasks in queue without considering the volume of concurrent tasks). -- ___ Python tracker <http://bugs.python.org/issue29603> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29603] More informative Queue class: new method that returns number of unfinished tasks
slytomcat added the comment: Not exactly there are 3 cases: If qsize <> 0 it means there is no idle consumers threads, all of them must be busy: we need to create one more. No doubt. If qsize = 0 it means one of two cases: - all consumers threads are busy: we need to create one more - some or all consumers threads are idle: no need create new one. But there is no way to distinguish two last cases. That's why I consider that (unfinished() <= num_threads) gives clear key for decision. -- ___ Python tracker <http://bugs.python.org/issue29603> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29603] More informative Queue class: new method that returns number of unfinished tasks
slytomcat added the comment: One more problem that adjusting of number of threads is performed exactly after placing the new task in the queue. In in some cases we can find that qsuze <> 0 but it doesn't mean that there is no idle threads. It can be equal to 1 (just queued task) as no threads manage to get it yet (don't forgot about GIL). So even qsuze <> 0 dosn't mean that we need to create a new thread. But (unfinished() <= num_threads) works perfect in this case also. -- ___ Python tracker <http://bugs.python.org/issue29603> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29603] More informative Queue class: new method that returns number of unfinished tasks
slytomcat added the comment: num_threads - unfinished() = the estimation for number of idle threads. -- ___ Python tracker <http://bugs.python.org/issue29603> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29603] More informative Queue class: new method that returns number of unfinished tasks
slytomcat added the comment: I can't fully agree with this: >Queue objects are primarily about managing a queue of inputs. >Monitoring and managing consumers is another (mostly orthogonal) realm. Monitoring of consumers is already added via task_done() and join() methods. At least this two methods allows to understand that all consumers are in idle state. The unfinished() - is just functional extension to this existing functionality. >For other users, the new method is problematic because it is distracting >and prone to misuse (potentially out-of-date upon return and potentially >meaningless if task_done isn't being used). I believe the new method >will cause more problems than it fixes. This sounds reasonable because I also understand that this method is not usable in all cases as in most cases task_done is not used. Probably my idea for method unfinished is really not so good... Actually I've manage to find unpublished Queue.unfinished_tasks variable just in several minutes when I tried to find solution for threaded PoolExecutor. Hope that any curious developer can find it too. I don't mind if you close this CR. Thanks for pleasant discussion. -- ___ Python tracker <http://bugs.python.org/issue29603> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue29603] More informative Queue class: new method that returns number of unfinished tasks
slytomcat added the comment: I've closed pull request on GitHub. -- stage: -> resolved status: open -> closed ___ Python tracker <http://bugs.python.org/issue29603> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
