[Twisted-Python] Running CPU bound function concurrently in twisted

2019-06-23 Thread Chengi Liu
Hi, I tried posting on SO, but havent been able to solve the problem. https://stackoverflow.com/questions/56696562/running-long-blocking-calculations-in-parallel-in-twisted I have a very noob question. I have a function, which takes a second or two to process (CPU bounded) and then there is a fo

Re: [Twisted-Python] Running CPU bound function concurrently in twisted

2019-06-24 Thread Chengi Liu
Thanks for pointing that out. Is there a dead simply example on how to parallelize a function using ampoule. Again, I have a function.. I have a for loop.. the function takes in an argument from loop.. but the for loop is extremely parallelizable. Also, I was going thru this awesome twisted tutori

Re: [Twisted-Python] Running CPU bound function concurrently in twisted

2019-06-24 Thread Chengi Liu
Thanks Moshe & Meejah & Gelin for the suggestions and advice. This is super helpful. I think, I am able to move forward with this. Let me just summarize this.. My usecase is.. fetch the data.. and then you assemble the data. Fetching data is network bound, assembling data is like CPU bound. Can y

Re: [Twisted-Python] Running CPU bound function concurrently in twisted

2019-06-24 Thread Chengi Liu
The stack overflow link: https://stackoverflow.com/questions/56696562/running-long-blocking-calculations-in-parallel-in-twisted On Mon, Jun 24, 2019 at 7:26 PM Chengi Liu wrote: > Thanks Moshe & Meejah & Gelin for the suggestions and advice. This is > super helpful. > > I

Re: [Twisted-Python] Running CPU bound function concurrently in twisted

2019-06-24 Thread Chengi Liu
ow, what is great is that with all your help, I am able to make this flow non-blocking. But, would definitely like to exploit parallellism wherever feasible. On Mon, Jun 24, 2019 at 7:37 PM Moshe Zadka wrote: > On Mon, Jun 24, 2019, at 16:27, Chengi Liu wrote: > > cpu_res *= yield