this is how you queue tasks, and there's nothing wrong with it. The missing point here is 1. : there is a limit to the number of concurrent workers, which is how many you start from the commandline... and, BTW, even if you queue 10k tasks, you can't expect to start 10k scheduler processes on a single server and observe every task completed in a single loop. web2py's scheduler with a postgresql backend starts to show its limits with 30~50 workers on modest hardware, i.e. even if you launch more, they won't get speedier in regards of concurrent execution. Given that it seems that you need to ingest data and probably the ingested goes on the same database, there's another probable concurrency penalty to consider.
On Monday, August 24, 2015 at 12:42:58 AM UTC+2, Phillip wrote: > > 1.My goal is to use the scheduler to concurrently process as many files as > possible (I assume there is no limit to the number of processes and the > scheduler will prevent the server from dropping them). > > 2.My tasks are defined in a module, not model. > > 3.It is in the view that a potentially large number of items are selected > which triggers the callback and process below. > > 4.* I*n the function importData, the database is accessed and records > added. > > > for dataID in dataIDs: > > scheduler.queue_task(ImportData, [dataID], timeout=100) > > > Is there anything I am missing to properly and fully utilize the scheduler > for this purpose? > > Please post any caveats, corrections, or tips, and let me know if any > other information is needed. > > > Thanks, > > PV > -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
