I think the main point you need to assess if you have enough resources to do what you want. If your server can't keep up with the work you're asking it to do, then using scheduler or whatever won't bring any benefits. If instead you have a not-so-snappy user-interface because you don't have enough power to respond quickly to a user, then the scheduler is a right fit.
If you're seeing all the "hurdle" of the system on the db backend, then you're in the same situation as the first one, i.e. you need to "buy more juice". On Thursday, April 3, 2014 1:33:59 AM UTC+2, Francisco Betancourt wrote: > > Hello everyone. > > I need some help defining the architecture for an application. We already > have something in place, but the size of the the users data files is making > the site very unresponsive. The app works as follows: > > > 1. The user uploads a csv file > 2. The user file is read to a table for further processing > 3. The user previews the data I read from his file (to see if > everything is ok) > 4. If data is ok he clicks a button which will make us generate an xml > (using some of the data he uploaded) and send the xml to a web service > 5. The web service is not ours, and we must send the xml files one at > a time (though we can send hundreds simultaneously) > 6. The web service returns data, and we store that data into the db > 7. Once done we offer a print friendly version of the final data > > So currently we are doing the following: > > 1. Receive a file, and once saved we process it with web2py built in > import from csv. > 2. Once data has been read we show a view with all the rows and a > button to start the process > 3. Using js we send groups of 20 rows at a time through ajax, to be > processed (to xml) and sent to the web service > 4. Each ajax call returns js code to update a progress bar > > Originally this files were suppose to consist of hundreds hardly a > thousand row, but in the end we have files with 15000 rows and the average > is about 4000. Incredibly the view (even with 15000 rows and js) works > well. But the site becomes quite unresponsive (especially because there are > normally a dozen or so users doing the same thing simultaneously). > > We have already done as many optimizations as we know (use migrate=False > for db, call session.forget in ajax controller functions, byte compile the > code) but it still is way too slow. So we are making some heavy changes. I > want to try to use the scheduler and workers to do this job, but I fear it > will generate even more db queries and make things worse. So I would like > some suggestions on how to tackle this problem?? > > Is scheduler the way to go? An if so should I make a task for every xml > and webservice call (this seems like a bad idea to me). Or should I group > them in chunks of an arbitrary size (e.g. 50)? And if I make this change > will I be able to display the progress of the process? > > Thanks for all the help. Good day. > -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
