Hello everyone. I need some help defining the architecture for an application. We already have something in place, but the size of the the users data files is making the site very unresponsive. The app works as follows:
1. The user uploads a csv file 2. The user file is read to a table for further processing 3. The user previews the data I read from his file (to see if everything is ok) 4. If data is ok he clicks a button which will make us generate an xml (using some of the data he uploaded) and send the xml to a web service 5. The web service is not ours, and we must send the xml files one at a time (though we can send hundreds simultaneously) 6. The web service returns data, and we store that data into the db 7. Once done we offer a print friendly version of the final data So currently we are doing the following: 1. Receive a file, and once saved we process it with web2py built in import from csv. 2. Once data has been read we show a view with all the rows and a button to start the process 3. Using js we send groups of 20 rows at a time through ajax, to be processed (to xml) and sent to the web service 4. Each ajax call returns js code to update a progress bar Originally this files were suppose to consist of hundreds hardly a thousand row, but in the end we have files with 15000 rows and the average is about 4000. Incredibly the view (even with 15000 rows and js) works well. But the site becomes quite unresponsive (especially because there are normally a dozen or so users doing the same thing simultaneously). We have already done as many optimizations as we know (use migrate=False for db, call session.forget in ajax controller functions, byte compile the code) but it still is way too slow. So we are making some heavy changes. I want to try to use the scheduler and workers to do this job, but I fear it will generate even more db queries and make things worse. So I would like some suggestions on how to tackle this problem?? Is scheduler the way to go? An if so should I make a task for every xml and webservice call (this seems like a bad idea to me). Or should I group them in chunks of an arbitrary size (e.g. 50)? And if I make this change will I be able to display the progress of the process? Thanks for all the help. Good day. -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
