Didn't knew you could do this. But sadly users preview the data uploaded before the process and require the information to be stored for archival purposes for a good amount of time (say years) so I do need the db.
I already talked to the web service provider and told them just that, and that they should add a method to send multiple rows at a time (or all of them), to avoid the connection overhead per row. They told me they would evaluate it, but I don't see them quite convinced, and they told me I must increase the amount of requests from 20 to say 100 that their service should be able to handle it. I was going to try that out, but there's where I wanted to use the scheduler to program my task and run a good number of workers. El viernes, 4 de abril de 2014 12:34:27 UTC-6, Leonel Câmara escribió: > > May I ask why are you storing the csv file rows in the database if what > you want is to send it to the webservice for processing? > > If there are only a dozen or so users at a time this could easily fit in > memory. I would store everything in cache ram with the session_id as key > and delete it after processing or after a certain time was elapsed. This > should speed up your processing by several orders of magnitude. > > That said, sending 20 rows at a time to the webservice when the files have > many thousands of rows, will always be extremely slow. At the very least, > you will have latency, tcp connection establishing, http connection > establishing, etc. per call. All that adds up to a huge amount of time. > > I would say that this webservice is not adequate for your objectives if > this being slow is a problem. You would need a webservice that would let > you send much more rows at a time. > > I am assuming that the webservice is not part of your application, and > that your application is not between the ajax calls in the client and the > webservice. So there's not much optimization for you to do there. > -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
