woah. that's a lot of unconcerned copy-paste around the web. Did you even try to start "small" and then see what your issues would be ?
let's take every statement in order. I need to render a page that requires a number of REST API calls to > external web services that take a significant amount of time to return. > > that's not a "problem" for any serious programming language. The thing is, usually webservers put a hard timeout on the time it takes to generate a result, so if your "rest api calls" are taking more than -e.g.- 60 seconds, the right thing to do is to avoid using the webserver and let another "thing" to do it, and store the results somewhere your page can fetch them, without incurring in the timeout > What is the best approach to fire all of these calls simultaneously > instead of sequentially? > ajax would be my first and obvious choice. > > Should I try something like Celery: > https://groups.google.com/forum/#!topic/web2py/js0VAKK6Dlw > https://code.google.com/p/web2py-celery/ > > Seems like this approach wasn't fully developed. > that is the "old way" to let celery integrate with python. the newer celery is usable without any dependency on specific code (or so it's advertised to) > > I thought about using the web2py scheduler: > http://web2py.com/books/default/chapter/29/04/the-core#web2py-Scheduler > but this seemed more appropriate for background cronjobs (like email > batching), rather than live page rendering. > > Also, I'm not sure if it's truly asynchronous. > trying wouldn't hurt, since setting it up takes 3 lines of code. I can process 3000 tasks divided upon 5 workers in under 200 seconds, on modest hardware. If 15 tasks per second are not enough, sure, scheduler isn't for you, but you may very well being working on something at the "twitter" level....... you'd need some power-tools to let your application behave like it should. > > I realize this usually would be done client side (There are some security > reasons for doing this server-side rather than just JS ajax calls). > I don't see why you couldn't do this server-side, taken for granted the timeout considerations in the first sentence > > I've also read that due to the python UWSGI that true multi-threading > might not be achievable in web2py. Is this the case? > > I really don't know where you copy-pasted this: it's blatantly false, and web2py would have no practical use for nobody if that was the case > Am I going about this the wrong way and should I use a hammer (node.js) > instead of a nice powertool drill (web2py)? > > Don't know really. <ironic mode on> You could need node.js workers attached to a hive powered by pig cluster fetching data continously from cassandra, storing temporal results on mongodb, searching them with elasticsearch or solr, decentralizing your resultsets on foundationdb, keeping scores with redis backed by lua scripts, fetching them asyncronously with goroutines and denormalizing them in couchdb.... then do a graph analysis on neo4j, passing messages around with the help of rabbitmq. All of it running on a severely optimized kernel recompilation of freebsd done by yourself to power your own fleet of ARM physical servers. Additionally, you may use websockets provided by pusher.com and adopt a CDN to speed things up even further than approaching amazon S3, or even better cloudflare. You can serve the layout from web2py, and that would be pretty much about your application. <ironic mode off> Sorry, I was feeling a taddle bit poetic. Nothing in web2py is stopping you from using ANY other tool to do the job, but premature optimization is the root of all evil. Start small and go from there, without copy-pasting the entire web of buzzword you can find around. See for yourself. -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
