On Tue, Aug 19, 2014 at 5:43 AM, manjunath <[email protected]> wrote: > we have the following scenario and we are greatly stuck to scale it and > cannot find the problem > > our processflow is as follows: > -request received from client and headers are collected > -initiate a async.waterfall >>>get some data from redis >>>build some object based on redis >>>build an array of tasks for async.parallel to send requests to 3-5 servers >>>run a async.parallel using above tasks >>>>>if the async.parallel got some results in any of the request sent in >>>>> parallel, send that to client and end(header is set to connection:close >>>>> for >>>>> client) >>>>>if not then run an async.waterfall to send requests to different set of >>>>> servers one after another in a synchronous manner, we are waiting for one >>>>> to >>>>> close before other one is initiated(wait.for is used with fibers) >>>>>>>same here if there is some result from request in waterfall then send >>>>>>> it to client and end > > we are trying to hit this to concurrency at 200(having plans to increase to > 2000) but still failing, we are facing gradual increase in the response time > from our system > we are using cluster also in this app, we have a guess this might be due to > sockets issue, can anyone help us out.
Looks like a fun problem! Zones is perfect for this use-case. They can be used to encapsulate a number of parallel async activities, and the first to complete can return a result for the Zone, and it will tear down all the other ongoing activities... see https://github.com/strongloop/zone. You could even try it... if it works for you, who cares if it has bugs that you don't hit, right? :-) And we could use the feedback. But, bleeding edge concurrency primitives aside, a few suggestions: - very unlikely that cluster is your problem, it doesn't have any effect (you are on 0.10, right? you didn't say, node version is important) except making sure listening sockets are shared across the cluster, shouldn't do anything with outgoing client connections - easy way to verify: run without cluster.... always a good idea anyhow, if it doesn't work standalone, it won't work with cluster. You don't have any way to cancel your parallel actions, that you mention, so when the first of a set of parallel actions completes.... all the others keep going.... so with fast incoming connections, every complete request will "leak" (for a little while) some set of incomplete outgoing connections... a brutal multiplier effect, made worse the more "irregular" the response time is... incoming requests/sec will be as fast as the fastest outgoing request, but accumulation of un-needed outgoing connections will be based on the slowest of the outgoing response times. Ouch. if you have fd limits (remove them), you can hit those. if you have connection pool max size limits (outgoing mysql connections, for example, or the HTTP outgoing concurrent request limit which defaults to 5, http://nodejs.org/api/http.html#http_agent_maxsockets) your dead-but-ongoing requests can slam into those barriers fast with the multiplier effect you have. This might be capping your incoming request completion rate. You don't mention logging... you should count current number of incoming and outgoing incomplete requests, and log those counts. Consider using vasync, it is more inspectable, might make it much easier to dump current state. Also, it appears you may benefit from its barrier(). For "interrupting", I'd suggest finding a way with all outgoing requests to cancel, to teardown the connection and terminate it early, if possible, returning an "interrupted" error result to async ASAP after being cancelled. Be interesting to hear what you did after you figure this out! Cheers, Sam -- Job board: http://jobs.nodejs.org/ New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/CACmrRmSeuXVo8faGNq80KyhAgkV1h6iohNvSxdc7rPTSYvindg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
