Thanks everyone, I very much appreciate the help. I will give both of these a shot! Node.js can be damn confusing at times.
Timothy Harding RPCV Vanuatu Skype: hardi...@gmail.com +1 (541) 632-6623 On Tue, Aug 25, 2015 at 1:57 AM, Lars Helge Øverland <larshe...@gmail.com> wrote: > Hi Tim, > > you can use promises to handle the async requests, or simply use > synchronous requests. No definite answers here but I find using synchronous > requests easier to work with and user interface responsiveness is not an > issue for server side ad-hoc scripts like this. > > The urllib-sync library will wrap urllib and provide sync behavior: > > https://www.npmjs.com/package/urllib-sync > > Install with npm: > > npm install urllib-sync > > Import: > > var urlsync = require('urllib-sync'); > > Then do synchronous requests/responses like this: > > var response = urlsync.request(url, postOptions); > > > > > Lars > > > > > > > > On Tue, Aug 25, 2015 at 10:35 AM, Mark Polak <m...@thedutchies.com> wrote: > >> Hey Timothy, >> >> Is there a reason why you would not want to wait for the response? You >> have figured out you're sort of flooding the server with a bunch of >> requests. >> >> You can still send them in batches of say 5, wait till those are done >> then send the next 5, etc. >> >> Node now supports Promises so you could transform your requests into >> promises and handle them in batches by using Promise.all(). >> >> The code below i just wrote up quickly without running it so you might >> need to see if it works and adjust it, but it was more to give you an idea. >> >> Regards, >> >> Mark >> >> function doRequests(dataElementsToUpdate) { >> //Map over the items >> return dataElementsToUpdate.map(function (dataElementToUpdate) { >> //Create a promise for each item and call resolve when that >> request is done. >> return new Promise(function (resolve, reject) { >> >> var postData = JSON.stringify(dataElementsToUpdate[i]); >> serverConfigPOST.dhis2server.path = >> "/dhis/api/sharing?type=dataElement&id=" + dataElementsToUpdate[i]. >> object.id; >> serverConfigPOST.dhis2server.headers = { >> 'Content-Type': 'application/json', >> 'Content-Length': postData.length >> }; >> console.log(serverConfigPOST); >> >> var req = https.request(serverConfigPOST.dhis2server, >> function(res) { >> console.log('STATUS: ' + res.statusCode); >> console.log('HEADERS: ' + >> JSON.stringify(res.headers)); >> res.setEncoding('utf8'); >> res.on('data', function (chunk) { >> console.log('BODY: ' + chunk); >> }); >> res.on('end', function () { >> resolve({item: dataElementsToUpdate[i].object, >> success: true); >> }); >> }); >> >> req.on('error', function(e) { >> //console.log('problem with request: ' + e.message); >> resolve({item: dataElementsToUpdate[i].object, >> success: false, message: e.message); >> }); >> }); >> }); >> } >> >> function updateSharingInBatches() { >> //Pick 5 elements to process next >> var itemsToProcess = dataElementsToUpdate.splice(0, 5); >> >> //Only continue if there are more elements >> if (itemsToProcess.length) { >> //Wait til the passed batch is done and excute the next after >> return Promise.all(doRequests(itemsToProcess)) >> .then(function (responses) { >> updateSharingInBatches(dataElementsToUpdate); >> }) >> //If you want to >> .catch(function (responses) { >> console.log('One or more did not work'); >> //If you wish to continue with the batches you can just >> call >> //updateSharingInBatches(dataElementsToUpdate); >> //Or you can try run the failed one again. :) >> }) >> } >> } >> >> //Run the whole thing >> updateSharingInBatches(dataElementsToUpdate); >> >> >> >> On Mon, Aug 24, 2015 at 6:45 PM, Timothy Harding <hardi...@gmail.com> >> wrote: >> >>> Thanks Pierre, >>> >>> Not sure how I would set up the recursive loop in javascript, I had >>> heard about async from stack overflow as well, I was thinking of trying >>> that, but I kind of like how quickly this works, not needing to wait for >>> the response from one request before sending the next. The DHIS2 hardware I >>> am working with just doesn't seem to keep up after a certain number of >>> requests however, hence my attempt to slow them down. >>> >>> Timothy Harding >>> RPCV Vanuatu >>> Skype: hardi...@gmail.com >>> +1 (541) 632-6623 >>> >>> On Mon, Aug 24, 2015 at 9:39 AM, Pierre Dane <pie...@jembi.org> wrote: >>> >>>> might be best to try a recursive loop using setInterval or >>>> setTimeout with a configurable value (I.e. a method calling itself) . or >>>> you could look at the async library to make things synchronous . >>>> >>>> >>>> On Monday, August 24, 2015, Timothy Harding <hardi...@gmail.com> wrote: >>>> >>>>> Hello DHIS2 Devs, >>>>> >>>>> I've got a script that works pretty well for updating a bunch of >>>>> sharing settings for various objects using node.js. The only problem is, I >>>>> am getting timeouts when the object update counts get too high (500+). >>>>> >>>>> I am a bit new to the asynchronous nature of node.js, and I like that >>>>> I can bundle a lot of updates into one without needing to worry that I >>>>> need >>>>> a response for one before sending the request for the next. The downside >>>>> is, I have no idea where to put the sleep timer to slow down the volume of >>>>> requests. Everywhere I've put it so far increases the timeouts, rather >>>>> than >>>>> decreases them. I know there are some node.js experts out there on the >>>>> DHIS2 dev list so, perhaps someone has run into something similar: >>>>> >>>>> var serverConfigPOST = serverConfig; >>>>> serverConfigPOST.dhis2server.method = "POST"; >>>>> var i=0; >>>>> for(i=0; i < dataElementsToUpdate.length; i++){ >>>>> >>>>> Tried here >>>>> var postData = JSON.stringify(dataElementsToUpdate[i]); >>>>> serverConfigPOST.dhis2server.path = >>>>> "/dhis/api/sharing?type=dataElement&id=" + dataElementsToUpdate[i]. >>>>> object.id; >>>>> serverConfigPOST.dhis2server.headers = { >>>>> 'Content-Type': 'application/json', >>>>> 'Content-Length': postData.length >>>>> }; >>>>> console.log(serverConfigPOST); >>>>> var req = https.request(serverConfigPOST.dhis2server, function(res) { >>>>> console.log('STATUS: ' + res.statusCode); >>>>> console.log('HEADERS: ' + JSON.stringify(res.headers)); >>>>> res.setEncoding('utf8'); >>>>> res.on('data', function (chunk) { >>>>> console.log('BODY: ' + chunk); >>>>> }); >>>>> }); >>>>> req.on('error', function(e) { >>>>> //console.log('problem with request: ' + e.message); >>>>> }); >>>>> // write data to request body >>>>> req.write(postData); >>>>> req.end(); >>>>> >>>>> Tried here >>>>> // if(i==20){ >>>>> // break; >>>>> // } >>>>> } >>>>> >>>>> Not quite sure how to slow this one down, I may just have to run/rerun >>>>> it with different starting and stopping i values in the mean time. >>>>> >>>>> Timothy Harding >>>>> RPCV Vanuatu >>>>> Skype: hardi...@gmail.com >>>>> +1 (541) 632-6623 >>>>> >>>> >>>> >>>> -- >>>> Pierre Dane >>>> >>>> Development Manager >>>> Jembi Health Systems >>>> >>>> >>> >>> _______________________________________________ >>> Mailing list: https://launchpad.net/~dhis2-devs >>> Post to : dhis2-devs@lists.launchpad.net >>> Unsubscribe : https://launchpad.net/~dhis2-devs >>> More help : https://help.launchpad.net/ListHelp >>> >>> >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~dhis2-devs >> Post to : dhis2-devs@lists.launchpad.net >> Unsubscribe : https://launchpad.net/~dhis2-devs >> More help : https://help.launchpad.net/ListHelp >> >> > > > -- > Lars Helge Øverland > Lead developer, DHIS 2 > University of Oslo > Skype: larshelgeoverland > http://www.dhis2.org <https://www.dhis2.org> > >
_______________________________________________ Mailing list: https://launchpad.net/~dhis2-devs Post to : dhis2-devs@lists.launchpad.net Unsubscribe : https://launchpad.net/~dhis2-devs More help : https://help.launchpad.net/ListHelp
