So HBC actually does that already--it dumps tweets into a LinkedBlockingQueue. Right now, I'm doing `(loop [tweet (.take queue)]...`, which, I think, essentially amounts to what you're suggesting, but I could be misunderstanding you.
There's a distinct possibility that all the reconnections are caused by my home's internet connection--my local ssh connections get dropped constantly, which suggests there might be a problem somewhere. I figured trying to optimize my code couldn't hurt, though. On Thu, Dec 11, 2014 at 11:43 AM, László Török <ltoro...@gmail.com> wrote: > Hi Sam, > > have you tried putting the incoming (hashtag,tweet) tuples into a queue > and have another thread pull them out and upload them to couchdb? > > I'm unfamiliar with HBC, but I assume it has a callback-based API, so you > should be able to have multiple callbacks/connections/streams feed the same > queue and have a single thread do the upload (and maybe batch if necessary). > > I don't see refs being a particularly good fit for this problem, but I > could be wrong. > > > 2014-12-11 16:18 GMT+00:00 Sam Raker <sam.ra...@gmail.com>: > >> I've got some code that's using Twitter's HoseBirdClient to pull tweets >> from the public stream, which I then preprocess and store with CouchDB. >> Right now, my HBC client is being forced to reconnect more than I'd like, >> which occasionally causes my app to hang, for reasons I'm not entirely >> clear on. Regardless, some preliminary research on HBC suggests that the >> reconnections are being caused by my code failing to keep up with the >> endpoint, which in turn suggests that my processing+uploading is taking too >> long. I tried wrapping the processing+uploading part in futures, which >> definitely sped things up, but caused 409 errors when uploading to >> CouchDB--briefly, Couch requires any update operation to include a >> git-style "rev" string, and if the rev you provide isn't the most recent >> one, it throws a 409 at you. I'm organizing things by hashtag, so tweets >> with multiple copies of the same hashtag, or series of tweets with the same >> hashtag are the culprit--future A gets the current doc from Couch, >> processes it, and uses the rev it got from the currently-existing doc, >> while future B does the same thing, but finishes first, so now future A has >> an outdated rev, and that causes the 409. >> >> The vague solution I've come up with involves using a map to store the >> rev values, with the last step of the processing/uploading function being >> to store the rev number Clutch helpfully returns to you after a successful >> update. From what I can tell, refs are the way to go, since each future is >> effectively a separate thread. My questions are as follows: >> 1) Would I have to store the map-of-refs in a ref? >> 2) Is this even feasible? Would the timing work out? >> 3) With the addition of all this dereferencing and `dosync`+`alter`-ing, >> would this actually end up speeding things up all that much? >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > László Török > -- > Checkout http://www.lollyrewards.com/ > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to a topic in the > Google Groups "Clojure" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/clojure/zRy6dm1Vmcs/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
