I have been using http-agents to grab a bunch of pages and then process them, my initial solutions involved partitioning a sequence of urls and then awaiting for that group before moving on.
(I'm using clojure-1.1.0 and clojure-contrib-1.1.0) However the problem with that approach is that when one of the responses is slow they all have to wait. To get around that I can loop and query each running agent with done? etc, but that is a hassle. I figured I'd use the :handler function instead, when the handler is called I can deal with the response body and then fire off another agent. However the agent's state hasn't changed internally to ::done yet so not all the accessors work (like status, for example). No problem I thought just queue another action behind the initial http-agent request. As I am sending the actions from the same thread they are guaranteed to run sequentially so my second action will have access to the final agent state and then all the accessors will work, etc. However there is another wrinkle; both string and result call await and you can't await in an action. I want to do the following; (let [agnt (http-agent "http://slashdot.org/";)] (send agnt (fn [state] ;; printing for demonstration but could write to file, post to another queue, etc. (println "Done:" (done? *agent*) "Status:" (status *agent*)) ;; the following causes an error as string calls await which is bad in an action (println (string *agent*)) state))) This seems fairly idiomatic to me but I end up with agnt having the "can't await in action" error. After I redefine result and string (the only two that await in http-agent) to be the following; (in-ns clojure.contrib.http.agent) (defn string "Returns the HTTP response body as a string, using the given encoding. If no encoding is given, uses the encoding specified in the server headers, or clojure.contrib.duck-streams/*default-encoding* if it is not specified." ([http-agnt] ;; may have to wait for Content-Encoding (if (not (done? http-agnt)) (await http-agnt)) (string http-agnt (or (.getContentEncoding #^HttpURLConnection (::connection @http- agnt)) duck/*default-encoding*))) ([http-agnt #^String encoding] (.toString (get-byte-buffer http-agnt) encoding))) (defn result "Returns the value returned by the :handler function of the HTTP agent; blocks until the HTTP request is completed. The default handler function returns a ByteArrayOutputStream." [http-agnt] (if (not (done? http-agnt)) (await http-agnt)) (::result @http-agnt)) Then everything is fine and the code above works. Interestingly this doesn't change the documented behaviour of result as result still blocks until complete but now only if it needs to which means I am free to call result in a later action. Finally my question, is my usage of actions idiomatic and I have just stumbled onto an unusual usage pattern for http-agent or is there some reason that await is always called in string and result that I am unaware of? -- sim -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
