Hey Everyone,
I've been using lamina channels to create queues to download files/web
pages in parallel. I'm using clj-http as my client as it supports proxies.
So basically I create a channel, then use map* to fetch process and queue
up pages in the channel. My fetch-url-with-proxy function, displayed below,
takes a proxy-agent (an agent with a vector containing the ip and port of a
proxy) and then fetches the url. The problem I'm having is that this does
not actually appear to run in parallel. The process seems to hang 10 to 15
url's in.
Anyone have any ideas?
Regards,
Folcon
(defn fetch-url
"If you give a wrong url it returns nil"
([url]
(fetch-url url nil nil))
([url host port]
(. java.lang.Thread sleep (rand 1000))
(try-times 3
(try
(printf (str url "\n"))
(if (or (nil? host) (nil? port))
(html/html-resource (java.io.StringReader. (:body (client/get
url))))
(html/html-resource (java.io.StringReader. (:body (client/get url
{:proxy-host host :proxy-port port})))))
(catch java.net.MalformedURLException e)))))
(defn fetch-url-with-proxy [url]
(try
(try-times 3
(letfn [(fetch-fn [host-port url result]
(let [[host port] host-port]
(deliver result (fetch-url url host port))
(printf (str "Retrieved: " url "\n"))
host-port))]
(let [result (promise)]
(send-off (agent-from-pool proxy-pool) fetch-fn url
result)
@result)))
(catch java.lang.Exception e
(to-log :error (str "Exception with url: " url " trace: " e)))))
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
