Depending on what you want to do with the result of the scraping, Spark may not be the best framework for your use case. Take a look at a general Akka application.
On Sun, Sep 7, 2014 at 12:15 AM, Sandeep Singh <sand...@techaddict.me> wrote: > Hi all, > > I am Implementing a Crawler, Scraper. The It should be able to process the > request for crawling & scraping, within few seconds of submitting the > job(around 1mil/sec), for rest I can take some time(scheduled evenly all > over the day). What is the best way to implement this? > > Thanks. > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Crawler-and-Scraper-with-different-priorities-tp13645.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
