The short answer is yes. How you do it depends on a number of factors. Assuming you want to build an RDD from the responses and then analyze the responses using Spark core (not Spark Streaming), here is one simple way to do it: 1) Implement a class or function that connects to a web service and returns a list of responses. This code has no dependency on Spark. It will be the same whether you are using Spark or not. Obviously, you have to be take into account memory and latency requirements. 2) Call sc.parallelize on the list obtained in step 1.
This is not the most efficient way of doing it, but hopefully gives you an idea. Mohammed -----Original Message----- From: kasparfischer [mailto:kaspar.fisc...@dreizak.com] Sent: Wednesday, June 3, 2015 12:49 AM To: user@spark.apache.org Subject: Make HTTP requests from within Spark Hi everybody, I'm new to Spark, apologies if my question is very basic. I have a need to send millions of requests to a web service and analyse and store the responses in an RDD. I can easy express the analysing part using Spark's filter/map/etc. primitives but I don't know how to make the requests. Is that something I can do from within Spark? Or Spark Streaming? Or does it conflict with the way Spark works? I've found a similar question but am not sure whether the answer applies here: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-use-Spark-Streaming-from-an-HTTP-api-tp12330.html Any clarifications or pointers would be super helpful! Thanks, Kaspar -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Make-HTTP-requests-from-within-Spark-tp23129.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
