Its a variable in spark-1.0.0/*/storagre/BlockManagerMaster.scala class. The return data of AskDriverWithReply() method for the getPeers() method.
Basically, it is a Seq[ArrayBuffer]: ArraySeq(ArrayBuffer(BlockManagerId(1, s1, 47006, 0), BlockManagerId(0, s1, 34625, 0)), ArrayBuffer(BlockManagerId(1, s1, 47006, 0), BlockManagerId(0, s2, 34625, 0)), ArrayBuffer(BlockManagerId(1, s1, 47006, 0), BlockManagerId(0, s2, 34625, 0)), ArrayBuffer(BlockManagerId(1, s1, 47006, 0), BlockManagerId(0, s2, 34625, 0)), ArrayBuffer(BlockManagerId(<driver>, karthik, 51051, 0), BlockManagerId(1, s1, 47006, 0))) On Sun, Oct 12, 2014 at 12:59 PM, @Sanjiv Singh [via Apache Spark User List] <ml-node+s1001560n16231...@n3.nabble.com> wrote: > Hi Karthik, > > Can you provide us more detail of dataset "data" that you wanted to > parallelize with > > SparkContext.parallelize(data); > > > > > Regards, > Sanjiv Singh > > > Regards > Sanjiv Singh > Mob : +091 9990-447-339 > > On Sun, Oct 12, 2014 at 11:45 AM, rapelly kartheek <[hidden email] > <http://user/SendEmail.jtp?type=node&node=16231&i=0>> wrote: > >> Hi, >> >> I am trying to write a String that is not an rdd to HDFS. This data is a >> variable in Spark Scheduler code. None of the spark File operations are >> working because my data is not rdd. >> >> So, I tried using SparkContext.parallelize(data). But it throws error: >> >> [error] >> /home/karthik/spark-1.0.0/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala:265: >> not found: value SparkContext >> [error] SparkContext.parallelize(result) >> [error] ^ >> [error] one error found >> >> I realized that this data is part of the Scheduler. So, the Sparkcontext >> would not have got created yet. >> >> Any help in "writing scheduler variable data to HDFS" is appreciated!! >> >> -Karthik >> > > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > > http://apache-spark-user-list.1001560.n3.nabble.com/How-to-convert-a-non-rdd-data-to-rdd-tp16230p16231.html > To unsubscribe from How to convert a non-rdd data to rdd., click here > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=16230&code=a2FydGhlZWsubWJtc0BnbWFpbC5jb218MTYyMzB8LTE1NjA1NDM4NDM=> > . > NAML > <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >
