I have a file "badFullIPs.csv" of bad IP addresses used for filtering. In
yarn-client mode, I simply read it off the edge node, transform it, and then
broadcast it:
val badIPs = fromFile(edgeDir + "badfullIPs.csv")
val badIPsLines = badIPs.getLines
val badIpSet = badIPsLines.toSet
val badIPsBC = sc.broadcast(badIpSet)
badIPs.close
How can I accomplish this in yarn-cluster mode?
Jon
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-broadcast-a-variable-read-from-a-file-in-yarn-cluster-mode-tp21524.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
