Hm, now I am also seeing this problem.
The essence of my code is:
final JavaSparkContext sparkContext = new JavaSparkContext(sparkConf);
JavaStreamingContextFactory streamingContextFactory = new
JavaStreamingContextFactory() {
@Override
public JavaStreamingContext create() {
if I reduce the app to the following code then I don't see the exception. It
creates the hadoop files but they are empty! The DStream doesn't get written
out to the files!
def main(args: Array[String]) {
try {
val properties = getProperties("settings.properties")
StreamingExamples
Look this is the whole program. I am not trying to serialize the JobConf.
def main(args: Array[String]) {
try {
val properties = getProperties("settings.properties")
StreamingExamples.setStreamingLogLevels()
val zkQuorum = properties.get("zookeeper.list").toString()
v
Somewhere, your function has a reference to the Hadoop JobConf object
and is trying to send that to the workers. It's not in this code you
pasted so must be from something slightly different?
It shouldn't need to send that around and in fact it can't be
serialized as you see. If you need a Hadoop
