For posterity, I found the root cause and filed a JIRA:
https://issues.apache.org/jira/browse/SPARK-21960. I plan to open a pull
request with the minor fix.
From: Karthik Palaniappan
Sent: Friday, September 1, 2017 9:49 AM
To: Akhil Das
Cc: user@spark.apache.org
Any ideas @Tathagata? I'd be happy to contribute a patch if you can point me in
the right direction.
From: Karthik Palaniappan
Sent: Friday, August 25, 2017 9:15 AM
To: Akhil Das
Cc: user@spark.apache.org; t...@databricks.com
Subject: RE: [Spark Stre
I definitely agree that dynamic allocation is useful, that's why I asked the
question :p
More specifically, does spark plan to solve the problems with DRA for
structured streaming mentioned in that Cloudera article?
If folks can give me pointers on where to start, I'd be happy to implement
s
explicitly set it to 0 after
hitting that error.
Setting executor cores > 1 seems like reasonable advice in general, but that
shouldn’t be my issue here, right?
From: Akhil Das<mailto:ak...@hacked.work>
Sent: Thursday, August 24, 2017 2:34 AM
To: Karthik Palaniappan<mailto:karthik...@hot
I ran the HdfsWordCount example using this command:
spark-submit run-example \
--conf spark.streaming.dynamicAllocation.enabled=true \
--conf spark.executor.instances=0 \
--conf spark.dynamicAllocation.enabled=false \
--conf spark.master=yarn \
--conf spark.submit.deployMode=client \
o
rg/jira/browse/SPARK-12133. Is that actually
a supported feature? Or was that just an experiment? I had trouble getting this
to work, but I'll follow up in a different thread.
Also, does Structured Streaming have its own dynamic allocation algorithm?
Thanks,
Karthik Palaniappan
