I was able to hack this on my similar setup issue by running (on the driver)
$ sudo hostname ip
Where ip is the same value set in the "spark.driver.host" property. This
isn't a solution I would use universally and hope the someone can fix this
bug in the distribution.
Regards,
Mike
--
View
I'm attempting to code a Java only implementation accessing the
StreamingContext.fileStream method and am especially interested in setting
the boolean "newFilesOnly" to false. Unfortunately my code throws
exceptions:
Exception in thread "main" java.lang.InstantiationException
at
sun.reflec
Thanks very much for the pointer, which validated my initial approach. It
turns out that I was creating a tag for the abstract class
"InputFormat.class". Using "TextInputFormat.class" instead fixed my issue.
Regards,
Mike
--
View this message in context:
http://apache-spark-user-list.1001560.
Haoming
If the Spark UI states that one of the jobs is in the "Waiting" state, this
is a resources issue. You will need to set properties such as:
spark.executor.memory
spark.cores.max
Set these so that each instance only takes a portion of the available worker
memory and cores.
Regards,
Mike
Soroka,
You should be able to use the filestream() method of the
JavaStreamingContext. In case you need something more custom, the code
below is something I developed to provide the max functionality of the
Scala method, but implemented in Java.
//Set these to reflect your app and input format sp
Hafiz,
You can probably use the RDD.mapPartitionsWithIndex method.
Mike
On Tue, Dec 23, 2014 at 8:35 AM, Hafiz Mujadid [via Apache Spark User List]
wrote:
>
> hi dears!
>
> Is there some efficient way to drop first line of an RDD[String]?
>
> any suggestion?
>
> Thanks
>
> -
I'm also interested in the solution to this.
Thanks,
Mike
On Mon, Dec 29, 2014 at 12:01 PM, hutashan [via Apache Spark User List] <
ml-node+s1001560n20889...@n3.nabble.com> wrote:
>
> Hello All,
>
> I need to clean up app folder(include app downloaded jar) in spark under
> work folder.
> I have
