Re: [Yarn-Client]Can not access SparkUI

2015-10-26 Thread Earthson Lu
1:45:36,600 INFO org.apache.commons.httpclient.HttpMethodDirector: Retrying request --  Earthson Lu On October 26, 2015 at 15:30:21, Deng Ching-Mallete (och...@apache.org) wrote: Hi Earthson, Unfortunately, attachments aren't allowed in the list so they seemed to have been removed from you

Re: [Spark-1.4.0]jackson-databind conflict?

2015-06-14 Thread Earthson Lu
I’ve recompiled spark-1.4.0 with fasterxml-2.5.x, it works fine now:) --  Earthson Lu On June 12, 2015 at 23:24:32, Sean Owen (so...@cloudera.com) wrote: I see the same thing in an app that uses Jackson 2.5. Downgrading to 2.4 made it work. I meant to go back and figure out if there&#

Re: what is the best way to implement mini batches?

2014-12-15 Thread Earthson Lu
large batch for parallel inside each batch(It seems to be the way that SGD implemented in MLLib does?). --  Earthson Lu On December 16, 2014 at 04:02:22, Imran Rashid (im...@therashids.com) wrote: I'm a little confused by some of the responses.  It seems like there are two different issues

parallelize for a large Seq is extreamly slow.

2014-04-24 Thread Earthson Lu
spark.parallelize(word_mapping.value.toSeq).saveAsTextFile("hdfs://ns1/nlp/word_mapping") this line is too slow. There are about 2 million elements in word_mapping. *Is there a good style for writing a large collection to hdfs?* import org.apache.spark._ > import SparkContext._ > import scala.io