[jira] [Commented] (SPARK-13359) ArrayType(_, true) should also accept ArrayType(_, false) fix for branch-1.6

2016-02-18 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15153586#comment-15153586 ] Earthson Lu commented on SPARK-13359: - I see:) > ArrayType(_, true) shou

[jira] [Created] (SPARK-13359) ArrayType(_, true) should also accept ArrayType(_, false) fix for branch-1.6

2016-02-16 Thread Earthson Lu (JIRA)
Earthson Lu created SPARK-13359: --- Summary: ArrayType(_, true) should also accept ArrayType(_, false) fix for branch-1.6 Key: SPARK-13359 URL: https://issues.apache.org/jira/browse/SPARK-13359 Project

[jira] [Issue Comment Deleted] (SPARK-12746) ArrayType(_, true) should also accept ArrayType(_, false)

2016-01-28 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Earthson Lu updated SPARK-12746: Comment: was deleted (was: Hi Joseph, what is the status of nullability now? It seems someone has

[jira] [Commented] (SPARK-12746) ArrayType(_, true) should also accept ArrayType(_, false)

2016-01-28 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122870#comment-15122870 ] Earthson Lu commented on SPARK-12746: - Hi Joseph, what is the status of nullabi

[jira] [Commented] (SPARK-12746) ArrayType(_, true) should also accept ArrayType(_, false)

2016-01-28 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15122871#comment-15122871 ] Earthson Lu commented on SPARK-12746: - Hi Joseph, what is the status of nullabi

[jira] [Commented] (SPARK-12746) ArrayType(_, true) should also accept ArrayType(_, false)

2016-01-13 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097724#comment-15097724 ] Earthson Lu commented on SPARK-12746: - ok, i see:) If there's no nullabil

[jira] [Issue Comment Deleted] (SPARK-12746) ArrayType(_, true) should also accept ArrayType(_, false)

2016-01-13 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Earthson Lu updated SPARK-12746: Comment: was deleted (was: I was just wandering if you could do a review:) On Tue, Jan 12, 2016

[jira] [Commented] (SPARK-12746) ArrayType(_, true) should also accept ArrayType(_, false)

2016-01-13 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096120#comment-15096120 ] Earthson Lu commented on SPARK-12746: - I was just wandering if you could do a re

[jira] [Updated] (SPARK-12746) ArrayType(_, true) should also accept ArrayType(_, false)

2016-01-11 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Earthson Lu updated SPARK-12746: Shepherd: Joseph K. Bradley (was: Xiangrui Meng) > ArrayType(_, true) should also acc

[jira] [Updated] (SPARK-12746) ArrayType(_, true) should also accept ArrayType(_, false)

2016-01-11 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Earthson Lu updated SPARK-12746: Description: I see CountVectorizer has schema check for ArrayType which has ArrayType(StringType

[jira] [Commented] (SPARK-12746) ArrayType(_, true) should also accept ArrayType(_, false)

2016-01-10 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15091487#comment-15091487 ] Earthson Lu commented on SPARK-12746: - I could work on this:) I have some idea

[jira] [Comment Edited] (SPARK-12746) ArrayType(_, true) should also accept ArrayType(_, false)

2016-01-10 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15091487#comment-15091487 ] Earthson Lu edited comment on SPARK-12746 at 1/11/16 6:1

[jira] [Created] (SPARK-12746) ArrayType(_, true) should also accept ArrayType(_, false)

2016-01-10 Thread Earthson Lu (JIRA)
Earthson Lu created SPARK-12746: --- Summary: ArrayType(_, true) should also accept ArrayType(_, false) Key: SPARK-12746 URL: https://issues.apache.org/jira/browse/SPARK-12746 Project: Spark

[jira] [Comment Edited] (SPARK-6725) Model export/import for Pipeline API

2015-11-18 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15012815#comment-15012815 ] Earthson Lu edited comment on SPARK-6725 at 11/19/15 6:3

[jira] [Comment Edited] (SPARK-6725) Model export/import for Pipeline API

2015-11-18 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15012815#comment-15012815 ] Earthson Lu edited comment on SPARK-6725 at 11/19/15 5:1

[jira] [Commented] (SPARK-6725) Model export/import for Pipeline API

2015-11-18 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15012815#comment-15012815 ] Earthson Lu commented on SPARK-6725: I'm glad to give help:) > Model e

[jira] [Commented] (SPARK-6727) Model export/import for spark.ml: HashingTF

2015-11-17 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15010111#comment-15010111 ] Earthson Lu commented on SPARK-6727: It’s fine:)  I can give some help when the

[jira] [Issue Comment Deleted] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer

2015-11-16 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Earthson Lu updated SPARK-8332: --- Comment: was deleted (was: SparkUI not works when upgrade fasterxml.jackson to 2.5.3

[jira] [Commented] (SPARK-6790) Model export/import for spark.ml: LinearRegression

2015-11-11 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001739#comment-15001739 ] Earthson Lu commented on SPARK-6790: I'm sorry, It's PR for SPARK-

[jira] [Commented] (SPARK-6727) Model export/import for spark.ml: HashingTF

2015-11-11 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001599#comment-15001599 ] Earthson Lu commented on SPARK-6727: It seems that we could implement a def

[jira] [Commented] (SPARK-6727) Model export/import for spark.ml: HashingTF

2015-11-11 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001590#comment-15001590 ] Earthson Lu commented on SPARK-6727: Is the API ready? Can I work on this? >

[jira] [Commented] (SPARK-6725) Model export/import for Pipeline API

2015-11-11 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15001578#comment-15001578 ] Earthson Lu commented on SPARK-6725: Can we expect the this api be usable in s

[jira] [Commented] (SPARK-6726) Model export/import for spark.ml: LogisticRegression

2015-11-11 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000192#comment-15000192 ] Earthson Lu commented on SPARK-6726: Is the API ready for subtasks? I can do

Re: [Yarn-Client]Can not access SparkUI

2015-10-26 Thread Earthson Lu
1:45:36,600 INFO org.apache.commons.httpclient.HttpMethodDirector: Retrying request --  Earthson Lu On October 26, 2015 at 15:30:21, Deng Ching-Mallete (och...@apache.org) wrote: Hi Earthson, Unfortunately, attachments aren't allowed in the list so they seemed to have been removed from you

[jira] [Issue Comment Deleted] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer

2015-10-19 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Earthson Lu updated SPARK-8332: --- Comment: was deleted (was: SparkUI not works when upgrade fasterxml.jackson to 2.5.3

[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer

2015-10-19 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964563#comment-14964563 ] Earthson Lu commented on SPARK-8332: SparkUI not works when upgrade fasterxml.jac

[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer

2015-10-19 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964562#comment-14964562 ] Earthson Lu commented on SPARK-8332: SparkUI not works when upgrade fasterxml.jac

[jira] [Comment Edited] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer

2015-07-22 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14636919#comment-14636919 ] Earthson Lu edited comment on SPARK-8332 at 7/22/15 1:4

[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer

2015-07-22 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14636919#comment-14636919 ] Earthson Lu commented on SPARK-8332: I recompiled spark with fasterxml.jackson 2

Re: [Spark-1.4.0]jackson-databind conflict?

2015-06-14 Thread Earthson Lu
I’ve recompiled spark-1.4.0 with fasterxml-2.5.x, it works fine now:) --  Earthson Lu On June 12, 2015 at 23:24:32, Sean Owen (so...@cloudera.com) wrote: I see the same thing in an app that uses Jackson 2.5. Downgrading to 2.4 made it work. I meant to go back and figure out if there&#

[jira] [Comment Edited] (SPARK-6465) GenericRowWithSchema: KryoException: Class cannot be created (missing no-arg constructor):

2015-03-24 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14377490#comment-14377490 ] Earthson Lu edited comment on SPARK-6465 at 3/25/15 5:2

[jira] [Comment Edited] (SPARK-6465) GenericRowWithSchema: KryoException: Class cannot be created (missing no-arg constructor):

2015-03-24 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14377490#comment-14377490 ] Earthson Lu edited comment on SPARK-6465 at 3/25/15 5:2

[jira] [Commented] (SPARK-6465) GenericRowWithSchema: KryoException: Class cannot be created (missing no-arg constructor):

2015-03-24 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14377490#comment-14377490 ] Earthson Lu commented on SPARK-6465: I'm confused. https://github.com/apa

[jira] [Created] (SPARK-6465) GenericRowWithSchema: KryoException: Class cannot be created (missing no-arg constructor):

2015-03-23 Thread Earthson Lu (JIRA)
Earthson Lu created SPARK-6465: -- Summary: GenericRowWithSchema: KryoException: Class cannot be created (missing no-arg constructor): Key: SPARK-6465 URL: https://issues.apache.org/jira/browse/SPARK-6465

Re: what is the best way to implement mini batches?

2014-12-15 Thread Earthson Lu
large batch for parallel inside each batch(It seems to be the way that SGD implemented in MLLib does?). --  Earthson Lu On December 16, 2014 at 04:02:22, Imran Rashid (im...@therashids.com) wrote: I'm a little confused by some of the responses.  It seems like there are two different issues

parallelize for a large Seq is extreamly slow.

2014-04-24 Thread Earthson Lu
spark.parallelize(word_mapping.value.toSeq).saveAsTextFile("hdfs://ns1/nlp/word_mapping") this line is too slow. There are about 2 million elements in word_mapping. *Is there a good style for writing a large collection to hdfs?* import org.apache.spark._ > import SparkContext._ > import scala.io