[jira] [Commented] (SPARK-10903) Make sqlContext global

2015-10-19 Thread Sun Rui (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964672#comment-14964672 ] Sun Rui commented on SPARK-10903: - I figured out the reason. You have to export the S3 me

[jira] [Updated] (SPARK-11199) Improve R context management story and add getOrCreate

2015-10-19 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu updated SPARK-11199: --- Assignee: Felix Cheung > Improve R context management story and add getOrCreate > ---

[jira] [Commented] (SPARK-9299) percentile and percentile_approx aggregate functions

2015-10-19 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964656#comment-14964656 ] Jeff Zhang commented on SPARK-9299: --- Link with SPARK-6761 as they can share the same alg

[jira] [Closed] (SPARK-8071) Run PySpark dataframe.rollup/cube test failed

2015-10-19 Thread Davies Liu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Davies Liu closed SPARK-8071. - Resolution: Won't Fix > Run PySpark dataframe.rollup/cube test failed > --

[jira] [Created] (SPARK-11203) UDF doesn't support charType column and lit function doesn't allow charType as argument

2015-10-19 Thread M Bharat lal (JIRA)
M Bharat lal created SPARK-11203: Summary: UDF doesn't support charType column and lit function doesn't allow charType as argument Key: SPARK-11203 URL: https://issues.apache.org/jira/browse/SPARK-11203

[jira] [Commented] (SPARK-10903) Make sqlContext global

2015-10-19 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964642#comment-14964642 ] Felix Cheung commented on SPARK-10903: -- Working from SparkR package version in this

[jira] [Updated] (SPARK-11192) Spark sql seems to leak org.apache.spark.sql.execution.ui.SQLTaskMetrics objects over time

2015-10-19 Thread Blake Livingston (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Blake Livingston updated SPARK-11192: - Summary: Spark sql seems to leak org.apache.spark.sql.execution.ui.SQLTaskMetrics objects

[jira] [Updated] (SPARK-11192) When graphite metric sink is enabled, spark sql leaks org.apache.spark.sql.execution.ui.SQLTaskMetrics objects over time

2015-10-19 Thread Blake Livingston (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Blake Livingston updated SPARK-11192: - Description: Noticed that slowly, over the course of a day or two, heap memory usage on a

[jira] [Updated] (SPARK-11202) Unsupported dataType

2015-10-19 Thread whc (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] whc updated SPARK-11202: Component/s: SQL > Unsupported dataType > > > Key: SPARK-11202 >

[jira] [Assigned] (SPARK-11201) StreamContext.getOrCreate is broken is yarn-client mode

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11201: Assignee: (was: Apache Spark) > StreamContext.getOrCreate is broken is yarn-client mod

[jira] [Commented] (SPARK-11201) StreamContext.getOrCreate is broken is yarn-client mode

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964608#comment-14964608 ] Apache Spark commented on SPARK-11201: -- User 'harishreedharan' has created a pull re

[jira] [Assigned] (SPARK-11201) StreamContext.getOrCreate is broken is yarn-client mode

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11201: Assignee: Apache Spark > StreamContext.getOrCreate is broken is yarn-client mode > ---

[jira] [Created] (SPARK-11202) Unsupported dataType

2015-10-19 Thread whc (JIRA)
whc created SPARK-11202: --- Summary: Unsupported dataType Key: SPARK-11202 URL: https://issues.apache.org/jira/browse/SPARK-11202 Project: Spark Issue Type: Bug Affects Versions: 1.5.0 Re

[jira] [Commented] (SPARK-2654) Leveled logging in PySpark

2015-10-19 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964586#comment-14964586 ] Jeff Zhang commented on SPARK-2654: --- [~davies] I think currently spark-core also don't h

[jira] [Commented] (SPARK-10903) Make sqlContext global

2015-10-19 Thread Sun Rui (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964574#comment-14964574 ] Sun Rui commented on SPARK-10903: - Yes, I did the experiment by typing code in the shell.

[jira] [Updated] (SPARK-10384) Univariate statistics as UDAFs

2015-10-19 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-10384: Description: It would be nice to define univariate statistics as UDAFs. This JIRA discusses genera

[jira] [Commented] (SPARK-10903) Make sqlContext global

2015-10-19 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964570#comment-14964570 ] Felix Cheung commented on SPARK-10903: -- Something to do with the namespace {code} >

[jira] [Updated] (SPARK-10384) Univariate statistics as UDAFs

2015-10-19 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-10384: Description: It would be nice to define univariate statistics as UDAFs. This JIRA discusses genera

[jira] [Closed] (SPARK-10861) Univariate Statistics: Adding range support as UDAF

2015-10-19 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin closed SPARK-10861. --- Resolution: Won't Fix Assignee: Jihong MA I don't think this range function provides enough va

[jira] [Issue Comment Deleted] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer

2015-10-19 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Earthson Lu updated SPARK-8332: --- Comment: was deleted (was: SparkUI not works when upgrade fasterxml.jackson to 2.5.3) > NoSuchMethodE

[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer

2015-10-19 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964563#comment-14964563 ] Earthson Lu commented on SPARK-8332: SparkUI not works when upgrade fasterxml.jackson

[jira] [Commented] (SPARK-8332) NoSuchMethodError: com.fasterxml.jackson.module.scala.deser.BigDecimalDeserializer

2015-10-19 Thread Earthson Lu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964562#comment-14964562 ] Earthson Lu commented on SPARK-8332: SparkUI not works when upgrade fasterxml.jackson

[jira] [Updated] (SPARK-11201) StreamContext.getOrCreate is broken is yarn-client mode

2015-10-19 Thread Hari Shreedharan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Shreedharan updated SPARK-11201: - Description: If {{StreamingContext.getOrCreate}} (or the constructors that create the Had

[jira] [Created] (SPARK-11201) StreamContext.getOrCreate is broken is yarn-client mode

2015-10-19 Thread Hari Shreedharan (JIRA)
Hari Shreedharan created SPARK-11201: Summary: StreamContext.getOrCreate is broken is yarn-client mode Key: SPARK-11201 URL: https://issues.apache.org/jira/browse/SPARK-11201 Project: Spark

[jira] [Commented] (SPARK-6517) Bisecting k-means clustering

2015-10-19 Thread Jeff Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964550#comment-14964550 ] Jeff Zhang commented on SPARK-6517: --- Is the work still going on ? If not, I'd like to he

[jira] [Commented] (SPARK-10903) Make sqlContext global

2015-10-19 Thread Sun Rui (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964552#comment-14964552 ] Sun Rui commented on SPARK-10903: - I don't know what's the difference:) > Make sqlContex

[jira] [Commented] (SPARK-10903) Make sqlContext global

2015-10-19 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964551#comment-14964551 ] Felix Cheung commented on SPARK-10903: -- And when revert the code in SparkR package i

[jira] [Updated] (SPARK-11200) NettyRpcEnv endless message "cannot send ${message} because RpcEnv is closed"

2015-10-19 Thread hujiayin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hujiayin updated SPARK-11200: - Description: The endless messages "cannot send ${message}because RpcEnv is closed" are pop up after star

[jira] [Updated] (SPARK-11200) NettyRpcEnv endless message "cannot send ${message} because RpcEnv is closed"

2015-10-19 Thread hujiayin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hujiayin updated SPARK-11200: - Description: The endless messages "cannot send ${message}because RpcEnv is closed" are pop up after star

[jira] [Created] (SPARK-11200) NettyRpcEnv endless message "cannot send ${message} because RpcEnv is closed"

2015-10-19 Thread hujiayin (JIRA)
hujiayin created SPARK-11200: Summary: NettyRpcEnv endless message "cannot send ${message} because RpcEnv is closed" Key: SPARK-11200 URL: https://issues.apache.org/jira/browse/SPARK-11200 Project: Spark

[jira] [Commented] (SPARK-10903) Make sqlContext global

2015-10-19 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964543#comment-14964543 ] Felix Cheung commented on SPARK-10903: -- It says 'no methods found'. btw, I think I k

[jira] [Commented] (SPARK-10903) Make sqlContext global

2015-10-19 Thread Sun Rui (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964539#comment-14964539 ] Sun Rui commented on SPARK-10903: - Could you dump the following output? methods("createDa

[jira] [Commented] (SPARK-10903) Make sqlContext global

2015-10-19 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964531#comment-14964531 ] Felix Cheung commented on SPARK-10903: -- That's puzzling. Any tips on how to debug th

[jira] [Commented] (SPARK-10903) Make sqlContext global

2015-10-19 Thread Sun Rui (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964525#comment-14964525 ] Sun Rui commented on SPARK-10903: - That's strange. It worked on my machine. > createDataF

[jira] [Comment Edited] (SPARK-10903) Make sqlContext global

2015-10-19 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964507#comment-14964507 ] Felix Cheung edited comment on SPARK-10903 at 10/20/15 4:40 AM: ---

[jira] [Commented] (SPARK-10903) Make sqlContext global

2015-10-19 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964507#comment-14964507 ] Felix Cheung commented on SPARK-10903: -- Same with this {code} createDataFrame <- fun

[jira] [Commented] (SPARK-10903) Make sqlContext global

2015-10-19 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964504#comment-14964504 ] Felix Cheung commented on SPARK-10903: -- I was sure I tried that but double-checking

[jira] [Updated] (SPARK-11180) Support BooleanType in DataFrame.na.fill

2015-10-19 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-11180: Assignee: Rishabh Bhardwaj > Support BooleanType in DataFrame.na.fill > ---

[jira] [Commented] (SPARK-10903) Make sqlContext global

2015-10-19 Thread Sun Rui (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964494#comment-14964494 ] Sun Rui commented on SPARK-10903: - [~felixcheung], I did some experiment and found that S

[jira] [Commented] (SPARK-11190) SparkR support for cassandra collection types.

2015-10-19 Thread Sun Rui (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964397#comment-14964397 ] Sun Rui commented on SPARK-11190: - yes, please try it on the latest master branch. > Spa

[jira] [Commented] (SPARK-9297) covar_pop and covar_samp aggregate functions

2015-10-19 Thread Jihong MA (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964370#comment-14964370 ] Jihong MA commented on SPARK-9297: -- I will work on this and prepare a PR using Imperative

[jira] [Comment Edited] (SPARK-11103) Filter applied on Merged Parquet shema with new column fail with (java.lang.IllegalArgumentException: Column [column_name] was not found in schema!)

2015-10-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964337#comment-14964337 ] Hyukjin Kwon edited comment on SPARK-11103 at 10/20/15 1:06 AM: ---

[jira] [Commented] (SPARK-11103) Filter applied on Merged Parquet shema with new column fail with (java.lang.IllegalArgumentException: Column [column_name] was not found in schema!)

2015-10-19 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964337#comment-14964337 ] Hyukjin Kwon commented on SPARK-11103: -- [~lian cheng] This looks clearly an issue an

[jira] [Updated] (SPARK-11199) Improve R context management story and add getOrCreate

2015-10-19 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-11199: - Component/s: (was: R) SparkR > Improve R context management story and add ge

[jira] [Commented] (SPARK-10903) Make sqlContext global

2015-10-19 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964333#comment-14964333 ] Felix Cheung commented on SPARK-10903: -- Created SPARK-11199 to track context initial

[jira] [Commented] (SPARK-11199) Improve R context management story and add getOrCreate

2015-10-19 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964332#comment-14964332 ] Felix Cheung commented on SPARK-11199: -- I would work on this after SPARK-10903. [~fa

[jira] [Created] (SPARK-11199) Improve R context management story and add getOrCreate

2015-10-19 Thread Felix Cheung (JIRA)
Felix Cheung created SPARK-11199: Summary: Improve R context management story and add getOrCreate Key: SPARK-11199 URL: https://issues.apache.org/jira/browse/SPARK-11199 Project: Spark Issue

[jira] [Commented] (SPARK-10903) Make sqlContext global

2015-10-19 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964326#comment-14964326 ] Felix Cheung commented on SPARK-10903: -- Looked into and tested with a few approaches

[jira] [Comment Edited] (SPARK-10903) Make sqlContext global

2015-10-19 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964326#comment-14964326 ] Felix Cheung edited comment on SPARK-10903 at 10/20/15 12:46 AM: --

[jira] [Created] (SPARK-11198) Support record de-aggregation in KinesisReceiver

2015-10-19 Thread Xiangrui Meng (JIRA)
Xiangrui Meng created SPARK-11198: - Summary: Support record de-aggregation in KinesisReceiver Key: SPARK-11198 URL: https://issues.apache.org/jira/browse/SPARK-11198 Project: Spark Issue Type

[jira] [Updated] (SPARK-10680) Flaky test: network.RequestTimeoutIntegrationSuite.timeoutInactiveRequests

2015-10-19 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-10680: Target Version/s: 1.6.0 (was: 1.5.0, 1.6.0) > Flaky test: network.RequestTimeoutIntegrationSuite.t

[jira] [Resolved] (SPARK-11088) Optimize DataSourceStrategy.mergeWithPartitionValues

2015-10-19 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust resolved SPARK-11088. -- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 9104 [http

[jira] [Comment Edited] (SPARK-6527) sc.binaryFiles can not access files on s3

2015-10-19 Thread bin wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964267#comment-14964267 ] bin wang edited comment on SPARK-6527 at 10/19/15 11:52 PM: [~

[jira] [Commented] (SPARK-6527) sc.binaryFiles can not access files on s3

2015-10-19 Thread bin wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964267#comment-14964267 ] bin wang commented on SPARK-6527: - [~zhaozhang], this errors happens to me too while I am

[jira] [Assigned] (SPARK-11197) Run SQL query on files directly without create a table

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11197: Assignee: Davies Liu (was: Apache Spark) > Run SQL query on files directly without create

[jira] [Assigned] (SPARK-11197) Run SQL query on files directly without create a table

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11197: Assignee: Apache Spark (was: Davies Liu) > Run SQL query on files directly without create

[jira] [Commented] (SPARK-11197) Run SQL query on files directly without create a table

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964262#comment-14964262 ] Apache Spark commented on SPARK-11197: -- User 'davies' has created a pull request for

[jira] [Created] (SPARK-11197) Run SQL query on files directly without create a table

2015-10-19 Thread Davies Liu (JIRA)
Davies Liu created SPARK-11197: -- Summary: Run SQL query on files directly without create a table Key: SPARK-11197 URL: https://issues.apache.org/jira/browse/SPARK-11197 Project: Spark Issue Type

[jira] [Resolved] (SPARK-11120) maxNumExecutorFailures defaults to 3 under dynamic allocation

2015-10-19 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-11120. --- Resolution: Fixed Assignee: Ryan Williams Fix Version/s: 1.6.0 > maxNumExecutorFailur

[jira] [Updated] (SPARK-11120) maxNumExecutorFailures defaults to 3 under dynamic allocation

2015-10-19 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-11120: -- Target Version/s: 1.6.0 > maxNumExecutorFailures defaults to 3 under dynamic allocation > -

[jira] [Resolved] (SPARK-9708) Spark should create local temporary directories in Mesos sandbox when launched with Mesos

2015-10-19 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-9708. -- Resolution: Fixed Assignee: Chris Bannister Fix Version/s: 1.6.0 Target Vers

[jira] [Created] (SPARK-11196) Support for equality and pushdown of filters on some UDTs

2015-10-19 Thread Michael Armbrust (JIRA)
Michael Armbrust created SPARK-11196: Summary: Support for equality and pushdown of filters on some UDTs Key: SPARK-11196 URL: https://issues.apache.org/jira/browse/SPARK-11196 Project: Spark

[jira] [Resolved] (SPARK-11114) Add getOrCreate for SparkContext/SQLContext for Python

2015-10-19 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-4. --- Resolution: Fixed Fix Version/s: 1.6.0 > Add getOrCreate for SparkContext/SQLContext for Pytho

[jira] [Resolved] (SPARK-11051) NullPointerException when action called on localCheckpointed RDD (that was checkpointed before)

2015-10-19 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-11051. --- Resolution: Fixed Fix Version/s: 1.6.0 1.5.2 > NullPointerException when ac

[jira] [Assigned] (SPARK-10861) Univariate Statistics: Adding range support as UDAF

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10861: Assignee: Apache Spark > Univariate Statistics: Adding range support as UDAF > ---

[jira] [Assigned] (SPARK-10861) Univariate Statistics: Adding range support as UDAF

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10861: Assignee: (was: Apache Spark) > Univariate Statistics: Adding range support as UDAF >

[jira] [Commented] (SPARK-10861) Univariate Statistics: Adding range support as UDAF

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964216#comment-14964216 ] Apache Spark commented on SPARK-10861: -- User 'JihongMA' has created a pull request f

[jira] [Commented] (SPARK-1086) Some corner case during HA master switching?

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964215#comment-14964215 ] Apache Spark commented on SPARK-1086: - User 'JihongMA' has created a pull request for

[jira] [Resolved] (SPARK-11063) Spark TaskSetManager doesn't use Receiver's scheduling executors

2015-10-19 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-11063. --- Resolution: Fixed Assignee: Shixiong Zhu Fix Version/s: 1.5.2 The fix for 1.5

[jira] [Updated] (SPARK-11063) Spark TaskSetManager doesn't use Receiver's scheduling executors

2015-10-19 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-11063: -- Fix Version/s: 1.6.0 > Spark TaskSetManager doesn't use Receiver's scheduling executors > -

[jira] [Updated] (SPARK-10151) Support invocation of hive macro

2015-10-19 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen updated SPARK-10151: --- Issue Type: New Feature (was: Bug) > Support invocation of hive macro >

[jira] [Created] (SPARK-11195) Exception thrown on executor throws ClassNotFound on driver

2015-10-19 Thread Hurshal Patel (JIRA)
Hurshal Patel created SPARK-11195: - Summary: Exception thrown on executor throws ClassNotFound on driver Key: SPARK-11195 URL: https://issues.apache.org/jira/browse/SPARK-11195 Project: Spark

[jira] [Updated] (SPARK-10641) skewness and kurtosis support

2015-10-19 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-10641: -- Shepherd: Xiangrui Meng > skewness and kurtosis support > - > >

[jira] [Commented] (SPARK-11194) Use a single URLClassLoader for jars added through SQL's "ADD JAR" command

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964156#comment-14964156 ] Apache Spark commented on SPARK-11194: -- User 'yhuai' has created a pull request for

[jira] [Updated] (SPARK-10978) Allow PrunedFilterScan to eliminate predicates from further evaluation

2015-10-19 Thread Michael Armbrust (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Armbrust updated SPARK-10978: - Target Version/s: 1.6.0 > Allow PrunedFilterScan to eliminate predicates from further eva

[jira] [Commented] (SPARK-11194) Use a single URLClassLoader for jars added through SQL's "ADD JAR" command

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964113#comment-14964113 ] Apache Spark commented on SPARK-11194: -- User 'yhuai' has created a pull request for

[jira] [Assigned] (SPARK-11194) Use a single URLClassLoader for jars added through SQL's "ADD JAR" command

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11194: Assignee: Yin Huai (was: Apache Spark) > Use a single URLClassLoader for jars added throu

[jira] [Assigned] (SPARK-11194) Use a single URLClassLoader for jars added through SQL's "ADD JAR" command

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11194: Assignee: Apache Spark (was: Yin Huai) > Use a single URLClassLoader for jars added throu

[jira] [Updated] (SPARK-11194) Use a single URLClassLoader for jars added through SQL's "ADD JAR" command

2015-10-19 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-11194: - Description: Right now, we stack a new URLClassLoader when a user add a jar through SQL's add jar comman

[jira] [Updated] (SPARK-11180) Support BooleanType in DataFrame.na.fill

2015-10-19 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-11180: Summary: Support BooleanType in DataFrame.na.fill (was: DataFrame.na.fill does not support Boolea

[jira] [Resolved] (SPARK-11180) DataFrame.na.fill does not support Boolean Type:

2015-10-19 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-11180. - Resolution: Fixed Fix Version/s: 1.6.0 > DataFrame.na.fill does not support Boolean Type:

[jira] [Updated] (SPARK-11180) DataFrame.na.fill does not support Boolean Type:

2015-10-19 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-11180: Description: Currently DataFrame.na.fill does not support Boolean primitive type. We have use ca

[jira] [Created] (SPARK-11194) Use a single URLClassLoader for jars added through SQL's "ADD JAR" command

2015-10-19 Thread Yin Huai (JIRA)
Yin Huai created SPARK-11194: Summary: Use a single URLClassLoader for jars added through SQL's "ADD JAR" command Key: SPARK-11194 URL: https://issues.apache.org/jira/browse/SPARK-11194 Project: Spark

[jira] [Assigned] (SPARK-11184) Declare most of .mllib code not-Experimental

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11184: Assignee: Apache Spark > Declare most of .mllib code not-Experimental > --

[jira] [Assigned] (SPARK-11184) Declare most of .mllib code not-Experimental

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11184: Assignee: (was: Apache Spark) > Declare most of .mllib code not-Experimental > ---

[jira] [Commented] (SPARK-11184) Declare most of .mllib code not-Experimental

2015-10-19 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964077#comment-14964077 ] Apache Spark commented on SPARK-11184: -- User 'srowen' has created a pull request for

[jira] [Commented] (SPARK-10645) Bivariate Statistics: Spearman's Correlation support as UDAF

2015-10-19 Thread Arvind Surve (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964068#comment-14964068 ] Arvind Surve commented on SPARK-10645: -- Spearman's correlation coefficient (SpCoeff)

[jira] [Updated] (SPARK-11192) When graphite metric sink is enabled, spark sql leaks org.apache.spark.sql.execution.ui.SQLTaskMetrics objects over time

2015-10-19 Thread Blake Livingston (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Blake Livingston updated SPARK-11192: - Description: Noticed that slowly, over the course of a day or two, heap memory usage on a

[jira] [Created] (SPARK-11192) When graphite metric sink is enabled, spark sql leaks org.apache.spark.sql.execution.ui.SQLTaskMetrics objects over time

2015-10-19 Thread Blake Livingston (JIRA)
Blake Livingston created SPARK-11192: Summary: When graphite metric sink is enabled, spark sql leaks org.apache.spark.sql.execution.ui.SQLTaskMetrics objects over time Key: SPARK-11192 URL: https://issues.apac

[jira] [Created] (SPARK-11193) Spark 1.5+ Kinesis Streaming - ClassCastException when starting KinesisReceiver

2015-10-19 Thread Phil Kallos (JIRA)
Phil Kallos created SPARK-11193: --- Summary: Spark 1.5+ Kinesis Streaming - ClassCastException when starting KinesisReceiver Key: SPARK-11193 URL: https://issues.apache.org/jira/browse/SPARK-11193 Project

[jira] [Commented] (SPARK-11191) [1.5] Can't create UDF's using hive thrift service

2015-10-19 Thread David Ross (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964048#comment-14964048 ] David Ross commented on SPARK-11191: I will add that the exact same thing happens whe

[jira] [Commented] (SPARK-11016) Spark fails when running with a task that requires a more recent version of RoaringBitmaps

2015-10-19 Thread Charles Allen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964043#comment-14964043 ] Charles Allen commented on SPARK-11016: --- [~srowen] I confirmed locally that https:/

[jira] [Commented] (SPARK-10754) table and column name are case sensitive when json Dataframe was registered as tempTable using JavaSparkContext.

2015-10-19 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964041#comment-14964041 ] Yin Huai commented on SPARK-10754: -- Can you use {{HiveContext}}, which set {{spark.sql.c

[jira] [Commented] (SPARK-11190) SparkR support for cassandra collection types.

2015-10-19 Thread Shivaram Venkataraman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14964032#comment-14964032 ] Shivaram Venkataraman commented on SPARK-11190: --- cc [~sunrui] Could you try

[jira] [Created] (SPARK-11191) [1.5] Can't create UDF's using hive thrift service

2015-10-19 Thread David Ross (JIRA)
David Ross created SPARK-11191: -- Summary: [1.5] Can't create UDF's using hive thrift service Key: SPARK-11191 URL: https://issues.apache.org/jira/browse/SPARK-11191 Project: Spark Issue Type: Bu

[jira] [Updated] (SPARK-10955) Warn if dynamic allocation is enabled for Streaming jobs

2015-10-19 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-10955: -- Description: Spark streaming can be tricky with dynamic allocation and can lose data if not use

[jira] [Resolved] (SPARK-11027) Better group distinct columns in query compilation

2015-10-19 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai resolved SPARK-11027. -- Resolution: Won't Fix > Better group distinct columns in query compilation > --

[jira] [Commented] (SPARK-11027) Better group distinct columns in query compilation

2015-10-19 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14963990#comment-14963990 ] Yin Huai commented on SPARK-11027: -- As pointed out by [~joshrosen] (see https://github.c

[jira] [Updated] (SPARK-10955) Warn if dynamic allocation is enabled for Streaming jobs

2015-10-19 Thread Tathagata Das (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das updated SPARK-10955: -- Summary: Warn if dynamic allocation is enabled for Streaming jobs (was: Disable dynamic alloca

[jira] [Created] (SPARK-11190) SparkR support for cassandra collection types.

2015-10-19 Thread Bilind Hajer (JIRA)
Bilind Hajer created SPARK-11190: Summary: SparkR support for cassandra collection types. Key: SPARK-11190 URL: https://issues.apache.org/jira/browse/SPARK-11190 Project: Spark Issue Type: B

  1   2   >