+1 Ran PageRank on standalone mode with 4 nodes and noticed a speedup after the specific commits that were in RC2 but not RC1:
c247b6a Dec 10 [SPARK-12155][SPARK-12253] Fix executor OOM in unified memory management 05e441e Dec 9 [SPARK-12165][SPARK-12189] Fix bugs in eviction of storage memory by execution Also jobs that triggered these issues now run successfully. 2015-12-14 10:45 GMT-08:00 Reynold Xin <r...@databricks.com>: > +1 > > Tested some dataframe operations on my Mac. > > > On Saturday, December 12, 2015, Michael Armbrust <mich...@databricks.com> > wrote: > >> Please vote on releasing the following candidate as Apache Spark version >> 1.6.0! >> >> The vote is open until Tuesday, December 15, 2015 at 6:00 UTC and passes >> if a majority of at least 3 +1 PMC votes are cast. >> >> [ ] +1 Release this package as Apache Spark 1.6.0 >> [ ] -1 Do not release this package because ... >> >> To learn more about Apache Spark, please see http://spark.apache.org/ >> >> The tag to be voted on is *v1.6.0-rc2 >> (23f8dfd45187cb8f2216328ab907ddb5fbdffd0b) >> <https://github.com/apache/spark/tree/v1.6.0-rc2>* >> >> The release files, including signatures, digests, etc. can be found at: >> http://people.apache.org/~pwendell/spark-releases/spark-1.6.0-rc2-bin/ >> >> Release artifacts are signed with the following key: >> https://people.apache.org/keys/committer/pwendell.asc >> >> The staging repository for this release can be found at: >> https://repository.apache.org/content/repositories/orgapachespark-1169/ >> >> The test repository (versioned as v1.6.0-rc2) for this release can be >> found at: >> https://repository.apache.org/content/repositories/orgapachespark-1168/ >> >> The documentation corresponding to this release can be found at: >> http://people.apache.org/~pwendell/spark-releases/spark-1.6.0-rc2-docs/ >> >> ======================================= >> == How can I help test this release? == >> ======================================= >> If you are a Spark user, you can help us test this release by taking an >> existing Spark workload and running on this release candidate, then >> reporting any regressions. >> >> ================================================ >> == What justifies a -1 vote for this release? == >> ================================================ >> This vote is happening towards the end of the 1.6 QA period, so -1 votes >> should only occur for significant regressions from 1.5. Bugs already >> present in 1.5, minor regressions, or bugs related to new features will not >> block this release. >> >> =============================================================== >> == What should happen to JIRA tickets still targeting 1.6.0? == >> =============================================================== >> 1. It is OK for documentation patches to target 1.6.0 and still go into >> branch-1.6, since documentations will be published separately from the >> release. >> 2. New features for non-alpha-modules should target 1.7+. >> 3. Non-blocker bug fixes should target 1.6.1 or 1.7.0, or drop the target >> version. >> >> >> ================================================== >> == Major changes to help you focus your testing == >> ================================================== >> >> Spark 1.6.0 PreviewNotable changes since 1.6 RC1Spark Streaming >> >> - SPARK-2629 <https://issues.apache.org/jira/browse/SPARK-2629> >> trackStateByKey has been renamed to mapWithState >> >> Spark SQL >> >> - SPARK-12165 <https://issues.apache.org/jira/browse/SPARK-12165> >> SPARK-12189 <https://issues.apache.org/jira/browse/SPARK-12189> Fix >> bugs in eviction of storage memory by execution. >> - SPARK-12258 <https://issues.apache.org/jira/browse/SPARK-12258> correct >> passing null into ScalaUDF >> >> Notable Features Since 1.5Spark SQL >> >> - SPARK-11787 <https://issues.apache.org/jira/browse/SPARK-11787> Parquet >> Performance - Improve Parquet scan performance when using flat >> schemas. >> - SPARK-10810 <https://issues.apache.org/jira/browse/SPARK-10810> >> Session Management - Isolated devault database (i.e USE mydb) even on >> shared clusters. >> - SPARK-9999 <https://issues.apache.org/jira/browse/SPARK-9999> Dataset >> API - A type-safe API (similar to RDDs) that performs many operations >> on serialized binary data and code generation (i.e. Project Tungsten). >> - SPARK-10000 <https://issues.apache.org/jira/browse/SPARK-10000> Unified >> Memory Management - Shared memory for execution and caching instead >> of exclusive division of the regions. >> - SPARK-11197 <https://issues.apache.org/jira/browse/SPARK-11197> SQL >> Queries on Files - Concise syntax for running SQL queries over files >> of any supported format without registering a table. >> - SPARK-11745 <https://issues.apache.org/jira/browse/SPARK-11745> Reading >> non-standard JSON files - Added options to read non-standard JSON >> files (e.g. single-quotes, unquoted attributes) >> - SPARK-10412 <https://issues.apache.org/jira/browse/SPARK-10412> >> Per-operator >> Metrics for SQL Execution - Display statistics on a peroperator basis >> for memory usage and spilled data size. >> - SPARK-11329 <https://issues.apache.org/jira/browse/SPARK-11329> Star >> (*) expansion for StructTypes - Makes it easier to nest and unest >> arbitrary numbers of columns >> - SPARK-10917 <https://issues.apache.org/jira/browse/SPARK-10917>, >> SPARK-11149 <https://issues.apache.org/jira/browse/SPARK-11149> In-memory >> Columnar Cache Performance - Significant (up to 14x) speed up when >> caching data that contains complex types in DataFrames or SQL. >> - SPARK-11111 <https://issues.apache.org/jira/browse/SPARK-11111> Fast >> null-safe joins - Joins using null-safe equality (<=>) will now >> execute using SortMergeJoin instead of computing a cartisian product. >> - SPARK-11389 <https://issues.apache.org/jira/browse/SPARK-11389> SQL >> Execution Using Off-Heap Memory - Support for configuring query >> execution to occur using off-heap memory to avoid GC overhead >> - SPARK-10978 <https://issues.apache.org/jira/browse/SPARK-10978> >> Datasource >> API Avoid Double Filter - When implemeting a datasource with filter >> pushdown, developers can now tell Spark SQL to avoid double evaluating a >> pushed-down filter. >> - SPARK-4849 <https://issues.apache.org/jira/browse/SPARK-4849> Advanced >> Layout of Cached Data - storing partitioning and ordering schemes in >> In-memory table scan, and adding distributeBy and localSort to DF API >> - SPARK-9858 <https://issues.apache.org/jira/browse/SPARK-9858> Adaptive >> query execution - Intial support for automatically selecting the >> number of reducers for joins and aggregations. >> - SPARK-9241 <https://issues.apache.org/jira/browse/SPARK-9241> Improved >> query planner for queries having distinct aggregations - Query plans >> of distinct aggregations are more robust when distinct columns have high >> cardinality. >> >> Spark Streaming >> >> - API Updates >> - SPARK-2629 <https://issues.apache.org/jira/browse/SPARK-2629> New >> improved state management - mapWithState - a DStream >> transformation for stateful stream processing, supercedes >> updateStateByKey in functionality and performance. >> - SPARK-11198 <https://issues.apache.org/jira/browse/SPARK-11198> >> Kinesis >> record deaggregation - Kinesis streams have been upgraded to use >> KCL 1.4.0 and supports transparent deaggregation of KPL-aggregated >> records. >> - SPARK-10891 <https://issues.apache.org/jira/browse/SPARK-10891> >> Kinesis >> message handler function - Allows arbitraray function to be >> applied to a Kinesis record in the Kinesis receiver before to customize >> what data is to be stored in memory. >> - SPARK-6328 <https://issues.apache.org/jira/browse/SPARK-6328> Python >> Streamng Listener API - Get streaming statistics (scheduling >> delays, batch processing times, etc.) in streaming. >> >> >> - UI Improvements >> - Made failures visible in the streaming tab, in the timelines, >> batch list, and batch details page. >> - Made output operations visible in the streaming tab as progress >> bars. >> >> MLlibNew algorithms/models >> >> - SPARK-8518 <https://issues.apache.org/jira/browse/SPARK-8518> Survival >> analysis - Log-linear model for survival analysis >> - SPARK-9834 <https://issues.apache.org/jira/browse/SPARK-9834> Normal >> equation for least squares - Normal equation solver, providing R-like >> model summary statistics >> - SPARK-3147 <https://issues.apache.org/jira/browse/SPARK-3147> Online >> hypothesis testing - A/B testing in the Spark Streaming framework >> - SPARK-9930 <https://issues.apache.org/jira/browse/SPARK-9930> New >> feature transformers - ChiSqSelector, QuantileDiscretizer, SQL >> transformer >> - SPARK-6517 <https://issues.apache.org/jira/browse/SPARK-6517> Bisecting >> K-Means clustering - Fast top-down clustering variant of K-Means >> >> API improvements >> >> - ML Pipelines >> - SPARK-6725 <https://issues.apache.org/jira/browse/SPARK-6725> >> Pipeline >> persistence - Save/load for ML Pipelines, with partial coverage of >> spark.ml algorithms >> - SPARK-5565 <https://issues.apache.org/jira/browse/SPARK-5565> LDA >> in ML Pipelines - API for Latent Dirichlet Allocation in ML >> Pipelines >> - R API >> - SPARK-9836 <https://issues.apache.org/jira/browse/SPARK-9836> R-like >> statistics for GLMs - (Partial) R-like stats for ordinary least >> squares via summary(model) >> - SPARK-9681 <https://issues.apache.org/jira/browse/SPARK-9681> >> Feature >> interactions in R formula - Interaction operator ":" in R formula >> - Python API - Many improvements to Python API to approach feature >> parity >> >> Misc improvements >> >> - SPARK-7685 <https://issues.apache.org/jira/browse/SPARK-7685>, >> SPARK-9642 <https://issues.apache.org/jira/browse/SPARK-9642> Instance >> weights for GLMs - Logistic and Linear Regression can take instance >> weights >> - SPARK-10384 <https://issues.apache.org/jira/browse/SPARK-10384>, >> SPARK-10385 <https://issues.apache.org/jira/browse/SPARK-10385> Univariate >> and bivariate statistics in DataFrames - Variance, stddev, >> correlations, etc. >> - SPARK-10117 <https://issues.apache.org/jira/browse/SPARK-10117> LIBSVM >> data source - LIBSVM as a SQL data sourceDocumentation improvements >> - >> >>
