Ted - technically I think you are correct, although I wouldn't recommend disabling this lock. This lock is not expensive (acquired once per task, as are many other locks already). Also, we've seen some cases where Hadoop concurrency bugs ended up requiring multiple fixes - concurrency of client access is not well tested in the Hadoop codebase since most of the Hadoop tools to not use concurrent access. So in general it's good to be conservative in what we expect of the Hadoop client libraries.
If you'd like to discuss this further, please fork a new thread, since this is a vote thread. Thanks! On Fri, Jul 25, 2014 at 10:14 PM, Ted Yu <yuzhih...@gmail.com> wrote: > HADOOP-10456 is fixed in hadoop 2.4.1 > > Does this mean that synchronization > on HadoopRDD.CONFIGURATION_INSTANTIATION_LOCK can be bypassed for hadoop > 2.4.1 ? > > Cheers > > > On Fri, Jul 25, 2014 at 6:00 PM, Patrick Wendell <pwend...@gmail.com> wrote: > >> The most important issue in this release is actually an ammendment to >> an earlier fix. The original fix caused a deadlock which was a >> regression from 1.0.0->1.0.1: >> >> Issue: >> https://issues.apache.org/jira/browse/SPARK-1097 >> >> 1.0.1 Fix: >> https://github.com/apache/spark/pull/1273/files (had a deadlock) >> >> 1.0.2 Fix: >> https://github.com/apache/spark/pull/1409/files >> >> I failed to correctly label this on JIRA, but I've updated it! >> >> On Fri, Jul 25, 2014 at 5:35 PM, Michael Armbrust >> <mich...@databricks.com> wrote: >> > That query is looking at "Fix Version" not "Target Version". The fact >> that >> > the first one is still open is only because the bug is not resolved in >> > master. It is fixed in 1.0.2. The second one is partially fixed in >> 1.0.2, >> > but is not worth blocking the release for. >> > >> > >> > On Fri, Jul 25, 2014 at 4:23 PM, Nicholas Chammas < >> > nicholas.cham...@gmail.com> wrote: >> > >> >> TD, there are a couple of unresolved issues slated for 1.0.2 >> >> < >> >> >> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SPARK%20AND%20fixVersion%20%3D%201.0.2%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20priority%20DESC >> >> >. >> >> Should they be edited somehow? >> >> >> >> >> >> On Fri, Jul 25, 2014 at 7:08 PM, Tathagata Das < >> >> tathagata.das1...@gmail.com> >> >> wrote: >> >> >> >> > Please vote on releasing the following candidate as Apache Spark >> version >> >> > 1.0.2. >> >> > >> >> > This release fixes a number of bugs in Spark 1.0.1. >> >> > Some of the notable ones are >> >> > - SPARK-2452: Known issue is Spark 1.0.1 caused by attempted fix for >> >> > SPARK-1199. The fix was reverted for 1.0.2. >> >> > - SPARK-2576: NoClassDefFoundError when executing Spark QL query on >> >> > HDFS CSV file. >> >> > The full list is at http://s.apache.org/9NJ >> >> > >> >> > The tag to be voted on is v1.0.2-rc1 (commit 8fb6f00e): >> >> > >> >> > >> >> >> https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=8fb6f00e195fb258f3f70f04756e07c259a2351f >> >> > >> >> > The release files, including signatures, digests, etc can be found at: >> >> > http://people.apache.org/~tdas/spark-1.0.2-rc1/ >> >> > >> >> > Release artifacts are signed with the following key: >> >> > https://people.apache.org/keys/committer/tdas.asc >> >> > >> >> > The staging repository for this release can be found at: >> >> > >> https://repository.apache.org/content/repositories/orgapachespark-1024/ >> >> > >> >> > The documentation corresponding to this release can be found at: >> >> > http://people.apache.org/~tdas/spark-1.0.2-rc1-docs/ >> >> > >> >> > Please vote on releasing this package as Apache Spark 1.0.2! >> >> > >> >> > The vote is open until Tuesday, July 29, at 23:00 UTC and passes if >> >> > a majority of at least 3 +1 PMC votes are cast. >> >> > [ ] +1 Release this package as Apache Spark 1.0.2 >> >> > [ ] -1 Do not release this package because ... >> >> > >> >> > To learn more about Apache Spark, please see >> >> > http://spark.apache.org/ >> >> > >> >> >>
