GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/3113
[SPARK-3971][SQL] Backport #2843 to branch-1.1
This PR backports #2843 to branch-1.1. The key difference is that this one
doesn't support Hive 0.13.1 and thus always returns `0.12.0` when
`spark.sql.hive.version` is queried.
5 other commits on which #2843 depends were also backported, they are:
- #2887 (for `SessionState` lifecycle control)
- #2214, #2823 & #3060 (for major test suite refactoring and bug fixes), and
- #2493 (for reading `spark.sql.*` configurations).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liancheng/spark get-info-for-1.1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3113.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3113
----
commit a445d63b87c376f9559d5bd24e24636c0d01b3e0
Author: Michael Armbrust <[email protected]>
Date: 2014-09-23T19:27:12Z
[SPARK-3646][SQL] Copy SQL configuration from SparkConf when a SQLContext
is created.
This will allow us to take advantage of things like the spark.defaults file.
Author: Michael Armbrust <[email protected]>
Closes #2493 from marmbrus/copySparkConf and squashes the following commits:
0bd1377 [Michael Armbrust] Copy SQL configuration from SparkConf when a
SQLContext is created.
Conflicts:
sql/core/src/main/scala/org/apache/spark/sql/test/TestSQLContext.scala
commit a0efc66eab28675a58ba717ecb2249276d013992
Author: Cheng Lian <[email protected]>
Date: 2014-10-13T20:50:27Z
[SPARK-3809][SQL] Fixes test suites in hive-thriftserver
As scwf pointed out, `HiveThriftServer2Suite` isn't effective anymore after
the Thrift server was made a daemon. On the other hand, these test suites were
known flaky, PR #2214 tried to fix them but failed because of unknown Jenkins
build error. This PR fixes both sets of issues.
In this PR, instead of watching `start-thriftserver.sh` output, the test
code start a `tail` process to watch the log file. A `Thread.sleep` has to be
introduced because the `kill` command used in `stop-thriftserver.sh` is not
synchronous.
As for the root cause of the mysterious Jenkins build failure. Please refer
to [this
comment](https://github.com/apache/spark/pull/2675#issuecomment-58464189) below
for details.
----
(Copied from PR description of #2214)
This PR fixes two issues of `HiveThriftServer2Suite` and brings 1
enhancement:
1. Although metastore, warehouse directories and listening port are
randomly chosen, all test cases share the same configuration. Due to parallel
test execution, one of the two test case is doomed to fail
2. We caught any exceptions thrown from a test case and print diagnosis
information, but forgot to re-throw the exception...
3. When the forked server process ends prematurely (e.g., fails to start),
the `serverRunning` promise is completed with a failure, preventing the test
code to keep waiting until timeout.
So, embarrassingly, this test suite was failing continuously for several
days but no one had ever noticed it... Fortunately no bugs in the production
code were covered under the hood.
Author: Cheng Lian <[email protected]>
Author: wangfei <[email protected]>
Closes #2675 from liancheng/fix-thriftserver-tests and squashes the
following commits:
1c384b7 [Cheng Lian] Minor code cleanup, restore the logging level hack in
TestHive.scala
7805c33 [wangfei] reset SPARK_TESTING to avoid loading Log4J
configurations in testing class paths
af2b5a9 [Cheng Lian] Removes log level hacks from TestHiveContext
d116405 [wangfei] make sure that log4j level is INFO
ee92a82 [Cheng Lian] Relaxes timeout
7fd6757 [Cheng Lian] Fixes test suites in hive-thriftserver
Conflicts:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala
commit ce0af002a2e7cd8d40684729335cd37ea56423fe
Author: Cheng Lian <[email protected]>
Date: 2014-10-16T23:15:55Z
[SQL] Fixes the race condition that may cause test failure
The removed `Future` was used to end the test case as soon as the Spark SQL
CLI process exits. When the process exits prematurely, this mechanism prevents
the test case to wait until timeout. But it also creates a race condition: when
`foundAllExpectedAnswers.tryFailure` is called, there are chances that the last
expected output line of the CLI process hasn't been caught by the main logics
of the test code, thus fails the test case.
Removing this `Future` doesn't affect correctness.
Author: Cheng Lian <[email protected]>
Closes #2823 from liancheng/clean-clisuite and squashes the following
commits:
489a97c [Cheng Lian] Fixes the race condition that may cause test failure
commit 755c9b4cb425a4a22ae77fb10adf90f27c42807d
Author: Cheng Lian <[email protected]>
Date: 2014-11-01T22:03:11Z
[SPARK-4037][SQL] Removes the SessionState instance created in
HiveThriftServer2
`HiveThriftServer2` creates a global singleton `SessionState` instance and
overrides `HiveContext` to inject the `SessionState` object. This messes up
`SessionState` initialization and causes problems.
This PR replaces the global `SessionState` with `HiveContext.sessionState`
to avoid the initialization conflict. Also `HiveContext` reuses existing
started `SessionState` if any (this is required by `SparkSQLCLIDriver`, which
uses specialized `CliSessionState`).
Author: Cheng Lian <[email protected]>
Closes #2887 from liancheng/spark-4037 and squashes the following commits:
8446675 [Cheng Lian] Removes redundant Driver initialization
a28fef5 [Cheng Lian] Avoid starting HiveContext.sessionState multiple times
49b1c5b [Cheng Lian] Reuses existing started SessionState if any
3cd6fab [Cheng Lian] Fixes SPARK-4037
Conflicts:
sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLEnv.scala
sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala
commit e395f976d1ce4b5e9f9582c470a9ab07253fef07
Author: Cheng Lian <[email protected]>
Date: 2014-11-02T23:15:52Z
[SQL] Fixes race condition in CliSuite
`CliSuite` has been flaky for a while, this PR tries to improve this
situation by fixing a race condition in `CliSuite`. The `captureOutput`
function is used to capture both stdout and stderr output of the forked
external process in two background threads and search for expected strings, but
wasn't been properly synchronized before.
Author: Cheng Lian <[email protected]>
Closes #3060 from liancheng/fix-cli-suite and squashes the following
commits:
a70569c [Cheng Lian] Fixes race condition in CliSuite
commit 72bffb156433abe469ba893eb2115414355578a1
Author: Cheng Lian <[email protected]>
Date: 2014-11-05T15:24:53Z
Provides Spark and Hive version in HiveThriftServer2 for branch-1.1
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
