Impala Public Jenkins has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/21160 )
Change subject: IMPALA-12921, IMPALA-12985: Support running Impala with locally
built Ranger
......................................................................
IMPALA-12921, IMPALA-12985: Support running Impala with locally built Ranger
The goals and non-goals of this patch could be summarized as follows.
Goals:
- Add changes to the minicluster configuration that allow a non-default
version of Ranger (possibly built locally) to run in the context of
the minicluster, and to be used as the authorization server by
Impala.
- Switch to the new constructor when instantiating
RangerAccessRequestImpl. This resolves IMPALA-12985 and also makes
Impala compatible with Apache Ranger if RangerAccessRequestImpl from
Apache Ranger is consumed.
- Prepare Ranger and Impala patches as supplemental material to verify
what authorization-related tests could be passed if Apache Ranger is
the authorization provider. Merging IMPALA-12921_addendum.diff to
the Impala repository is not in the scope of this patch in that the
diff file changes the behavior of Impala and thus more discussion is
required if we'd like to merge it in the future.
Non-goals:
- Set up any automation for building Ranger from source.
- Pass all Impala authorization-related tests with a non-default
version of Ranger.
Instructions on running Impala with locally built Ranger:
Suppose the Ranger project is under the folder $RANGER_SRC_DIR. We could
execute the following to build Apache Ranger for easy reference. By
default, the compressed tarball is produced under
$RANGER_SRC_DIR/target.
mvn clean compile -B -nsu -DskipCheck=true -Dcheckstyle.skip=true \
package install -DskipITs -DskipTests -Dmaven.javadoc.skip=true
After building Ranger, we need to build Impala's Java code so that
Impala's Java code could consume the locally produced Ranger classes. We
will need to export the following environment variables before building
Impala. This prevents bootstrap_toolchain.py from trying to download the
compressed Ranger tarball.
1. export RANGER_VERSION_OVERRIDE=\
$(mvn -f $RANGER_SRC_DIR/pom.xml -q help:evaluate \
-Dexpression=project.version -DforceStdout)
2. export RANGER_HOME_OVERRIDE=$RANGER_SRC_DIR/target/\
ranger-${RANGER_VERSION_OVERRIDE}-admin
It then suffices to execute the following to point
Impala to the locally built Ranger server before starting Impala.
1. source $IMPALA_HOME/bin/impala-config.sh
2. tar zxv -f $RANGER_SRC_DIR/target/\
ranger-${IMPALA_RANGER_VERSION}-admin.tar.gz \
-C $RANGER_SRC_DIR/target/
3. $IMPALA_HOME/bin/create-test-configuration.sh
4. $IMPALA_HOME/bin/create-test-configuration.sh \
-create_ranger_policy_db
5. $IMPALA_HOME/testdata/bin/run-ranger.sh
(run-all.sh has to be executed instead if other underlying services
have not been started)
6. $IMPALA_HOME/testdata/bin/setup-ranger.sh
Testing:
- Manually verified that we could point Impala to a locally built
Apache Ranger on the master branch (with tip being
https://github.com/apache/ranger/commit/4abb993).
- Manually verified that with RANGER-4771.diff and
IMPALA-12921_addendum.diff, only 3 authorization-related tests
failed. They failed because the resource type of 'storage-type' is
not supported in Apache Ranger yet and thus the test cases added in
IMPALA-10436 could fail.
- Manually verified that the log files of Apache and CDP Ranger's Admin
server could be created under ${RANGER_LOG_DIR} after we start the
Ranger service.
- Verified that this patch passed the core tests when CDP Ranger is
used.
Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
Reviewed-on: http://gerrit.cloudera.org:8080/21160
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
---
M README-build.md
M bin/bootstrap_toolchain.py
M bin/create-test-configuration.sh
M bin/impala-config.sh
M bin/rat_exclude_files.txt
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java
M
fe/src/main/java/org/apache/impala/authorization/ranger/RangerImpaladAuthorizationManager.java
M testdata/bin/setup-ranger.sh
A testdata/cluster/ranger/IMPALA-12921_addendum.diff
A testdata/cluster/ranger/RANGER-4771.diff
A testdata/cluster/ranger/README
R testdata/cluster/ranger/setup/all_database_policy_revised.json.template
A testdata/cluster/ranger/setup/impala_group_non_owner_2.json
M testdata/cluster/ranger/setup/impala_user_non_owner.json.template
A testdata/cluster/ranger/setup/impala_user_non_owner_2.json.template
M testdata/cluster/ranger/setup/impala_user_owner.json.template
16 files changed, 385 insertions(+), 16 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/21160
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I268d6d4d6e371da7497aac8d12f78178d57c6f27
Gerrit-Change-Number: 21160
Gerrit-PatchSet: 18
Gerrit-Owner: Fang-Yu Rao <[email protected]>
Gerrit-Reviewer: Aman Sinha <[email protected]>
Gerrit-Reviewer: Fang-Yu Rao <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: John Sherman <[email protected]>
Gerrit-Reviewer: Laszlo Gaal <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
