[
https://issues.apache.org/jira/browse/HIVE-21954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-21954:
--------------------------------
Description:
In HIVE-21940, a postgres metastore related issue has been fixed, and a local
reproduction has been provided.
{code}
export QTEST_LEAVE_FILES=true
docker kill metastore-test-postgres-install
docker rm metastore-test-postgres-install
cd standalone-metastore
mvn verify -DskipITests=false -Dit.test=ITestPostgres#install -Dtest=nosuch
-Dmetastore.itest.no.stop.container=true
cd ..
mvn test -Dtest.output.overwrite=true -Pitests,hadoop-2 -pl itests/qtest
-Dtest=TestCliDriver -Dqfile=partition_params_postgres.q
-Dhive.metastore.rawstore.impl=org.apache.hadoop.hive.metastore.ObjectStore
{code}
The problem with this solution is that data/conf/hive-site.xml has to be edited
manually. My proposal is to introduce a property (-Dmetastore.db=postgres),
which can take care of the parameters on the fly. 2 supported solutions could
be:
1. simple parameters: -Dmetastore.db=postgres
In this case, tests depend on settings from ITestPostgres class (password, db,
etc.)
2. verbose but flexible parameters: [see hive-site.xml HIVE-21940's repro
patch|https://issues.apache.org/jira/secure/attachment/12973534/HIVE-21940.repro.patch]
In the long term, we should consider running this kind of tests in precommit
phase, so maybe -Dmetastore.db=postgres could start metastore db automatically.
Also we should consider running some qtests on various metastores. I would not
pick randomly, but choose some "metastore-heavy" ones instead.
was:
In HIVE-21940, a postgres metastore related issue has been fixed, and a local
reproduction has been provided.
{code}
export QTEST_LEAVE_FILES=true
docker kill metastore-test-postgres-install
docker rm metastore-test-postgres-install
cd standalone-metastore
mvn verify -DskipITests=false -Dit.test=ITestPostgres#install -Dtest=nosuch
-Dmetastore.itest.no.stop.container=true
cd ..
mvn test -Dtest.output.overwrite=true -Pitests,hadoop-2 -pl itests/qtest
-Dtest=TestCliDriver -Dqfile=partition_params_postgres.q
-Dhive.metastore.rawstore.impl=org.apache.hadoop.hive.metastore.ObjectStore
{code}
The problem with this solution is that data/conf/hive-site.xml has to be edited
manually. My proposal is to introduce a property (-Dmetastore.db=postgres),
which can take care of the parameters on the fly. 2 supported solutions could
be:
1. simple parameters: -Dmetastore.db=postgres
In this case, tests depend on settings from ITestPostgres class (password, db,
etc.)
2. verbose but flexible parameters: [see hive-site.xml HIVE-21940's repro
patch|https://issues.apache.org/jira/secure/attachment/12973534/HIVE-21940.repro.patch]
In the first implementation, I would not start metastore db automatically
(which is done be 'mvn verify ...'), but it's still under planning.
In the long term, we should consider running this kind of tests in precommit
phase, so maybe -Dmetastore.db=postgres could start metastore db automatically.
Also we should consider running some qtests on various metastores. I would not
pick randomly, but choose some "metastore-heavy" ones instead.
> QTest: support for running qtests on various metastore DBs
> ----------------------------------------------------------
>
> Key: HIVE-21954
> URL: https://issues.apache.org/jira/browse/HIVE-21954
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore, Testing Infrastructure
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21954.01.patch, HIVE-21954.02.patch,
> HIVE-21954.03.patch, HIVE-21954.03.patch, HIVE-21954.03.patch,
> HIVE-21954.04.patch, HIVE-21954.05.patch, HIVE-21954.07.patch,
> HIVE-21954.07.patch, HIVE-21954.08.patch, HIVE-21954.09.patch,
> HIVE-21954.10.patch, HIVE-21954.10.patch
>
>
> In HIVE-21940, a postgres metastore related issue has been fixed, and a local
> reproduction has been provided.
> {code}
> export QTEST_LEAVE_FILES=true
> docker kill metastore-test-postgres-install
> docker rm metastore-test-postgres-install
> cd standalone-metastore
> mvn verify -DskipITests=false -Dit.test=ITestPostgres#install -Dtest=nosuch
> -Dmetastore.itest.no.stop.container=true
> cd ..
> mvn test -Dtest.output.overwrite=true -Pitests,hadoop-2 -pl itests/qtest
> -Dtest=TestCliDriver -Dqfile=partition_params_postgres.q
> -Dhive.metastore.rawstore.impl=org.apache.hadoop.hive.metastore.ObjectStore
> {code}
> The problem with this solution is that data/conf/hive-site.xml has to be
> edited manually. My proposal is to introduce a property
> (-Dmetastore.db=postgres), which can take care of the parameters on the fly.
> 2 supported solutions could be:
> 1. simple parameters: -Dmetastore.db=postgres
> In this case, tests depend on settings from ITestPostgres class (password,
> db, etc.)
> 2. verbose but flexible parameters: [see hive-site.xml HIVE-21940's repro
> patch|https://issues.apache.org/jira/secure/attachment/12973534/HIVE-21940.repro.patch]
>
>
> In the long term, we should consider running this kind of tests in precommit
> phase, so maybe -Dmetastore.db=postgres could start metastore db
> automatically. Also we should consider running some qtests on various
> metastores. I would not pick randomly, but choose some "metastore-heavy" ones
> instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
