Markus Kemper created SQOOP-3129:
------------------------------------
Summary: Sqoop Metastore job command does not honor -D generic
arguments
Key: SQOOP-3129
URL: https://issues.apache.org/jira/browse/SQOOP-3129
Project: Sqoop
Issue Type: Bug
Reporter: Markus Kemper
Not 100% sure that this is a defect but, it seems like one to me.
The Sqoop job help suggests that generic args are supported however I am not
able to get them to work, see test case below
*Sqoop Help*
{noformat}
$ sqoop job -help
Output:
17/01/17 08:13:12 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.8.3
usage: sqoop job [GENERIC-ARGS] [JOB-ARGS] [-- [<tool-name>] [TOOL-ARGS]]
{noformat}
*Test Case*
{noformat}
#################
# STEP 01 - Setup Table and Data
#################
export MYCONN=jdbc:mysql://mysql.cloudera.com:3306/db_coe
export MYUSER=sqoop
export MYPSWD=cloudera
sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query
"drop table t1"
sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query
"create table t1 (c1 int, c2 varchar(10))"
sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query
"insert into t1 values (1, 'some data')"
sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query
"select * from t1"
----------------------------
| c1 | c2 |
----------------------------
| 1 | some data |
----------------------------
#################
# STEP 02 - Import Data Baseline (default)
#################
sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table t1
--target-dir /user/root/t1 --delete-target-dir --num-mappers 1
yarn application -status application_1481911879790_0093 | head -6
17/01/02 07:03:46 INFO client.RMProxy: Connecting to ResourceManager at
host1.cloudera.com/NN.NN.NNN.NNN:8032
Application Report :
Application-Id : application_1481911879790_0093
Application-Name : t1.jar
Application-Type : MAPREDUCE
User : root
Queue : root.users.root
#################
# STEP 03 - Import Data (-Dmapreduce.job.queuename=MY_SQOOP_QUEUE
-Dmapreduce.job.name=MY_SQOOP_JOB)
#################
sqoop import -Dmapreduce.job.queuename=MY_SQOOP_QUEUE
-Dmapreduce.job.name=MY_SQOOP_JOB --connect $MYCONN --username $MYUSER
--password $MYPSWD --table t1 --target-dir /user/root/t1 --delete-target-dir
--num-mappers 1
yarn application -status application_1481911879790_0094 | head -6
17/01/02 07:07:00 INFO client.RMProxy: Connecting to ResourceManager at
host1.cloudera.com/NN.NN.NNN.NNN:8032
Application Report :
Application-Id : application_1481911879790_0094
Application-Name : MY_SQOOP_JOB
Application-Type : MAPREDUCE
User : root
Queue : root.MY_SQOOP_QUEUE
#################
# STEP 04 - Import Data using Sqoop1 Metastore Job (default)
#################
sqoop job --create my_sqoop_meta_job -- import --connect $MYCONN --username
$MYUSER --password $MYPSWD --table t1 --target-dir /user/root/t1
--delete-target-dir --num-mappers 1
sqoop job --list
Available jobs:
my_sqoop_meta_job
sqoop job --exec my_sqoop_meta_job
yarn application -status application_1481911879790_0095 | head -6
17/01/02 07:16:13 INFO client.RMProxy: Connecting to ResourceManager at
host1.cloudera.com/NN.NN.NNN.NNN:8032
Application Report :
Application-Id : application_1481911879790_0095
Application-Name : t1.jar
Application-Type : MAPREDUCE
User : root
Queue : root.users.root
#################
# STEP 06 - Review Sqoop Job Help
#################
$ sqoop job -help
Output:
17/01/17 08:13:12 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.8.3
usage: sqoop job [GENERIC-ARGS] [JOB-ARGS] [-- [<tool-name>] [TOOL-ARGS]]
Job management arguments:
--create <job-id> Create a new saved job
--delete <job-id> Delete a saved job
--exec <job-id> Run a saved job
--help Print usage instructions
--list List saved jobs
--meta-connect <jdbc-uri> Specify JDBC connect string for the
metastore
--show <job-id> Show the parameters for a saved job
--verbose Print more information while working
Generic Hadoop command-line arguments:
(must preceed any tool-specific arguments)
Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
#################
# STEP 07 - Import Data using Sqoop1 Metastore Job
(-Dmapreduce.job.queuename=MY_SQOOP_QUEUE -Dmapreduce.job.name=MY_SQOOP_JOB)
#################
sqoop job -Dmapreduce.job.queuename=MY_SQOOP_META_QUEUE
-Dmapreduce.job.name=MY_SQOOP_META_JOB --exec my_sqoop_meta_job
yarn application -status application_1481911879790_0096 | head -6
17/01/02 07:19:39 INFO client.RMProxy: Connecting to ResourceManager at
host1.cloudera.com/NN.NN.NNN.NNN:8032
Application Report :
Application-Id : application_1481911879790_0096
Application-Name : t1.jar
Application-Type : MAPREDUCE
User : root
Queue : root.users.root
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
