[ https://issues.apache.org/jira/browse/SQOOP-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Venkatesan updated SQOOP-3240: ------------------------------------- Description: Hive/Hadoop row count was not matching on sqoop import of oracle table with direct mode, it's perfectly matching without direct mode. See the below examples. 1. Row Count is varying based on fetch-size 10000 and direct mode. 2. Row count is not consistent, sometimes it is 630869 and sometimes 639924 etc., 3. We are pulling data with direct mode for 1 year with monthly partition, data mismatch happening for couple of months not for all months. 4. Hive/Hadoop table Count looks good, If we import the data from oracle without direct mode. Actual row count in oracle table for year 2015 and month 12 is : 609924, we are getting more records on sqoop import with direct mode : 630869. hive table count is matching with oracle count on sqoop import *without direct* mode : 609924 *1. sqoop import without direct mode, Row Count : 609924* sqoop import -Dmapredue.job.queuename=queue -Doraoop.import.partitions=oracle_table_partition_201512 --connect jdbc:oracle:thin:@//xxxxx.yyyyy.com:2408/zzzz --username user_id -P --table oracle_db.oracle_table --split-by split_column --hive-import --hive-table hive_db.hive_table --where "TO_DATE(oracle_timestamp_column) BETWEEN TO_DATE('12/01/2015','mm/dd/yyyy') AND TO_DATE('12/31/2015','mm/dd/yyyy')" --fields-terminated-by '\001' --lines-terminated-by '\n' --as-textfile -m 4 *2. sqoop import with direct mode, Row Count : 630869* sqoop import -Dmapredue.job.queuename=queue -Doraoop.import.partitions=oracle_table_partition_201512 --connect jdbc:oracle:thin:@//xxxxx.yyyyy.com:2408/zzzz --username user_id -P --table oracle_db.oracle_table --split-by split_column --hive-import --hive-table hive_db.hive_table --where "TO_DATE(oracle_timestamp_column) BETWEEN TO_DATE('12/01/2015','mm/dd/yyyy') AND TO_DATE('12/31/2015','mm/dd/yyyy')" --fields-terminated-by '\001' --lines-terminated-by '\n' --as-textfile -m 4 --direct was: Hive/Hadoop row count was not matching on sqoop import of oracle table with direct mode, it's perfectly matching without direct mode. See the below examples. 1. Row Count is varying based on fetch-size 10000 and direct mode. 2. Row count is not consistent, sometimes it is 630869 and sometimes 639924 etc., 3. We are pulling data with direct mode for 1 year with monthly partition, data mismatch happening for couple of months not for all months. 4. Hive/Hadoop table Count looks good, If we import the data from oracle without direct mode. Actual row count in oracle table for year 2015 and month 12 is : 609924, we are getting more records on sqoop import with direct mode : 630869. hive table count is matching with oracle count on sqoop import *without direct* mode : 609924 *1. sqoop import without direct mode, Row Count : 609924* sqoop import -Dmapredue.job.queuename=queue -Doraoop.import.partitions=oracle_table_partition_201512 --connect jdbc:oracle:thin:@//xxxxx.yyyyy.com:2408/zzzz --username user_id -P --table oracle_db.oracle_table --split-by split_column --hive-import --hive-table hive_db.hive_table --where "TO_DATE(oracle_timestamp_column) BETWEEN TO_DATE('12/01/2015','mm/dd/yyyy') AND TO_DATE('12/31/2015','mm/dd/yyyy')" --fields-terminated-by '\001' --lines-terminated-by '\n' --as-textfile -m 4 *2. sqoop import with direct mode, Row Count : 630869* sqoop import -Dmapredue.job.queuename=queue -Doraoop.import.partitions=oracle_table_partition_201512 --connect jdbc:oracle:thin:@//xxxxx.yyyyy.com:2408/zzzz --username user_id -P --table oracle_db.oracle_table --split-by split_column --hive-import --hive-table hive_db.hive_table --where "TO_DATE(oracle_timestamp_column) BETWEEN TO_DATE('12/01/2015','mm/dd/yyyy') AND TO_DATE('12/31/2015','mm/dd/yyyy')" --fields-terminated-by '\001' --lines-terminated-by '\n' --as-textfile -m 4 *--direct* > Hive table Row count mismatch(more records in hadoop/hive than oracle) when > importing data from oracle using sqoop import with direct mode > ------------------------------------------------------------------------------------------------------------------------------------------ > > Key: SQOOP-3240 > URL: https://issues.apache.org/jira/browse/SQOOP-3240 > Project: Sqoop > Issue Type: Bug > Components: connectors/oracle > Affects Versions: 1.4.6 > Environment: Sqoop 1.4.6-cdh5.8.2 > Oracle 11i > oracle table Sqoop import with direct mode > Reporter: Suresh Venkatesan > Priority: Minor > > Hive/Hadoop row count was not matching on sqoop import of oracle table with > direct mode, it's perfectly matching without direct mode. > See the below examples. > 1. Row Count is varying based on fetch-size 10000 and direct mode. > 2. Row count is not consistent, sometimes it is 630869 and sometimes 639924 > etc., > 3. We are pulling data with direct mode for 1 year with monthly partition, > data mismatch happening for couple of months not for all months. > 4. Hive/Hadoop table Count looks good, If we import the data from oracle > without direct mode. > Actual row count in oracle table for year 2015 and month 12 is : 609924, we > are getting more records on sqoop import with direct mode : 630869. hive > table count is matching with oracle count on sqoop import *without direct* > mode : 609924 > *1. sqoop import without direct mode, Row Count : 609924* > sqoop import -Dmapredue.job.queuename=queue > -Doraoop.import.partitions=oracle_table_partition_201512 --connect > jdbc:oracle:thin:@//xxxxx.yyyyy.com:2408/zzzz --username user_id -P --table > oracle_db.oracle_table --split-by split_column --hive-import --hive-table > hive_db.hive_table --where "TO_DATE(oracle_timestamp_column) BETWEEN > TO_DATE('12/01/2015','mm/dd/yyyy') AND TO_DATE('12/31/2015','mm/dd/yyyy')" > --fields-terminated-by '\001' --lines-terminated-by '\n' --as-textfile -m 4 > *2. sqoop import with direct mode, Row Count : 630869* > sqoop import -Dmapredue.job.queuename=queue > -Doraoop.import.partitions=oracle_table_partition_201512 --connect > jdbc:oracle:thin:@//xxxxx.yyyyy.com:2408/zzzz --username user_id -P --table > oracle_db.oracle_table --split-by split_column --hive-import --hive-table > hive_db.hive_table --where "TO_DATE(oracle_timestamp_column) BETWEEN > TO_DATE('12/01/2015','mm/dd/yyyy') AND TO_DATE('12/31/2015','mm/dd/yyyy')" > --fields-terminated-by '\001' --lines-terminated-by '\n' --as-textfile -m 4 > --direct -- This message was sent by Atlassian JIRA (v6.4.14#64029)