[ 
https://issues.apache.org/jira/browse/SQOOP-3317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16447063#comment-16447063
 ] 

Sri Kumaran Thirupathy commented on SQOOP-3317:
-----------------------------------------------

[~BoglarkaEgyed] 

01. Source = Oracle OL*T*P DB

02. Command used, 

--validate --validator org.apache.sqoop.validation.RowCountValidator \
--validation-threshold \
org.apache.sqoop.validation.AbsoluteValidationThreshold \
--validation-failurehandler \
org.apache.sqoop.validation.AbortOnFailureHandler

03. Issue: sqoop.validation.RowCountValidator is pulling source count after 
Mapper job completes and verifying with the retrived records. 

i) When the Sqoop Mapper started, the source count was 8427534. 

ii) After Mapper completed, when the Sqoop validator retrived count from 
Source, the source count is 8427566. 

In OLTP DB, the count varies every second. The validator can use Mapper 
input/output count to verify/validate the transaction. 

Sqoop Logs:

Map-Reduce Framework
Map *input* records=8427534
Map *output* records=8427534

[main] DEBUG org.apache.sqoop.validation.RowCountValidator - Validating data 
using row counts: *Source [8427566]* with *Target[8427534]*
[main] DEBUG org.apache.sqoop.validation.AbsoluteValidationThreshold - Absolute 
Validation threshold comparing 8427566 with 8427534

04. How to use Percentage Tolerant? 

 

> org.apache.sqoop.validation.RowCountValidator in live RDBMS system
> ------------------------------------------------------------------
>
>                 Key: SQOOP-3317
>                 URL: https://issues.apache.org/jira/browse/SQOOP-3317
>             Project: Sqoop
>          Issue Type: Bug
>            Reporter: Sri Kumaran Thirupathy
>            Priority: Major
>
> org.apache.sqoop.validation.RowCountValidator is retrieving count from Source 
> after the MR completes. This fails in live RDBMS case.
> org.apache.sqoop.validation.RowCountValidator can retrive count during MR 
> execution phase.  
> Also, How to use Percentage Tolerant? Reference: 
> [https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to