----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/11320/#review33144 -----------------------------------------------------------
Updating patch as it missed commit last time. - Venkat Ranganathan On May 30, 2013, 5:11 p.m., Shuaishuai Nie wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/11320/ > ----------------------------------------------------------- > > (Updated May 30, 2013, 5:11 p.m.) > > > Review request for Sqoop and Venkat Ranganathan. > > > Bugs: SQOOP-1056 > https://issues.apache.org/jira/browse/SQOOP-1056 > > > Repository: sqoop-trunk > > > Description > ------- > > Implement a pluggable way for handling connection failures, and/or > intermittent errors in Sqoop. This is especially crucial in environments > where the probability of connections getting reset or throttled is high. > In case of intermittent failures in Sqoop, due to connection losses or server > throttling, Sqoop does not recover from those failures. As a result, the > running Sqoop task would eventually fail, and a new task is started. In those > cases, Sqoop does not always guarantee that tasks can safely be restarted. > For example, if part of the records is already committed to the database, > then restarting the task would result in some failures like primary key > violations. Even for Sqoop jobs which commit the records only at the end of > the task, any failures towards the end of the task would involve reprocessing > the whole range of split owned by the task, and any progress is lost. (More > detail please refer to the document attached at SQOOP-1056) > > > Diffs > ----- > > src/java/org/apache/sqoop/manager/SQLServerManager.java eea5a1d > src/java/org/apache/sqoop/mapreduce/SQLServerAsyncDBExecThread.java > PRE-CREATION > src/java/org/apache/sqoop/mapreduce/SQLServerExportDBExecThread.java > PRE-CREATION > > src/java/org/apache/sqoop/mapreduce/SQLServerResilientExportOutputFormat.java > PRE-CREATION > > src/java/org/apache/sqoop/mapreduce/SQLServerResilientUpdateOutputFormat.java > PRE-CREATION > src/java/org/apache/sqoop/mapreduce/SQLServerUpdateDBExecThread.java > PRE-CREATION > src/java/org/apache/sqoop/mapreduce/db/BasicRetrySQLFailureHandler.java > PRE-CREATION > src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java 7b9e4d2 > src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java > 4b44244 > src/java/org/apache/sqoop/mapreduce/db/SQLFailureHandler.java PRE-CREATION > > src/java/org/apache/sqoop/mapreduce/db/SQLServerConnectionFailureHandler.java > PRE-CREATION > src/java/org/apache/sqoop/mapreduce/db/SQLServerDBInputFormat.java > PRE-CREATION > src/java/org/apache/sqoop/mapreduce/db/SQLServerDBRecordReader.java > PRE-CREATION > src/java/org/apache/sqoop/mapreduce/sqlserver/SqlServerRecordReader.java > 2c08f12 > > Diff: https://reviews.apache.org/r/11320/diff/ > > > Testing > ------- > > Tested with fault injection on SQL server integration tests from SQOOP-1035 > > > Thanks, > > Shuaishuai Nie > >
