[
https://issues.apache.org/jira/browse/SQOOP-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16127254#comment-16127254
]
Sandish Kumar HN commented on SQOOP-2411:
-----------------------------------------
Hi [~anna.szonyi],
Yes, We can't solve this issue from SQOOP side.
if you
see[here|https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/mapreduce/MySQLDumpMapper.java#L437]
.p.waitFor(); is getting time from MYSQL, even the status code is coming from
MYSQL side. The error is coming for the larger dataset and it fails when the
running time is longer than the waiting net-read-timeout. I think we can close
this issue. Correct me if I'm wrong?.
> Sqoop using '--direct' option fails with mysqldump exit code 2 and 3
> --------------------------------------------------------------------
>
> Key: SQOOP-2411
> URL: https://issues.apache.org/jira/browse/SQOOP-2411
> Project: Sqoop
> Issue Type: Bug
> Components: connectors/mysql
> Affects Versions: 1.4.6
> Environment: Amazon EMR
> Reporter: Karthick H
> Assignee: Sandish Kumar HN
> Priority: Critical
>
> I am running Sqoop in AWS EMR. I am trying to copy a table ~10 GB from MySQL
> into HDFS.
> I get the following exception
> 15/07/06 12:19:07 INFO mapreduce.Job: Task Id :
> attempt_1435664372091_0048_m_000000_2, Status : FAILED
> Error: java.io.IOException: mysqldump terminated with status 3
> at org.apache.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:485)
> at org.apache.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:49)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:152)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:773)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:170)
> 15/07/06 12:19:07 INFO mapreduce.Job: Task Id :
> attempt_1435664372091_0048_m_000005_2, Status : FAILED
> Error: java.io.IOException: mysqldump terminated with status 2
> at org.apache.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:485)
> at org.apache.sqoop.mapreduce.MySQLDumpMapper.map(MySQLDumpMapper.java:49)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:152)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:773)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:170)
> 15/07/06 12:19:08 INFO mapreduce.Job: map 0% reduce 0%
> 15/07/06 12:19:20 INFO mapreduce.Job: map 25% reduce 0%
> 15/07/06 12:19:22 INFO mapreduce.Job: map 38% reduce 0%
> 15/07/06 12:19:23 INFO mapreduce.Job: map 50% reduce 0%
> 15/07/06 12:19:24 INFO mapreduce.Job: map 75% reduce 0%
> 15/07/06 12:19:25 INFO mapreduce.Job: map 100% reduce 0%
> 15/07/06 12:23:11 INFO mapreduce.Job: Job job_1435664372091_0048 failed with
> state FAILED due to: Task failed task_1435664372091_0048_m_000000
> Job failed as tasks failed. failedMaps:1 failedReduces:0
> 15/07/06 12:23:11 INFO mapreduce.Job: Counters: 8
> Job Counters
> Failed map tasks=28
> Launched map tasks=28
> Other local map tasks=28
> Total time spent by all maps in occupied slots (ms)=34760760
> Total time spent by all reduces in occupied slots (ms)=0
> Total time spent by all map tasks (ms)=5793460
> Total vcore-seconds taken by all map tasks=5793460
> Total megabyte-seconds taken by all map tasks=8342582400
> 15/07/06 12:23:11 WARN mapreduce.Counters: Group FileSystemCounters is
> deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
> 15/07/06 12:23:11 INFO mapreduce.ImportJobBase: Transferred 0 bytes in
> 829.8697 seconds (0 bytes/sec)
> 15/07/06 12:23:11 WARN mapreduce.Counters: Group
> org.apache.hadoop.mapred.Task$Counter is deprecated. Use
> org.apache.hadoop.mapreduce.TaskCounter instead
> 15/07/06 12:23:11 INFO mapreduce.ImportJobBase: Retrieved 0 records.
> 15/07/06 12:23:11 ERROR tool.ImportTool: Error during import: Import job
> failed!
> If I run with out '--direct' option, I get the communication exception as in
> https://issues.cloudera.org/browse/SQOOP-186
> I have set 'net-write-timeout' and 'net-read-timeout' values in MySQL to 6000.
> My Sqoop command looks like this
> sqoop import -D mapred.task.timeout=0 --fields-terminated-by '\t'
> --escaped-by '\\' --optionally-enclosed-by '\"' --bindir ./ --connect
> jdbc:mysql://<remote ip>/<mysql db> --username tuser --password tuser --table
> table1 --target-dir=/base/table1 --split-by id -m 8 --direct
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
