[
https://issues.apache.org/jira/browse/HIVE-20064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532031#comment-16532031
]
gavin_shi commented on HIVE-20064:
----------------------------------
i can not understand that mr is not really supported anymore,I'm just a
beginner in hive
> Data cannot be inserted into a dynamic partition
> ------------------------------------------------
>
> Key: HIVE-20064
> URL: https://issues.apache.org/jira/browse/HIVE-20064
> Project: Hive
> Issue Type: Bug
> Components: SQL
> Reporter: gavin_shi
> Priority: Blocker
>
> Data cannot be inserted into a dynamic partition,The relevant table
> information is as follows：
> The source table:
> {code:java}
> //代码占位符
> create table customer(id string,name string,gender string,state string)
> partitioned by(country string);
> {code}
> The source table data：
>
> {code:java}
> //代码占位符
> +--------------+----------------+------------------+-----------------+-------------------+--+
> | customer.id | customer.name | customer.gender | customer.state |
> customer.country |
> +--------------+----------------+------------------+-----------------+-------------------+--+
> | 1 | gavin | 1 | shanghai | china |
> +--------------+----------------+------------------+-----------------+-------------------+--+
> {code}
>
> The target table:
>
> {code:java}
> //代码占位符
> create table customer_bak(id string,name string,gender string,state string)
> partitioned by(country string);
> {code}
>
> Now I use dynamic techniques to insert data into the customer_bak table：
> {code:java}
> //代码占位符
> set hive.exec.dynamic.partition=true;
> set hive.exec.dynamic.partition.mode=nonstrict;
> insert into table customer_bak partition(country) select
> id,name,gender,state,country from customer;
> {code}
> The program is always executing and neither success nor failure is shown. The
> information in the hive log is as follows:
> {code:java}
> //代码占位符
> 2018-06-28T16:01:22,900 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Driver: Executing
> command(queryId=hadoop_20180628160122_f80c5f8e-98a0-4286-bc55-c25efd7a60bc):
> insert into table customer_bak partition(country) select
> id,name,gender,state,country from customer
> 2018-06-28T16:01:22,901 WARN [HiveServer2-Background-Pool: Thread-97]
> ql.Driver: Hive-on-MR is deprecated in Hive 2 and may not be available in the
> future versions. Consider using a different execution engine (i.e. tez,
> spark) or using Hive 1.X releases.
> 2018-06-28T16:01:22,902 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Driver: WARNING: Hive-on-MR is deprecated in Hive 2 and may not be
> available in the future versions. Consider using a different execution engine
> (i.e. tez, spark) or using Hive 1.X releases.
> 2018-06-28T16:01:22,902 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Driver: Query ID =
> hadoop_20180628160122_f80c5f8e-98a0-4286-bc55-c25efd7a60bc
> 2018-06-28T16:01:22,903 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Driver: Total jobs = 3
> 2018-06-28T16:01:22,904 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Driver: Launching Job 1 out of 3
> 2018-06-28T16:01:22,907 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Driver: Starting task [Stage-1:MAPRED] in serial mode
> 2018-06-28T16:01:22,908 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Task: Number of reduce tasks is set to 0 since there's no reduce operator
> 2018-06-28T16:01:22,929 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Context: New scratch dir is
> hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3
> 2018-06-28T16:01:22,942 INFO [HiveServer2-Background-Pool: Thread-97]
> mr.ExecDriver: Using org.apache.hadoop.hive.ql.io.CombineHiveInputFormat
> 2018-06-28T16:01:22,943 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Utilities: Processing alias customer
> 2018-06-28T16:01:22,943 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Utilities: Adding 2 inputs; the first input is
> hdfs://hadoop-cluster/user/hive/warehouse/customer/country=china
> 2018-06-28T16:01:22,943 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Utilities: Content Summary not cached for
> hdfs://hadoop-cluster/user/hive/warehouse/customer/country=china
> 2018-06-28T16:01:22,945 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Utilities: Content Summary not cached for
> hdfs://hadoop-cluster/user/hive/warehouse/customer/country=usa
> 2018-06-28T16:01:22,978 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Utilities: Changed input file
> hdfs://hadoop-cluster/user/hive/warehouse/customer/country=usa to empty file
> hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10003/408baabf-a30d-4c00-a196-70e14e172036
> (false)
> 2018-06-28T16:01:22,979 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Context: New scratch dir is
> hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3
> 2018-06-28T16:01:22,996 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.SerializationUtilities: Serializing MapWork using kryo
> 2018-06-28T16:01:23,055 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Utilities: Serialized plan (via FILE) - name: null size: 4.86KB
> 2018-06-28T16:01:29,318 INFO [HiveServer2-Background-Pool: Thread-97]
> fs.FSStatsPublisher: created :
> hdfs://hadoop-cluster/user/hive/warehouse/customer_bak/.hive-staging_hive_2018-06-28_16-01-22_402_5116918946052746369-1/-ext-10001
> 2018-06-28T16:01:29,395 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Utilities: PLAN PATH =
> hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10004/2cb63e68-2c77-4554-817a-0e25d700cf0f/map.xml
> 2018-06-28T16:01:29,395 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Utilities: PLAN PATH =
> hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10004/2cb63e68-2c77-4554-817a-0e25d700cf0f/reduce.xml
> 2018-06-28T16:01:29,398 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Utilities: No plan file found:
> hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10004/2cb63e68-2c77-4554-817a-0e25d700cf0f/reduce.xml
> 2018-06-28T16:01:29,406 INFO [HiveServer2-Background-Pool: Thread-97]
> client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
> 2018-06-28T16:01:29,413 WARN [HiveServer2-Background-Pool: Thread-97]
> mapreduce.JobSubmitter: Hadoop command-line option parsing not performed.
> Implement the Tool interface and execute your application with ToolRunner to
> remedy this.
> 2018-06-28T16:01:31,721 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Utilities: PLAN PATH =
> hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10004/2cb63e68-2c77-4554-817a-0e25d700cf0f/map.xml
> 2018-06-28T16:01:31,722 INFO [HiveServer2-Background-Pool: Thread-97]
> io.CombineHiveInputFormat: Total number of paths: 2, launching 1 threads to
> check non-combinable ones.
> 2018-06-28T16:01:31,724 INFO [HiveServer2-Background-Pool: Thread-97]
> io.CombineHiveInputFormat: CombineHiveInputSplit creating pool for
> hdfs://hadoop-cluster/user/hive/warehouse/customer/country=china; using
> filter path hdfs://hadoop-cluster/user/hive/warehouse/customer/country=china
> 2018-06-28T16:01:31,725 INFO [HiveServer2-Background-Pool: Thread-97]
> io.CombineHiveInputFormat: CombineHiveInputSplit: pool is already created for
> hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10003/408baabf-a30d-4c00-a196-70e14e172036;
> using filter path
> hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10003/408baabf-a30d-4c00-a196-70e14e172036
> 2018-06-28T16:01:31,738 INFO [HiveServer2-Background-Pool: Thread-97]
> input.FileInputFormat: Total input paths to process : 2
> 2018-06-28T16:01:31,742 INFO [HiveServer2-Background-Pool: Thread-97]
> input.CombineFileInputFormat: DEBUG: Terminated node allocation with :
> CompletedNodes: 3, size left: 0
> 2018-06-28T16:01:31,742 INFO [HiveServer2-Background-Pool: Thread-97]
> io.CombineHiveInputFormat: number of splits 1
> 2018-06-28T16:01:31,743 INFO [HiveServer2-Background-Pool: Thread-97]
> io.CombineHiveInputFormat: Number of all splits 1
> 2018-06-28T16:01:31,853 INFO [HiveServer2-Background-Pool: Thread-97]
> mapreduce.JobSubmitter: number of splits:1
> 2018-06-28T16:01:31,982 INFO [HiveServer2-Background-Pool: Thread-97]
> mapreduce.JobSubmitter: Submitting tokens for job: job_1530599349924_0007
> 2018-06-28T16:01:32,220 INFO [HiveServer2-Background-Pool: Thread-97]
> impl.YarnClientImpl: Submitted application application_1530599349924_0007
> 2018-06-28T16:01:32,226 INFO [HiveServer2-Background-Pool: Thread-97]
> mapreduce.Job: The url to track the job:
> http://cluster3:8088/proxy/application_1530599349924_0007/
> 2018-06-28T16:01:32,227 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Task: Starting Job = job_1530599349924_0007, Tracking URL =
> http://cluster3:8088/proxy/application_1530599349924_0007/
> 2018-06-28T16:01:32,227 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Task: Kill Command = /home/hadoop/app/hadoop_2.6/bin/hadoop job -kill
> job_1530599349924_0007
> 2018-06-28T16:01:56,835 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Task: Hadoop job information for Stage-1: number of mappers: 1; number
> of reducers: 0
> 2018-06-28T16:01:56,933 WARN [HiveServer2-Background-Pool: Thread-97]
> mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is
> deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
> 2018-06-28T16:01:56,934 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Task: 2018-06-28 16:01:56,933 Stage-1 map = 0%, reduce = 0%
> 2018-06-28T16:02:05,922 INFO [HiveServer2-Handler-Pool: Thread-30]
> conf.HiveConf: Using the default value passed in for log id:
> 07fd131c-6590-4081-b7ed-2f0ae5b16551
> 2018-06-28T16:02:05,922 INFO [HiveServer2-Handler-Pool: Thread-30]
> session.SessionState: Updating thread name to
> 07fd131c-6590-4081-b7ed-2f0ae5b16551 HiveServer2-Handler-Pool: Thread-30
> 2018-06-28T16:02:05,924 INFO [07fd131c-6590-4081-b7ed-2f0ae5b16551
> HiveServer2-Handler-Pool: Thread-30] conf.HiveConf: Using the default value
> passed in for log id: 07fd131c-6590-4081-b7ed-2f0ae5b16551
> 2018-06-28T16:02:05,925 INFO [HiveServer2-Handler-Pool: Thread-30]
> session.SessionState: Resetting thread name to HiveServer2-Handler-Pool:
> Thread-30
> 2018-06-28T16:02:35,131 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Task: 2018-06-28 16:02:35,130 Stage-1 map = 100%, reduce = 0%,
> Cumulative CPU 2.82 sec
> 2018-06-28T16:02:39,527 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Task: MapReduce Total cumulative CPU time: 2 seconds 820 msec
> 2018-06-28T16:02:39,607 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Task: Ended Job = job_1530599349924_0007
> 2018-06-28T16:02:39,643 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.FileSinkOperator: Moving tmp dir:
> hdfs://hadoop-cluster/user/hive/warehouse/customer_bak/.hive-staging_hive_2018-06-28_16-01-22_402_5116918946052746369-1/_tmp.-ext-10002
> to:
> hdfs://hadoop-cluster/user/hive/warehouse/customer_bak/.hive-staging_hive_2018-06-28_16-01-22_402_5116918946052746369-1/-ext-10002
> 2018-06-28T16:02:39,669 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Driver: Starting task [Stage-7:CONDITIONAL] in serial mode
> 2018-06-28T16:02:39,688 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Task: Stage-4 is selected by condition resolver.
> 2018-06-28T16:02:39,689 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Task: Stage-3 is filtered out by condition resolver.
> 2018-06-28T16:02:39,689 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Task: Stage-5 is filtered out by condition resolver.
> 2018-06-28T16:02:39,690 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Driver: Starting task [Stage-4:MOVE] in serial mode
> 2018-06-28T16:02:39,691 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Task: Moving data to directory
> hdfs://hadoop-cluster/user/hive/warehouse/customer_bak/.hive-staging_hive_2018-06-28_16-01-22_402_5116918946052746369-1/-ext-10000
> from
> hdfs://hadoop-cluster/user/hive/warehouse/customer_bak/.hive-staging_hive_2018-06-28_16-01-22_402_5116918946052746369-1/-ext-10002
> 2018-06-28T16:02:39,785 INFO [HiveServer2-Background-Pool: Thread-97]
> ql.Driver: Starting task [Stage-0:MOVE] in serial mode
> 2018-06-28T16:02:39,786 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Task: Loading data to table default.customer_bak partition
> (country=null) from
> hdfs://hadoop-cluster/user/hive/warehouse/customer_bak/.hive-staging_hive_2018-06-28_16-01-22_402_5116918946052746369-1/-ext-10000
> 2018-06-28T16:02:39,871 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.MoveTask: Partition is: {country=null}
> 2018-06-28T16:02:39,879 INFO [HiveServer2-Background-Pool: Thread-97]
> exec.Task:
> {code}
> Why doesn't the program continue？
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
