[
https://issues.apache.org/jira/browse/KUDU-2830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854968#comment-16854968
]
Grant Henke edited comment on KUDU-2830 at 6/3/19 7:21 PM:
-----------------------------------------------------------
This is the formatted full trace text from the example summary above:
{noformat}
Traces:
[0ms] querying master,
[0ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[0ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[0ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[44ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
capacity),
[44ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.22% of capacity),
[59ms] querying master,
[59ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[60ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[97ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
capacity),
[97ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.22% of capacity),
[119ms] querying master,
[119ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[120ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[160ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
capacity),
[160ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.22% of capacity),
[218ms] querying master,
[218ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[218ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[262ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
capacity),
[262ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.22% of capacity),
[479ms] querying master,
[479ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[480ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[516ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
capacity),
[516ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.22% of capacity),
[719ms] querying master,
[719ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[720ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[758ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
capacity),
[758ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.22% of capacity),
[879ms] querying master,
[879ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[879ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[916ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
capacity),
[916ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.22% of capacity),
[1060ms] querying master,
[1060ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[1060ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[1103ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of
capacity),
[1103ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.23% of capacity),
[4480ms] querying master,
[4480ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[4480ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[4524ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of
capacity),
[4524ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.34% of capacity),
[7020ms] querying master,
[7020ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[7020ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[7061ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of
capacity),
[7061ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.42% of capacity),
[9820ms] querying master,
[9820ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[9821ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[9860ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of
capacity),
[9860ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.51% of capacity),
[13100ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[13138ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response
Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.63%
of capacity),
[13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.63% of capacity),
[16920ms] querying master,
[16920ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[16920ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[16957ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response
Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.71%
of capacity),
[16957ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.71% of capacity),
[18959ms] querying master,
[18959ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[18960ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[19002ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response
Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.75%
of capacity),
[19002ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.75% of capacity),
[22439ms] querying master,
[22439ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[22440ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[22476ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response
Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.82%
of capacity),
[22476ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.82% of capacity),
[24520ms] querying master,
[24520ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[24520ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[24562ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response
Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.87%
of capacity),
[24562ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.87% of capacity),
[25079ms] querying master,
[25079ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[25080ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[25117ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response
Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.88%
of capacity),
[25117ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.88% of capacity),
[28739ms] querying master,
[28740ms] trace too long, truncated{noformat}
The summary shows the _MasterRefresh_ for every _SERVICE_UNAVAILABLE_, which is
clearly
[https://github.com/apache/kudu/commit/2b5b7372f27a1009209bd30f86b2a725e9ba58eb]Then
it also shows the timeout is caused by _SERVICE_UNAVAILABLE_ (due to memory
pressure)
Another example it looks like it would illuminate well is
https://issues.apache.org/jira/browse/KUDU-2343
was (Author: granthenke):
This is the formatted full trace text from the example summary above:
{noformat}
Traces:
[0ms] querying master,
[0ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[0ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[0ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[44ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
capacity),
[44ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.22% of capacity),
[59ms] querying master,
[59ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[60ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[97ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
capacity),
[97ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.22% of capacity),
[119ms] querying master,
[119ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[120ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[160ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
capacity),
[160ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.22% of capacity),
[218ms] querying master,
[218ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[218ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[262ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
capacity),
[262ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.22% of capacity),
[479ms] querying master,
[479ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[480ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[516ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
capacity),
[516ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.22% of capacity),
[719ms] querying master,
[719ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[720ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[758ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
capacity),
[758ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.22% of capacity),
[879ms] querying master,
[879ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[879ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[916ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
capacity),
[916ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.22% of capacity),
[1060ms] querying master,
[1060ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[1060ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[1103ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of
capacity),
[1103ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.23% of capacity),
[4480ms] querying master,
[4480ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[4480ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[4524ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of
capacity),
[4524ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.34% of capacity),
[7020ms] querying master,
[7020ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[7020ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[7061ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of
capacity),
[7061ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.42% of capacity),
[9820ms] querying master,
[9820ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[9821ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[9860ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of
capacity),
[9860ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.51% of capacity),
[13100ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[13138ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response
Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.63%
of capacity),
[13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.63% of capacity),
[16920ms] querying master,
[16920ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[16920ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[16957ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response
Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.71%
of capacity),
[16957ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.71% of capacity),
[18959ms] querying master,
[18959ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[18960ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[19002ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response
Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.75%
of capacity),
[19002ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.75% of capacity),
[22439ms] querying master,
[22439ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[22440ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[22476ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response
Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.82%
of capacity),
[22476ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.82% of capacity),
[24520ms] querying master,
[24520ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[24520ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[24562ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response
Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.87%
of capacity),
[24562ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.87% of capacity),
[25079ms] querying master,
[25079ms] Sub rpc: GetTableLocations sending RPC to server
master-vc1320.halxg.cloudera.com:7051,
[25080ms] Sub rpc: GetTableLocations received from server
master-vc1320.halxg.cloudera.com:7051 response OK,
[25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d,
[25117ms] received from server 09d6bf7a02124145b43f43cb7a667b3d response
Service unavailable: Service unavailable: Soft memory limit exceeded (at 95.88%
of capacity),
[25117ms] delaying RPC due to Service unavailable: Service unavailable: Soft
memory limit exceeded (at 95.88% of capacity),
[28739ms] querying master,
[28740ms] trace too long, truncated{noformat}
> Improve RPC traces on RPC timeout
> ---------------------------------
>
> Key: KUDU-2830
> URL: https://issues.apache.org/jira/browse/KUDU-2830
> Project: Kudu
> Issue Type: Improvement
> Reporter: Grant Henke
> Assignee: Grant Henke
> Priority: Major
> Labels: backup
>
> When RPCs are retried repeatedly and then eventually a timeout is caused, the
> logged RPC trace includes the entire nested RPC call stack. This is often way
> to much to read and usually includes a lot of repeated information. Perhaps
> we could use various levels of detail and summarization based on the log
> level to make things more useful.
> Below is a sample RPC trace:
> {noformat}
> 19/01/22 06:01:56 WARN scheduler.TaskSetManager: Lost task 1125.0 in stage
> 1.0 (TID 1152, vc1308.halxg.cloudera.com, executor 9):
> java.lang.RuntimeException: failed to write 516 rows from DataFrame to Kudu;
> sample errors: Timed out: can not complete before timeout:
> Batch{operations=516, tablet="c31055f8d3f74ee3b52687e7aff5b68a"
> [0x0000000300000003, <end>), ignoreAllDuplicateRows=false,
> rpc=KuduRpc(method=Write, tablet=c31055f8d3f74ee3b52687e7aff5b68a,
> attempt=35, DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms]
> querying master, [0ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from
> server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms]
> delaying RPC due to Service unavailable: Service unavailable: Soft memory
> limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub
> rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [97ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms]
> querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [160ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms]
> querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [262ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms]
> querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [516ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms]
> querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [758ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms]
> querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [916ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms]
> querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of
> capacity), [1103ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms]
> querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of
> capacity), [4524ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms]
> querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of
> capacity), [7061ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms]
> querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of
> capacity), [9860ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received
> from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable:
> Service unavailable: Soft memory limit exceeded (at 95.63% of capacity),
> [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft
> memory limit exceeded (at 95.63% of capacity), [16920ms] querying master,
> [16920ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of
> capacity), [16957ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms]
> querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of
> capacity), [19002ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms]
> querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of
> capacity), [22476ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms]
> querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of
> capacity), [24562ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms]
> querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of
> capacity), [25117ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms]
> querying master, [28740ms] trace too long, truncated)}Timed out: can not
> complete before timeout: Batch{operations=516,
> tablet="c31055f8d3f74ee3b52687e7aff5b68a" [0x0000000300000003, <end>),
> ignoreAllDuplicateRows=false, rpc=KuduRpc(method=Write,
> tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35,
> DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master,
> [0ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from
> server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms]
> delaying RPC due to Service unavailable: Service unavailable: Soft memory
> limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub
> rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [97ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms]
> querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [160ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms]
> querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [262ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms]
> querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [516ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms]
> querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [758ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms]
> querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [916ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms]
> querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of
> capacity), [1103ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms]
> querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of
> capacity), [4524ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms]
> querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of
> capacity), [7061ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms]
> querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of
> capacity), [9860ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received
> from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable:
> Service unavailable: Soft memory limit exceeded (at 95.63% of capacity),
> [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft
> memory limit exceeded (at 95.63% of capacity), [16920ms] querying master,
> [16920ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of
> capacity), [16957ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms]
> querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of
> capacity), [19002ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms]
> querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of
> capacity), [22476ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms]
> querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of
> capacity), [24562ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms]
> querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of
> capacity), [25117ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms]
> querying master, [28740ms] trace too long, truncated)}Timed out: can not
> complete before timeout: Batch{operations=516,
> tablet="c31055f8d3f74ee3b52687e7aff5b68a" [0x0000000300000003, <end>),
> ignoreAllDuplicateRows=false, rpc=KuduRpc(method=Write,
> tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35,
> DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master,
> [0ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from
> server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms]
> delaying RPC due to Service unavailable: Service unavailable: Soft memory
> limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub
> rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [97ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms]
> querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [160ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms]
> querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [262ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms]
> querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [516ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms]
> querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [758ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms]
> querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [916ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms]
> querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of
> capacity), [1103ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms]
> querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of
> capacity), [4524ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms]
> querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of
> capacity), [7061ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms]
> querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of
> capacity), [9860ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received
> from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable:
> Service unavailable: Soft memory limit exceeded (at 95.63% of capacity),
> [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft
> memory limit exceeded (at 95.63% of capacity), [16920ms] querying master,
> [16920ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of
> capacity), [16957ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms]
> querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of
> capacity), [19002ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms]
> querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of
> capacity), [22476ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms]
> querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of
> capacity), [24562ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms]
> querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of
> capacity), [25117ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms]
> querying master, [28740ms] trace too long, truncated)}Timed out: can not
> complete before timeout: Batch{operations=516,
> tablet="c31055f8d3f74ee3b52687e7aff5b68a" [0x0000000300000003, <end>),
> ignoreAllDuplicateRows=false, rpc=KuduRpc(method=Write,
> tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35,
> DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master,
> [0ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from
> server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms]
> delaying RPC due to Service unavailable: Service unavailable: Soft memory
> limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub
> rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [97ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms]
> querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [160ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms]
> querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [262ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms]
> querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [516ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms]
> querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [758ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms]
> querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [916ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms]
> querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of
> capacity), [1103ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms]
> querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of
> capacity), [4524ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms]
> querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of
> capacity), [7061ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms]
> querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of
> capacity), [9860ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received
> from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable:
> Service unavailable: Soft memory limit exceeded (at 95.63% of capacity),
> [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft
> memory limit exceeded (at 95.63% of capacity), [16920ms] querying master,
> [16920ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of
> capacity), [16957ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms]
> querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of
> capacity), [19002ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms]
> querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of
> capacity), [22476ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms]
> querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of
> capacity), [24562ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms]
> querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of
> capacity), [25117ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms]
> querying master, [28740ms] trace too long, truncated)}Timed out: can not
> complete before timeout: Batch{operations=516,
> tablet="c31055f8d3f74ee3b52687e7aff5b68a" [0x0000000300000003, <end>),
> ignoreAllDuplicateRows=false, rpc=KuduRpc(method=Write,
> tablet=c31055f8d3f74ee3b52687e7aff5b68a, attempt=35,
> DeadlineTracker(timeout=30000, elapsed=28820), Traces: [0ms] querying master,
> [0ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [0ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK, [0ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [44ms] received from
> server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [44ms]
> delaying RPC due to Service unavailable: Service unavailable: Soft memory
> limit exceeded (at 95.22% of capacity), [59ms] querying master, [59ms] Sub
> rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [60ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [60ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [97ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [97ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [119ms]
> querying master, [119ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [120ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [120ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [160ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [160ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [218ms]
> querying master, [218ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [218ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [219ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [262ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [262ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [479ms]
> querying master, [479ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [516ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [516ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [719ms]
> querying master, [719ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [720ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [720ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [758ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [758ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [879ms]
> querying master, [879ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [879ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [879ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [916ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.22% of
> capacity), [916ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.22% of capacity), [1060ms]
> querying master, [1060ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [1060ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [1060ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [1103ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.23% of
> capacity), [1103ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.23% of capacity), [4480ms]
> querying master, [4480ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [4480ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [4480ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [4524ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.34% of
> capacity), [4524ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.34% of capacity), [7020ms]
> querying master, [7020ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [7020ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [7020ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [7061ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.42% of
> capacity), [7061ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.42% of capacity), [9820ms]
> querying master, [9820ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [9821ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [9821ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [9860ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.51% of
> capacity), [9860ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.51% of capacity), [13100ms]
> sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [13138ms] received
> from server 09d6bf7a02124145b43f43cb7a667b3d response Service unavailable:
> Service unavailable: Soft memory limit exceeded (at 95.63% of capacity),
> [13138ms] delaying RPC due to Service unavailable: Service unavailable: Soft
> memory limit exceeded (at 95.63% of capacity), [16920ms] querying master,
> [16920ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [16920ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [16920ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [16957ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.71% of
> capacity), [16957ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.71% of capacity), [18959ms]
> querying master, [18959ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [18960ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [18960ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [19002ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.75% of
> capacity), [19002ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.75% of capacity), [22439ms]
> querying master, [22439ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [22440ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [22440ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [22476ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.82% of
> capacity), [22476ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.82% of capacity), [24520ms]
> querying master, [24520ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [24520ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [24520ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [24562ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.87% of
> capacity), [24562ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.87% of capacity), [25079ms]
> querying master, [25079ms] Sub rpc: GetTableLocations sending RPC to server
> master-vc1320.halxg.cloudera.com:7051, [25080ms] Sub rpc: GetTableLocations
> received from server master-vc1320.halxg.cloudera.com:7051 response OK,
> [25080ms] sending RPC to server 09d6bf7a02124145b43f43cb7a667b3d, [25117ms]
> received from server 09d6bf7a02124145b43f43cb7a667b3d response Service
> unavailable: Service unavailable: Soft memory limit exceeded (at 95.88% of
> capacity), [25117ms] delaying RPC due to Service unavailable: Service
> unavailable: Soft memory limit exceeded (at 95.88% of capacity), [28739ms]
> querying master, [28740ms] trace too long, truncated)}
> at
> org.apache.kudu.spark.kudu.KuduContext$$anonfun$writeRows$1.apply(KuduContext.scala:311)
> at
> org.apache.kudu.spark.kudu.KuduContext$$anonfun$writeRows$1.apply(KuduContext.scala:299)
> at
> org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)
> at
> org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)
> at
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
> at
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:109)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
