[
https://issues.apache.org/jira/browse/HADOOP-14770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yonger resolved HADOOP-14770.
-----------------------------
Resolution: Duplicate
Thanks Steve. when we apply the read input policy with random in our workload
after upgrade to Hadoop 2.8.1, it works as my expect, connections not destroyed
every time but reused.
> S3A http connection in s3a driver not reuse in Spark application
> ----------------------------------------------------------------
>
> Key: HADOOP-14770
> URL: https://issues.apache.org/jira/browse/HADOOP-14770
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 2.7.3
> Reporter: Yonger
> Assignee: Yonger
> Priority: Minor
>
> I print out connection stats every 2 s when running Spark application against
> s3-compatible storage:
> {code}
> ESTAB 0 0 ::ffff:10.0.2.36:44446
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44454
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44374
> ::ffff:10.0.2.254:80
> ESTAB 159724 0 ::ffff:10.0.2.36:44436
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44448
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44338
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44438
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44414
> ::ffff:10.0.2.254:80
> ESTAB 0 480 ::ffff:10.0.2.36:44450
> ::ffff:10.0.2.254:80 timer:(on,170ms,0)
> ESTAB 0 0 ::ffff:10.0.2.36:44442
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44390
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44326
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44452
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44394
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44444
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44456
> ::ffff:10.0.2.254:80
> ======================
> ESTAB 0 0 ::ffff:10.0.2.36:44508
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44476
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44524
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44374
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44500
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44504
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44512
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44506
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44464
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44518
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44510
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44442
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44526
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44472
> ::ffff:10.0.2.254:80
> ESTAB 0 0 ::ffff:10.0.2.36:44466
> ::ffff:10.0.2.254:80
> {code}
> the connection in the above of "=" and below were changed all the time. But
> this haven't seen in MR application.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
