Hi all, I'm using Hive with the s3native FS. Today, I noticed that hive locks up after 4 queries that directly access S3 (select * from mytable limit 10). With debug logging on, I get the following output:
2011-07-06 15:54:31,459 DEBUG s3native.NativeS3FileSystem (NativeS3FileSystem.java:getFileStatus(393)) - getFileStatus retrieving metadata for key 'tmp/hive-mapred/hive_2011-07-06_15-54-29_881_4253697128840334916/-mr-10000' 2011-07-06 15:54:31,459 DEBUG httpclient.RestS3Service (RestS3Service.java:getObjectImpl(1511)) - Retrieving Head information for bucket XXXXXXXX and object tmp/hive-mapred/hive_2011-07-06_15-54-29_881_4253697128840334916/-mr-10000 2011-07-06 15:54:31,460 DEBUG service.Jets3tProperties (Jets3tProperties.java:getBoolProperty(314)) - s3service.disable-dns-buckets=false 2011-07-06 15:54:31,460 DEBUG httpclient.RestS3Service (RestS3Service.java:setupConnection(811)) - S3 URL: https://XXXXXXXX:443/tmp%2Fhive-mapred%2Fhive_2011-07-06_15-54-29_881_4253697128840334916%2F-mr-10000 2011-07-06 15:54:31,460 DEBUG httpclient.RestS3Service (RestS3Service.java:performRequest(334)) - Performing HEAD request for 'https://XXXXXXXXXXXXXX/tmp%2Fhive-mapred%2Fhive_2011-07-06_15-54-29_881_4253697128840334916%2F-mr-10000', expecting response code 200 2011-07-06 15:54:31,461 DEBUG httpclient.RestS3Service (RestS3Service.java:buildAuthorizationString(872)) - Adding authorization for AWS Access Key 'XXXXXXXXXXXXX'. 2011-07-06 15:54:31,461 DEBUG httpclient.RestS3Service (RestS3Service.java:buildAuthorizationString(922)) - Canonical string ('|' is a newline): HEAD|||Wed, 06 Jul 2011 15:54:31 GMT|/XXXXXXXXX/tmp%2Fhive-mapred%2Fhive_2011-07-06_15-54-29_881_4253697128840334916%2F-mr-10000 2011-07-06 15:54:31,461 DEBUG httpclient.HttpClient (HttpClient.java:executeMethod(322)) - enter HttpClient.executeMethod(HttpMethod) 2011-07-06 15:54:31,462 DEBUG httpclient.HttpClient (HttpClient.java:executeMethod(373)) - enter HttpClient.executeMethod(HostConfiguration,HttpMethod,HttpState) 2011-07-06 15:54:31,462 DEBUG httpclient.MultiThreadedHttpConnectionManager (MultiThreadedHttpConnectionManager.java:getConnectionWithTimeout(383)) - enter HttpConnectionManager.getConnectionWithTimeout(HostConfiguration, long) 2011-07-06 15:54:31,462 DEBUG httpclient.MultiThreadedHttpConnectionManager (MultiThreadedHttpConnectionManager.java:getConnectionWithTimeout(390)) - HttpConnectionManager.getConnection: config = HostConfiguration[host=https://XXXXXXXXX.s3.amazonaws.com], timeout = 0 2011-07-06 15:54:31,462 DEBUG httpclient.MultiThreadedHttpConnectionManager (MultiThreadedHttpConnectionManager.java:getHostPool(775)) - enter HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration) 2011-07-06 15:54:31,463 DEBUG httpclient.MultiThreadedHttpConnectionManager (MultiThreadedHttpConnectionManager.java:doGetConnection(494)) - Unable to get a connection, waiting..., hostConfig=HostConfiguration[host=https://XXXXXXXXXX.s3.amazonaws.com] Does anyone know if I can do anything to prevent this? It looks like connections are not returned correctly to the pool.. // Wouter
