Are you using hive 0.6? ... may be fixed in the latest version. Also I wonder why these thrift libraries are being used ... is this normal hive operation, or can you do something to avoid using thrift?
-Ajo On Fri, Feb 11, 2011 at 12:05 AM, vaibhav negi <sssssssena...@gmail.com>wrote: > > Hi all, > > I am loading data into hive tables by connecting to hiveserver through > thrift api using "load data local inpath ... " query . > Hive server is running as a background process for days . After some 4-5 > days hiveserver throws this error : > > 11/02/11 11:20:02 WARN server.TThreadPoolServer: Transport error occurred > during acceptance of message. > org.apache.thrift.transport.TTransportException: java.net.SocketException: > Too many open files > at > org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:124) > at > org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35) > at > org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31) > at > org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:184) > at > org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:388) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:156) > Caused by: java.net.SocketException: Too many open files > at java.net.PlainSocketImpl.socketAccept(Native Method) > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) > at java.net.ServerSocket.implAccept(ServerSocket.java:453) > at java.net.ServerSocket.accept(ServerSocket.java:421) > at > org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:119) > ... 9 more > > I checked network connections using netstat and it show about 1000 active > network connections, most of them to hadoop datanode . Output is attached as > netstat.out > > after i restarted hiveserver network connections falls to about 20 . output > is attached as netstat_after_HS_restart.out > > Is this a problem with hive or hadoop configuration ? > > Vaibhav Negi >
