You could try to save the information in a tab separated file and try to load that into hive.
-Ajo On Tue, Feb 15, 2011 at 1:24 AM, vaibhav negi <sssssssena...@gmail.com>wrote: > Hi , > > I am using hive version 0.7 ( as suggested by lib/hive-hwi-0.7.0.war) . I > checked out code from svn two weeks back . I use thrift api's to load data > into hive programmatically . > > Vaibhav Negi > > > > On Sat, Feb 12, 2011 at 6:06 AM, Viral Bajaria <viral.baja...@gmail.com>wrote: > >> We use hive through the thrift api too. I have seen the above error when >> someone accidentally connects to the thrift server port through multiple >> instances of our hive-wrapper code. We open a bunch of hive thrift ports to >> achieve the automation of running multiple hive queries in parallel i.e. >> continuous aggregations. >> >> you need to make sure you are not facing a similar issue because the hive >> thrift server as far as i know is single threaded up until hive 0.5.0 (the >> version that i use). Not too sure if that's been changed in 0.6.0 or higher. >> >> -Viral >> >> On Fri, Feb 11, 2011 at 7:06 AM, Ajo Fod <ajo....@gmail.com> wrote: >> >>> Are you using hive 0.6? ... may be fixed in the latest version. >>> >>> Also I wonder why these thrift libraries are being used ... is this >>> normal hive operation, or can you do something to avoid using thrift? >>> >>> -Ajo >>> >>> >>> On Fri, Feb 11, 2011 at 12:05 AM, vaibhav negi >>> <sssssssena...@gmail.com>wrote: >>> >>>> >>>> Hi all, >>>> >>>> I am loading data into hive tables by connecting to hiveserver through >>>> thrift api using "load data local inpath ... " query . >>>> Hive server is running as a background process for days . After some 4-5 >>>> days hiveserver throws this error : >>>> >>>> 11/02/11 11:20:02 WARN server.TThreadPoolServer: Transport error >>>> occurred during acceptance of message. >>>> org.apache.thrift.transport.TTransportException: >>>> java.net.SocketException: Too many open files >>>> at >>>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:124) >>>> at >>>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35) >>>> at >>>> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31) >>>> at >>>> org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:184) >>>> at >>>> org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:388) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156) >>>> Caused by: java.net.SocketException: Too many open files >>>> at java.net.PlainSocketImpl.socketAccept(Native Method) >>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) >>>> at java.net.ServerSocket.implAccept(ServerSocket.java:453) >>>> at java.net.ServerSocket.accept(ServerSocket.java:421) >>>> at >>>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:119) >>>> ... 9 more >>>> >>>> I checked network connections using netstat and it show about 1000 >>>> active network connections, most of them to hadoop datanode . Output is >>>> attached as netstat.out >>>> >>>> after i restarted hiveserver network connections falls to about 20 . >>>> output is attached as netstat_after_HS_restart.out >>>> >>>> Is this a problem with hive or hadoop configuration ? >>>> >>>> Vaibhav Negi >>>> >>> >>> >> >
