Basically the question is:
Does Hive checks files in location before creating table?
Because if I move files away before creating tables it works and after
table is created I can move files back and all works :)
Margus (margusja) Roo
http://margus.roo.ee
skype: margusja
+372 51 48 780
On 26/02/16 16:40, Margus Roo wrote:
Hi
I try to create external table and in the location there are 8960
small files
And I am getting every time something like that:
GC pool 'PS MarkSweep' had collection(s): count=1 time=1672ms
GC pool 'PS Scavenge' had collection(s): count=1 time=45ms
2016-02-26 15:18:29,721 INFO
[org.apache.hadoop.util.JvmPauseMonitor$Monitor@3af36922]:
util.JvmPauseMonitor (JvmPauseMonitor.java:run(195)) - Detected pause
in JVM or host machine (eg GC): pause of approximately 1995ms
GC pool 'PS MarkSweep' had collection(s): count=1 time=1936ms
And all 16 cpu cores are in 100% and all 16G memory almost gone.
Helps only if I restart hive server.
I use Hive 1.2.1 from HDP-2.3 by packaged by Hortonworks
hive.tez.java.opts=-server -Djava.net.preferIPv4Stack=true
-XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB
-XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps
