I'm beginning to suspect this myself. We have a import job which has
many smaller files. We've been merging them into a single log file and
partitioning by day however I've seen this and other errors (usually
memory related errors) posted by hive and the load fails.
Our latest error has been not having enough partitions pernode set in
hive (set to 1000 currently). When increasing this setting it gives the
same error however I've noticed by loading fewer logs I'm avoiding
dynamic partition errors (and thus the job failing).
I have to keep reminding myself NOT to think of hive/hadoop like a
database (though that is my background ::grinz::)
If you find the solution to this I'd be very interested. It's hit me
from time to time as well :-)
Thanks!
On 07/13/2011 04:36 AM, labtrax wrote:
It seems that the more dynamic partitions are imported the fewer I am able to
import respectively the smaller the files have to be.
Any clues?
[snip]
but some mapreduce jobs finshed sucessfully some not. They all do run with
these 3 up to about 9 errors on different datanodes.
I set the hive.exec.max.created.files to 1000000, but still most querys
end up with FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask
--
NEU: FreePhone - kostenlos mobil telefonieren!
Jetzt informieren: http://www.gmx.net/de/go/freephone
