Re: figuring out the right setting for dfs.datanode.max.xcievers

2011-03-30 Thread Edward Capriolo
On Wed, Mar 30, 2011 at 1:38 PM, Igor Tatarinov wrote: > I haven't found a good description on this setting and the costs in setting > it too high. Hope somebody can explain. > I have about a year's worth of data partitioned by date. Using 10 nodes and > setting xcievers to 5000, I can only save i

figuring out the right setting for dfs.datanode.max.xcievers

2011-03-30 Thread Igor Tatarinov
I haven't found a good description on this setting and the costs in setting it too high. Hope somebody can explain. I have about a year's worth of data partitioned by date. Using 10 nodes and setting xcievers to 5000, I can only save into 100 or so partitions. As a result, I have to do 4 rounds of