On Dec 14, 2008, at 4:47 PM, Ricky Ho wrote:
Yes, I am referring to the "key" INPUT INTO the map() function and
the "key" EMITTED FROM the reduce() function. Can someone explain
why do we need a "key" in these cases and what is the proper use of
it ?
It was a design choice and could have been done as:
R1 -> map -> K,V -> reduce -> R2
instead of
K1,V1 -> map -> K2,V2 -> reduce -> K3,V3
but since the input of the reduce is sorted on K2, the output of the
reduce is also typically sorted and therefore keyed. Since jobs are
often chained together, it makes sense to make the reduce input match
the map input. Of course everything you could do with the first option
is possible with the second using either K1 = R1 or V1 = R1. Having
the keys is often convenient...
Who determines what the "key" should be ? (by the corresponding
"InputFormat" implementation class) ?
The InputFormat makes the choice.
In this case, what is the key in the map() call ? (name of the
input file) ?
TextInputFormat uses the byte offset as the key and the line as the
value.
What if the reduce() function emits multiple <key, value> entries or
not emitting any entry at all ? Is this considered OK ?
Yes.
What if the reduce() function emits a <key, value> entry whose key
is not the same as the input key parameter to the reduce()
function ? Is this OK ?
Yes, although the reduce output is not re-sorted, so the results won't
be sorted unless K3 is a subset of K2.
If there is a two Map/Reduce cycle chained together. Is the "key"
input into the 2nd round map() function determined by the "key"
emitted from the 1st round reduce() function ?
Yes.
-- Owen
