<[EMAIL PROTECTED]> wrote in message news:[EMAIL PROTECTED] > (This Google article suggestion comes from the CleverCS site): > > http://labs.google.com/papers/mapreduce-osdi04.pdf > > "MapReduce is a programming model and an associated implementation for > processing and generating large data sets. Users specify a map function > that processes a key/value pair to generate a set of intermediate > key/value pairs, and a reduce function that merges all intermediate > values associated with the same intermediate key."
Summarizing groups (and sometimes simultaneously by subgroups) has been a standard operation for decades in both statistics packages and database report generators. Python has various versions of the map and reduce operations that they use. Its dicts can easily be used to group items with the same key. What is somewhat specific to Google is the need to generate *multiple* key-value pairs from each input document. What MapReduce does that is somewhat innovative is automatically parallelize the computation to run fault-tolerantly on a cluster of up to 1000s of machines with machine slowdowns and failures 'common'. And indeed, that is the reason to use the system with trivial map or reduce functions, as they sometimes do. What this does show is that Google could be regarded as a cluster supercomputing company, with Web search as the visible development application. Terry J. Reedy -- http://mail.python.org/mailman/listinfo/python-list
