Chaining the three phases gives a complete MapReduce job: reduce_phase(shuffle(map_phase(...))). One more optimization matters at scale, the combiner. It runs the reducer locally on each mapper's output before the shuffle, shrinking the data that has to move across the network. Because the word-count reducer is associative, combining first yields the exact same final answer with far less shuffle traffic.
Build the driver and combiner:
mapreduce(inputs, mapper, reducer): run all three phases and return the result dict.combine(pairs, reducer): group the pairs and pre-reduce each key, returning a list of (key, reduced) pairs ready to shuffle again.def map_phase(inputs, mapper):
pairs = []
for item in inputs:
pairs.extend(mapper(item))
return pairs
def shuffle(pairs):
groups = {}
for k, v in pairs:
groups.setdefault(k, []).append(v)
return groups
def reduce_phase(groups, reducer):
return {k: reducer(k, vs) for k, vs in groups.items()}
def wc_map(line):
return [(w, 1) for w in line.split()]
def wc_reduce(key, values):
return sum(values)
def mapreduce(inputs, mapper, reducer):
"""Run map -> shuffle -> reduce end to end."""
# TODO: return reduce_phase(shuffle(map_phase(inputs, mapper)), reducer)
pass
def combine(pairs, reducer):
"""Locally pre-reduce each key before the shuffle."""
# TODO:
# grouped = shuffle(pairs)
# return [(k, reducer(k, vs)) for k, vs in grouped.items()]
pass
The editor and the test runner need a wide screen. Open this page on a desktop browser and your progress will be waiting.
Chaining the three phases gives a complete MapReduce job: reduce_phase(shuffle(map_phase(...))). One more optimization matters at scale, the combiner. It runs the reducer locally on each mapper's output before the shuffle, shrinking the data that has to move across the network. Because the word-count reducer is associative, combining first yields the exact same final answer with far less shuffle traffic.
Build the driver and combiner:
mapreduce(inputs, mapper, reducer): run all three phases and return the result dict.combine(pairs, reducer): group the pairs and pre-reduce each key, returning a list of (key, reduced) pairs ready to shuffle again.Press Run Tests or ⌘↩ to check your solution.