MAP-REDUCE的程序和系统英文文献和中文翻译
System Services The system provides three categories of services to facilitate the running of a Map-Reduce program across a large number of machines, as shown in Figure 2. (i) Distributed file system: the input and output are stored in a distributed file system to be globally accessible. (ii) Automatically distributing tasks: the system instantiates. Map and Reduce tasks across a large number of machines, partitions and assigns the input to inpidual Map tasks, then coordinates the copying of intermediate results (stored on local disks) from the machines running Map tasks to machines running Reduce tasks. A typical optimization is to co-locate a Map task to a machine (or rack) that also holds a copy of the assigned input partition in the distributed file system. (iii) Fault tolerance: the system monitors the task executions and re-executes tasks if they fail (due to machine crashes etc.). Since the input to a Reduce task may come from any of the Map tasks, it is necessary to wait for all Map tasks to finish before launching Reduce tasks. Near the end of the Map or the Reduce phase, the system schedules backup executions of the remaining in-progress tasks to protect against “stragglers”, which are machines taking anunusually long time to complete tasks due to permanent or transient hardware or software problems.
Advanced Features Figure 2 shows the many advanced components surrounding and supporting the Mapper and the Reducer computations. Typically,the default implementations of these components can be replaced with customized implementations. InputFormat extracts the input record (in_key, in_value) from the input (default: extracting a line from a text file). This gives part of the functionality of a database access method. Partitioner computes a Reduce task ID from mid_key, determining which local file to append an intermediate result (default: hash code(mid_key) modulo R). One may perform range based partitioning instead. The local files containing intermediate results are copied to the machine running the corresponding Reduce task. Then Sorter sorts the intermediate results to complete the group-by operation (default: merge sort). After the reduce computation, OutputFormat formats output results for writing to output files (default: out_key tab out_value per line).The network traffic of copying can be reduc by implementing Combiner, which is a partial reduce function, to be called on the intermediate results local on inpidual Map task machines. This is an optimization for reduce operations that are associative and commutative. For example, in the above word counting example, we can use the Reduce function as the Combiner.
下一篇：评价推荐系统英文文献和中文翻译