Explicit Control in a Batch-Aware Distributed File System Export

@misc{citeulike:5014726, abstract = {We present the design, implementation, and evaluation of the Batch-Aware Distributed File System (BAD-FS), a system designed to orchestrate large, I/O-intensive batch workloads on remote computing clusters distributed across the wide area. BAD-FS consists of two novel components: a storage layer which exposes control of traditionally fixed policies such as caching, consistency, and replication; and a scheduler that exploits this control as needed for different users and workloads. By extracting these controls from the storage layer and placing them in an external scheduler, BAD-FS manages both storage and computation in a coordinated way while gracefully dealing with cache consistency, fault-tolerance, and space management issues in an application-specific manner. Using both microbenchmarks and real applications, we demonstrate the performance benefits of explicit control, delivering excellent end-to-end performance across the wide-area. }, author = {Bent, John and Thain, Douglas and Arpaci-Dusseau, Andrea C. and Arpaci-Dusseau, Remzi H. and Livny, Miron}, citeulike-article-id = {5014726}, citeulike-linkout-0 = {http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.127.3550}, keywords = {batch, distributed, filesystem, mosa, processing}, posted-at = {2009-11-07 00:57:27}, priority = {2}, title = {Explicit Control in a Batch-Aware Distributed File System}, url = {http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.127.3550} }

TY - ELEC ID - citeulike:5014726 L3 - citeulike-article-id:5014726 N2 - We present the design, implementation, and evaluation of the Batch-Aware Distributed File System (BAD-FS), a system designed to orchestrate large, I/O-intensive batch workloads on remote computing clusters distributed across the wide area. BAD-FS consists of two novel components: a storage layer which exposes control of traditionally fixed policies such as caching, consistency, and replication; and a scheduler that exploits this control as needed for different users and workloads. By extracting these controls from the storage layer and placing them in an external scheduler, BAD-FS manages both storage and computation in a coordinated way while gracefully dealing with cache consistency, fault-tolerance, and space management issues in an application-specific manner. Using both microbenchmarks and real applications, we demonstrate the performance benefits of explicit control, delivering excellent end-to-end performance across the wide-area. TI - Explicit Control in a Batch-Aware Distributed File System KW - batch KW - distributed KW - filesystem KW - mosa KW - processing AU - Bent, John AU - Thain, Douglas AU - Arpaci-Dusseau, Andrea AU - Arpaci-Dusseau, Remzi AU - Livny, Miron UR - http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.127.3550 ER -