Swift: A language for distributed parallel scripting

Scientists, engineers, and statisticians must execute domain-specific application programs many times on large collections of file-based data. This activity requires complex orchestration and data management as data is passed to, from, and among application invocations. Distributed and parallel computing resources can accelerate such processing, but their use further increases programming complexity. The Swift parallel scripting language reduces these complexities by making file system structures accessible via language constructs and by allowing ordinary application programs to be composed into powerful parallel scripts that can efficiently utilize parallel and distributed resources. We present Swift’s implicitly parallel and deterministic programming model, which applies external applications to file collections using a functional style that abstracts and simplifies distributed parallel execution. ⺠The Swift parallel scripting language is described in detail. ⺠Swift provides an implicitly parallel functional data flow computation model that simplifies parallel scripting. ⺠Swift scripts describe parallel execution patterns for running independent serial or parallel programs. ⺠Swift scales from multicore workstations to petascale systems, using parallel and distributed resources, including clusters, grids, and clouds.