The desire to store, retrieve, communicate, and process data sets of unprecedented size at ever greater rates and ever decreasing cost is the prime motivating force for the intense focus of manpower and money on the development of parallel computing. The intractable problems of computational science and engineering (e.g., weather prediction, circuit simulation, genetic transmission, thermonuclear device modeling) are among the leading motivating applications for high-end parallel computing and helped to launch the federal High Performance Computing and Communication (HPCC) initiative of the early 1990's and the Accelerated Strategic Computing Initiative (ASCI) and Scientific Simulation Program (SSP) of the late 1990's. However, it is by now widely appreciated that existing codes based on classical algorithms and "flat" data structures often run at only a few percent of peak performance when ported blindly to advanced parallel architectures.
This short course will look at parallel architecture from the point of view of an algorithms designer. We concentrate on parallel numerical (i.e., floating-point) algorithms, since they typically possess strong structure that can be exploited in the form of temporal and spatial locality. In addition to the usual study of the convergence and arithmetic complexity of algorithms, we will study their computation-to-communication ratios and data access patterns in time (e.g., communication granularity) and space (e.g., communication topology).
Topics touched upon in pursuit of the overall agenda include: parallelization metrics, performance metrics, scalability and the ``laws'' of Amdahl and Gustafson, models of parallel computation, parallelization paradigms, communication templates and libraries, memory hierarchy, cache coherence, strategies for latency tolerance, and data dependencies typically arising in high-performance applications. However, this course neither pretends to be a substitute for nor requires as a prerequisite a proper course in computer architecture.
Suggested prerequisites: