Chapter 1: Motivation and History

I. Introduction, motivation, terminology

• What is parallel computation?
  • An informal definition: multiple CPUs cooperating concurrently to solve one problem.
  • So we are not interested in fine-grained, instruction-level parallelism here.
  • And we are focusing on systems that solve a single problem.
  • Parallel computation assumes an underlying parallel machine
    • a cluster of workstations
    • a Grid (dynamic collection of distributed resources)
    • an SMP server
    • a desktop box or an embedded system with a multi-core processor

  • Are these examples of parallel computation?

    • SETI@home (and numerous other examples of screensaver science). Yes, although these are embarrassingly parallel
    • A large electronic banking/ATM system. No. This is a classic distributed computing application.
    • A large web server, serving up multiple pages at the same time. No. These are separate tasks, although certainly some coordination is needed between them.
    • Molecular dynamics, climate simulations, etc. Yes! These are examples of probably the most important set of motivating applications, namely those from computational science and engineering
    • Parallel merge sort. Sure. There are parallel versions of many classic algorithms. However, we will see that the opportunities for parallelism in many classic algorithms (e.g., sorting, searching, graph theory) are somewhat limited.
    • A single app on a single workstation. No. But if you look closely enough you can always find concurrency under the hood of any modern microprocessor.
    • Class discussion: is the operating system of an SMP box a parallel program?

• Some historical perspectives:
  • Supercomputing meant large, expensive, and highly specialized, often application-specific machines.
  • Most SC companies went out of business/bankrupt in the 80s/90s. Why?
  • Major revolution: the use of commodity off-the-shelf components (COTS), especially microprocessors.
  • Major cost determinant: communication network architecture.
  • Open debates: What architecture? What kind of processors? What kind of networks? s, because of diminishing returns of Moore's law, ever-increasing need for performance, memory and bandwidth,
• State of the art:
  • Real supercomputers have thousands of processors (machines with up to 128K processors in operation, e.g. IBM BlueGene/L).
  • Processor architecture varies dramatically (high-end desktop processors, embedded low-power processors, vector processors, GPUs, symmetric/assymetric multicore/multithreaded processors, tak-level parallel game machines...)
  • Emphasis used to be on performance only, interest has shifted to other factors, including power, reliability and maintenance cost.
  • Parallel programming as a concept is now pervasive (sensor nets, desktop SMPs, multicore systems, embedded multiprocessors, clusters of all sorts and sizes, supercomputers).