Introduction to Experimentation, Measurement, and Performance Evaluation in CS Research

What role do experimental approaches play in CS research?

An overview of Jain

  1. Select appropriate evaluation techniques, performance metrics, and workloads for a system.

  2. Conduct performance measurements correctly.

  3. Use proper statistical techniques to compare several alternatives.

  4. Design measurement and simulation experiments to provide the most information with the least effort.

  5. Perform simulations correctly.

  6. Use simple queueing models to analyze the performance of systems.

What is performance? What do we want to measure?

Common mistakes [Jain, 2.1]

A systematic approach [Jain, 2.2]

  1. State goals; define the system
  2. List services and outcomes. What does the system do, and what are the system's responses?
  3. Select metrics
  4. List parameters
  5. Select factors to study (a factor is a parameter that will be varied)
  6. Select evaluation technique, e.g., simulation, analysis, measurement
  7. Select workload
  8. Design experiments
  9. Analyze and interpret data
  10. Present results

CS 5014, C. J. Ribbens, 09/17/2001