While performance for a simple program can be measured as execution speed (i.e., time command), it’s insufficient for complex systems like servers

• Other factors like how long they run and how they handle multiple requests become critical

Latency vs Throughput

• Latency (or response time): is the time taken to complete a single task or respond to a single request
  • Measure of “how fast” an individual operation is
  • How many milliseconds it takes for a web page to start loading after you click a link
• Throughput: total number of tasks or requests completed in a given unit of time
  • Measure of how much work gets done overall
  • How many web pages a server can deliver per second

The Relationship

• Positive Correlation: in a simple, sequential system, improving one improves the other

  • Faster CPU will reduce the latency of each request, which in turn allows the server to handle more requests per second (higher throughput)

    

• Negative Correlation: when parallelism is introduced (i.e., processing multiple requests concurrently), a trade-off often appears

  • Throughput typically improves because the system is doing more work simultaneously

  • Latency for each individual request might increase (get worse) due to the overhead of managing parallel tasks (i.e., context switching, contention for shared resources like CPU or memory)

    

Airplane Analogy

Concorde vs Boeing 747

• Latency: Concorde is faster
• Throughput: 747 carries more passengers per hour

Limits of Parallelism

Adding more parallel threads does not infinitely improve throughput

• Initially, throughput increases close to linearly, but then the gains slow down due to overhead
• Eventually, adding more threads can actually degrade overall performance because the system spends more time managing threads than doing useful work