Informatics 43 — Week 9 Lecture Notes

Covers Lecture 17 & Lecture 18: "What process do we follow to develop the software? (Parts 1 & 2)"

Table of Contents

1. Where We Are — From Testing to Process
2. Why We Need Processes — The Difficulties of Software Engineering
3. What Is a Model, and What Is It For?
4. The Software Life Cycle Models — The Catalog
5. Build-and-Fix
6. Waterfall
7. Rapid Prototyping
8. Incremental
9. Spiral
10. Rational Unified Process (RUP)
11. Extreme Programming (XP)
12. The Fourteen XP Practices
13. From XP to Agile — The Agile Manifesto
14. Agile Principles, Ziv's Law, and Scrum
15. Agile — Strengths, Weaknesses, and Adoption
16. The Big Picture — Comparing All Eight Models
17. Quiz 9 / Final Exam Study Guide
18. Wrap-Up and Memory Framework

1. Where We Are — From Testing to Process

Weeks 7 and 8 lived inside a single activity: quality assurance and testing. The previous lecture closed out white-box testing — using program graphs derived from source code to derive test cases, choosing those cases to hit different types of coverage (node, edge, loop, condition, path), and surveying the other QA approaches (inspections/reviews, formal methods, static analysis).

Week 9 zooms all the way out. We stop asking how do we do one activity well and start asking how the whole effort is organized:

The full arc of the course is now complete, and process models are the meta-layer that orchestrates everything beneath them:

Two lectures cover the territory. Lecture 17 (Part 1) motivates why we need processes (software engineering is intrinsically hard), defines what a process model is, and walks through the first five models — build-and-fix, waterfall, rapid prototyping, incremental, spiral — and introduces RUP. Lecture 18 (Part 2) drills into RUP, then Extreme Programming (XP) and Agile, and frames the whole unit around a historical shift: from rigid, heavyweight, document-driven processes to lightweight, agile ones.

2. Why We Need Processes — The Difficulties of Software Engineering

The starting premise is blunt: software engineering is hard, and we need processes to help. The lecture grounds this in the classic argument from Fred Brooks's essay "No Silver Bullet" — software has a set of essential qualities (essential difficulties) that no tool or trick can wish away.

On top of these four, the lecture adds a fifth motivator: software can have potentially drastic or disastrous consequences (the failure case studies from Week 7 — Boeing 737 Max, Therac-style hazards, Y2K — are the backdrop here).

Processes as a remedy. Because the difficulties are essential, the response is discipline, not a magic tool. Software is engineered via a defined process, which exists to:

• make sure we do the right things, do things right, and do not forget anything;
• and to be fitted to the situation — different processes suit different kinds of software, companies, and situations.

⚠️ Brooks's punchline is a likely quiz/exam point: a process is not a silver bullet [Brooks, "No Silver Bullet"]. Brooks argued that no single development — in technology or management — would by itself produce an order-of-magnitude improvement in software productivity, reliability, or simplicity within a decade. Processes help us manage the essential difficulties; they do not eliminate them.

3. What Is a Model, and What Is It For?

The lecture first disambiguates the word "model," because it carries two everyday meanings:

So a software process model is a simplified representation of how software development proceeds — it abstracts away detail to capture the high-level shape of the work.

What is a process model for? It can play two roles, and the same model can do both:

The prescriptive/descriptive distinction is worth keeping straight: a process model on a whiteboard that dictates "requirements before design before code" is being used prescriptively; a researcher who watches how a real team works and draws the resulting flow is using a model descriptively. Most real models are used both ways.

4. The Software Life Cycle Models — The Catalog

The course covers eight life cycle models. Listed roughly in historical order, they also trace a broad arc from heavyweight/rigid to lightweight/agile:

The lecture's organizing storyline (made explicit in the Lecture 18 summary) is a shift over time: from rigid, heavyweight, document-driven processes (waterfall, RUP) toward lightweight, agile ones (XP, Scrum, Agile). Build-and-fix sits before the arc even starts — it is the absence of process that everything else is reacting against.

5. Build-and-Fix

Build-and-fix is the degenerate case: there is no requirements phase, no design phase, no plan. You just write code and keep patching it.

The lifecycle:

The slide diagram draws the loop with two kinds of arrows: solid arrows for development and dotted arrows for maintenance (the "modify until client is satisfied" box loops back on itself, and maintenance arrows feed from operations back into modification).

Build-and-fix is the baseline against which every other model is an improvement. Its single virtue — that you start coding immediately — is also why it collapses on anything large: with no requirements and no design, complexity, changeability, and the need to coordinate many developers have nowhere to be managed.

6. Waterfall

The waterfall model imposes a strict, ordered sequence of phases. Each phase is fully completed (and documented) before the next begins, like water flowing down a series of steps.

The phases:

In the slide diagram, solid (development) arrows step strictly downward through the phases, while dotted (maintenance) arrows run back upward — and a "Changed requirements" box feeds back into the phases to represent maintenance after the system is operational.

⚠️ Memorable quote (likely exam material) — Clay Shirky, on the Healthcare.gov rollout (2013): "The waterfall model amounts to a pledge by all parties not to learn anything while doing the actual work." The criticism is that waterfall freezes all the learning into the requirements phase up front, and forbids the team from adapting once they discover — as they always do — that the plan was wrong.

7. Rapid Prototyping

Rapid prototyping addresses waterfall's "limited user input" weakness. Instead of locking down requirements on paper first, you build and discard a simple prototype — a quick, throwaway version whose only job is to elicit and validate what the users actually want. Once it has done that job, you throw it away and develop the real system.

The lifecycle:

The "build and discard simple prototype" box replaces the requirements phase at the head of an otherwise waterfall-like flow; the rest of the diagram uses the same development/maintenance arrows and "Changed requirements" feedback.

The two quotes on the slide capture why prototyping works:

"People don't know what they want until you show it to them." — Steve Jobs

"People don't know what they want until they see what they don't want." — Humphrey's Law

The key word is discard. The prototype is a learning device, not the first version of the product. Building it cheaply and being willing to throw it away is what lets the team surface hidden or misunderstood requirements before committing to a full, expensive build.

8. Incremental

The incremental model delivers the system in pieces (increments) rather than all at once. You do the requirements and high-level architecture once, then repeatedly build, test, and ship one increment at a time.

The lifecycle:

The central "FOR EACH INCREMENT" box reads: Perform detailed design, implementation, and integration. Test. Deliver to client. A second slide visualizes this as a descending staircase of analysis → design → implementation → testing → deployment → maintenance, each staircase producing Release 1, Release 2, Release 3, … Release n.

Case study — Microsoft evolves its process:

Incremental is the first model where the client sees working software early and often. That early feedback is its big advantage — and it is the same instinct that XP and Agile push to its limit a few models later.

9. Spiral

The spiral model (Boehm) is risk-driven. Development proceeds in loops around a spiral; at the start of each loop you do risk analysis, and the steps taken in that loop are variable, depending on what the risk analysis found (a loop might be a rapid prototype, or a specification, or a design, or an implementation — then a "verify" step). The riskiest, most uncertain parts are tackled first.

The slide diagram shows a spiral of expanding loops, each beginning with Risk analysis and ending with Verify, with the inner activities labeled Rapid prototype? / Specification? / Design? / Implementation? — the question marks emphasizing that which activity happens is decided by the risk analysis, not fixed in advance.

Boehm's Top 10 software risks (the six shown), with management techniques:

(Adapted from Hans Schaefer's "Software Risk Management: A Calculated Gamble.")

The spiral's defining move is front-loading risk. Waterfall does the easy, well-understood requirements work first and discovers the hard problems late; the spiral deliberately attacks the scary, uncertain parts first so a doomed project fails early rather than after years of investment.

10. Rational Unified Process (RUP)

RUP is a heavyweight, highly-structured iterative process defined by three characteristics:

• Use case driven — development is organized around use cases (what users want to accomplish).
• Architecture centric — the system architecture is a central, first-class artifact.
• Iterative and incremental — built in repeated iterations, each adding increments of capability.

The RUP model is a 2-D structure (the famous "humps" diagram):

Two captions on the diagram state the core idea:

• "Workflows group activities logically" — each workflow is a coherent kind of work (e.g., everything to do with requirements).
• "In an iteration, you walk through all workflows" — within a single iteration you touch every workflow, but the amount of effort in each shifts across the phases (the colored "humps" are tall for Requirements early on, tall for Implementation in the middle, tall for Deployment near the end).

The lecture also shows three detailed RUP workflows as examples of how much guidance RUP provides: Architectural Analysis, Analysis & Design, and Use Case Analysis (each is its own elaborate diagram of roles, activities, and artifacts).

RUP is the high-water mark of the heavyweight tradition: enormous, prescriptive, tool-supported, and thorough. Its strengths (guidance, tooling) and its weaknesses (heavyweight, complicated) are two sides of the same coin — and the heaviness of processes like RUP is exactly what XP and Agile were created to revolt against.

11. Extreme Programming (XP)

XP (Kent Beck) is a deliberately "extreme" reaction against waterfall and other heavyweight/traditional processes. The lecture lists the problems with those traditional processes that XP is reacting to:

• Lengthy development times
• Inability to cope with changing requirements
• Assumption that requirements are completely understood before the project begins
• Too much reliance on heroic development effort
• Complex methodology
• Waste / duplication of effort

(The slide literally crosses these out with a giant red X to signal: XP rejects all of this.)

XP process steps:

1. Determine the desired features (as user stories).
2. Estimate, prioritize, refine those stories.
3. Implement / deliver each task — typically using test-driven development and pair programming.
4. Follow the XP values and principles throughout (see §12).

The spirit of XP: take the things that traditional processes treat as occasional or end-of-project activities — testing, integration, design review, customer feedback — and do them all the time, in tiny increments. "Extreme" means dialing every good practice up to maximum frequency.

12. The Fourteen XP Practices

XP is defined by a set of mutually reinforcing practices and principles:

The TDD cycle (red-green-refactor):

XP's nested feedback loops run at every timescale at once:

For the exam you do not need to memorize all fourteen. Know that XP is a bundle of lightweight practices, and be able to name the headline ones — pair programming, test-driven development, small releases, continuous integration, simple design — and explain how they reinforce the XP goal of frequent, low-risk, feedback-rich delivery.

13. From XP to Agile — The Agile Manifesto

Agile is the broader philosophy that XP helped launch. It includes and evolved from Extreme Programming, Scrum, Crystal, Kanban, and others.

Its founding document is The Agile Manifesto (2001), framed as "We have come to value…" a set of four trade-offs:

⚠️ The four value statements are prime exam material. The crucial nuance: the manifesto says the items on the left are valued more, but "while there is value in the items on the right." Agile does not say documentation or plans are worthless — it says that when they conflict with working software and responsiveness, the left wins.

14. Agile Principles, Ziv's Law, and Scrum

Some Agile principles (the lecture's selection):

• Welcome change in requirements — even late in development.
• Business people and developers work together daily throughout the project.
• Build projects around motivated individuals — trust them to get the job done.
• Face-to-face conversation is the best way to convey information.
• Documentation is just a means to an end — not the goal.
• Continuous delivery of working software.

Ziv's Law — the theoretical justification for the agile stance:

Ziv's Law says uncertainty is inherent and inevitable in software engineering processes and products. This is the direct rebuttal to waterfall's core assumption (that requirements can be completely understood up front): if uncertainty can never be eliminated, then a process that embraces change will beat one that tries to freeze the plan.

Scrum — the most popular agile framework. The lecture introduces it via a short "Scrum in 10 minutes" video rather than deep slides, but the vocabulary it covers is worth knowing:

Note for the exam: Scrum got a video, not a detailed lecture — so the expectation is recognition of the vocabulary and that Scrum is an agile framework, not a deep walkthrough of the ceremonies.

15. Agile — Strengths, Weaknesses, and Adoption

Industry adoption (two surveys shown side by side):

Stack Overflow Developer Survey 2018 — "Which methodologies do developers use?" (select all that apply, 58,981 responses):

Kassab et al., "Software Testing: The State of the Practice," IEEE Software, 2017 — the single methodology that best described each project:

The data tells the whole story of the unit: agile and Scrum dominate practice, while the heavyweight, document-driven processes (waterfall, RUP/Unified Process) have shrunk to a small minority. This is the empirical evidence behind the "shift from heavyweight to lightweight" arc.

16. The Big Picture — Comparing All Eight Models

17. Quiz 9 / Final Exam Study Guide

This was the last content unit of the course — the next session is the course wrap-up, final exam review, and an ungraded mini practice quiz. Everything below is fair game.

17.1 Vocabulary you must be able to define

17.2 The four essential difficulties (Brooks)

Complexity, Conformity, Changeability, Invisibility — why software engineering is intrinsically hard. Plus: processes are not a silver bullet (Brooks).

17.3 The eight models, in order

Build-and-fix → Waterfall → Rapid prototyping → Incremental → Spiral → RUP → XP → Agile. Be able to give each one's core idea, one strength, and one weakness, and to match a model to a scenario (e.g., uncertain complex requirements → spiral or prototyping; stable well-understood requirements → waterfall; small team, changing requirements → agile/XP; safety-critical, large team → not plain agile).

17.4 The quotes (memorize the wording)

17.5 RUP specifics

Three characteristics (use-case driven, architecture-centric, iterative and incremental); the four phases (Inception, Elaboration, Construction, Transition); the idea that in one iteration you walk through all workflows; strengths (guidance, tooling, risk-driven) vs. weaknesses (heavyweight, complicated).

17.6 XP and Agile specifics

XP is a reaction to heavyweight processes; know the headline practices (pair programming, TDD, small releases, continuous integration, simple design) and the TDD red-green-refactor cycle. Know the four Agile Manifesto values (and the nuance that the right-hand items still have value), a few agile principles, and that Scrum is the most popular agile framework.

17.7 The big arc + the data

Be able to state the shift from heavyweight/document-driven to lightweight/agile, and cite the survey evidence (Agile ~85%, Scrum the most common single methodology; waterfall/RUP now a minority).

18. Wrap-Up and Memory Framework

Week 9 knowledge structure
|
+-- The question this week
|   +-- "What process do we follow to develop the software?"
|   +-- A software life cycle model = a HIGH-LEVEL process
|       (model = simplified representation, not "ideal")
|   +-- Used PRESCRIPTIVELY (enforce rules) or DESCRIPTIVELY (characterize)
|
+-- Why we need processes (Lecture 17)
|   +-- SE is hard -- Brooks's essential difficulties:
|   |     Complexity, Conformity, Changeability, Invisibility
|   |     (+ potentially disastrous consequences)
|   +-- Process = do the right things / do them right / forget nothing
|   +-- NOT a silver bullet  [Brooks, "No Silver Bullet"]
|
+-- The five "classic" models (Lecture 17)
|   +-- Build-and-fix : code -> patch; tiny programs only
|   +-- Waterfall     : Req->Design->Impl->Integration->Test->Ops->Retire
|   |                   rigorous + documented; rigid; bad w/ ambiguity
|   |                   (Healthcare.gov; Shirky quote)
|   +-- Rapid prototyping : build & DISCARD prototype to learn reqs
|   |                       (Jobs; Humphrey's Law)
|   +-- Incremental   : deliver in pieces (Release 1..n)
|   |                   (MS: Win8 "Ta-da" -> Win10 iterative + A/B)
|   +-- Spiral        : RISK-DRIVEN; riskiest parts first; variable steps
|                       (Boehm's top-10 risks); end may not be known
|
+-- Heavyweight peak (Lecture 17->18)
|   +-- RUP : use-case driven + architecture-centric + iterative/incremental
|             4 phases: Inception, Elaboration, Construction, Transition
|             workflows x iterations ("humps")
|             strong: guidance/tooling | weak: HEAVYWEIGHT, complicated
|
+-- The lightweight revolt (Lecture 18)
|   +-- XP : extreme reaction to heavyweight processes
|   |        14 practices (pair programming, TDD, small releases,
|   |        continuous integration, simple design, ...)
|   |        steps: stories -> estimate/prioritize -> deliver -> values
|   +-- Agile : umbrella (XP, Scrum, Crystal, Kanban, ...)
|   |   +-- Manifesto (2001): individuals>process, working sw>docs,
|   |   |   collaboration>contracts, responding>plan (left valued MORE)
|   |   +-- Ziv's Law: uncertainty is inherent & inevitable
|   |   +-- Scrum: sprints, backlogs, standups, burndown, retrospectives
|   +-- Strengths: satisfaction, adaptable, good small teams
|       Weaknesses: few docs, unstable reqs, tech debt;
|       not for large / safety-critical / no-customer projects
|
+-- The big arc + data
    +-- SHIFT: rigid/heavyweight/document-driven -> lightweight/agile
    +-- Surveys: Agile ~85% (SO 2018); Scrum #1 single method (Kassab 2017);
        waterfall + RUP now a minority

One-sentence summary

Because software engineering is intrinsically hard (Brooks's complexity, conformity, changeability, invisibility) and no process is a silver bullet, teams adopt a software life cycle model — a high-level process used prescriptively or descriptively — and these have evolved along a clear arc from rigid, heavyweight, document-driven approaches (build-and-fix → waterfall → rapid prototyping → incremental → spiral → RUP) toward lightweight, feedback-driven, change-embracing ones (XP → Agile/Scrum), a shift now confirmed in practice by surveys where agile and Scrum dominate and waterfall and RUP have become the minority.