On Waterfall

What is Waterfall?

A lot of software people sneer at Waterfall, like it's some barbaric practice. It's not - it represents the pinnacle of humanity's efforts to coordinate almost all sorts of group projects. It so happens that software is one of the exceptions, which is why we got the Agile Manifesto and a million books and blog posts like this one. The purpose of this essay is discuss the practical application of how and when to use one or the other.

Software Engineering's Original Sin

I maintain that the first big mistake of software enginering as an industry was calling ourselves "engineers". We assumed that the methods we use to build skyscrapers would work for building software programs, and that turned out to be wrong. Agile is the answer to, "Well what should we do instead?"

The first and most important line of the Agile Manifesto was: People over Process. In practical terms, that means that when there's a conflict between the people doing the work and the process by which they're doing the work, the process gets changed?. The first question to answer is, why don't we follow that advice for skyscrapers?

Imagine the construction site:

The crew shows up on the first day to dig the foundation. The plans ask for a hole 20 feet deep, but the foreman of the digging crew decides that's not nearly deep enough. "Why should I follow these plans", she asks, "when they were drawn up by some fool who's never dug a hole? They can't see what kind of soil we're working with. They don't know as much about holes as I do. I think I'll dig it 40 feet."

This - the idea that the team doing the work can decide how to do the work - is a recipe for disaster if you're building a bridge, raising a skyscraper, or trying to send a rocket to the moon. But in at least some cases, it turns out to be the right approach for software.

Why is Software Different?

This is a complicated answer - part of it has to do with the fact that software is very new and the tools change often and stuff like that - but I maintain that the main and most important difference is that software is, or at least can be, modular.

We build software in discrete pieces that are loosely coupled with each other, and we do that intentionally because it gives us flexibility in how each module is built. That's what allows us to use Agile in the first place, and why people building something modular complain when they aren't afforded that flexibility. If we could build skycrapers in modules, so that it was cheap and easy to swap out the cement in the foundation after the building was complete, we would, and then maybe we would let the crew that digs the hole decide how deep to dig.

This sounds obvious - loose coupling, modularity, hexagonal architecture, SOLID, the value of this is all well understood - but the critical thing to point out is, if we use Agile instead of waterfall because we can make software modular, it follows that we must use waterfall instead of Agile for the times when we can't make it modular, or for the software-adjacent tasks that can't be decomposed the way a well-architected application can.

(Important note: I'm not suggesting that the people designing skyscrapers don't know about or use modular design when possible; the point of this essay is to examine the times when software teams, who live and breathe Agile because almost all of their work is Features, need a more Waterfall approach and why it's not evil.)

How to tell the difference

By convention (at least I follow this convention, and I wish everyone else did), the things we need Waterfall for are called "Projects" and the things we can use Agile for are called "Features". Sometimes, the distinction is obvious. Adding "dark mode" to the app is a feature and we don't need a Project Manager for it. Moving the company from Pittsburgh to Portland is a Project, and can't be accomplished by asking each team to do their own bit in their own way.

Things that are in the middle, well, the approach ought to be somewhere in the middle too. In practice, most companies I've been involved with have only two processes, and everything that isn't one has to be the other, and have very little flexibility in dealing with something that's mostly feature work but requires a little bit of Project Management. I think the best solution to that is, when looking at a Project, ask: which parts of this are as loosely-coupled to the whole as the internal logic of a microservice? and use Agile for those.

tl;dr

Waterfall is predicated on the idea that someone needs to have thought through the details and coordinated them before the project starts, and for team A to deviate from the plan will necessarily cause problems for team B's portion of the work. Agile describes the opposite process, in which the high-level plan is intentionally vague and teams A and B are responsible for figuring out the details of their respective modules of the work. Agile is generally easier and better when it doesn't cause problems, so we use it wherever we can, and what we call "good architecture" in software is largely a matter of making things loosely-coupled enough to defer as much of the decision-making to the teams as possible.