Scientific Method

Imagine for a moment that you’re an auto mechanic with a knack for attracting eccentric customers with unusual problems. Which of these scenarios would you rather deal with? We’ll assume that you’re paid the same amount in either scenario and you’re about to go on vacation, so you’re not really interested in a challenge, you just want to get the car fixed as efficiently as possible.

• Scenario 1: Alice tows a car in to your shop. She’s been tinkering with it for a month, but she doesn’t really know what she’s doing and she can’t get it working. Your job is to figure out why it won’t start and get it back in working condition.
• Scenario 2: Alice brings in a working car and has you do an oil change and general inspection. You pronounce it in good condition, print up an invoice, and wish her a good day. She then drives the car across the street and tinkers with it for fifteen minutes. Sheepishly, she comes back into your shop – it won’t start anymore. Your job is to figure out why and get it back in working condition so she can go home.

I know what I prefer – scenario 2, by a long shot. Why? Simply put, the fearless but incompetent Alice has been able to make far fewer changes to the car. There are only so many (non-destructive) ways you can get a car from a working state to a non-starting state in fifteen minutes (e.g., disconnect the starter motor, drain the gasoline, disable the battery). In scenario 1, she could have done all sorts of awful things to the engine over a whole month in an effort to fix it, creating new problems on top of the existing problems. In scenario 2, probably only one thing, maybe two or three, has gotten broken. Further, as the mechanic, I just looked the car over thoroughly thirty minutes ago, so I’ll be in an excellent position to spot what changed.

One Thing at a Time

This imaginary anecdote suggests a valuable rule in software development: change one thing at a time. This applies to debugging, but it also applies to preventing debugging. In other words, when initially writing software, good developers don’t sit down, carefully write 5,000 lines of code, then run them and see what happens. The inevitable result of that development pattern is days on end of infuriating debugging, as the programmer tries to untangle chains of complexity even worse than those in Alice’s first car. So many variables have changed that it’s impossible to know what factor (or, not unlikely in this case, set of factors) is causing each problem.

A better way involves splitting the work to be done into small changes. For each change, we identify what the change is supposed to do (i.e., create a hypothesis), then we try to write the code, then we experimentally test it to make sure it works. That involves not only making sure any new functionality works, but also checking to make sure any similar functionality that the change might have touched hasn’t stopped working! If everything looks right, we have reasonable certainty that this piece is good, and we can move on. If something goes wrong, our change is more like that made to Alice’s second car: the number of things that could have gone wrong is quite small, and we can even look to see exactly what changed (this is even easier than with a car, since we don’t have to rely on our memory: a version control system combined with a diff tool can identify exactly which characters across a large program changed since the last version).

Even if we’re debugging some code that’s been written all at once, or any code at all that we didn’t write recently and can’t easily split into individual changes to test, we can still benefit from doing one thing at a time. In trying to fix Alice’s first car, we wouldn’t go under the hood, quickly take a wrench to everything that doesn’t look quite right, and then try to start it. We’d begin by ensuring we really couldn’t start the car and it’s not just that Alice forgot how to turn the key or something (analogous to reproducing the software bug). Then we’d search methodically for issues and fix them one at a time, seeing if each fixed the problem.

Note: I am not a qualified mechanic, but I would guess that given the circumstances, most mechanics would want to inspect the car fully and repair everything that looked off before trying to start it, lest Alice have, e.g., drained the oil and not replaced it, or changed something that could cause an explosion on turning the key. Nevertheless, the overall process is the same, we just have to delay the full empirical testing until the end and use other kinds of observation to do the test at each stage (say, checking things against specs in the car’s maintenance manual or what we expect them to look like). Fortunately, few of us face real-life problems that pose such risks from a quick test!

In Real Life

Most of us don’t run our lives as a series of experiments, nor probably should we (at most, we might do some tightly controlled experiments if we’re having a complicated medical problem or troubleshooting a broken device). That said, we don’t have to go all the way every time. Just the simple act of doing things intentionally and incrementally can be extremely helpful.

If you’re trying to improve your cooking, don’t make new recipes or completely change old ones, change one or just a couple of things in your usual recipes and see what happens. If you change everything, you will paradoxically learn less – “I like all of the instructions taken together in X recipe better than those in Y recipe” is far less helpful than “I like these vegetables better if I cook them slower.”

Try to focus on one new skill or activity at a time; that way, you have a much clearer idea whether you actually like it and it actually improves your life.

The methodology I suggest in Just Getting Started is essentially the scientific method of working in action. You get started by changing one thing and experimenting with the results, then you use those results to inform your next changes.