I know I fixed this bug before. But how?

Sometimes we spend considerable time tracking down a bug, and eventually find a fix, only to meet the same problem a couple of months later, by which time we have forgotten the solution, and have to repeat the process.

Gandalf: I have a vague memory of this place

Here are some suggestions for how to remember (or not forget) the solutions to problems we have already met in the past.

Store in wetware (our brains)

For bugs that we meet quite often (e.g. “undefined is not a function”) then we might not need to take any extra steps. Sooner or later, we will learn to understand exactly what this means, and resolving the issue will become second nature.

If the error message is helpful (e.g. “This route only accepts POST requests”) then the solution is clear. We don’t need to take any special effort here.

If the pain is immediate, i.e. we make a mistake and are instantly punished for it, then we may learn permanently how to avoid it. “Pain is a good teacher.”

But the suggestions below are for those trickier issues which (for whatever reason) take some time to debug, and which occur so rarely that we don’t always remember the solutions.

Make the solution public

Let’s say you got the error “socket hang up” when using the web3 library, so you googled for a solution. The first result was a GitHub issue and the second result was a StackOverflow question. But neither page actually provided a solution to your problem.

Eventually, after more searching and testing, you did find a solution.

How to remember it? Write the solution on the GitHub issue or on the StackOverflow question.

The next time you (or someone on your team, or someone far away) meets the same problem, the first google search should yield the answer you need to find!

(True story: On multiple occasions I have googled and found exactly the answer I needed, and only then noticed that the post was written by me, some years before. That realisation inspired this suggestion.)

Remember in-house

Maybe you don’t want to store the solution publicly. Perhaps it’s a problem specific to your project, or the environment you are running in.

In such cases we have found it useful to store a TROUBLESHOOTING.md in our git repository.

For example, one of our documents notes the various troubles we encountered when we built our first Xcode project. We quote each error message, and then follow it with the solution.

> Value for SWIFT_VERSION cannot be empty
Solution: Go to `[Target] Build Settings > Swift Language Version` and set it to `Swift 4.2`
> Your development team, "______", does not support the Network Extensions capability
Solution: Use the company developer account, instead of your personal account.
> Unlock [device] to Continue.  Xcode cannot launch [project] on [device] because the device is locked.
Solution: Leave XCode running.  Unplug the device's USB cable and plug it back in.

In future, when one of our developers encounters an error message, they can do a quick search of our project, to see if the problem has already been documented.

In the above example, we actually put this Troubleshooting section at the bottom of our IOS_BUILD.md document, and we encouraged developers to add new entries for each problem they encountered. We now have 18 of them! And this is an invaluable resource, avoiding lots of googling and trial and error when someone is setting up a new build.

(Eventually, we tried to ensure that the build steps covered all potential issues ahead of time, but we still found the searchable error messages were valuable for devs who had missed a step, who had pre-existing setups, or who encountered issues during an upgrade.)

Keep private notes

Sometimes I don’t have time to write beautiful documentation for the team. But I still keep some notes for myself.

I have a NOTES.md file next to README.md but I don’t commit this file to git. Instead I just drop lots of notes and snippets in here.

Later, when I am confident that something is really useful, I will clean it up, and add it into the README.md or some other suitable place.

By using a private notes file, the information is not lost, but it’s also not making too much noise for other people.

This concludes the suggestions for how to remember/record solutions to tough issues. But here are some extra tips…

Tip: Track down what the actual solution was

You might have tried five different things while you explored to fix a problem.

When it eventually starts working again, you don’t know if it was only the last thing you tried which fixed it, or perhaps one or more of the other things were necessary too.

It might take some extra time to rewind and test each thing in isolation, or in different combinations. But I would argue that this is often a good investment of time.

By testing what happens if you try this, or that, or these two things together, you can gain a real understanding of the issue, and the system. By spending a bit more time you can really learn something, clear the ambiguity from your head, and generate a precise solution rather than a fuzzy workaround.

And during the process, you might end up writing a unit/system test that will help to catch or debug this issue if it reappears in the future. (If the test is too messy to run regularly, I might comment it out or hide it behind a switch, but at least the code is available if it’s ever needed again.)

It might seem like working out the details is something you can come back to later, but usually the best time to do a thorough investigation is when you have all the pieces in your mind already. Leaving it for a later date means you start off cold and it will take longer.

The best time to do a thorough investigation is when you have all the pieces in your mind already.

Of course deadlines might get in the way of this. So your mileage may vary. It can help to set a deadline: “I will explore this for 30 minutes, but after that I need to get back on track.”

Tip: Ensure your testing cycle is smooth

Having a fast and automated development-test cycle is important for such investigations.

For example, it’s good to have a test suite which runs as soon as you save the file, or a webpage which refreshes as soon as you switch window.

For the former, I often use nodemonto act after a change is detected:

"start:watch": "nodemon -w src -e '*' -d 2 -x 'yarn run start'",

If you are spending time to execute the compilation and perform the testing by hand, then the repetition might make you bored of the task, leading you to abandon it. Letting the machine do that will leave your mind free to focus on the higher level problem.

This is a general recommendation for any project, not just for solving one specific issue.

Tip: When things are fubar, throw an error with a sensible message

This is more related to prevention, or pre-emptively helping the developer to find a solution.

Let me demonstrate with an example. The getVehicle() function is supposed to return a vehicle when called. But what happens if you pass it invalid arguments?

// Bad code: If the type is invalid, the caller gets back a null.
// Some time later the null value will cause an error, but the
// source of the problem will not be clear!
function getVehicle(type, opts) {
  if (type === 'car') {
    return new Car(opts);
  } else if (type === 'bicycle') {
    return new Bicycle(opts);
  }
  return null;
}
// Better code: If the type is invalid, the program immediately
// informs the developer, with a clear message and stack trace
// that can direct the developer to a solution.
function getVehicle(type, opts) {
  if (type === 'car') {
    return new Car(opts);
  } else if (type === 'bicycle') {
    return new Bicycle(opts);
  }
  throw new Error(`Vehicle type '${type}' not recognized`);
}

How does this help? Imagine a developer calls getVehicle('bike').

The good code above will emit a nice clear error message "Vehicle type 'bike' is not recognized" which the developer can easily understand and fix.

But the bad code will delay the problem, and later emit something like "Cannot read property 'accelerate' of null" which will leave the developer scratching their head. The stack trace of this error will show a different part of the program than the part where the invalid argument was actually passed.

This situation is quite common. Sometimes I see code that returns an empty string, thinking it is fulfilling the contract, but the contract was really to return a valid string, and it failed to do that!

Of course, there are some situations where a null value or an empty value are appropriate. But if the developer is expecting something of a certain type (e.g. a Vehicle above, or a non-empty string) and the function cannot fulfil that expectation, then instead of silently failing to fulfil its duty, the function should immediately throw an error explaining what is wrong with the input.

This might just make life easier for developers in the future…

(An even kinder error message might say "Vehicle type 'bike' is not recognized. Valid types are 'car' and 'bicycle'.". That sort of response might be appropriate for an API server where the caller is not able to see the code or the list of valid types so easily.)

Anecdote: Putting obstacles in front of junior devs

We had one situation where a junior dev would meet a problem, ask me for the solution, and then forget it, and ask the same question again a couple of weeks later.

By the fourth time, I realised there wasn’t any learning happening, so at one point I arranged the developer’s work so that they would meet that problem three times in the same week. After that, the solution stuck, and this developer started to fix the problem by themself, and then eventually preemptively prevent it.

Similarly, when we had a new developer join, we had to create two deployments in one month. So I arranged to make both deployments in the same week. The idea was that if they encountered problems, by encountering the problem twice rather than once, the solutions would be remembered.

Is it cruel to set up obstacles for other devs? Well I thought it might be a form of kindness. I might not be around the next time they face that problem!

We also started the policy (mentioned earlier) of noting down the issues we solved, so that we can refer back to our notes in future, and this had some success. And, as mentioned earlier, this seems to be most useful for those issues which take time or are disruptive to fix, but which occur so rarely that we forget the solutions in the meantime.