Kelechi Nwankwoala

Aspiring polymath

Execution Failures are Still Failures (Calibrate Surprise by Making Weird Excuses)

“Almost only counts in horseshoes and hand grenades” — Frank Robinson

When I was studying for the GRE, I observed a rationalization process in my mind. It happened most often when I screwed up a small detail on a problem I felt I understood. If, for example, I quickly calculated 27-26 +1 as 0 rather than 2, my brain would sooth me by saying, “Don’t worry. That was just a one-off mistake.”

When I was feeling particularly dedicated, I’d fight the rationalization on its first premise. Wait dude, maybe this is not just a one-off mistake. Maybe it’s secretly a deep mistake, an error exposing systemic deficiencies and conceptual failings. Then I would buckle down and pay attention.

This method had some shortcomings. For one, it required repeated deliberation. Deep or not? Deep or not? Deep or not? In practice, whether I made the appropriate decision was a toss up, especially if I was tired. It was hard to sustain the willpower needed for careful error analysis, and the analysis needed to be quite careful to even approximate usefulness. Consider the complexity of the following examples.

You are playing in a recreational online chess match with ample time left. The position is sharp, but you have the initiative and an inevitable checkmate in three moves. You begin the movement by moving your rook, but at the last moment your mouse slips and you click the wrong square. Your rook is taken and then your king is checked and then you lose the game. Deep mistake? (No? What if you are a professional player? What if it was a championship game?)

You are lifting weights at the gym and trying to beat your personal deadlift record. 405 lb. Feeling nervous, you walk up to the bar, grip, and pull. As you grunt, you feel the weight resisting you. You begin to hit a wall. Desperate, you gather all of your strength into one last attempt, rounding your back and breaking your usually perfect form. With that final burst, you get the bar up to your hips and then drop it loudly. You did it! You feel great. Deep mistake?

You are very drunk at a bar that is a 10 minute drive from your house. You decide to drive. You do not hit or hurt anyone on your way back. Deep mistake?

You are memorizing a 100 line poem about ancestry. As you get to line eighty-eight, instead of saying “I flop on my bunk and stare at 47 black faces across the space”, you say “I flop on my bunk and stare at 47 pictures across the space.” This is the only mistake you make. Deep mistake?

You are writing python code. Working quickly you write reversed_str = string.reversed() forgetting that the .reversed() function is in place. Later, when you try to use the reversed_str variable you get an error that you quickly and easily fix (after a facepalm). Deep mistake?

These are quite simple examples. Yet I expect people to have many different (and strong!) intuitions about which constitutes a deep and/or trivial mistake. I myself, thinking as clearly as I ever do, am confused about why some of these matter to me and others don’t. It’s some function of your threshold for success, how much you value succeeding, how consequential you think the mistake will be in the long run, and how frequently you think the mistake will occur, but I find the exact formulation hard to describe. There’s no clean mapping between success and depth. You can think a mistake is deep even when you succeed or label it trivial though it caused you to fail. Consider too that success is hyper legible in these examples. In some cases, you won’t even be sure you succeeded.

But, to be honest, focusing on the label is missing the point. What’s more important is the argument’s frame, our broadly shared assumption that the label “trivial” or “one-off” buys you the right to gloss over a mistake. What most of us really mean when we say a mistake is a one-off is I can safely stop contemplating this.

No mistake, no matter how fringe or small, should ever be ignored. You don’t have to try to fix it, but you do have to give it at least five minutes of thought. You need to train yourself to react when you see evidence you deviated from reality.

You cannot expect a mistake. A mistake you expect and choose is no longer a mistake. You chose it. A mistake you expect and avoid is no longer a mistake. You didn’t make it. And a mistake you expect but cannot avoid is not a mistake. It is not a choice. (You may have made a mistake leading up to that choice, but making the forced choice is not in itself a mistake.) Every true mistake then is a surprise, and one does not gain permission to ignore a surprise by highlighting how out of distribution it is! It’s unusualness is a sign you should pay more attention!

But you’ll find that’s what a lot of people end up doing, myself included. If they cannot convince you the mistake doesn’t matter, than they will say it doesn’t happen often. That is a totally valid objection from the perspective of trying to figure out if it is worthwhile to spend resources preventing the mistake from happening again but fundamentally premature in 99% of cases. You cannot dismiss the object before you have seen it. You cannot reason about what you have not yet accepted.

Ideally brains would do this kind of impartial reasoning:

  • Woah something is different here → what exactly is this beautiful difference? → ok I think I have a better handle on it now → Ok, should I worry about it?

Instead they tend to do this:

  • Woah something is different here → should I worry about it? → no you’re so smart lol. It’s all good.

I get that this is hard. There’s some kind of process in us that loves to normalize and accommodate. Recently, to properly calibrate my surprise, I’ve been experimenting with mentally inserting strange explanations for unwanted results. Like so:

I was working on a math problem and suddenly the late Wittgenstein walked into my room covered in mud. Just as I was calculating the largest integer x such that 20!/2^x has a remainder 0, Witt grabbed my hand and crossed out two 2s from my prime factorization. My answer suddenly became sixteen instead of eighteen! Then he forced me to circle in the incorrect answer, disappearing before I could protest. It was so crazy! I’ve been thinking about it ever since.

Such an anecdote deserves at least five minutes of my time.

2024

Back to top ↑