TL;DR A big part of programming is about handling the “unhappy” paths. As is the case in life. Lower your expectations and expect the unexpected.
Time flies by when you are busy with one task after another. A lot happened in the past six months since my last retrospection, 1 Year at Gusto — the 15 things I learned. Among all, one thing, in particular, stands out. It’s related to a bug — an expensive, painful, and soul-sucking bug.
I introduced the bug and caught it a few weeks after it went live. The bug was dead stupid, and the fix was easy. But its damage was quite significant. A good portion of production data was polluted. It took my manager and me a few weeks, including some late nights and weekends, to correct the data. Customers were confused, and so was our support team. Many support tickets were created as a result.
Responsible for the bug, I felt awful, horrible, and sick to my stomach whenever I thought of the confusion, chaos, and waste of time the bug caused. I have never felt so ashamed and deeply sorry for a mistake. The skies were grey.
Reflection
When I first reflected on this incident, I thought very hard about what caused it and how to prevent similar mistakes from happening. My reflection wasn’t fruitful because the bug was literally a stupid mistake. I couldn’t think of anything other than “don’t make stupid mistakes.”
My manager, Nick, reflected with a completely different perspective: knowing there would be bugs in the code, what could we have done differently to minimize the damage?
No matter how hard we try, we will inevitably have bugs in our code. Sometimes it’s because of false assumptions. Sometimes it’s because of our lack of domain knowledge. Sometimes it’s because of stupid mistakes. The truth is, we are imperfect human beings who make mistakes.
When the code is mission critical, we shouldn’t bet against reality and assume that there will be no bugs in our code. Instead, we should operate under the assumption that there will be bugs.
Knowing there are bugs in the code, now what?
If we assumed there were bugs in my code, we could have done at least two things differently to minimize their damage.
- Since my code replaced an old system with a new one, we could have had the two systems run in parallel for a while, monitored the new system and compared its results with the old one before committing to the new system.
- I should have paid closer attention to how my code was working after it rolled out so if anything went wrong, I would catch it early.
Expect the unexpected
A small conversation that happened during my next project was a perfect example of using the expect-the-unexpected mindset. The project was an integration with a third-party service. Before it rolled out, Sam, our product specialist from the customer support team, reached out to me to learn more about the project. The first thing Sam asked was: “if something went wrong, how could the support team tell if it’s our system or the third-party service misbehaving?”
I was shocked when I heard the question. Coming from the support team, Sam spends a decent amount of time helping confused customers. The fundamental cause of confused customers is because unexpected cases happen and cause the system to misbehave. To some degree, Sam’s job is about expecting and preparing for the unexpected.
I was amazed by how naturally Sam operated with the mindset of “knowing something will go wrong, how can I prepare for it?”
On the contrary, when I thought of software development, my instinct was: “here is the happy path for this feature, what do I need to do to make it happen?”
As experience showed, again and again, implementing the happy path is never the hard part. Discovering and covering all the unhappy paths, all the things that can go wrong, is what makes software development hard.
80% of the time in software development is about discovering and covering the "unhappy" paths.
— Sihui Huang (@sihui_io) October 2, 2017
Under this hypothesis, as developers, we should allocate energy and time for discovering and covering the unhappy paths. In fact, during planning, instead of focusing on the perfect happy path, we should pay special attention to and brainstorm what can go wrong.
Why is software development often frustrating?
Through this lens, lots of my frustrations about software development make sense. I estimated timelines mostly based on the time needed for implementing the happy path. During the course of development, I would discover some unhappy paths I needed to take care of which added challenges for meeting the deadline I set. The further I was in the development process, the more edge cases I discovered. As the deadline approached, the amount of remaining time decreased, more unexpected cases were discovered, and the pressure increased.
All these factors made meeting the deadline almost impossible. But since I dislike falling short of my own expectations, I would try to knock out those edge cases as soon as possible so I could finish implementing the happy path.
Obviously, this process was frustrating. During software development, the harder I tried to make things happen, the more inclined I was to hack things together and the less I cared about actually understanding the code or the problem, which in turn made making progress harder. And the process is error-prone. The more pressure I had, the easier it was to overlook cases and make mistakes.
The wrong expectation
Essentially, I experienced frustrations and disappointments because I set the wrong expectation. My expectation was based on the wrong thing — the happy path.
There are two things I can do to improve the process.
- Set better expectations. Expect things to go wrong, because they will, and save room and energy for the unexpected.
- As I progress, I should adjust timelines based on reality, instead of trying to force things to happen. A quote from a colleague: “Deadlines are just estimations.” Failure of meeting a deadline is a failure of estimation, not a failure of the developers themselves.
What about life?
“Rather than thinking, ‘I’m right.’ I started to ask myself, ‘How do I know I’m right?’” — Ray Dalio
Zooming out, I see similar patterns in life. Many times I have gotten frustrated, disappointed, and lost thinking things were not working. I would start questioning my approach. But in reality, most of the time, things were working, just not in the pace or form I expected.
For example, when I was writing the Design Pattern in Life and Ruby Series. The first couple weeks, nothing happened and I thought maybe no one cares and it’s not worth the effort. But months later, it started to get some attention and I started receiving very encouraging emails from readers.
The same patterns have emerged many times in my life. Whenever I start something new, I always go through a phase of frustration and disappointment. Reflecting on this, I believe most disappointments come from a misalignment between expectations and reality. When that happens, I tend to assume I failed. But what really fails is my expectation, not my execution nor myself.
Two things I can do to make my life easier.
- Lower my expectations. Assume things will go wrong and be prepared for it mentally.
- Adjust my expectations according to reality. When things don’t happen according to plan, instead of declaring myself as a failure, consider I might just have the wrong expectations.
The subtle art of not giving a f*ck
One of my favorite books of the year is The Subtle Art of Not Giving a F*ck by Mark Manson. I felt a great sense of relief after reading the book.
When things don't go as planned, I tend to think it’s either my fault or the fault of others. So I will either get angry at myself or others. But in reality, there are a million ways a thing can play out and it’s against all the odds that it will play out exactly the way I expect. Things are supposed to go wrong, and that’s fine.
If there’s a problem in my life, I will get a bit freaked out thinking something is wrong. In reality, lots of times life just sucks in general. It never goes as planned. There will always be ups and downs. And that’s fine. Nothing is wrong. That’s how life is supposed to be. Imagine if everything went as planned; wouldn’t that be a bit boring?
“Life is essentially an endless series of problems. The solution to one problem is merely the creation of another.” ― Mark Manson
The Japanese even have a word for it: Wabi-Sabi - The Art Of Imperfection.
When you expect the unexpected
Here comes a question: if a path is "unhappy" because it’s unexpected, when you expect the unexpected, don't that turn the path into a happy one? ?
Which is why I (as the main release + rollback person) never release anything near the end of day/week or near lunch 😀
Expect things to go wrong, and have enough time to find out and fix issues is a great practice
Yup, nothing worse than having to fix production bugs over weekends. ?
Ouch! As a QA person, I can relate so much. Our team had a bug and after the hot fix we did endless regression testing to prove it works. Payback is a bitch.
If your org doesn’t have a QA, I suggest to get one. Else, pair with your PM/Support or anyone for that matter.
I had a talk last week why should we care about QA. Basically, I said we want to provide value to others and avoid hurting them too. Cheesy but true.
Love your blog btw.
Thanks, Karlo! We don’t have a QA team at my company. I think mainly it’s because we want to keep the feedback loop tight and the development cycle lean.
We try to rely on tests as much as possible. And for customer-facing features, we normally have internal test fests.