One Damn Machine After Another

The coffee tastes like someone brewed a shovelful of dirt. The bunkhouse stinks of exertion and your boots are still wet from yesterday’s toil. You and your partner sit under a lantern tending to the saw. Six feet of steel with huge teeth and tiny rakers that act like little spoons when they’re deep in the middle of a tree. You oil the saw because you hear it makes the job easier.

You walk with the saw bouncing on your shoulders. A sharp axe swings from your hip, and a sledge hammer and a couple of steel wedges make up the last bit of required kit. If it’s a good day you will fully process two trees. “Processing a tree” includes falling it (cutting it down), limbing it (cutting off all of the limbs), bucking it (cutting it into sections), and hauling out the pieces. But first you have to pick a tree.

A foolish man inside of a notch.

You finally get to the stand and look up. You want one that’s leaning the way you want it to. You want a healthy tree (they fall more predictably). You want one that has clear escape routes in case something goes wrong. There’s an art to selecting a tree that you have to earn.

You pick one and set down the extra gear. The first part of falling a tree is to cut your notch. You and your partner take turns chiseling into the tree with your axes. If you do your job right, the tree will fall in the direction of this cut. 

It’s hard work with a rhythm. Tack, tack, tack. Once you get to the heartwood the sound changes to a thunk. Tack, tack, thunk. The forest sings with the quiet work of men and steel.

You breathe in the scent of wood chips and walk to the back side of the tree. Where you put the back cut is very, very important. If you get it wrong, you could die. If you place it too high, the hinge will be too thick and the tree could tear. If you place it too low, the tree could spin - or worse, it could split vertically up the trunk as it’s falling. But you put the cut in the right spot because you know what you’re doing.

Old time woodchoppers called the saw a “misery whip”. It might be a misery whip, but it’s yours. You and your partner get to each side of the saw, hands on the handles. Push-pull. Your job is to make your partner’s job easier. The first few inches are tough until the track is set. You have to keep it level so you don’t torque the blade. Push-pull. You can tell the species of the tree by the scent of the sawdust. Today it’s pine. 

As you get deeper into the tree, you can feel its weight binding on the blade, so you tap in a steel wedge. It resonates with a satisfying ping. Ping, ping, ping. 

You and your partner continue your dance and start to hear the tree talking to you. It pops, creaks, and groans as it realizes what’s about to happen. The tree begins to go. You back out into the escape path that you cleared earlier and keep your eyes up as it falls - it can kick back if you did anything wrong. This time it doesn’t. It lands with a deep bass that sounds like someone dropped the sky. You exhale, grateful to be alive. And now the hard work begins.

You and your partner grab your axes and walk the tree, taking off every limb you pass. Others join in. The group rolls the tree, always standing on the uphill side. Once it’s limbed, you have to buck it (cut it into sections). More people join. Push-pull. Wedges. Ping. Push-pull. Sawdust clings to your sweaty face. Push-pull. It’s in your lungs. Push-pull.

And then it’s time to get the logs out. You hook up each one to a team of horses and the teamsters pull it to the skid. This tree was close to the landing (where they collect all of the trees), so you’ll need to blow up the stump with dynamite. After that, if there’s time, you’ll get ready for tree number two.


Scenes like this repeated themselves daily in early 1900s America. The job wasn’t too different from how trees were cut down in the 1800s. Or the 1700s. To be a lumberjack was a dangerous, backbreaking job. There was a craft and an art to the job, and those who didn’t learn quickly (or were unlucky) ended up paying for it with their lives. 

By 1940 there were about 170,000 people in the country doing this kind of work (source, page 25). Today there are around 23,000. No one who cuts trees for a living uses hand saws, but there are still people who do it using chainsaws. Most logging these days is done using machines called “feller bunchers”. These beasts cut the tree at the base, rotate it, and then feed it through, limbing and bucking as it goes. The two man team of 1940 could manage 2 trees on a good day; feller bunchers can process around 100 trees an hour.

A feller buncher processing a tree.

So how did we get from axes and hand saws to a rotating buzzsaw on tank tracks? My intuition was that early lumberjacks would’ve leapt at the opportunity to do less physical work, to cut more trees, and to earn more money. But my intuition was wrong.

Chainsaw from 1938.

The first chainsaws appeared sometime between 1920 and 1930 in the US. These early machines were massive two man devices that weighed up to 130 lbs. The form factor matched what the workforce was used to, but they weren’t ready for mass adoption. 

The first chains required frequent resharpening, which slowed down productivity. They would break down regularly too, and would spit hot oil and gas at their operators. Many lumberjacks felt that the machines didn’t cut much better than a regular cross-cut saw, and the maintenance wasn’t worth it. They preferred to use the tools that they’d always used. They didn’t want to futz with engines and sharpen chains. They were woodsmen, not mechanics.

Over time chainsaws got lighter and more reliable. But they didn’t see widespread adoption in America until WWII. During and after the war, demand for lumber skyrocketed, and logging companies scrambled to try out the new machines. Initial productivity gains were modest in absolute terms: instead of 2 trees, a crew might get through 3 or 4. Still, a doubling of output was a sign of the promise of these machines.

Two innovations would catapult chainsaws from curiosities to must-have tools. First, they got lighter and were able to be carried and used by a single person. By the 1950s they weighed about 60 lbs. A single man with a chainsaw could now fall and buck a tree by himself. This used to require an entire team of fallers and buckers (source). 

The second invention that caused chainsaws to take off was Joe Cox’s “Chipper” style chain. Early chains were “scratcher” style. They would scratch the surface of the wood, scooping wood out and going a little deeper each time the chain rotated around the bar. This dulled the teeth very quickly, which caused crews to stop and resharpen far more often than they wanted to. 

Joe had some curious hobbies - one of which was watching timberwood beetles feeding on wood. Rather than burrowing into the wood, the beetle moved its mandibles left to right as it ate, chipping wood along the way. Joe designed a new chain based on this action. It was more durable and required a lot less maintenance than the existing chains. And it was this combination of a more reliable chain and lighter saws that allowed chainsaws to gain widespread adoption throughout the 50s.

For a decade or so, logging didn’t change much. There were changes to how trees were moved from place to place, and saws continued to get lighter and more reliable. But the tools that the men in the field used were basically the same. By the late 1960s the first patents for the feller buncher were filed. By the 70s, prototypes were out and cutting down trees. By the 90s, we arrived at the kinds of machines that are more or less still in use today. In less than a career’s length of time, timber harvesting went from large crews of men with axes, to large crews of men with chainsaws, to small crews of highly trained machine operators driving up to trees and cutting them down.

To some it felt like “just one damn machine after another.” (source) They saw the craft of hand-falling a tree replaced by people sitting in air conditioned cabins fiddling with joysticks. 

Others embraced the technology and were retrained to use the newfangled machines. The transition wasn’t smooth and the benefits weren't equally spread. The number of frontline workers dropped by 90% while overall production increased.

Knocker uppers, waking people up before clocks were widespread.

History is littered with jobs that no longer exist (e.g., the knocker-upper). But this story isn’t one where technology completely obliterated a job. We still need someone to cut down trees. And while the shift to mechanization did put people out of work and it did change whole communities as a result, the biggest shift was that fewer people could do far more work than before. The center of gravity shifted from the weight of your muscles to the dexterity of your fingers. From your arms to your brain. It didn’t obviate the skill required to do the job, it just changed where you placed the emphasis.

Software is following a similar arc. We used to have to punch physical cards in order to get computers to do anything useful (the axe-wielding days). Then we got terminals, GUIs, and IDEs where we could debug, test, and introspect the code at runtime (one man chainsaws). And now with AI coding tools, we can do a day’s worth of work in less than an hour (chipper chains). The current tools still spit oil and gas at their operators; they’re still heavy. But lighter and more reliable saws are coming. Feller bunchers are on their way as well. The march of “one damn machine after another” is underway. 

I don’t want you to read this and worry too much about your job. Logging is constrained by the number of trees available for harvest and the overall demand for wood. Those exact pressures don’t exist in software. Demand is high and continuing to grow - but so too is output. The most likely scenario is that demand for SWEs stagnates in the near term and contracts over the medium to long term. The job you will be doing 3 years from now will be unrecognizable to someone who retired 3 years ago.

You’re probably going to be fine as long as you adapt to the times. But if you’re still holding an axe, you really ought to pick up a chainsaw. They’re here and they’re getting better every week.

A science-based approach to debugging code

There are a ton of ways to solve any individual problem. In this post we’re going to talk about the one I’ve had the most success with: a science-based approach.

Generally, when a scientist goes to figure out how the world works, they start by observing world. Then they make a guess about how it’s working. Then they set up some kind of experiment to collect evidence for/against their theory. Richard Feynman says it with more authority than I ever could.

Now I’m going to discuss how we would look for a new law. In general, we look for a new law by the following process. First, we guess it...Then we compute the consequences of the guess…And then we compare the computation results…with observations to see if it works.

If it disagrees with experiment, it’s wrong. In that simple statement is the key to science. It doesn’t make any difference how beautiful your guess is, it doesn’t matter how smart you are who made the guess, or what his name is…If it disagrees with experiment, it’s wrong. That’s all there is to it.

In the quote above, Feynman was talking about how physicists figure out how the world works. But that same way of thinking can be applied to problems in software. If you don’t believe me, let’s take a look at an example I took from a recent exchange with a colleague.

I was covering for the other team lead when one of the engineers came to me for help.

“I can’t figure this thing out. Can you help?”

”Sure,” I say. “What’s going on?”

He summarizes the problem:

“When reloading the large workspace after auto-arranging the windows, there is a hole where a window should be, and several windows are on top of one another when they shouldn’t be.”

A workspace is simply a collection of windows, their internal state, and their physical position on the screen. A native window is a type of window. Auto-arranging is a way of taking all of the windows on the monitor and forcing them to fill all of the available space on the monitor. Our product manages several different types of windows, and we were only seeing this bug manifest on one particular window type, after triggering auto-arrange.

What follows is a reconstructed transcript of the rest of our conversation:

D: It looks like the bug only presents on native windows. It was reported on the large workspace (40+ windows).
B: Huh. Why do you think that’s happening?
D: Well. When saving a large number of windows to the workspace, we are putting the windows in the wrong place because of a race condition in the code. There is a specific number of windows that makes this bug more likely to happen.
B: Alright. Why do you think that?
D: Well, I did a little experiment.

Before going over the results of his experiment, I want to pause. I’ve gotten a couple of very important bits of information from him. First, he told me his observations. Second, he told me his guess. And next, he’s going to tell me the experiments he ran to confirm/disconfirm his guess. I used this example in the talk I gave to the company. When I was prepping it, I asked for permission to use it. After he said yes, he told me that he’d heard a talk about using the scientific method to debug problems, and he was trying to apply that style to this problem. 👍👍

Now, for the results of his experiment and the rest of our conversation:

Scenario Result
2 Windows No bug
4 Windows No bug
16 Windows No bug
25 Windows No bug
20 Windows No bug
21 Windows BUG

D: I think the problem is in [this other place in the code that you don’t need to understand to get the point I’m illustrating]. Something about a large number of windows is causing the bounds of the native window to be set incorrectly. It doesn’t happen on Electron windows. It looks like 21 windows appears to be the tipping point.
B: Does it happen when you just auto-arrange a bunch of native windows?
D: No
B: I’m not convinced. I think it's in the workspace management code. When we recreate a workspace, we just take the bounds from storage and pass them off to [the other part of the code that is irrelevant]. If your bug doesn’t happen when you trigger auto-arrange, I doubt it’s in [the other part of the code].
D: Okay, let's look at the data in the workspace.
B: Data looks fine.
B: Wait. There are fractional pixels for some of the windows. That’s strange. Don’t we round those when we move windows?
D: Maybe?
B: Wait. All of your tests were on even numbers. except 21. and 25. I don’t think it’s 21. I think it’s N, where N is a number that doesn’t divide evenly into the pixel-width of the monitor. Try 3.
D: It happens with 3.
D: Let me try 5. It happens with 5. Let me try 6. It doesn’t happen with 6. I think we’ve found the bug.
B: :)

So his initial guess was wrong. My initial guess was wrong. But by looking at evidence in favor of/contradicting our hypotheses, we were able to figure out where the bug was. One of the reasons we were able to solve this problem so quickly is that back in the day I’d seen some weird behavior when we tried to move a window to a fractional place in space (e.g., x: 283.76). It turns out that the win32 API expects coordinates to be integers, not floating point numbers.

But the other reason we were able to get to the bottom if this is because of the process he was following before we started talking. We only spoke for maybe 20 minutes. Before that, he only spent a couple of hours diagnosing the problem. There’s a lot that we can take from this example, but I really want to focus on his process.

First, he observed the problem. Second, he guessed what was causing the bug. Then he came up with an experiment to test his idea. And then he kept iterating. Once he felt like he had enough data, he looked at it and couldn’t make sense of it, so he came to me. Because he was already set up to approach the problem from a scientific angle, we were able to iterate rapidly and we ultimately solved the problem.

Using this approach doesn’t guarantee instant or even quick success. You won’t always be right on your first guess. Or your twelfth. But if you keep your mind open to the possibility that you’re wrong and you set yourself up to iterate quickly, this approach will get you to where you need to be. It makes a ton of intuitive sense to me, and it’s been by far the most effective way that I’ve been able to solve problems.

The next several posts will be about some common debugging mistakes that I’ve observed. I’ve tried to make them as generic as possible so that they are useful no matter which approach you use to solve problems. First, we’ll discuss perhaps the most important part of solving a problem: defining it.

How to Be a Better Debugger - a Series

One of my favorite things to ask interviewees is this: imagine the ideal software engineer. This engineer has 3 discrete skills:

  1. Their ability to communicate with others.

  2. Their ability to debug code.

  3. Their technical expertise and experience.

I then ask the candidate to rank those skills in order of importance. Then I ask them to rank themselves on each of the skills. The way the candidate answers the question tells you a little bit about how self-aware they are; it also tells you what the ideal software engineer looks like in their head. The order doesn’t matter so much as their rationale for the order.

I think the order above is correct. Also, note that I called them skills and not qualities. Skills are things that can be learned and improved. Communication is at the top because of how difficult it is to learn to do well. Yes, you can become a better communicator. But I think the results depend a little more on aptitude than the other two on the list. As for expertise and experience - I think it’s nice to have. But you can gain experience by doing a job poorly. I have experience in woodworking. I’m not a good woodworker. Debugging — debugging is huge. Depending on the maturity level of the project, I’d wager that you spend 35%-75% of your coding time doing some kind of debugging. In an interview, I’m more interested in the person who tells me about an interesting problem that they solved than the person who rambles on about why Redux is clearly the best way to do state management in React apps.

That might be because I’m self taught. I got into this field because I enjoyed the problems. I never studied computer science or software engineering, so early on in my career I wrote a lot of code that didn’t work. Because of that, I got decent at figuring out why stuff wouldn’t work. But it wasn’t until I got onto my current project that I really honed my skills as a debugger.

I’m currently in a team lead role on a project I helped start. It’s a multi-process javascript framework and architecture designed for creating enterprise grade workflows. It’s very complex. Any given problem can be in the DOM, in the layer above, in the layer below, or somewhere in the communication layer. The product itself is only about 3 years old. We currently have 11 very smart engineers working on it, most of whom have been with us for less than a year. Because the problem space is so big, so complex, and so novel, I often end up serving as pair-programmer, rubber duck, and observer. Over the course of several months, I found myself giving the same or similar advice to different people. At some point, I had an epiphany in the form of a series of thoughts.

“Not everyone solves problems like I do”.
“ 😱”
“They’d be more efficient if they did.”
”Maybe I should do a talk about debugging.”

A couple of weeks later, I did a talk about debugging for the whole company. I want to be clear - I don’t think that the approach I will outline is the best way to solve every problem. However, if you combine the general approach and avoid some of the pitfalls that we will go over, you will be markedly better at debugging code.

By the end of this series, you should feel confident enough to parachute into a section of code with nothing but a stack trace or a description of what should happen and what is happening. From there, you’ll be able to make a guess, gather data, test your guess, and iterate quickly.

Up next: A Scientificish Approach to Debugging Code.