Do you remember when software was expensive to make? When you couldn’t just snap your fingers and make some code appear with a full test suite? Crafting code this way was hard, but it was rewarding. I would be so proud when my snippets of code all worked together to solve the problem at hand. No software lives forever, but I would have been sad if that code had to be thrown away. I spent time on it, I thought hard about it, and I made it exactly the way I thought it needed to be. But now, code is cheap.
Code is Cheap
AI can generate code as fast as you can dream it up. The effort is trivial and the cost is laughable compared to the amount of time we used to spend writing code. We can take a moment of silence to be sad about this new reality of our precious craft…
Sit with your sadness for a bit…
Okay, the moment is over. Now that we got that out of the way, let’s talk about what this means for how we build software today.
A New Approach to Debugging
I’m going to drill into one area of change I’m excited about, which is that this unlocks new ways of debugging. For example, AI can instantly create tailored, disposable debugging utilities. This speeds up debugging because we can iterate more quickly on our hypotheses about the root cause.
The current approach many developers take with AI-assisted debugging does not work for complex bugs. Most developers I work with tell their AI: “Find the bug and fix it.” This would be great if it worked, and it actually does for simple bugs. Problem solved. But in my experience with more messy issues, this leaves room for too many assumptions to be made by the AI. It implements a “fix” that does not work. You say: “This doesn’t work, try again”, and the cycle is repeated. It’s an exercise in falling down rabbit holes, hoping AI stumbled into the right one. And if not, you have to drag it back out and start over.
Hypothesis Driven Debugging
A better approach is to use AI to do “hypothesis-driven debugging.”
- Gather context
- Generate hypotheses
- Rank and select one
- Build throwaway utilities to verify hypothesis
- AI runs experiment, or hands off to human if it is not capable
- If issue is not fixed, return to #2. If it is, implement the fix and remove the throwaway utilities.
This is a good process run on your own, but doing it with AI is incredibly helpful. For one, it can generate a multitude of hypotheses quickly. It can help rank them by likelihood * effort to test to determine the hypothesis that is best to start with. AI is also great for the experimentation piece. Depending on the hypothesis, AI might be able to just run the experiment for you and report out whether the hypothesis was correct or not. But if it can’t, it can give you the tools and detailed instructions to run the experiment yourself, and then to feed the results back into AI for analysis.
A Real Example
Let me give you a real example where I used this. We had a bug in the front-end of a software product I support. On one of the pages, we use query parameters to prefill a complicated, multi-step form that is generated based on various sources of data. The bug we faced was that the prefill would intermittently fail, but not always in the same place. Developers could never reproduce it on their own machines. Forcing a slow connection would not reproduce the issue. There seemed to be some similarities in the underlying data for the affected cases, but we couldn’t pin anything down.
Eventually, I asked AI to make some hypotheses about what could be causing this, and pointed it toward areas of the code that I thought might contain the culprit. One of the hypotheses was that this was the result of a complex race condition. This seemed reasonable because there are 6 different API requests that feed into the form one way or another. Based on this hypothesis, I had AI create some throwaway code that allowed me to toggle delays for the various API calls from the query parameters. For example, it looked something like this:
https://myapp.com/complex-form-page
?productsApiDelayMs=1000
&ordersApiDelayMs=2000
&cartApiDelayMs=500
This made for a great experience, because I could easily try different combinations of request speeds and eventually identified the race conditions that were causing the bugs. There were actually 4 separate failure points that needed to be fixed.
Objections
At this point you may be objecting, why not just use the network tab in the developer tools to throttle individual requests? This is a reasonable objection, because I absolutely could have done that. My reasons are speed and feel. I would argue that making the throwaway code is actually faster. This would not be true pre-AI, but now it is. I don’t throttle individual requests in the developer tools too often, so it would take me some time to do that. Especially for 6 separate requests and configuring them all differently to get precisely the order I want. But my custom utility, made by AI, is perfectly suited for what I am trying to do. It’s fast, and easy to see what I have configured. Which leads to my second point, which is that the custom utility just feels better. Dev tools has a lot in it. It can feel a bit overwhelming. Our application also makes a LOT of requests, and to target the ones I want in dev tools, I would need to remember the URL paths to filter on for each of my API requests, which could take some time. But with the AI approach, I can just say: “make a query param for each API request involved in initializing this form that allows me to delay it in milliseconds.” Then I can plug those query params into the URL as demonstrated in the code snippet above. I believe this is a better experience than using the developer tools for request throttling. And I think there are many cases where these one-off tools would provide a better experience than standard tooling.
Why is Nobody Doing This?
So why am I not seeing more developers doing this? I think part of it is habit. We all just reach for what we’re used to, whether that’s IDE debuggers, dev tools, peppering the code with console logs, or something else. I think another reason is that code still feels precious. We’re used to perfecting it, refactoring it, reviewing it, writing tests for it. We put a lot of time into code. So it feels wasteful to create code destined to be tossed away almost immediately. But code is cheap now. As developers we need to understand the new implications of disposable code, and change our habits accordingly.
Conclusion
Creating AI-generated throwaway debugging utilities is a new, fast methodology for quickly hypothesizing and experimenting to find the root cause of defects. To be clear, I’m not advocating that this should replace all existing methods of debugging. In many cases the existing tooling should be the preferred approach. It is simply another tool in our developer toolbox, and one that I don’t see many developers reaching for.
If you want to give it a try for yourself, I distilled this into a skill file for coding agents. You can view that here: Gist | Hypothesis-Driven Development Skill