Software engineers, for all of their deep rational view of the universe, tend to sometimes approach debugging like soothsayers inspecting entrails trying to divine what the hell is going on.
The scene played out like this:
Bug report: XyZZy is happening
Engineer stares at bug report, stares at software, stares at test case that doesn’t reproduce it.
Engineer then tries things that make sense.
Bug doesn’t go away
Then Engineer tries random acts of coding to see if the bug goes away.
And his peers start saying: Wow, yeah I think I saw something like that. Did you try the toggle the frommlebit?
And the engineer says: But this has nothing to do with the frommle bit, this is tied to pluxion submodule.
And another peer says: Trying adjusting the memory size of the core structure.
And the engineer says: But my code never touches that core structure!
And another peer says: We’ve had success with this kind of problem that we don’t understand when we altered the return value of the function..
And the engineer says: But this makes no sense.
And they all respond like a Greek chorus: But it sometimes works.
At this point in time the engineer is hoping and praying that just moving the code around will cause the bug to go away.
My buddy and I used to call that last stage of debugging “Voodoo debugging”.
You don’t understand the problem, you don’t understand why the bug is happening, you have a deadline to meet, so you try random things hoping that the bug will just go away.
And the sad truth is that the simple act of moving things around may cause the bug report to go away. The bug may reappear later in some other poor bastards bug queue. And the Voodoo debugging will continue as people try this and try that.
In the pre-cloud era of computing, when software engineers believed that it was their divine right to understand how it all fit together, this was viewed as a moral failing. Back in the day we could understand how the entire system from the silicon up worked and we could use that information to thoroughly debug and understand the problem.
Those who lacked the moral fiber and intellectual rigor to do real software engineering practice Voodoo debugging. The rest of us understood how the entire system fit together and resolved the issue down to it’s root cause.
Then I moved to Zynga. And at Zynga it became apparent that Voodoo debugging wasn’t just a product of a lack of moral fiber or intellectual rigor it was a rational response to the environment.
I talk about this a lot in an answer on Quora. The essential elements of the arguments are that systems are in such continuous flux that understanding how the system is working in its entirety is impossible and worse by the time you understand it, the system has changed.
Bugs come and bugs go. And bugs have an element of Voodoo magic about them. Trying to understand bugs and resolving them is impossible. We are now in the era of trying to resolve issues without fixing them. We have become doctors who offer palliatives rather than cures in the hope that the problem will just go away.
And so I embrace my inner Voodoo doctor. I will try the magic cants, I will pray that the next update of the API I am using fixes the problem, and I will internalize the truth that finding the root cause is an illusion my earlier me had.
Leave a Reply