What constitutes debugging?
An overview of debugging practices based on live-streaming coding sessions
Debugging is an inevitable and critical part of software development. However the process of debugging often remains invisible and misunderstood by managers which leads to sub-optimal team productivity and decreased product quality.
One of the reasons why debugging is misunderstood is that it is difficult to measure. A debugging session usually involves a single developer working on an issue. When the bug arises as a result of a task that the developer is working on, no one else will know that there was a bug at all since the developer will only commit code to the central repository after fixing any discovered bugs. Other times a developer might work on a bug which is reported as an issue in the issue tracking system. They will eventually submit a fix, but the process that they take to discover the fix is not observed by anyone.
The authors of this study found a way around this: they watched live-coding streams on Youtube and Twitch where developers code or debug open-source projects. The streams are unedited which allowed the authors to observe and annotate the devs' activities.
The authors define debugging as the process of localization, understanding and fixing of defects in programs. They watched 30h of videos and extracted about 15h of debugging episodes. Here are the key findings of the study.
Debugging is time intensive
In the authors' dataset roughly 50% of the time was allocated to debugging. This surprisingly big number is confirmed in other studies which try to answer this specific question. Overall estimates from multiple studies claim that the range is between 35%-50%.
Majority of the time is spent on the complex bugs
Around 79% of the total debugging time is dedicated to resolving the hardest 26% of bugs.
Fresh vs. Committed bugs
The study categorizes bugs into two main types:
Fresh Bugs: Defects triggered by the developer during the ongoing development process
Committed Bugs: Defects that are already reported in an issue tracker.
It turned out that committed bugs take significantly longer (median of 29 minutes) to resolve compared to fresh bugs (median of 3 minutes).
Types of debugging activities
Debugging involves various activities which authors grouped as follows:
Code editing - takes 41% of the time
Running/testing the code - 29%
Inspecting state through logs or debuggers - 16%.
Navigating code - 15%
Consulting resources - 6%
Miscellaneous - 4%
During a debugging episode developers would switch several times between the various tasks. They would have on average 3 instances of editing, 3 instances of testing and 3 instances of navigating. In 84% of the cases they tested the program by running it manually and not through tests.
Inspecting program state occurred in only 40% of debugging episodes. When inspecting program state developers would use log statements in 70% of the cases and breakpoints in only 30%.
Examples
Here are two examples of debugging sessions. The first one is from a developer of the Kap screen recorder:
The developer starts a debugging session at 11:34 and tries to understand a bug unsuccessfully for 22 minutes at which point he moves on to another bug. During this debugging episode he tried to use print debugging (console.log in this case) to understand the location of the issue. This involved several iterations of adding log statements (editing), manually reproducing the bug (testing) and then looking at the logs to see which parts of the code were executed (inspecting). One can clearly see that with better tooling this process could have gone much faster.
The next debugging episode is from a developer working on Firefox:
This is a two-hour video where for the most part the developer is working on a single bug. The bug is relatively easy to reproduce but still it takes a lot of time for the developer to understand which parts of the code are related to the bug and what the correct behavior should be. During the debugging session he reproduces the bug manually several times, refers to web standards and compares how this behavior works in other browsers. He used print debugging, but also a debugger to help them locate the issue. We can see that even though the developer is experienced and has good knowledge of the code base it still takes him a lot of time to locate and understand the bug. Once he understood it, the fix was a few lines of code. The developer is using the available tooling well, but the tooling is not helpful enough and so time is still being wasted in mundane activities.
What does this all mean?
If you are managing a software development organization, here’s how you can leverage these insights to significantly enhance your team’s productivity and product quality:
Prioritize debugging practices
Given the significant proportion of time developers spend debugging (up to half of their active coding time), your organization stands to achieve substantial improvements in productivity and software quality by focusing on it. Treat debugging as a critical activity. Train your developers on better practices. Incentivize them to share their debugging experience through pair-programming sessions or in other ways.
Focus on the hardest bugs
The majority of debugging effort (79%) goes into resolving the hardest 26% of bugs. This insight implies that management initiatives aimed solely at automating simple debugging tasks or introducing basic AI-assisted tools may not significantly impact overall debugging efficiency. Instead, support your developers with tools and processes specifically designed to tackle complex and challenging bugs effectively.
Reduce reliance on print-debugging
Many developers heavily rely on print-debugging (using print statements to track program execution), despite its known inefficiencies. Instead you should provide access to advanced debugging tools, such as time-travel debuggers, and offer targeted training. This can greatly reduce debugging time, and improve the speed and effectiveness of defect resolution.
Prevent fresh bugs from becoming committed bugs
Committed bugs take significantly more time to resolve than fresh bugs. A proactive approach involves catching fresh bugs early by implementing more rigorous code reviews, among other practices. Reviewers should be given proper tools which allow them to inspect the code and its execution in greater detail than just by looking at it.
Significant potential impact
If you focus on debugging you can deliver substantial returns. A separate 2013 study found that adopting advanced debugging tools, such as time-travel debuggers, can lead to a 26% reduction in debugging time on average, translating to a savings of approximately $81 billion of annually for year 2013. Said in another way if you introduce better debugging practices in a team of five developers you will get the same or better results as hiring an additional developer.
If you prioritize debugging, provide appropriate tools, and foster targeted training, you can significantly elevate your team's productivity, software quality, and ultimately, your organization's success.
References
Alaboudi, A., & LaToza, T. D. (2023). What constitutes debugging? An exploratory study of debugging episodes. Empirical Software Engineering, 28(117). https://doi.org/10.1007/s10664-023-10352-5
Britton, T., Jeng, L., Carver, G., Cheak, P., & Katzenellenbogen, T. (2013). Reversible debugging software. Judge Bus. School, Univ. Cambridge, Cambridge, UK, Tech. Rep, 229, 2013.