Debugging tools survey
The reason for working on the Signpost project was that I assert that programmers need better tool support, especially in the area of debugging where often human memory cannot keep up with the amount of information that needs to be digested and retained. It would appear however that most of the knowledge we hold to be true about tool usage is based upon anecdotal evidence. I therefore sought to discover what tools were used, and how frequently by asking programmers directly.
A questionnaire based several debugging techniques/tools, with the addition of print statements, colleagues (including Web and UseNET discussions) and knowledge bases was written and placed on the web. Programmers were asked to rate how often they used the techniques listed on a scale of 1 through 9, where 1 was used all the time and 9 was hardly ever used. In addition, 0 could be used to indicate that this technique was never used and X if the technique was unknown to the programmer. For final collation, values 1, 2 and 3 were used to indicate frequent usage, 4, 5 and 6 for moderate usage, 7, 8 and 9 for infrequent usage. Using these value ranges averaged out the variability in the respondents’ evaluation of tool use and made the graphs below much more readable. Free-text spaces were also available for comments on the tools listed (and to give the ability for further techniques to be submitted) as well as a space for a “wish-list”.
In total, 85 responses were received, whereupon we segregated the replies into two groups: academic users and commercial users. These groups contained 29 and 44 replies respectively. Replies were elicited for the academic group by using an internal computing staff email list at the University of Kent whereas commercial users were asked to respond and self-select from a UseNET posting to groups dedicated to programming (e.g. comp.lang.c, comp.lang.c++, comp.lang.java). In addition, 12 users wished to remain anonymous but were including in the commercial users group as their responses originated for the UseNET posting. The results shown below confirmed some of the beliefs we have over tool use but also showed some other interesting findings.
 |
 |
One of the immediately obvious results of the survey is that within both groups, print statements are more widely used that any other technique: including debuggers. Comments from the respondents corroborated anecdotal evidence that programmers prefer print statements as they don’t have an additional skill to learn and the level of information they require is tailorable to their current situation.
As a subset of questions, we asked programmers to also rate the usage of a debugger’s features.
 |
 |
Both usage patterns show that programmers use debuggers more for watching flow of control than for watching changes in variables’ states. Not only is this shown by the usage of breakpoints and single stepping over variable watches, but also by the increased usage of the call stack. Of the areas that are not used frequently, comments show that overall programmers trust the operation and optimizations of compilers and therefore rarely stray into a debugger’s assembler view. Additionally, the usage of data-driven breakpoints is surprisingly low with 10% of commercial programmers not even knowing how to use them. A few programmers commented that they didn't’t use data breakpoints because of the performance impact , but on the whole it is an example of inefficient use of features.
Among the academic users, a disappointingly small number identified static checking as a technique that can be used to find potential defects. The questionnaire was deliberately vague in the descriptions of the techniques to try and elicit tools that programmers know they use. It is, of course, known that the majority of programmers implicitly use some type of static checking within their compiler, assuming that they take notice of the warnings that are given, but this was overlooked by most of the respondents within academic departments. Some of these respondents are most probably students but (the posting went to all staff and postgraduate students) but it's still disappointing. The other group of programmers however were much more perceptive, showing an increased usage accompanied with comments indicating that they knew their compilers incorporated static checking and a separate tool was not needed.
Of dynamic bounds checking, one of the interesting properties was the difference in use between academic and commercial users. Although most users had at least heard of tools like Purify, the number of academic programmers that have never used such a tool was twice as large as commercial programmers. Cost was certainly one contributing factor but unavailability of a tool (not only bounds checking) on a platform or language was mentioned much more frequently. Writing a debugging tool takes both considerable knowledge and effort leading to the situation where tools that are available for one language or platform may not be available elsewhere. Consequently, tool developers target large established user-bases and charge substantial sums of money to be able to justify a product that has relatively few users. Within the UNIX community free tools are common: debuggers like Mark Russell’s UPS are just as good, if not better, than commercial tools. However, authors can lose interest or not have the time to perform maintenance and the tool either dies or its usefulness dwindles. Additionally, commercial tool developers patent the technologies used within their products and can therefore exclude any similar free tools.
Respondents within the survey have clearly not found a use for visualization tools within the debugging process. Although comments were received about the lack of availability on their language or platform of choice, many programmers had never encountered techniques such as algorithm animation, communication (or call) animation, or change impact analysis. More programmers understood the use of architecture overviews but it would appear that high level design aspects were not important at the level of debugging the respondents were working on.
The use of execution histories had surprisingly low usage for an activity that we consider essential in the debugging process. In order to understand a defect and to write an effective fix, a program’s behavior has to be understood – information that can be gathered using execution histories at a variety of levels. It would seem however that rather than using tools to generate execution histories, respondents of the survey were using print statements to gather the information instead.
The last two questions on the survey addressed the subject of using shared knowledge. Thankfully, all respondents of the survey at least knew of debuggers and asking colleagues (which were our “sanity check” questions) apart from one person who, perhaps inadvertently, answered “unknown” to using anyone else as source of debugging information! Survey results showed that both sets of programmers asked other people more frequently than consulting knowledge bases but no comments within these areas were submitted. This demonstrates that promoting frequent use of knowledge bases within the debugging process is possible, but due to the lack of comments we cannot be sure that the respondents came across the same problems in using knowledge bases as we outlined in the overview
When looking at the comments received, one of the reasons behind the poor take up rate of debugging tools is that programmers believe that the time taken to learn how to use a tool is unproductive – even though virtually all the tools/techniques surveyed claim an improvement in time taken to debug certain defects. This may be partly explained through the fact that programmers tend to learn new techniques only when they have to, and when the necessity arises there simply isn't’t the time to spare.
Evidence also suggests that because programmers only use debuggers infrequently, they are unfamiliar with their full functionality. This can lead to the programmer engaging in unproductive behavior (e.g. stepping through a loop a number of times rather than using a data driven breakpoint) or being forced into features that they don’t care about (e.g. assembler level debugging). The flip-side to this is that programmers are unaware of, or do not have access to, tools that can help in specific situations (e.g. Purify in memory type defects).
Although the above criticisms are aimed at programmer’s behavior and not specifically against the available tools, there is still room from improvement in the area of debugging tools. Programmers want tools that are simple and intuitive to use and supply information about where and why a defect is happening. Most tools, although competent in the tasks they carry out, have complex interfaces or usage, and require the programmer to make that final jump from the information presented by the tool to the underlying cause of the defect.
This survey was carried out again in the summer of 2002 with more than 200 respondent to similar results as those above.