Test protocols generally have a rather bad reputation and at best are said to be a paper tiger. It’s wrong that test protocols are seen this way; rather they are real workhorses in software development.
Before I examine the test protocol in detail, I would like you as a reader to think of your current status quo of your software testing (probably without using test protocols).
• What is the aim of your testing activities?
• Are there clearly defined test runs with start and end?
• Which tools do you use?
• How do you deal with the test results?
• Do you communicate positive and negative test results?
Take some quiet time to think about the above questions, but I can give at least a few suggestions.
If you use a test management tool like TestLink, the test results probably look like that:
If you have entered the found problems into an issue tracker, then the result may look like that:
Or you do not use an issue tracker at all and send bugs by email or collect them in Excel sheets?
Especially, in my opinion, the point bug tracker as the tool for the documentation of the results should be thought once critically. They are highly structured and definitely no issue is lost in them, but they mean quite some overhead in data entry, especially with many details.
I also find that bug trackers tend to weaken the communication. One should be aware that the own organization is influenced only by the presence of certain features in the bug tracker. For example, a tester is busy with the testing of a feature, in which he submits an issue to the bug tracker. A developer is probably notified by email and the chances are definitely given that he thereby feels obliged to view this bug immediately. The communication between developer and tester then primarily happens in writing comments via the bug tracker. This can, but need not be a desired and good process.
So, at best, you now have thought about your tools and methodologies. Memorize your thoughts!
Technique of the Test protocol
Now I would like to examine the test protocol and how it might be able to change the status quo of software testing. I refer mainly to the adoption of test protocols in Quality Spy, but many concepts are generally valid, not just for Quality Spy.
First, I think that there must be a clear definition of the test run. A test run always refers to one software version and has a clearly defined goal and of course a distinct start and end.
In the phase of the test execution, all found notable issues are chronologically inserted in a protocol. That is done regardless of whether a test plan is used or not.
A test run should normally last no longer than one day, but at least 30 minutes. Only after the test run is completed, information (for example, found errors) flows back into the organization.
The process is not too complex, the following diagram illustrates this:
Side note for experts: If you are convinced by the concept of test sessions, you can divide the whole test activity into in any number of test sessions!
As a preparation step, the test scope must be defined. This can be quite short like in this example:
The protocol in Quality Spy has a classic Word document look & feel and supports fast typing of entries during the test:
An entry in a test protocol is basically quite similar to an entry in a bug tracker, but there are some differences. The only “information fields” are a classification as "error", "warning", "other" or "positive", the issue text and images.
Despite the document look and feel, the entries are still stored in a structured format, which is very important for the subsequent use.
After completion of the test, this protocol might be edited somewhat, for example duplicate entries might be deleted, but very often the raw protocol from testing can be used directly. It should not be forgotten to summarize the test results in short key points ("totally unstable", "Feature XYZ blocked, but otherwise all right").
Now, the question is how to handle the results and what to do next, after the test is done.
Initially it is a wonderful idea to communicate about the test results (Tester to developer, tester to project manager, tester to customer). The document format is useful for such purposes, because it is much more pleasant to read than an Excel spreadsheet or bug tracker entries. Newer versions of Quality Spy even allow presenting the results in a slides format that is similar to MS PowerPoint and is even more communication endorsing.
Of cause, now it’s the time to fix the issues. Quality Spy offers several options which make sense for different scenarios and organizational tastes:
• Integrated Workflow (“Lightweight Bug Tracking”)
• Export in a Bug Tracker
• Export as Document (RTF) e.g. for dispatch by E-Mail
• Publishing in WordPress-Blog
A very simple but effective workflow is integrated, which works with the status Accepted (AC) -> Fixed (FX) -> Checked (CK):
Quality Spy allows filtering the protocol by error type and status, which is already quite comfortable to work with.
The key concept is to handle and fix all issues in the test protocol “at once”. If this is done right, a lot of administration work can be saved compared to bug trackers, hence the attribute “Lightweight”. From my personal experience I can say that errors in a test protocol appear much less malicious and as a developer it makes even fun to fix them.
However, this approach also requires harsh discipline and sets certain requirements to the organization; I will explain this more in a separate article “Lightweight Bug Tracking".
Currently the export in a bug tracker is possible for Mantis BT (a famous open-source system), but more can be easily added via the plug-in model. As the bug tracker requires more information such as priority and error severity, this must be maintained before submitting it to the bug tracker, but this can be done by mass assignment, so that it doesn’t take too much effort. In the bug tracker, an accordingly more extensive workflow can be possibly carried out.
The RTF export is well suited if one works together with someone adhoc and uses neither a shared bug tracker nor Quality Spy as a common tool.
Furthermore it is also possible to publish the test protocol in a WordPress blog (using a Quality Spy plug-in). This is useful to increase the visibility of the test results, mainly for people who otherwise do not have or want the access to these test results. I will describe this topic also in a separate article.
This is really more than just a paper tiger, right? Now think about your status quo once again, I hope you have thought about it initially. What could test protocols change about it?
A test protocol is certainly not a wonder weapon, but in many projects it has been proven as a solid workhorse for me. Even in Word format – in the Stone Age era before Quality Spy existed.