Suppose a test team in a software project is highly motivated, it produces an impressive amount of unit tests, thus creates the most stable system, but the customer is still unsatisfied as the features do not work as he imagines them. Or the team is testing precisely the smallest layout error and the project manager becomes desperate for the escalating costs. Or even worse: a software has been extensively tested, but the core functions have been forgotten or the compatibility to an important target platform was not verified. Or even much worse: the test has not found critical errors because the used methodology was inappropriate.
All these are examples of non-effective testing. Hardly a test procedure will be perfect, but such difficulties can be significantly reduced when some upfront planning is done, which is called a test strategy.
If such a test strategy is composed for the first time, a basic template is useful. For the past many years I use the following points for orientation:
• What (Test subject)
• Acceptance criteria
For some projects, defining a strategy can be easy, usually if a similar project was already completed. However, one should think carefully about the methodologies to be used while creating the strategy, scrutinize what will they result and which fit well for the specific context.
Example Entity Framework Sync
Entity Framework Sync is a library to enhance Entity Framework with synchronization features. I've used the following test strategy for it:
Here you can see number of project specifics. Unlike a website or an application, this library is created in such a way that it can be installed in many applications.
As it is a data access component, it is quite critical for the application, since an error might cause data loss. Therefore, this component must be fundamentally stable, whereas here mainly unit tests are used to test the individual functions.
However, thereby usability is not at all ensured in real applications. For this, test applications must be constructed. If possible, the library should be checked directly with available real applications.
Further to test this library, I have carried out a risk-based analysis. Something that I have not done yet in any other project. This is a very dry, but immensely structured approach which has worked wonderfully in this project and has identified important weaknesses, but is also quite expensive.
In addition to the actual test strategy, I personally like to use a visual representation which is a kind of cross between test strategy and test plan.
For this library, it is important to design the core aspects of the test.
Entity Framework provides several "styles" on how to program it. Therefore, the functionality must be tested effectively for each "style".
Also, it is useless if the functionality runs in a primitive data model, but not in a data model that uses all the possibilities of Entity Framework.
Example Quality Spy
Quality Spy itself can be considered as a good example for a test strategy. Since it is a desktop application, a completely different test approach is useful than for a basic component like EF Sync.
Here, the main test is done manually. It is important that the GUI is tested on different operating systems. In Windows 8, even the functionality should be completely tested once again since there the application runs under .NET 4.5 instead of .NET 4, which differs in detail.
Of course, the application should not be full of bugs, but it must not be tested so intensively for stability as in EF sync. Instead, it is important here to check the suitability for different scenarios.
Also for Quality Spy, I have made a drawing, which summarizes the test:
As a whole, a proper planning is important in testing as well as in development. A meaningful test strategy forms the foundation for effective software testing.