All bespoke software projects start with a problem or a reason for existing. This reason helps build the requirements (or scope) for the project and at some point it’s going to be important to check that the end result matches these expectations. But should software testing just be a one-off task at the end of a project, or can it be used more strategically on a continuous basis?
Why should we test software?
Software testing is an investment that ensures a higher quality, more resilient, and better performing product. It saves money in the long run and leads to a more successful outcome.
Software products are complex systems made up of many component parts. When changes are made in one area there can be unexpected effects in a seemingly unrelated area. At scale, and when moving at speed, this quickly becomes unmanageable, unless you have a good testing strategy in place.
A good software testing strategy will:
1. Ensure requirements are met
By clearly defining what success looks like early in the process it becomes clear when a piece of software is behaving as expected. A thorough testing strategy gives product stakeholders confidence that requirements are being met and that the intended business value is unlocked.
2. Prevent previous errors from reoccurring
Software systems are complex and it’s possible for previously fixed issues to resurface when other changes are introduced. This type of bug is known as a ‘regression’. Testing helps reduce the occurrences of regression bugs.
3. Catch bugs early
Bugs in code are unfortunately inevitable and no amount of software testing will completely eradicate them. Effective testing does however reduce the number of bugs that get experienced by end users in production. Identifying and dealing with issues earlier in the production cycle is also easier and cheaper than resolving them later on or after release.
4. Allow product teams to move quickly
When testing is embedded into a product team, the risk of introducing bugs is reduced and so the team can have confidence to move with more velocity, knowing that quality isn’t going to suffer. This allows them to deliver more business value, quicker.
In summary, software systems are complex and a good testing strategy will result in better, more resilient products, delivered more quickly.
How can you test a digital product?
Testing is largely broken down into two categories.
Manual testing
This is any testing that is completed by a person. It involves directly interacting with the platform in the same way that the target audience will, normally via the user interface (UI). If sufficiently planned and conducted by skilled testers this will typically be the most valuable testing phase as it can be more holistic. As well as checking functional requirements a person can simultaneously catch aesthetic issues, performance glitches and other user experience improvements.
When manually testing, it’s important to create a Test Suite that covers as much of the product’s functionality as possible. For each discrete workflow or scenario you need to create a Test Case which should be well documented so that someone can replicate the test in the future. We find that keeping Test Case descriptions to be text based is helpful. Whilst it’s tempting to include screenshots, these often get out of date quickly and increase the cost of maintaining your Test Suite.
Tools like TestLodge can help you manage your manual tests. They create a repeatable process around manual testing and can even be configured to automatically raise issues in project management tools like Asana, GitHub and Jira when Test Cases fail.
Automated testing
Automated testing is a suite of techniques that programmatically ensures software is working as it should, without human intervention. This includes functional tests of smaller blocks of code, to how parts of the system interact, to more emergent properties like security or performance.
Because there is no direct reliance on a person in the team, automated tests can be run much more quickly and are typically run more frequently than manual tests. It’s usual for automated testing to be run any time a developer makes a change as part of a broader CI/CD pipeline.
What are the different types of software testing?
The following are examples of ways to test a piece of software:
Functional testing
This core testing type ensures the software delivers on the promised requirements. This is an example of manual testing where testers meticulously follow predefined test cases based on requirements, mimicking real-world user interactions.
Non-functional testing
This is a catch-all category for all testing activity that isn’t directly related to ensuring functional requirements are met. This is often a mix of manual and automated testing and covers things such as:
- Security – Scanning 3rd-party packages for known vulnerabilities, running penetration tests (aka pentest) and other activities aimed at hardening the security of a product.
- Performance – Ensuring a website loads quickly and the UI is fast and responsive. Profiling long running processes and database queries.
- Accessibility (aka a11y) – Auditing a product for compliance to accessibility standards like WCAG AA and AAA
Negative testing
A form of manual testing where testers use their understanding of the software to try and break it. This can include following unexpected pathways through the user interface, entering known invalid data or looking for other ways to throw errors. This requires a particular skill and, when combined with functional testing, gives confidence that both the expected and unexpected paths of the software are robust.
Unit testing
An automated form of software testing, unit tests focus on individual units of code (functions, modules) to ensure they function as intended. By isolating these discrete units of code, developers can make assertions about how they should behave and unit tests can then check these are true.
Integration testing
Another example of automated testing, integration testing verifies how different smaller modules of code interact and communicate with each other. This catches issues arising from dependencies or compatibility problems.
User testing
User testing is a specific type of software testing that focuses on the usability and user experience (UX) of a product. It involves real users from the target audience interacting with the software, website, app, or prototype in realistic conditions. It should be conducted early in the software process to validate ideas and approaches prior to expensive development being undertaken. It should also become part of the regular rhythm of the continuous improvement cycle to help refine and improve the product over time.
Acceptance testing
The final stage of testing before a release. Here, key stakeholders (client or internal product owner) validate if the software meets their acceptance criteria. This ensures the product aligns with their needs and expectations.
When working with a product studio, like Rareloop, who become an extension of your team this is a critical part of the testing process that only the client can do and should not be outsourced.
What sorts of digital products should you test?
All of them! But that’s not to say that the testing strategy should be the same across the board. No amount of testing is going to guarantee that issues don’t arise and it’s a balance of risk and reward.
For a website where the main aim is communication and functionality is relatively light it’s wise to lean more on manual testing. Many of the issues are likely to be visual in nature and it’s more cost effective to catch these with a skilled person. There could still be value in automating the testing of any specific integrations or unique workflows though. Leaning on automated testing for things like security, performance and accessibility will also allow you to monitor quality over time.
When it comes to bespoke web software or mobile apps the product is likely to have a much more complex set of functional requirements. As such leaning more on automated testing will dramatically improve the product’s quality. A high coverage automated testing suite that is built into the development workflow will reduce the amount of manual testing required, saving time and money.
How else can you manage quality?
Testing is a great way to manage the quality of a software product but it should be considered among other tools and processes. The following are other ways in which software quality can be managed:
- Design QA – ensuring designers have a voice in the Quality Assurance process to keep visual quality high and to make sure that UX and usability are maintained
- Pair programming – enable developers to working together in pairs to solve challenges
- Code reviews – encourage code reviews from a peer on all code changes before they are accepted in the software
- Good CI/CD pipeline and development process – leverage automation to ensure your automated tests are automatically run on all code changes. You can also prevent code changes being accepted if any tests fail
- Linting code – linting is the act of enforcing a style guide on your code. The aim is to have a consistent style across a team of developers. Consistency makes code changes easier to reason about and makes it easier to spot potential errors
- Monitoring – once software is in the wild it should be monitored to ensure it’s operating as expected. This can include things like uptime, resource usage, query volumes, response times or anything else critical to the success of the product.
Conclusion
The right testing strategy will differ from project to project and organisation to organisation. It’s important to have a mix of manual and automated tests and to balance the risks present in a product versus the time and budget needed to mitigate these via testing. Software testing should also be a continuous process built into all aspects of the software lifecycle.
If you need help implementing a resilient testing strategy, or want advice on how testing can embed a culture of quality whilst moving at speed, get in touch!
