By Benjamin Augustin and Frank Boylan
Software quality is a measure of the degree to which a mobile product meets its specified requirements and users' expectations, as well as the extent to which it is free of defects and represents a stable and maintainable system.
Here, we look at the key ways to ensure exceptional software quality on mobile including the pyramid of testing, performance and stability metrics, scalability and maintainability.
Unit, Integration & UI: The Pyramid of Testing in Mobile
Unit tests are a core part of a mobile developer’s arsenal, allowing us to capture and test individual components in isolation. Later down the line, this will help you detect bugs early, which is crucial before a release or hotfix, and also acts as a safety net – so you can be sure that internal architectural changes won’t break existing functionality.
You’ll often hear people talk about the pyramid of testing, with Unit testing sitting at the base. This is where you test each unit and component within your application. This level of testing is vital for maintainability because it makes it easier for people to work with your code – they know that if they change something, it won’t break any existing software, making changes much faster to test and integrate.
Next, you have integration tests which are about how the component you’re developing interplays with others. At this stage, there will be fewer individual tests because you’re looking at multiple elements working together, rather than every single isolated component.
At the top, you have User Interface (UI) testing. This tests your app from a user flow perspective, looking at how users interact with the product and validating that each user journey functions in the way it’s supposed to and that the product does what it’s designed to do.
The idea of the pyramid is that as you progress along the testing journey, the complexity and cost of implementation go up. Therefore, you can make your Unit tests as thorough as possible, then you will need fewer Integration tests and fewer UI tests.
This is vital in mobile development as, due to the complexity of different parts and how they interact with one another, UI tests can be complicated to execute because they have to be run on real devices or simulators.
UI testing for mobile products focuses on validating user interface elements, usability, user flows and journeys. These types of tests usually involve mimicking user behaviour to catch bugs that are specific to the user interface or experience. Delivering a polished interface that has been tested in every way it would be used is vital to ensuring the quality of your user experience. Ultimately, if a component isn’t going to work in isolation then it can’t be tested in isolation.
Testing & Continuous Delivery in Mobile Teams
Taking a continuous integration/delivery approach within your team means running your unit and integration tests with every single pull request (PR) cycle. Therefore, you want the unit test to be fast enough that at least part of it can be run as part of the local developer cycle.
When it comes to UI tests, because these are longer and potentially more expensive to run (if you run them on device farms or similar) then you would ideally run them either as part of your release cycle or once a day.
Although some larger companies have put in huge amounts of effort to be able to run UI testing to a certain degree on the PR cycle, your ability to run these types of tests depends on how much time and investment you’re realistically able to make as an organisation.
Performance & Stability Metrics to Determine Software Quality
The quest for perfect performance and stability for users is never-ending. Users are always going to expect perfection – and every time new hardware is released, it pushes the limits of existing apps.
Here are some key performance and stability metrics to consider as part of your testing and feedback process.
Crash rate
When it comes to software quality, the crash rate of an app is what impacts users the most – there is nothing more frustrating than a crash causing an abrupt end to a user’s experience, which is why this is the most common reason for bad reviews.
Therefore, maintaining a low crash rate should be one of your mobile development team’s key goals. This comes back to having a complete test suite as outlined above, as the more journeys you can cover in testing, the less likely you are to release updates that result in crashes.
Startup time
Other key performance metrics to consider when it comes to software quality are time-based ones, such as startup time or time to interaction. Startup time is a key part of your user experience – if your app takes too long to load, users will bounce off before interacting with it.
As the average attention span of users decreases, load time becomes increasingly vital. Studies have found that you have a maximum of five seconds to show an interactable element before a user dismisses the app as not working.
Of course, there are some external factors at play here such as network speed, so it’s important to focus on factors which you can control. For example, depending on your audience you might have a high percentage of users on older devices, therefore you need to make sure you are still testing on these. An app may be working perfectly on your two-year-old device, but if 20% of your user base is using six-year-old devices then testing on these is crucial too.
Time to interaction
High levels of responsiveness are key indicators of software quality. Studies show that when a user engages with your app, interaction time has to be less than a second – otherwise, users suspect something isn’t working. This can mean that users have less confidence in your product, as they don’t feel like they can trust their inputs are being registered.
User reviews
One of the easiest metrics to overlook when it comes to your mobile app is user feedback – this is a great indicator of how stable your app is. It’s good practice to regularly analyse user reviews, and feedback to your team to identify where the user pain points are. User reviews can also indicate wider issues that may have been overlooked in testing and can be really descriptive, providing key test points for the QA team.
Change Failure Rate
Change Failure Rate is the rate of failure every time you make a change or perform a release. Failure can mean lots of different things – introducing a bug, a crash, or not meeting user expectations in some way.
Monitoring how often these things happen when you make a change indicates how stable your overall application development process is. If you focus on lowering your Change Failure Rate, then as you're moving forward, you can have more confidence that you can introduce change without introducing failure.
Battery usage
Battery usage is a metric that’s unique to mobile development. If an app is functioning on a device, then it’s functioning on battery power. Depending on what your app is doing, it can severely impact the battery life of a user’s device – and users can see this in their device settings. If your app is responsible for using 40% of a user’s battery then they’re just going to delete it – unless you’re providing an essential service.
API calls can have a huge impact on battery usage, but there are plenty of ways to reduce this. For example, if you don’t require real-time interaction then you can ensure data fetching happens at the same time as other applications which is better for a device’s battery life.
Often, people build apps just like websites with the mentality that if I need something, I query the API and I get it. Therefore, to reduce battery usage the key questions to ask as a team are:
- Which parts of the app need to be real-time and interactive?
- Which parts could be fetched ahead of time?
- Can we have pre-ached information in the app?
- What is the optimal time to cache information?
Data Storage
Another unique factor to consider when developing for mobile is how much storage space your app uses on a device. People with newer devices are typically going to have more storage than older devices, therefore this is something you need to consider if you are looking to target a wider demographic.
Caching is a great way to make things run faster and reduce battery usage, but if you are caching too much information then you may be taking up too much space on a user’s device. Therefore, it’s important to find this balance.
Raising Code Coverage & Reducing Code Churn
Code coverage refers to how much of your code is being covered by your tests. In the context of mobile, certain areas of code might not be as easy to test as others. For example, when you’re interacting with the UI path, you're really close to a framework that you don't have much control over because you're working on Android and iOS code. This makes these elements harder to test, meaning your code coverage may not be as high as you want.
If you can't raise your code coverage, then you will want to move complexity away from the areas that have low coverage. This way, you can keep confidence levels high when it comes to releases
Code churn is a way of measuring how often code is rewritten or changed. If you have a high level of code churn, this indicates maintainability issues as it suggests that you have a lot of interdependencies within your development process. Ideally, you want to write as much code as you can in isolation, so that when you introduce a new feature you don't have to go back and make major changes to ensure it works. Not only is this inefficient, but it also introduces more risks of bugs.
Scalability of Your Mobile Software
When you’re scaling a product, you might experience a rapid boom in your user base the same way you would experience a sudden increase in website traffic. Essentially, this means that a lot more people are trying to access your back end – and you need to ensure your systems and processes can handle it.
Planning for Increased API calls
API calls are a key factor here. For example, we worked with a newspaper app that wanted to make sure all devices received the new edition every day. That meant a boom in users would result in an increase in API calls to their endpoint. As all the API calls were happening automatically at a given time, we needed to prevent them from inadvertently organising a denial of service attack on their own infrastructure.
The answer for us was not to acquire the normal endpoint for this specific type of automated syncing. Instead, we did it in a specific place that was resilient to high traffic and high load.
This is just one example of how to consider scalability when your app interacts with the rest of your infrastructure, ensuring you think about how it functions and planning for periods of rapid growth in advance.
Reducing Tech Debt
Tech debt refers to the implied cost incurred when businesses do not fix problems that will affect them in the future. Ultimately, accruing tech debt causes existing problems to get worse over time. And the longer debt builds up, the more costly it becomes to rectify – which can prove problematic when it comes to scaling your product.
The key thing is to find the right balance for your product and for your team. You need to ensure that you aren’t over-engineering things at the start, but also that you aren’t compromising your efficiency by having to continually respond to fixing unforeseen issues.
Maintaining legacy versions
Following a new release, you will have 100% of users updating to the newest version of your app. Therefore, you need to maintain and support legacy versions of your app. As you grow and scale, this becomes a compounding cost because the more versions you have out there, the more effort you have to expend ensuring you aren’t breaking compatibility with those older versions.
Eventually, if it’s costing too much, you will have to decide to stop supporting those older versions and force users to upgrade. However, when you do this, you're potentially losing a percentage of your user base. Therefore, it’s important to weigh up the impact of this versus the cost of maintaining older app versions.
Maintainability of Software
Documentation
Insufficient documentation is one of the main system maintenance issues mobile teams face. And the impact of poor documentation is amplified in teams with a high turnover rate or where knowledge silos are common.
Without appropriate documentation, you run the risk of key information being lost. Ultimately, this delays and prolongs critical bug fixing, reducing a team’s productivity and potentially causing critical quality issues for your app.
Code ownership
One of the biggest issues we’ve seen as teams scale is a lack of ownership around code. When you start scaling, you’ll likely have different teams working on different aspects of your product. If you’re scaling fast, your team might build the things they need, thereby creating pieces of code as they go that aren’t directly inside products.
Without ownership, if a bug happens in this piece of code later down the line and you don’t know who's responsible or able to fix it, then this can be a recipe for disaster. It can easily lead to a difficult team dynamic as people argue over ownership, slowing down the response to fix the bug.
Therefore, making sure that you have defined ownership of code is key. As a rule, if nobody is going to be maintaining it long-term, don't build it. This will drastically improve the maintainability of your system.
Modularity
Another hallmark of a system that's easy to maintain is modularity. This refers to keeping parts of a project separate or decoupled when they are related to and interact with each other. A clean, well-defined module, complete with clear responsibilities and dependencies. is easier to extend in the future.
Modularity involves your team agreeing to follow a clear procedural structure, ensuring everyone's on the same page with how it should look and how the different modules should interact with each other.
