What is upgrade testing?
Upgrade testing involves testing an upgrade to existing application to make sure that application is working without any issue when upgrade from existing version to new version. An application usually needs to be tested that an upgrade can be installed on an existing user’s device, and also to verify user’s data will be retained after upgrade to new version.
Why do we need upgrade testing?
Upgrade testing is very essential and important in mobile application testing. This process helps ensure that the application doesn’t crash after the upgrade, as the development team might have provided a mismatched version number. Besides, data retention with upgrades is critical in the exact preferences the user had saved in the previous versions to avoid hassles while using the app. For example, in e-commerce apps, the users save a considerable amount of data like payment details, delivery addresses, contact information, purchase history, message and much more, which must be appropriately retained with every upgrade.
When do we perform upgrade testing?
Upgrade testing should be a part in regression testing, where this is a case that need to run in each release. In fact, many things can happen with app updates. Based on the changes, some updates are obviously less likely to cause issues than others. Being informed about the changes going into the release can help identify the risks and areas to focus on.
How can we do upgrade testing?
Testing the application with many upgrades scenarios help to identify as much as user cases that could happen in UAT testing stage or in production environment. When testing mobile application, upgrade testing needs to be tested on device that has a previous version installed.
Here are some scenarios that need to be covered when perform app upgrade testing.
– Upgrading Multiple App Versions
Before testing an upgrade for a version will be released, selecting app versions are currently being used in production and see the percentage of the user base is using each version.
The scope of update testing depends on how many versions need to be tested. If the last app update was several months prior, chances are any active users are all on the latest version by now. If the last update was only a few days ago and updates were released every week prior to that, then there is a significant percentage of users on multiple versions. This will increase the scope a bit.
Assuming there are multiple versions to test, the logical place to start is with the version that has the highest usage percentage. Users who have auto-updates on would have the latest version, so this is likely the most used version. This is where upgrade testing needs to focus the most attention because issues with this upgrade would affect the biggest audience, and likely the most frequent users.
– OS Versions
After figured out the priority of app versions to test with the latest build, it’s time to test upgrade on multiple OS versions. Turning to analytics will once again give the data needed to decide where to focus testing. The OS version with the highest percentage of users, and then the app version with the highest percentage of users, will be the area to focus on. This would be the scenario to look at on multiple devices in different states. With the right set of test cases and diligent testing of upgrades for this set of users, it can be assured this user base won’t be affected.
With any OS update, it’s always best to start early. When a beta version is available, it’s good to do a sanity test on that OS version with each app update. Testing around that version then ramp up as it gets closer to a public release.
– Latest and Minimum
When testing app upgrades, another important test is the verification of upgrading with the minimum OS version and the most recently released OS version.
The minimum OS version that is supported is at high risk for issues because some new app features may not be supported by the minimum supported version. The issue could manifest itself in a crash after the update.
On the other hand, it should be considered with the latest OS version need to be tested. Depending on the season, it could be very few users or a very high percentage of users. Even if the latest Android OS update was only just released and few users have it, it is another area where the risk is high. The OS version wouldn’t have been tested in public and there are still likely to be tweaks and bug fixes being worked out. Obviously, issues such as these need to be discovered and mitigated before the user base of that OS increases rapidly.
– Important States and Scenarios
When testing app updates, there could be a number of states to look at depending on app functionality and desired behavior after an upgrade. Here are a few that would apply to a lot of apps:
+ Authentication – If application uses authentication in any way, it is imperative to ensure the user remains authenticated. Nobody enjoys launching an app and being required to authenticate after an update.
+ Customization – Depending on the app and the customization options, this could be a big one. Apps that include favorites or UI customization shouldn’t be lost.
+ Notifications – Verify that notifications are still received, and that notification settings have not been affected following the upgrade.
+ New or updated functionality – New screens, and especially those that use an updated third-party SDK version, should be viewed after an upgrade. Usually, the app won’t crash if it is a clean install but will crash after an upgrade.
What are defects can be found during upgrade testing?
During testing app upgrades, here are some popular defects can be found in both iOS and Android devices:
– Unable to install new/updated application version on top current/existing/old version during upgrades: after downloaded new version, click Install then it will show message such as “Unable to uninstall this version on top current version”, usually it’s caused by mismatch library using in application or system operation version has some updates between current and new application.
– App is crashed after upgrades: after installed new version on top current successfully, app is crashed immediately when open new app because new version has some configuration is missing/incorrect in code that leads to crashes.
– Data is not retained after upgrades: except some data is not required to retained as that is requirement, all user’s data should be retained after upgrades, for example after upgrades user is still able to read messages/emails in inbox or view ordered history or keep current point and membership level.
– Configuration/Setting is incorrect after upgrades: it is not good if asking user to login again every time after upgrades unless it is required as relating to high security application, or it should keep correct setting of default address as shipping address, or make sure that notifications are still received and that notification settings have not been affected following the upgrade, or apps that include favorites or UI customization shouldn’t be lost.
– Function is not working after upgrades: new function is not working or current function is broken as new version use an updated third-party SDK version, it should be checked if current functions are still working correctly and new function is working also to make sure not cause any crashes so that’s why we need to test upgrades in full regression testing stage.
What are challenges when perform upgrade testing?
As mentioned above, there are some difficulties when specify correct current version of application to upgrade. Usually, selecting app versions are currently being used in production or latest version if the last app update was several months ago. If the last update was only a few days ago and updates were released every week prior to that, then there is a significant testing on multiple versions. This will increase the versions need to test upgrades. When there are many versions need to be tested, it is good to test all versions but also need to identify which version should be focused on, mostly the version that has the highest usage percentage, because issues with this upgrade would affect the biggest audience and likely the most frequent users.
Besides, it is necessary to test upgrade on multiple OS versions because users are using many devices with differ kinds of OS versions. The challenge is if we have enough devices corresponding with many OS versions to test on it. If not then we have to specify which one should be focus on, for example the OS version with the highest percentage of users, then we need a device has that OS version. If we don’t have device with needed OS version, we have to upgrade that device to expected OS version. However, a device can have an upgrade OS version and not able to down-grade, so be careful when decide to upgrade OS version as there is no way back to previous version if we want to go back to test with old OS version. Ideally, testing team should have many kinds of devices with differ OS versions on each of them.
Moreover, it’s always best to start testing early with any OS update. When a beta version is available, it’s good to do a sanity test on that OS version with each app update to find any new defect if have as soon as possible. Finding defects on early testing stage will reduce the cost much better than fixing bugs when application is ready for production. Therefore, it is recommended to have a separation device using to test with latest OS update on beta version. If current device is not able to upgrade to latest OS update because the device is too old to support upgrades, then it is should considered to buy a new device able to upgrade, which is budget for that is the main thing need to be noticed.
Conclusion
When initiating testing for mobile application upgrades, it is advisable to prioritize identifying the areas posing the highest risk. Given the frequency at which mobile app updates are deployed, upgrade testing often constitutes one of the most frequently conducted test sets. As target versions evolve regularly, it is crucial to meticulously assess the version undergoing testing and to remain abreast of the latest app usage data.
References: https://blog.testlodge.com/mobile-apps-and-upgrade-testing/