NashTech Insights

A random crash on mobile app due to large database operations

Hoa Nguyen Thi
Hoa Nguyen Thi
Table of Contents
Mobile app with offline storage feature High Level Architecture

How large database operations made a mobile app crash randomly? How did we investigate this issue? What was our solution? What was the results and lessons learned? Following is our journey to solve this hard-to-reproduce issue and lessons we learned along the way.

The context

Investigation actions

Because it was a production issue that needed to be solved asap, we performed multiple actions at the same time:

Investigation results

Mobile app with offline storage feature
Mobile app with offline storage feature

After many long-hour investigation days, intensive back and forth communication with client, we found out the following results:

  • The Sentry integration still works fine with other intended testing crashes. So there is no problem with Sentry service.
  • The app occasionally crashes on one of our old device when running with UAT environment data. This is a hint to investigate more on low power devices and differences in data on production environment.
  • The app randomly crashes after a lot of user interactions in about 30 to 60 minutes. It consumes a very high amount of memory (approximately 1GB). This makes the app can be killed by the OS without a chance to send the crash report to Sentry.
  • The app has offline feature that implemented by using a local database syncing data with the server. Most of the code accessing database has optimized for performance with bulk delete, bulk update, and filtered query operations. However, there are some places in code that load all tables records from the database to in-memory variables. Then each screen filters the needed data from those variables. This seems to be a performance optimization strategy by in-memory caching that the over 2 year-old codebase had applied. Following is the simplified code to illustrate this strategy
```javascript code
// load all table data once after database has synced with backend system
const globalTableItems = databaseClient.getAllItems("tableName")

// filter data used for each displaying screen from in-memory global variables
const screenItems = globalTableItems.filter(item => isValidItemForThisScreen(item))
  • The app works well when the database is small like in development and staging environment. When crashing issue appeared in the UAT environment, we found some big database tables, including
    • a table having more than 30,000 records (approximate 7 MB database file storage)
    • a table having more than 500 large text records of logging APIs calls (approximate 15MB database file storage)
    • and a table with nearly 2,000 records
  • The inflated in-memory objects for those database records can lead to a huge in-memory consumption. Though, the actual used data for displaying on UI or other operations is quite small. This is the root cause of the high memory and CPU used in app that we measured out. This results in crash issue when the consuming memory exceed the limitation OS allows.
  • Another inspected code is in the implementation of bulk update large amount of records (1000 records) to the database. This code can also consume large memory at a time and cannot release objects in time.

The solution

With the investigation results, we have analyzed the app architecture and applied following changes to the code:

  • On the tables with large numbers of record, we filter the data using database query instead of filtering in-memory data. This helps to dramatically reduce memory and CPU used by the app.
```javascript code
// filter data used for each displaying screen from database directly
const screenItems = databaseClient.find("tableName", {condition: filterConditionForThisScreen()})
  • On the logging table having largest data storage, we cleared the whole table after reviewing the business needs with client. It turned out that this logging data was unnecessary due to redundant with integration of Sentry cloud service. The logging code also already reported as potential security issue due to logging data without consideration of masking sensitive information.
  • We reduced the bulk update size to reasonable values (from 1000 to 200 records). It causes no differences in human perception while reduces the memory consumption more quickly
  • Also, we optimized the backend API to only return necessary and valid data so that mobile app database become smaller

Impact of the solution

We tried our best to apply the smallest changes to the codebase due to the urgent of the production issue. However, the impact of the changes still quite large. Multiple screens using those global variables will be impacted. We need to test thoroughly.

Outcome of the solution

The outcome of the solution is quite tempting:

  • The random crash issue was fixed.
  • The app consumes less memory and CPU, runs more effectively without noticeable user experience trade-off.
  • The offline local database storage was optimized and reduced about 3.5 times from over 21 MB to about 6 MB

Lessons learned

Here are some lessons we learned through the progress of solving this issue

  • The large amount of data in read/write database operations can consume huge memory resource of mobile devices. It can result in crashing the app, especially in low performance devices. To avoid this, we can apply filter to read database operations, choose the reasonable bulk update size to write database operations.
  • Sometimes, mobile apps can be killed by the OS without any chance to send crash information to the cloud service. In our case, the app was killed due to consuming too much memory resource.
  • Performance optimization strategy need to be discussed, reviewed, and recorded carefully. We should consider business needs constraints, data volumn consumed by the mobile app, and collaborate well with backend system APIs.
  • Data logging needs to be taken more seriously for security, performance, and data consumption aspects.
Hoa Nguyen Thi

Hoa Nguyen Thi

With over 10 years of hands-on experience in the mobile app development industry, I am passionate and obsessed with creating high-quality software, especially mobile apps. My focus is not only on satisfying our users and customers with our products but also ensuring that our development teams, including myself as a part of them, feel proud of the work we produce, even after many years have passed.

Leave a Comment

Your email address will not be published. Required fields are marked *

Suggested Article

%d bloggers like this: