Maestro – Orchestrate your Mobile Testing Flow – A Comprehensive Guide

Mobile application development is experiencing unprecedented growth. Ensuring the quality and success of mobile apps requires meticulous testing and the use of automated testing tools. Selecting the right tools involves a set of factors to be considered like reducing flakiness, runtime, and simplicity. If you have a simple mobile project and need an easy way to automate UI testing, Maestro is an excellent choice.

Maestro, developed by mobile.dev, is the simplest and most effective keyword mobile automated test framework built on the learnings from its elder predecessors (Appium, Espresso, UIAutomator, XCTest). It allows you to easily define your automated test script in a simple flow that any stakeholders could understand the semantics of the scripts – the user journey or E2E scenarios of the app. As mobile applications continue to evolve, Maestro provides a seamless way to adapt and ensure the quality of mobile interfaces and functionalities. With its intuitive design, Maestro empowers testers to comprehensively assess the performance and behavior of mobile apps under various conditions. The framework, introduced in July 2022, has quickly gained recognition for its ability to streamline mobile UI testing processes, positioning itself as an indispensable tool for mobile development teams.

These items below are some key points that you would love while performing mobile automated test with Maestro:

• Ease of Use: Maestro saves you additional time for writing test script without requiring knowledge of a programming language. Everything works on the fly with a simple, declarative YAML language
• Focus on User Journey: Simple, declarative YAML syntax allows us to focus on the big picture of the user E2E journey – think in a flow – instead of being a little nitty-picky on getting the tests to work with a complicated programming language
• Minimal Learning Curve: The framework does not need any compilation process before executing the test, making it very easy to learn and use.
• Simple Keywords: Maestro provides a set of declarative keywords that are simple enough to remember. Non tech-savvy person could also contribute to write some test flows right away without knowledge for a specific programming language
• Continuous Mode: Maestro also supports the Contiuous Mode (similar to Hot Reload feature in many IDEs). The framework is able to continuously monitor the change in the YAML test files and can automatically rerun the tests when they detect any changes in the test files, which can enhances the developer, debugging experience
• Open Source: Maestro mobile automation framework comes with Apache 2.0 License, so all the features are free for commercial use and modification

• Community: The framework is actively driven by the community. The community is quite small currently but it has active, helpful support from both the Maestro team and the Maestro users. Developers actively listen to the user’s problems, questions, and ideas.
• Built-in tolerance to flakiness. UI elements will not always be where you expect them, screen tap will not always go through, etc. Maestro embraces the instability of mobile applications and devices and tries to counter it.
• Wait strategy: Maestro introduces some ways to reduce the flakiness of the test script at runtime by having a smart wait strategy. The user journey works and flows like a river stream. There is no need to use the evil sleep() function in each of your test steps before getting the elements. Maestro knows that it might take time to load the content on the browser (ie. over the network) so it optimizes the waiting time for the desired elements to be displayed.
• Supported Platforms: Maestro supports running E2E automated tests for both mobile native applications (Android, iOS), cross-platform application (React Native, Flutter) and Web Views on iOS and Android
• Setup and Configuration: Simple setup for Linux. Maestro is a single binary that when we write once, it works everywhere. However, it’s quite hard to perform the setup for development on Windows. I will have a separate article to go through the installation steps on the Windows OS.
• Extensibility: Ability to customize the workflow by injecting an external JavaScript code into the test. Below are some of the scenarios we could use JavaScript:
  • Use JavaScript to send, receive the HTTP requests and responses – using Maestro HTTP package
  • Customize the test flow logging
  • Apply page object model pattern on the test flow by organizing the UI elements’s locator in JS files. This will improve readability, abstraction, grouping of the script, reduce unnecessary duplication and improve test flow maintenance, which could save you lots of time modifying, debugging issues
• CI/CD Integration: Ability to integrate with any CI platform (Azure Pipelines, Jenkins, GitHub Actions,…)
• AI Support: Starting at the end of 2023, Maestro integrates AI with LLM technology to assist in creating YAML test flows, making the process even easier.

Let’s take a quick look at a simple Maestro flow running the Twitter app on Android

# flow_twitter_search.yaml
appId: com.twitter.android
name: Twitter Search Test
---
- launchApp
- tapOn: Search And Explore
- tapOn: Search Twitter
- inputText: "@mobile_dev"
- tapOn: mobile.dev
- tapOn: Follow
- assertVisible: Following

Below is a GIF from Maestro document showing how it works

In the above flow, Maestro launches the Twitter app then performs a search. Let’s break down the flow above:

At the first line, we use the command appId to define the application Id that will be launched

appId: com.twitter.android

You can notice 2 lines below, we have the triple hyphens marker “—“. This marker will separate the flow into two sections. The sections above its is the configuration part. And the part after that marker is your workflow.

appId: com.twitter.android
name: Twitter Search Test
---

3.1 Test Configuration

For the configuration section, we can configure the following items in the YAML file:

- appId: # (mandatory) Define the ID of the application under test. The ID can be easily be inspected by Maestro Studio 
- name: # (optional) Customize the Flow name. If we don't set this keyword value, the YAML file name will be used as the Flow name
- tags: # (optional) list of tags that this Flow is part of. More information regarding tags can be found here: https://maestro.mobile.dev/cli/tags 
- env: # (optional) map of environment variables that should be made available for this Flow
- onFlowStart: # (optional) This setup hook takes a list of commands to be executed before the initiation of the flow. It can be used to initialize data, seed database or execute any scripts to prepare for the main flow
- onFlowComplete: # (optinal) This teardown hook takes a list of commands to be executed upon completion of the flow. It can be used to destroy testing data or exeucte any scripts to cleanup the system
For more info, please refer to https://maestro.mobile.dev/api-reference/configuration/flow-configuration 

3.2 Main Workflow

Next, let’s delve into our main workflow

3.2.1 Launch application

Before performing anything on the app, we obviously have to launch the application by using the launchApp command.

- launchApp

The launchApp command above also accepts other parameters. We can launch the application with a clear state, a clear keychain, assign permission to the app (like location, permission) (ref: https://maestro.mobile.dev/api-reference/commands/launchapp)

3.2.2 Interact with elements

After launching the app, we use the following action commands to interact with our system under test – tap on the Search button, input the search phrase then tap on the search result to follow the page

- tapOn: Search And Explore
- tapOn: Search Twitter
- inputText: "@mobile_dev"
- tapOn: mobile.dev
- tapOn: Follow

3.2.3 Selectors

At this point, a question might pop up in our mind. How can Maestro locate the exact location of the element in the app under test? By default, we can pass a string as an argument to tell Maestro the whereabouts of the view/component/element. After receiving the argument, Maestro will look for any elements in the application having the EXACT desired text. But that would be not enough for our use cases. What if we want to look for the third item with the same text, or to search for an element with a relative point in the screen? Luckily, Maestro provides us more options to select elements like text, id, index, point,… We can also select the element based on another element’s position (like a button on top of another button). For more information, please refer to the Selectors document: https://maestro.mobile.dev/api-reference/selectors

3.2.4 Assertions

After interacting with the elements, the next thing we must do to complete a test is to add a validation point

- assertVisible: Following

In the above command, we tell Maestro to assert if any “Following” is displayed on the screen. Currently, Maestro provides us four commands to assert the app under test: assertVisible, assertNotVisible, assertTrue, assertFalse. Details of the effect of the command are listed in the below section “Maestro Keywords List”.

To run the test, we will execute this command

maestro test flow_twitter_search.yml

After going through the flow, we can see that the command is declarative, and simple enough for any stakeholders to understand the user’s E2E journey without caring too much about what works under the hood. Next, let’s go through some key features that make Maestro a maestro.

4.1 Maestro CLI

Maestro provides users set of CLI commands for multiple purposes. We can:

• Run defined YAML test flow on an emulator, real device or on Maestro Cloud
• Filter the test flow by tags
• Generate JUnit report after completing the Flow
• Record your test flow to demonstrate the flow to stakeholders
• Launch the test in Continuous Mode to continuously monitor the test run after each modification to YAML flow
• View the hierarchy of all the elements in the application
• Query the elements in the application
• Start Android emulator, and iOS Simulator by CLI command

4.2 Maestro Studio

Maestro Studio is your personal assistant to write your Maestro test flow. With Maestro Studio, we can seamlessly:

• Inspect the element in the application
• Interact with the element on the screen
• Search for the element
• Write and execute YAML commands without running the full-flow
• Automatically generating Maestro commands after interacting with the element
• Find Maestro commands easily using natural language. The feature is available from Maestro 1.32.0
• Export the commands written in Maestro Studio to the YAML file

To launch Maestro Studio, we will execute this Maestro CLI command

maestro studio

And below is a video from Maestro showing how it works

4.3 Maestro Cloud

Maestro Cloud is the easiest way to run Maestro Flows in CI. Since our Flows run in the cloud, there’s no need to configure any simulators or emulators on the local environment. All we need to do is just upload the application under test and the Test Flows to the cloud via one of Maestro’s CI integrations, then Maestro will handle the rest of the configuration for us.

With this service, we can

• Integrate with any CI providers (ref: https://cloud.mobile.dev/getting-started/integrate-into-ci)
• Integrate with CI providers to automatically trigger the test on schedule or per pull request
• Work with secret variables in order to ensure security while development/testing
• Check the test result, download the artifacts, and screenshots after running
• View test commands, hierarchy, logs, test screen record
• Configure the desired device configuration for the Emulator/Simulator
• Retry the failed test by using Intelligent retry mechanism to improve the reliability of test results (ref: https://cloud.mobile.dev/reference/retries)
• For more info, please refer to https://cloud.mobile.dev/

5. Maestro vs Appium

Until now, Appium is still considered as the popular first choice for mobile automated testing, let’s compare between Maestro and Appium

Below are lists of Maestro keywords to get you started on the flow

- launchApp: # Launches the app under test
- stopApp: # Stops current application
- tapOn: # Taps on a selected element
- doubleTapOn: # Double taps on a selected element
- inputText: # Inputs text
- eraseText: # Removes characters from the currently selected text field
- assertVisible: # Asserts whether an element is visible
- assertNotVisible: # Asserts whether an element is not visible
- copyTextFrom: # Copies text from an element and saves it in-memory
- pasteText: # Pastes any text copied with copyTextFrom into the currently focused field
- startRecording: # Starts a screen recording
- stopRecording: # Stops a running screen recording
- scroll: # Does a vertical scroll
- scrollUntilVisible: # Scrolls towards a direction until an element becomes visible in the view hierarchy
- waitForAnimationToEnd: # Waits until an ongoing animation/video is fully finished, and the screen becomes static
- extendedWaitUntil: # Waits until an element becomes visible within a specified amount of time
- back: # Navigates the user to the previous screen
- pressKey: # Presses a set of special keys
- runFlow: # Runs commands from another file
- runScript: # Runs a provided JavaScript file
- setLocation: # Applies a mock geolocation to the device
- travel: # Mocks the motion of the user, by specifying a set of points and a speed
- clearState: # Clears the application state
- clearKeychain: # Clears the entire iOS keychain
- hideKeyboard: # Hides the software keyboard
- swipe: # Makes the swipe gesture
- addMedia: # Adds media to the device’s gallery
- takeScreenshot: # Saves a screenshot in a PNG file
- assertTrue: # Asserts whether the given value is either true or non-empty
- repeat: # Repeats set of commands N times
- openLink: # Opens a link on a device
- evalScript: # Allows specifying JavaScript directly in the Maestro flow

7. Maestro Disadvantages

There are still many disadvantages while using the framework

• The framework was introduced in July 2022, so it is a quite new framework. It lacks of additional resources and it does not have strong community support like many old existing frameworks
• Maestro does not support running on real iOS device
• Maestro does not have the capability to handle complex testing flow. Maestro supports a limited set of vanilla JavaScript APIs that impacts the possibilities of creating complicated tests. For more details, the framework uses Rhino JS Engine under the hood to execute JavaScript, which has limited ES2015 / ES6 support. None of the Rhino’s APIs allow interaction with the OS but it offers enough functionality to write more sophisticated conditions or make HTTP requests. Please refer to this page for a full list of supported commands.
• To counter the above drawback, Maestro is slowly changing from using Rhino JS to GraalJS for JavaScript evaluation. GraalJS is fully ECMAScript 2022 compliant while Rhino JS only supports ES5. This feature is still experimental but the Maestro team has a plan to roll this out as a default JS engine if all goes well
• Limited execution time on the Maestro Cloud – only 3 minutes.
• Limited reporting capabilities. Reports are generated and stored in JUnit XML format
• Limited screen recording time – only 3 minutes. As of today, Maestro has provided the fix for the three-minutes screen recording limitation. If you are curious about their fixing approach, give them a clap at https://blog.mobile.dev/fixing-androids-3-minute-screen-recording-limitation-4839abc74aee
• It’s very cumbersome to set up Maestro on Windows currently as the framework natively supports Linux OS. So if we want to use it on Windows, we have to use WSL2 (Windows Subsystem for Linux) or a Linux virtual machine. It’s very time-consuming and effort-intensive to get it working on Windows. The configuration to get Maestro under WSL2 to connect with the simulator/device on the Windows machine requires some network knowledge to ensure the connectivity between them

Conclusion

To recap, in part 1 of this series, we have gone through the overview of the Maestro mobile UI testing framework: what it is, the advantages and disadvantages of using the framework, a simple flow in action, key features, some useful Maestro commands and we also compare between Maestro and Appium
In the next part of the series, I will go through the Maestro installation steps on the Windows OS.

https://maestro.mobile.dev

https://blog.mobile.dev

https://github.com/mobile-dev-inc/maestro

https://cloud.mobile.dev

https://www.testdevlab.com/blog/getting-started-with-maestro-mobile-ui-testing-framework