Run Cypress Test parallel without Cypress Cloud

Cypress is an excellent tool for end-to-end testing, offering the valuable feature of parallel test execution to optimize testing time. However, relying on Cypress Cloud for parallel execution requires a paid subscription. In this article, we will explore an alternative approach to running Cypress tests in parallel by utilizing the matrix feature available in GitHub Actions. This method not only saves costs but also enhances the efficiency of your test suite. The GitHub repo used in this post https://github.com/Binhlex/cypress-parallel/tree/master

Prerequisite

First, we need to install Cypress with the below command in Nodejs

npm i –D cypress

(Note: The current Cypress version is 12.13.0 at the time of writing.)

Additionally, create a set of sample tests for demonstration purposes. In this example, we will use six test specs from the default Cypress examples.

Run parallel tests in a single machine

In this section, I will show how we run parallel tests on a single machine.

Install package ‘Cypress-parallel’ https://github.com/tnicola/cypress-parallel

npm i cypress-parallel -D

Install ‘cypress-multi-reporters’ https://www.npmjs.com/package/cypress-multi-reporters

npm install cypress-multi-reporters --save-dev

Add two scripts below to package.json

"scripts": {
    "cy:run": "cypress run",
    "cy:parallel": "cypress-parallel -s cy:run -t 2 -d 'cypress/e2e/**/*.cy.js'"
  }

• -s: Script to run
• -t: number of threads
• -d: path of test specs

Once the setup is complete, you can run the tests sequentially or in parallel using cypress-parallel and analyze the timing and resource utilization.

Note: Cypress does not recommend running parallel tests on a single machine due to increased execution time and resource consumption. We will analyze the results to understand why it is not recommended.

Sequential test execution:

npm run cy:run

As seen in the picture above, it takes 42 seconds to complete 38 tests, let us now use cypress-parallel in two threads.

npm run cy:parallel

The test with two threads took around 54 seconds to complete. Now we repeat the test with four threads.

npx cypress-parallel -s cy:run -t 4 -d 'cypress/e2e/**/*.cy.js'

This time, the execution duration was 57 seconds, indicating that running cypress-parallel in a single machine did not assist to lower the execution time while running sequentially. Parallel execution took 54 seconds in two threads and 57 seconds in four threads, whereas sequential execution took just 42 seconds. Furthermore, during simultaneous testing, CPU utilization got up from 2% to 80-100%.

In conclusion, it will take more time and resources used if we run parallel tests in a single machine. On next section, I will show the way to run parallel test by utilizing GitHub Action.

Run parallel tests by utilizing GitHub Action

On GitHub Actions, we will take advantage of the matrix features. Create a workflow file (.yml) with two jobs install and tests in it: .githubworkflowsparallel-test.yml

name: cypress parallel
on:
  push:
    branches: master
jobs:
  install:
    runs-on: ubuntu-latest
  tests:
    runs-on: ubuntu-latest
    needs: install

Add the necessary steps into the install and test jobs.

name: cypress parallel
on:
  push:
    branches: master
jobs:
  install:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Install Cypress
        uses: cypress-io/github-action@v5
        with:
          runTests: false
  tests:
    runs-on: ubuntu-latest
    needs: install
    steps:
      - name: Checkout
        uses: actions/checkout@v3

The actions/checkout@v3 to checkout code from the repository, cypress-io/github-action@v5 sets up Cypress, installs dependencies and caches them between runs. In the install step it uses runTests: false since we only want to set up the Cypress, not run tests just yet.

We now leverage Github Action’s matrix feature to run tests in parallel; instead of using the flag –parallel from cypress-io/github-action@v5 as we usually do with Cypress Cloud, we specify the spec using the matrix feature. The spec files will be placed in matrix variable containers.

In the first run, I will only put two spec files actions and aliasing to matrix containers.

name: cypress parallel
on:
  push:
    branches: master
jobs:
  install:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Install Cypress
        uses: cypress-io/github-action@v5
        with:
          runTests: false
  tests:
    runs-on: ubuntu-latest
    needs: install
    strategy:
      fail-fast: false
      matrix:
        containers: [actions, aliasing]
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Run Test Parallel
        uses: cypress-io/github-action@v5
        with:
          spec: cypress/e2e/examples/${{matrix.containers}}.cy.js
          browser: chrome

Commit and Push the workflow to a repository on Github, the Github Actions will trigger the jobs, two instances of Ubuntu will be created and run a test in each instance.

Now we added all 6 tests to the workflow and push them to GitHub.

matrix:
     containers: [actions, aliasing, assertions, cypress_api, viewport, viewport]

The total execution time of 6 tests is around 43 seconds, which is comparable to the time spent when executing tests sequentially. The longest test took 18.38 seconds, which means it only took 18 seconds to execute all 6 tests in parallel.

For more information, Github allows running Github Actions for 2000 minutes and 20 concurrent jobs monthly for a free account. A job matrix can generate a maximum of 256 jobs per workflow run. This limit applies to both GitHub-hosted and self-hosted runners.

In the real world, we have a large number of features and test spec files in our test project. How can we organize and run them together? I’ll utilize the same six tests as previously, dividing them into two folders labeled ‘group1’ and ‘group2’.

And edit the workflow:

Conclusion

name: cypress parallel
on:
  push:
    branches: master
jobs:
  install:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Install Cypress
        uses: cypress-io/github-action@v5
        with:
          runTests: false
  tests:
    runs-on: ubuntu-latest
    needs: install
    strategy:
      fail-fast: false
      matrix:
        groups: [group1, group2]
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Run Test Parallel
        uses: cypress-io/github-action@v5
        with:
          spec: cypress/e2e/${{matrix.groups}}/*.cy.js
          browser: chrome

We defined variable groups in the stratety.matrix and pass the test folders as an array. At the spec section, we passed the variable matrix.groups to specify the test spec folders. When pushing this workflow to Github, it will create two Ubuntu instances, and each instance will run all tests in each folder.

We may run Cypress tests in parallel without using Cypress Cloud by utilizing the matrix features in GitHub Actions. This method not only saves money for your project but also shortens the execution time. The execution duration of each instance is determined by the amount of test spec files held in the folders (groups). As a result, it is critical to organize the number of test specs among directories in such a manner that their execution times are similar.