Build a full CI pipeline for a microservices project (monorepo) using GitHub Actions

In today world, continuous integration (CI) plays an important role in software projects. There are many tools that help developers create and manage CI pipelines such as Azure Pipeline, GitHub Actions, Jenkins, GitLab CI, CircleCI etc. In this post I will show you how we have built a full CI pipeline for a microservices project organized in a monorepo.

Generally, there are two ways to organize the source code for microservices projects.: multi-repo and monorepo. Multi-repo means that there are multiple repositories to host the project, each micro-service hosted in its own repo while monorepo means that we only use one repository to host all the source code for all the micro-services of the project, and each micro-services usually organized in its own folder. Each approach has its own pros and cons. But we do not go deep dive into this in this post.

Let talk about GitHub Action, GitHub Actions helps you automate your software development workflows from within GitHub. You can build, test, package, release, or deploy any projects on GitHub with a workflow. It’s total free for public projects in GitHub.

Now I am going to show to how we have used GitHub Action to build a full CI pipeline: build, test, run code analysis, run unit test, show unit test report, build docker image and push the docker image to GitHub Packages

In GitHub, all the GitHub Actions workflows need to be put in `/.github/workflows` folder. We can have multiple workflows for 1 repository. A workflow is a .yml file where we can code all the steps we needed. Let take a look at the excerpt below.

name: product service ci

on:
  push:
    branches: [ "main" ]
    paths:
      - "product/**"
      - ".github/workflows/actions/action.yaml"
      - ".github/workflows/product-ci.yaml"
  pull_request:
    branches: [ "main" ]
    paths:
      - "product/**"
      - ".github/workflows/actions/action.yaml"
      - ".github/workflows/product-ci.yaml"
      
  workflow_dispatch:

We use the `on` keyword to specify what event will trigger our workflow. Here we trigger the workflow when there are pushes to the main branch. As we organized the project in a single monorepo, we need to specify the paths, the workflow only run when there are changes in that paths. That mean developers push code to the order folders doesn’t trigger the workflow of the product. Next, we also would like to run the workflows in the pull requests to make sure the code changes in pull request pass all the requirement before being merged. Finally, with `workflow_dispatch` we allow the workflow can be trigger manually in GitHub UI

Next, we will define jobs in our workflow. One GitHub Actions workflow can contain many jobs which run parallel by default. Each job runs inside its own virtual machine (runner) specified by `run-on`. In our case we only need one job. In the job we can have many steps. Each step is either a shell script or an action. Steps are executed in order and are dependent on each other. Since each step is executed on the same runner, you can share data from one step to another. For example, you can have a step that builds your application followed by a step that tests the application that was built.

jobs:
  Build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0  # Shallow clones should be disabled for a better relevancy of analysis
      - uses: ./.github/workflows/actions
      - name: Run Maven Build Command
        run: mvn clean install -DskipTests -f product
      - name: Run Maven Test
        run: mvn test -f product
      - name: Unit Test Results
        uses: dorny/test-reporter@v1
        if: success() || failure()
        with:
          name: Product-Service-Unit-Test-Results
          path: "product/**/surefire-reports/*.xml"
          reporter: java-junit
      - name: Analyze with sonar cloud
        if: ${{ github.event.pull_request.head.repo.full_name == github.repository || github.ref == 'refs/heads/main' }}
        env:
          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
        run: mvn org.sonarsource.scanner.maven:sonar-maven-plugin:sonar -f product
      - name: Log in to the Container registry
        if: ${{ github.ref == 'refs/heads/main' }}
        uses: docker/login-action@v2
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - name: Build and push Docker images
        if: ${{ github.ref == 'refs/heads/main' }}
        uses: docker/build-push-action@v3
        with:
          context: ./product
          push: true
          tags: ghcr.io/nashtech-garage/yas-product:latest

The first step in our workflow is checkout the source code. This is done by using the action `actions/checkout` version 3. The next steps we reuse some actions defined in the https://github.com/nashtech-garage/yas/blob/main/.github/workflows/actions/action.yaml file which will setup Java SDK 17 and some caches to improve the build time. We build the source code by run Maven command; run test, and export test result to be showed in the UI. There is a limitation that if there are many GitHub Action workflows are triggered by one git push, the test report is showed only in the first workflow. The issue is reported here https://github.com/dorny/test-reporter/issues/67

We use SonarCloud to analyze the source code. SonarCloud is free for open-source projects. To authenticate with SonarCloud, we will need the SONAR_TOKEN. After register an account on SonarCloud and add our GitHub repo to SonarCloud, we can get the SONAR_TOKEN. This SONAR_TOKEN needs to be added to repository secret in GitHub. In the repository, go to Settings –> Security –> Secrets and variables –> Actions and add new repository secret. Because the security reason, the SONAR_TOKEN is not available in pull requests from forked repos. We added the `if:` statement so that this step only run on the main branch or pull requests created from within our repo not from a fork. The SonarCloud bot will add the scanning report to every pull request as image below.

The final steps are login to GitHub Packages, build and push the docker image to there. We only build and push docker image when the workflow is run in the main branch not on pull requests.

To improve the code quality of the project, we have configured that every pull request needs to pass certain conditions: build success, pass sonar gate and have at least 2 developers review and approved, otherwise the Merge button will be blocked.

You can see all of this in real life at https://github.com/nashtech-garage