Overview¶

A Continuous Integration (CI) aims at:

• verifying software on a day-to-day basis during development
• having a central server build and verify official releases without relying on the computer of a developer

A CI pipeline is a series of steps (called jobs) that must be performed in order to verify a new version of software. These steps are executed on a machine called a runner.

The chosen and supported CI solution is Gitlab.

This page gives general guidelines on how to set up a CI:

• Project organization
• Define your CI strategy
• CI infrastructure

Project Organization¶

The following diagram shows the dependencies between components in a Yocto-based project. The manifest is the entry point, and from the manifest are deduced needed components and their versions.

When an official version is released, tags are set on each component to identify unambiguously their versions:

However when your team is developing some components, they are not tagged but referenced by branches. If you are developing App-1 and Lib-2, the diagram would probably look like this:

CI Strategy¶

To set up your CI strategy:

• Define when the CI verifications should be run. This depends on the organization of your team (size, ...), on which components you are developing, on the size of the project, etc. Common examples are:

  • run periodically (every week, every night, every hour)
  • run automatically on events of the development workflow (such as push, merge request,...)
  • run on demand (eg: when components have reached a consistent state)

• Define what the CI should run. Common examples are:

  • verify compilation and build of all components
  • test the resulting programs
  • build/publish documentation
  • check for updates in external dependencies
  • validate syntax/format

On Welma we made the following choices:

• when:

  • on demand

• what: the following jobs are defined:

  • Build all demonstration images
  • Perform a CVEScan analysis of one image
  • Test the headless development image on all supported boards

Directories Shared between Pipelines¶

On a given runner, some directories can be shared and reused between pipelines if the path is outside the Gitlab project directory (eg: in $CI_BUILDS_DIR). It enables saving pipeline execution time and data transfer.

The following directories are worth sharing:

• DL_DIR:

  • Benefit: do not download all components at every build: save time and data transfer
  • Drawback: you won't notice if some component becomes unavailable

• SSTATE_DIR:

  • Benefit: do not rebuild all components at every build: save time
  • Drawback: you won't notice some (infrequent) issue of the build system

We recommend sharing these on a day-to-day basis, but erasing them periodically (eg: every month).

These directories can also be shared across different runners, for examples with NFS mounted directories.

CI Infrastructure¶

A typical topology of machines in a CI infrastructure is:

The arrows indicate interactions, and go from the initiator to the destination peer.

Organize your Runners¶

You need at least one runner to execute your pipeline. A runner is a machine (or a compartmentalized environment on a machine) that has computing and storage resources.

In some cases the use of several runners makes sense. Eg:

• a runner on a powerful machine (with many CPU cores, much RAM and fast disk storage), dedicated to intensive computing (such as Yocto builds).
• a runner on a machine connected to a specific device, dedicated to the testing of this very device.
• runners in a cluster, for parallel builds.

Jobs can be associated to runners via Gitlab's tags.

If you build often you may also consider sharing Yocto's SSTATE cache between runners via a NFS mount (SSTATE_DIR).

Set up a runner¶

Gitlab gives the instructions for setting up a runner on a dedicated machine:

• https://docs.gitlab.com/runner/install/linux-manually.html

The steps are:

• Install gitlab-runner software on a runner machine
• Authorize the runner to connect to Gitlab (via a registration token TOKEN)
• Authorize the runner to download needed repositories (eg: via SSH, or using CI_JOB_TOKEN)
• Start the runner

Installing a Runner on Ubuntu/amd64 via a deb package¶

Install the runner program:

curl -LJO "https://gitlab-runner-downloads.s3.amazonaws.com/latest/deb/gitlab-runner_amd64.deb"
sudo dpkg -i gitlab-runner_amd64.deb

Register it in Gitlab.com:

sudo gitlab-runner register --url https://gitlab.com --token $TOKEN

Gitlab reference: https://docs.gitlab.com/runner/

Installing a Local gitlab-runner (non root) (amd64)¶

• Install a local gitlab-runner on your PC:

  sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash
  sudo su - gitlab-runner
  mkdir bin
  curl -L --output ~/bin/gitlab-runner "https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64"
  chmod +x ~/bin/gitlab-runner
  

• Grant gitlab-runner SSH rights to git-clone the layers:

  • Have gitlab-runner create a SSH key and register it in Gitlab (as a deploy key) and in Bitbucket.
  • Register the keys of these remote servers in the runner's .ssh/known_hosts file:

    $ ssh -T git@gitlab.com
    The authenticity of host 'gitlab.com (2606:4700:90:0:f22e:fbec:5bed:a9b9)' can't be established.
    ED25519 key fingerprint is SHA256:eUXGGm1YGsMAS7vkcx6JOJdOGHPem5gQp4taiCfCLB8.
    This key is not known by any other names
    Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
    Warning: Permanently added 'gitlab.com' (ED25519) to the list of known hosts.
    
    $ ssh -T git@bitbucket.org
    ...
    $ ssh -T git@github.com
    ...
    

• Register the local runner in Gitlab:

  • Set up a runner at https://gitlab.com and get a registration token (TOKEN)
  • ./bin/gitlab-runner register --url https://gitlab.com --token $TOKEN
  • Select an executor: shell

• Start your local runner (as user gitlab-runner)

  • ./bin/gitlab-runner run

• Look at jobs running (when submitted or scheduled):

  Checking for jobs... received   job=3337025322 repo_url=https://gitlab.com/witekio/... runner=3aenWn7-
  Job succeeded                   duration_s=2.250673458 job=3337025322 project=... runner=3aenWn7-
  

Tips and Tricks, Troubleshooting¶

• Job's log exceeded limit of 4194304 bytes

This limit can be modified in the runner configuration file config.toml (/etc/gitlab-runner/config.toml or $HOME/.gitlab-runner/config.toml) with the output_limit parameter (in kilobytes):

[[runners]]
  output_limit = 50000

See also the Gitlab runners-section documentation.

• Activate package registry

The Package Registry is automatically enabled in Gitlab.

You can verify or modify this in your project: go to Settings > General > Visibility, project features, permissions > Package registry.

Ref: https://docs.gitlab.com/ee/user/packages/package_registry/