Technical Solution¶

Goals¶

The solution proposed here consists of leveraging the GitLab CI to build Yocto BSP while respecting the following guidelines:

• Support multiple CI configurations.
• Provide templates to simplify the declaration of a Yocto BSP build to a few lines.
• Support triggering builds from dependent repositories.
• Condition execution of CI builds.
• Allow parallelization of the builds.
• Allow users to participate by providing a mean to declare a build configuration.
• Ensure users can proceed with their builds without waiting for approval of a maintainer.

Architecture¶

Organization and content¶

A dedicated repository is used to host the CI. It contains a default branch and a set of special branches where users define their builds:

main (default branch)
├── .gitlab-ci.yml      GitLab CI configuration file maintaining the infrastructure.
├── containers          Container(s) supporting the CI infrastructure.
│   └── trigger-build
│       ├── build       Python utility to determine which branches to build.
│       └── Dockerfile
├── snippets            Reusable YAML snippets for CI configuration files.
│   ├── common.yml
│   └── welma.yml
└── templates           GitLab CI configurations externally usable as CI/CD configuration file.
    └── trigger.yml     Preset GitLab CI configuration to trigger this CI from external GitLab
                        repositories.

config/*
├── .ci-trigger.yml     GitLab CI rules to determine when to build this branch when externally 
│                       triggered (optional).
└── .gitlab-ci.yml      GitLab CI configuration defining what to do (e.g.: Yocto BSP build).

Configuration of the CI is done in the config/* branches, through the standard .gitlab-ci.yml file, where users can include YAML snippets from the default branch. This ensures sane defaults, simplifies the declaration of a Yocto BSP build, and provides complete flexibility as to how the build should happen.

Users can individually trigger each configured build by running the GitLab CI at the desired branch.

Automated building (optional) is achieved through external triggers from other GitLab repositories. To support this, we are providing:

• A Python script called build, part of a trigger-build container, managed by the CI configuration of the default branch.
• A preset GitLab CI configuration file (i.e.: meant as .gitlab-ci.yml) usable in external repositories...
• ... and leveraging include-able YAML snippets to support customization.

This architecture is self-contained, compliant with GitLab CI syntax and flows, and compatible with GitLab APIs, CLI, and UI.

Recommended branch naming¶

The name of each CI configuration branch should start with the configurable prefix (default: config/) used by the external trigger system.

We do recommend the following naming conventions and policies:

Those conventions ensure that:

• Developers can create ephemeral builds, maintain and customize them.
• Developers do not require approval for their CI build to run.
• Non-user-branches are under strict control (through merge requests).
• Testing of the CI build is made possible without impacting the main builds.

The trigger system¶

A Yocto BSP source being a collection of multiple Git repositories (e.g.: layers, recipes' source code) and in order to achieve automated builds, we require a mean to trigger a build when a change occurs in any of those repositories. We also desire to have information as to who and what triggered the build.

Note that we are using a dedicated GitLab repository to contain the CI configuration file defining the Yocto BSP build. One of the benefits is to provide a unique CI pipeline instead of duplicating it in each repository (and having to keep those in sync.)

As such, we have developed a mean to optionally trigger this dedicated CI repository, using the downstream pipelines functionality of GitLab CI.

How to setup the automated trigger¶

The repository that needs to trigger the CI is configured to use our trigger system. There are two options:

1. Go in the repository settings, from the sidebar: Settings / CI/CD / General pipelines, and set the CI/CD configuration file option to use the template provided by this project.
2. Create your own .gitlab-ci.yml file and optionally leverage the YAML snippets provided to define a trigger job in your pipeline.

The first solution allows not to pollute the source code of your repository (the CI configuration is externally located). The second accommodates cases where a GitLab CI pipeline already exits.

The one other action is to create a CI variable named TRIGGER_PROJECT_PATH, from the sidebar: Settings / CI/CD / Variables, defining where is the main CI to trigger.

How it works¶

As per the standard GitLab CI architecture, the repository's pipeline is executed upon various events during which we trigger another repository pointed by TRIGGER_PROJECT_PATH. This is implemented with two jobs, described here below:

We first determine which branches of our main CI we need to trigger. This is the role of the trigger-build container and the build Python script mentioned above. The result of this script is a YAML file, written in GitLab CI syntax, for use with the downstream pipelines feature of GitLab CI.

"Determine configurations":
  image: ${CI_REGISTRY}/${TRIGGER_PROJECT_PATH}/trigger-build:latest
  stage: "Trigger CI"
  script: build generated-pipeline.yaml
  artifacts:
    paths:
      - generated-pipeline.yaml

The build script will:

• Open the CI repository.
• Find all the branches whose name match a configurable prefix (default: config/).
• For each, create a GitLab CI trigger at that branch...
• ...and optionally adds the rules: and other settings defined in that branch's .ci-trigger.yml file (GitLab syntax).
• Define environment variables that can be passed down to provide information about the trigger event.
• Output the result as a standard GitLab CI configuration file.

The second part is simple, we execute that configuration file.

"Trigger builds":
  needs: [ "Determine configurations" ]
  stage: "Trigger CI"
  trigger:
    strategy: depend
    include:
      - job: "Determine configurations"
        artifact: generated-pipeline.yaml

How it looks like¶

Note the following:

• Here, we have a single configuration branch (config/kirkstone-next) with no special conditions as to when it should be built:
  • The .ci-trigger.yml file does not exists or does not define rules:.
  • No rules: in the branch's .gitlab-ci.yml CI configuration file.
• It only defines two jobs: build:qemuarm-welma and build:raspberrypi3-64-welma.
• The upstream / parent, is the repository triggering (here meta-welma).
• The downstream is the main CI repository.
• Due to the use of strategy: depend, the downstream failures are reported upstream.
• There is documented and visual link between the upstream and downstream repositories.

This means that you can leverage information about the build (for instance in merge requests) in the upstream repositories where your source code is changed and trace failures down to the CI pipeline (programmatically or through the UI).