                Finite Element Discretization Library
                               __
                   _ __ ___   / _|  ___  _ __ ___
                  | '_ ` _ \ | |_  / _ \| '_ ` _ \
                  | | | | | ||  _||  __/| | | | | |
                  |_| |_| |_||_|   \___||_| |_| |_|

                           https://mfem.org

This directory contains most of the GitLab CI configuration. MFEM runs both PR and nightly testing on GitLab.

Structure

Top level

The root configuration file is .gitlab-ci.yml at the root of MFEM repo. This file only defines one stage, in which we trigger several sub-pipelines.

We use sub-pipelines to isolate the test for one combination of machine and test type.

Machines typically include:

Quartz: Intel bi-socket x86
Lassen: Power9 + Nvidia GPU
Corona: AMD GPU

Test types include:

Build and test: Spack driven build of dependencies, mfem build, mfem test
Baseline: Script driven build of dependencies, thorough testing

⚠️ The sub-pipeline design allows to add a new machine or a new test type without altering the scheduling, execution and displaying of the others.

Sub-pipelines

Each file is this directory is the root configuration file for one sub-pipeline. The naming reflects the corresponding couple (machine, test_type).

Those files define the stages and the jobs for the sub-pipeline. They also contain any configuration that cannot be shared. For the most part though, the configuration is shared and is placed in .gitlab/configs.

We try to keep scripts out of the CI config and share them among similar jobs. They are gathered in .gitlab/scripts.

Scripts

Scripts specific to the CI only are in .gitlab/scripts. It is best practice to keep scripts outside the CI configuration (no bash scripts embedded in a yaml file) because it helps with readability, maintenance and also with transition to another CI system.

⚠️ Most of the scripts there are driven by environment variables and do not have a usage function. This should be improved.

More testing

Adding a new target to a build_and_test pipeline

build_and_test pipelines rely on Spack to install dependencies. Spack is driven by Uberenv which helps freezing Spack configuration: the goal being to point to specific commit in Spack and isolate its configuration so that it is not influenced by the user environment. More documentation about this can be found in tests/gitlab.

In the end, the MFEM target for which to build the dependencies is expressed with a spack spec of MFEM, within the limits permitted by the MFEM spack package.

In any build-and-test sub-pipeline a job basically consists in defining the spack spec to use. Adding a job on quartz for example resumes to:

<job_name>:
  variables:
    SPEC: "<spack_spec>"
  extends: .build_and_test_on_quartz

The remaining and non trivial work is to make sure this spec is working. To test a spec before adding it, or reproduce a CI configuration, please refer to tests/gitlab/reproduce-ci-jobs-interactively.md.

⚠️ It is assumed that the spack spec applies to mfem@develop. That's why in the CI all the specs start with the compiler or the variants to apply to mfem. The mechanism still works with a full spec.