Move Jenkins job configuration into the repository (#3893)

* Add a first attempt at explicitly using Jenkinsfiles. * A first attempt... * Put code in a script block. * Add first attempt at link check job pipeline. * Hopefully correct shell block. * Install git. * Install packages as root. * Use custom image that already has dependencies installed. * Try processing the JUnit results. * Fix syntax (hopefully). * Try to get the build to set its status on Github. * Try and see if I can get the snippet build to run too. * A first attempt at reviving the static code analysis build. * Refactor style check job a bit. * Try to clean up other files and have them set statuses. * Set status in script blocks. * Fix directory (this may not fix my problem). * Should I load in the script step? * Maybe I can just load it without a name. * Maybe I have my path wrong. * Will this work? Just a test... * Try using a plugin instead. * And if I define the function manually at the top? * Maybe this will fix the load. * Try to turn unstable into failed. * Hopefully fix documentation builds. * Fix script blocks. * Maybe fix static code analysis job. * Try to adapt PR number variable. * First attempt at cross-compilation job. * Try to fix some syntax. * Clean workspaces after build. * Try to put the matrix in the right place. * Another attempt at the matrix configuration. * Maybe I have to nest it deeper. * Maybe I have to clean always? * What we need is more tabbing. * Use try/catch to handle failed junit processing. * Better handling of environment variables. * Try a differernt approach than try/catch. * Try to get some more information about ccache. * Is it possible we could store the ccache at a higher level? * Maybe I have the variable name wrong. * Clean the cross-compilation workspace. * Try mounting the ccache so it can be shared across multiple jobs. * Always pull images. * We need to run on the same node. * Run on only one core. * Try building in the Docker container in a different way. * Do I have the order backwards? * Can I run anything at all in the container? * The static code analysis job isn't helpful. * Try to set the user of the docker container. * Rebuild the Docker container instead. * Always pull an updated image. * Download any necessary dependencies too. * Oops, use the correct CMake options. * Fix line break in the wrong place. * Make sure to use the correct architecture. * We can't use MATCHES, that is a regex. * Oops, we need to use STREQUAL. * Bump to an older version since newer versions don't have gfortran. * Try to run the tests on the target. * Correct syntax. * Okay, I'm not allowed to generate a stage name. * Try cleaning the workspace at the start of the build. * Okay, so I just can't depend on the workspace cleaning job, wonderful. * Try and add the passphrase correctly. * Fix path for memory checks. * Fix path to test. * Fix PR number variable. * Try to fix path for test copying. * Try to get the PR number correct. * Try and centralize where the link cache is stored. * Why is it being printed strangely? * Is there some weird restriction where this all has to be on one line? * Always publish the HTML, and fix a link. * Try to fix SSH host key check. * Make the reports directory. * Try to fix file parsing. * Try to enable ccache. * Try to set ccache directory correctly. * Try to get the full pipeline set up correctly for cross-compilation. * Fix path to test data. * Allow debug builds when cross-compiling. * Remember to unpack all the test data! * Fail tests when the data isn't there. * Maybe I can use find instead. * Double escape for backslash? * What if we just run the test? * Port Catch2 improvement for junit runner. See https://github.com/catchorg/Catch2/commit/c29e198eab0ccdb190495397854b937677385e2e. * Re-enable junit testing (hopefully it will work now). * Output directly to the xml file. * Try to clean up regex. * Try to set IN PROGRESS status. * Could it be called RUNNING? * I guess I don't get access to set jobs in progress through this API. * Fix regex for test name extraction. * Try to clean up Jenkinsfiles. * Fix parameter name. * Maybe fix syntax? * Does it work without keyword arguments? * Correctly accept named parameters. * Abort previous builds to reduce load on Jenkins. * Use optimization when compiling. * Fix syntax for abortPrevious. * Fix missing closing brace... * Update links in CI documentation and try to fix memory check job.
2025-03-12 08:34:59 -04:00
parent b5e3284cf5
commit a44fbe4906
13 changed files with 708 additions and 25 deletions
@@ -0,0 +1,161 @@
+// The cross-compilation job tests mlpack on a set of low-resource devices.
+// First, the cross-compilation host compiles mlpack tests for each
+// architecture, and then copies them to the destination host and runs them.
+pipeline
+{
+  agent
+  {
+    // Only use a node that has access to the target hosts.
+    label 'cross-compile'
+  }
+
+  options
+  {
+    // Only allow one build at a time of this job.
+    disableConcurrentBuilds(abortPrevious: true)
+  }
+
+  stages
+  {
+    stage('Set build as pending')
+    {
+      steps
+      {
+        // Set the build status...
+        script
+        {
+          u = load '.jenkins/utils.groovy'
+          u.startBuild("Cross-compilation Tests");
+        }
+
+        // Create a directory for our resulting reports.
+        sh'mkdir -p reports/'
+      }
+    }
+
+    stage('Cross-compile mlpack to different targets')
+    {
+      matrix
+      {
+        axes
+        {
+          axis
+          {
+            name 'target'
+            values 'couscous;rpi5;cortexa76'
+          }
+        }
+
+        stages
+        {
+          // Extract the hostname, the device, and the architecture.
+          stage('Extract parameters from build matrix')
+          {
+            steps
+            {
+              script
+              {
+                def components = env.target.split(';')
+
+                env.hostname = components[0]
+                env.device = components[1]
+                env.arch = components[2]
+                env.arch_upper = components[2].toUpperCase()
+              }
+            }
+          }
+
+          // Cross-compile mlpack tests.
+          stage('Cross-compilation tests')
+          {
+            agent
+            {
+              docker
+              {
+                image 'mlpack/mlpack-cross-compile-' + env.arch + ':latest'
+                alwaysPull true
+                reuseNode true
+              }
+            }
+            steps
+            {
+              sh '''
+                rm -rf build/
+                mkdir build/
+                cd build/
+                cmake \
+                    -DBUILD_TESTS=ON \
+                    -DARCH_NAME=${arch_upper} \
+                    -DCMAKE_CROSSCOMPILING=ON \
+                    -DCMAKE_TOOLCHAIN_FILE=../CMake/crosscompile-toolchain.cmake \
+                    -DTOOLCHAIN_PREFIX=$TOOLCHAIN_PREFIX \
+                    -DCMAKE_SYSROOT=$CMAKE_SYSROOT \
+                    -DDOWNLOAD_DEPENDENCIES=ON \
+                    ../
+                make mlpack_test;
+              '''
+
+              withCredentials([sshUserPrivateKey(
+                  credentialsId: 'mlpack-jenkins-cross-compile-rsa-key',
+                  keyFileVariable: 'KEY_FILE',
+                  passphraseVariable: 'PASSPHRASE')])
+              {
+                sh'''
+                  eval $(ssh-agent -s)
+                  echo ${PASSPHRASE} | SSH_ASKPASS=/bin/cat setsid -w ssh-add ${KEY_FILE}
+
+                  # Don't check the host keys, because they won't be saved in
+                  # this container anyway.
+                  mkdir -p ~/.ssh/
+                  echo 'Host *' >> ~/.ssh/config;
+                  echo '  StrictHostKeyChecking no' >> ~/.ssh/config;
+
+                  ssh jenkins@${hostname} -t mkdir -p test_${BRANCH_NAME}_${BUILD_ID}/
+                  scp build/bin/mlpack_test jenkins@${hostname}:test_${BRANCH_NAME}_${BUILD_ID}/
+                  scp -r src/mlpack/tests/data/* jenkins@${hostname}:test_${BRANCH_NAME}_${BUILD_ID}/
+                  # Unpack all compressed test data.
+                  ssh jenkins@${hostname} -t "
+                      cd test_${BRANCH_NAME}_${BUILD_ID};
+                      find ./ -iname '*.bz2' -exec tar xvf \\{\\} \\;"
+
+                  mkdir -p reports;
+                  ssh jenkins@${hostname} -t "
+                      cd test_${BRANCH_NAME}_${BUILD_ID};
+                      mkdir -p reports;
+                      ./mlpack_test -r junit -o reports/mlpack_test.junit.xml"
+
+                  # Clean up afterwards.
+                  scp jenkins@${hostname}:test_${BRANCH_NAME}_${BUILD_ID}/reports/mlpack_test.junit.xml reports/mlpack_test.${hostname}.junit.xml;
+                  ssh jenkins@${hostname} -t rm -rf test_${BRANCH_NAME}_${BUILD_ID}/;
+                '''
+              }
+            }
+          }
+        }
+      }
+    }
+  }
+
+  post
+  {
+    always
+    {
+      junit '**/reports/mlpack_test.*.junit.xml'
+
+      // Clean the workspace.
+      cleanWs(cleanWhenNotBuilt: true,
+              deleteDirs: true,
+              disableDeferredWipeout: true,
+              notFailBuild: true)
+
+      script
+      {
+        u.setBuildStatus(result: currentBuild.currentResult,
+                         context: "Cross-compilation Tests",
+                         successMessage: "Cross-compilation succeeded with no errors.",
+                         unstableMessage: "Cross-compilation build unstable.",
+                         failureMessage: "Cross-compilation failed.");
+      }
+    }
+  }
+}
@@ -0,0 +1,107 @@
+// The documentation link checker build will build the Markdown documentation in
+// doc/, and ensures that all of the links contained in the documentation are
+// valid.
+//
+// Note that a cache is maintained on Jenkins to avoid checking the same links
+// over and over again.
+pipeline
+{
+  // Run inside of the custom Docker image for style checking.
+  // Every docker agent has a 'link_cache/' directory in its Jenkins workspace
+  // for this job.
+  agent
+  {
+    docker
+    {
+      image 'mlpack/jenkins-mlpack-docbuild:latest'
+      alwaysPull true
+      args '-v /home/jenkins/link_cache/:/opt/link_cache/'
+    }
+  }
+
+  options
+  {
+    // Only allow one build at a time of this job.
+    disableConcurrentBuilds(abortPrevious: true)
+  }
+
+  stages
+  {
+    // First we have to check out the jenkins-conf repository, which contains
+    // the scripts that we will use for checking the style.
+    stage('Build documentation and check links')
+    {
+      steps
+      {
+        script
+        {
+          u = load '.jenkins/utils.groovy'
+          u.startBuild("Documentation Link Check");
+        }
+
+        sh '''
+          # Set $HOME because the Docker container may be running with a
+          # different uid.  Note that the container has /workspace/ as the
+          # working directory; we'll just reuse that as $HOME.
+          export HOME=/workspace/
+
+          # Print the size of the link cache.
+          if [ ! -f /opt/link_cache/link_cache.db ];
+          then
+            echo "Link cache does not exist!";
+          else
+            echo "Link cache current size:";
+            ls -lh /opt/link_cache/link_cache.db;
+          fi
+
+          # Skip the check if the documentation build script doesn't exist.
+          if [ ! -f scripts/build-docs.sh ];
+          then
+            exit 0;
+          fi
+
+          # This will fail if there are any issues converting the Markdown to
+          # kramdown, or if there is a linting or link-checking failure.
+          LINK_CACHE_FILE=/opt/link_cache/link_cache.db ./scripts/build-docs.sh;
+          build_doc_out=$?;
+          if [ $build_doc_out -ne 0 ];
+          then
+            echo "build-docs.sh failed!";
+            exit 1;
+          fi
+        '''
+      }
+    }
+  }
+
+  post
+  {
+    always
+    {
+      // Publish the generated HTML.
+      publishHTML([
+        allowMissing: false,
+        alwaysLinkToLastBuild: false,
+        keepAll: true,
+        reportDir: 'doc/html/',
+        reportFiles: 'index.html',
+        reportName: 'Build documentation']);
+
+      // Clean the workspace.
+      cleanWs(cleanWhenNotBuilt: true,
+              deleteDirs: true,
+              disableDeferredWipeout: true,
+              notFailBuild: true);
+
+      // Set the build status.
+      script
+      {
+        u.setBuildStatus(result: currentBuild.currentResult,
+                         context: "Documentation Link Check",
+                         successMessage: "All HTML links checked.",
+                         unstableMessage: "Problem with HTML links.",
+                         failureMessage: "HTML link check failure or problem.");
+      }
+    }
+  }
+}
@@ -0,0 +1,74 @@
+// The documentation snippet build will extract all C++ code snippets from the
+// Markdown documentation in doc/ and ensure that it compiles and runs without
+// error.
+pipeline
+{
+  // Run inside of the custom Docker image for documentation builds.
+  agent
+  {
+    docker
+    {
+      image 'mlpack/jenkins-mlpack-docbuild:latest'
+      alwaysPull true
+      args '-v /home/jenkins/ccache:/opt/ccache'
+    }
+  }
+
+  options
+  {
+    // Only allow one build at a time of this job.
+    disableConcurrentBuilds(abortPrevious: true)
+  }
+
+  stages
+  {
+    stage('Extract and build documentation snippets')
+    {
+      steps
+      {
+        script
+        {
+          u = load '.jenkins/utils.groovy'
+          u.startBuild("Documentation Snippet Build");
+        }
+
+        sh'''
+          export CCACHE_DIR=/opt/ccache/;
+          export CXX="ccache g++";
+          export CXXFLAGS="-O3 -DNDEBUG -fopenmp -I./src/ -I/usr/include/eigen3/";
+          export LDFLAGS="-fopenmp";
+          export OMP_NUM_THREADS=1;
+
+          ccache -p;
+          ls -l /opt/ccache/
+          ccache --zero-stats;
+          ./scripts/test-docs.sh doc/;
+          # Print ccache statistics.
+          ccache -s
+        '''
+      }
+    }
+  }
+
+  post
+  {
+    always
+    {
+      // Clean the workspace.
+      cleanWs(cleanWhenNotBuilt: true,
+              deleteDirs: true,
+              disableDeferredWipeout: true,
+              notFailBuild: true);
+
+      // Set the build status.
+      script
+      {
+        u.setBuildStatus(result: currentBuild.currentResult,
+                         context: "Documentation Snippet Build",
+                         successMessage: "All snippets built and run successfully.",
+                         unstableMessage: "Snippets build unstable..",
+                         failureMessage: "Snippet build or runtime failure.");
+      }
+    }
+  }
+}
@@ -0,0 +1,137 @@
+// The static code analysis build will analyze the mlpack codebase for any known
+// C++ issues.
+pipeline
+{
+  // Run inside of the custom Docker image for style checking.
+  agent
+  {
+    docker
+    {
+      image 'mlpack/jenkins-amd64-debian:latest'
+      alwaysPull true
+      args '-v /home/jenkins/ccache:/opt/ccache'
+    }
+  }
+
+  options
+  {
+    // Only allow one build at a time of this job.
+    disableConcurrentBuilds(abortPrevious: true)
+  }
+
+  stages
+  {
+    // First we have to check out the jenkins-conf repository, which contains
+    // the scripts that we will use for checking the style.
+    stage('Check out jenkins-conf repository')
+    {
+      steps
+      {
+        script
+        {
+          u = load '.jenkins/utils.groovy'
+          u.startBuild("Memory Checks");
+        }
+
+        sh '''
+          git clone https://github.com/mlpack/jenkins-conf
+        '''
+      }
+    }
+
+    // First build mlpack_test.
+    stage('Build mlpack')
+    {
+      steps
+      {
+        sh '''
+          export CCACHE_DIR=/opt/ccache/;
+          ccache --zero-stats
+
+          mkdir build
+          cd build
+          cmake -DDEBUG=ON -DBUILD_TESTS=ON -DDOWNLOAD_DEPENDENCIES=ON ..
+          make mlpack_test
+          cd ..
+
+          # Print ccache statistics.
+          ccache -s
+        '''
+      }
+    }
+
+    // Now run the memory checks.
+    stage('Run memory checks')
+    {
+      steps
+      {
+        // First get the number of the PR, as we will need to do that to see
+        // what files have changed.
+        script
+        {
+          if (env.BRANCH_NAME.startsWith('PR-'))
+          {
+            // Strip 'PR-' from the front.
+            env.PR_NUM = env.BRANCH_NAME.substring(3)
+          }
+        }
+
+        sh'''
+          # Move memory tests to the current directory.
+          cp jenkins-conf/memory/* .
+
+          # Get information about the current PR.
+          echo "PR number: ${PR_NUM}";
+          curl -o files.txt https://api.github.com/repos/mlpack/mlpack/pulls/${PR_NUM}/files
+          grep -o '^[ ]*"filename":.*' files.txt |\
+              sed -e 's/^[ ]*"filename": "//' -e 's/",//' |\
+              uniq |\
+              awk '/.cpp/ || /.hpp/' > filenames.txt;
+
+          # Debug print modified files, we try to run the memory check for those
+          # files only.
+          cat filenames.txt;
+
+          # Workaround for docker container where the ulimit is set to a
+          # strangely large number...
+          ulimit -n 1024;
+
+          # Run memory checks.
+          OMP_NUM_THREADS=1 ./run-mlpack-valgrind-tests.sh ||\
+              mkdir -p temp/test;
+
+          # Debug print tests to run.
+          cat testbins.txt;
+
+          # Cat the output...
+          ls -lh reports/tests/
+        '''
+      }
+    }
+  }
+
+  post
+  {
+    always
+    {
+      junit(allowEmptyResults: true,
+            testResults: '**/reports/tests/*.xml')
+
+      // Clean the workspace.
+      cleanWs(cleanWhenNotBuilt: true,
+              deleteDirs: true,
+              disableDeferredWipeout: true,
+              notFailBuild: true)
+
+      // Set the build status.
+      script
+      {
+        u.setBuildStatus(result: currentBuild.currentResult,
+                         context: "Memory Checks",
+                         successMessage: "No memory errors.",
+                         unstableMessage: "Build unstable.",
+                         failureMessage: "Memory check failure with valgrind.");
+      }
+    }
+  }
+}
@@ -0,0 +1,93 @@
+// The style checker build will check the style of all the code in the
+// repository.
+pipeline
+{
+  // Run inside of the custom Docker image for style checking.
+  agent
+  {
+    docker
+    {
+      image 'mlpack/jenkins-mlpack-style-checks:latest'
+      alwaysPull true
+    }
+  }
+
+  options
+  {
+    // Only allow one build at a time of this job.
+    disableConcurrentBuilds(abortPrevious: true)
+  }
+
+  stages
+  {
+    // First we have to check out the jenkins-conf repository, which contains
+    // the scripts that we will use for checking the style.
+    stage('Check out jenkins-conf repository')
+    {
+      steps
+      {
+        script
+        {
+          u = load '.jenkins/utils.groovy'
+          u.startBuild('Style Checks')
+        }
+
+        sh '''
+          git clone https://github.com/mlpack/jenkins-conf
+        '''
+      }
+    }
+
+    // Now we can run those scripts.
+    stage('Check code style')
+    {
+      steps
+      {
+        sh '''
+          mkdir -p reports
+          ./jenkins-conf/linter/lint.sh \
+              --root . \
+              --reports reports/cpplint.junit.xml \
+              --dir ./src/mlpack
+
+          # Print the results.
+          cat reports/cpplint.junit.xml
+        '''
+      }
+    }
+  }
+
+  post
+  {
+    // Mark unstable builds as failed.
+    unstable
+    {
+      script
+      {
+        error "Style check failure."
+      }
+    }
+
+    always
+    {
+      // Process the test results.
+      junit(allowEmptyResults: true,
+            testResults: '**/reports/cpplint.junit.xml')
+
+      // Clean the workspace.
+      cleanWs(cleanWhenNotBuilt: false,
+              deleteDirs: true,
+              disableDeferredWipeout: true,
+              notFailBuild: true)
+
+      script
+      {
+        u.setBuildStatus(result: currentBuild.currentResult,
+                         context: "Style Checks",
+                         successMessage: "No style issues.",
+                         unstableMessage: "Style issues found.",
+                         failureMessage: "Style issues found.");
+      }
+    }
+  }
+}
@@ -0,0 +1,66 @@
+// A simple utility to mark the build as pending on Github.
+def startBuild(String context)
+{
+  step([
+      $class: "GitHubCommitStatusSetter",
+      reposSource: [$class: "ManuallyEnteredRepositorySource",
+                    url: "https://github.com/mlpack/mlpack"],
+      contextSource: [$class: "ManuallyEnteredCommitContextSource",
+                      context: context ],
+      errorHandlers: [[$class: "ChangingBuildStatusErrorHandler",
+                       result: "UNSTABLE"]],
+      statusResultSource: [$class: "ConditionalStatusResultSource",
+                           results: [[$class: "AnyBuildResult",
+                                      message: "Building...",
+                                      state: "PENDING"]]]
+  ]);
+}
+
+// A simple utility to set the build status on Github for a commit.
+def setBuildStatus(Map paramsMap)
+{
+  // Extract arguments from the map.
+  def result = paramsMap.result;
+  def context = paramsMap.context;
+  def successMessage = paramsMap.successMessage;
+  def unstableMessage = paramsMap.unstableMessage;
+  def failureMessage = paramsMap.failureMessage;
+
+  def message = "(unknown Jenkins build result)";
+  def state = "FAILURE";
+  if (result == "FAILURE")
+  {
+    message = failureMessage;
+  }
+  else if (result == "UNSTABLE")
+  {
+    message = unstableMessage;
+    state = "UNSTABLE";
+  }
+  else if (result == "SUCCESS")
+  {
+    message = successMessage;
+    state = "SUCCESS";
+  }
+  else if (result == "ABORTED")
+  {
+    message = "Job aborted.";
+    state = "ERROR";
+  }
+
+  step([
+      $class: "GitHubCommitStatusSetter",
+      reposSource: [$class: "ManuallyEnteredRepositorySource",
+                    url: "https://github.com/mlpack/mlpack"],
+      contextSource: [$class: "ManuallyEnteredCommitContextSource",
+                      context: context ],
+      errorHandlers: [[$class: "ChangingBuildStatusErrorHandler",
+                       result: "UNSTABLE"]],
+      statusResultSource: [$class: "ConditionalStatusResultSource",
+                           results: [[$class: "AnyBuildResult",
+                                      message: message,
+                                      state: state]]]
+  ]);
+}
+
+return this
@@ -208,8 +208,10 @@ if (CMAKE_BUILD_TYPE STREQUAL "Debug" OR DEBUG)
  endif()

  # mlpack uses it's own mlpack::backtrace class based on Binary File Descriptor
-  # <bfd.h> and linux Dynamic Loader <libdl.h> and more portable version in future
-  if (CMAKE_SYSTEM_NAME STREQUAL "Linux")
+  # <bfd.h> and linux Dynamic Loader <libdl.h> and more portable version in
+  # future.  However, if we are cross-compiling, we cannot run the CMake tests
+  # for LibDL and BFD.
+  if (CMAKE_SYSTEM_NAME STREQUAL "Linux" AND NOT CMAKE_CROSSCOMPILING)
    find_package(Bfd)
    find_package(LibDL)
    if (LIBBFD_FOUND AND LIBDL_FOUND)
@@ -9,10 +9,10 @@ complex project, there are occasionally spurious failures or other unrelated
 problems.

 * [Basic compilation and test jobs](#basic-compilation-and-test-jobs)
- * [Documentation build and test](#documentation-build-and-test)
+ * [Documentation builds](#documentation-builds)
 * [Style checks](#style-checks)
 * [Cross-compilation checks](#cross-compilation-checks)
- * [Static code analysis checks](#static-code-analysis-checks)
+ * [Memory checks](#memory-checks)

 Also you can see the [list of CI infrastructure](#list-of-ci-infrastructure).

@@ -67,10 +67,10 @@ Similarly, the binding tests are also run using a local Github action.
   can parse it, sometimes we have to do strange things for some languages, and
   we can't use CTest directly.

-## Documentation build and test
+## Documentation builds

-The 'documentation build and test' job builds and tests *all* documentation,
-checking:
+The 'documentation link check' job and 'documentation snippet build' jobs build
+and tests *all* documentation, checking:

 * that all Markdown pages build and render properly;
 * that all HTML is valid;
@@ -78,7 +78,9 @@ checking:
 * that all code examples compile and run.

 All of the scripts to perform these builds are located in the `scripts/`
-directory, so that they can be run locally.
+directory, so that they can be run locally.  On CI, these are defined as two
+Jenkins jobs in `.jenkins/doc-link-check/Jenkinsfile` and
+`.jenkins/doc-snippet-build/Jenkinsfile`.

 * `./scripts/build-docs.sh`
   - Builds all documentation in `doc/` with the output directory `doc/html/`.
@@ -98,12 +100,14 @@ directory, so that they can be run locally.

 When writing new documentation, be sure to test it locally---going back and
 forth with the
-[job on Jenkins](http://ci.mlpack.org/job/pull-request%20documentation%20build%20and%20test/)
+[link checker job on Jenkins](http://ci.mlpack.org/job/mlpack%20documentation%20link%20check/)
+and [snippet build job on Jenkins](http://ci.mlpack.org/job/mlpack%20documentation%20snippet%20build/)
 can be very tedious.

 ## Style checks

-The [style checker job](http://ci.mlpack.org/job/pull-requests%20mlpack%20style%20checks/) runs on Jenkins.
+The [style checker job](http://ci.mlpack.org/job/mlpack%20style%20checks/) runs
+on Jenkins and is defined in `.jenkins/style-checks/Jenkinsfile`.

 * The [`lint.sh` script](https://github.com/mlpack/jenkins-conf/blob/master/linter/lint.sh) to check for C++ style issues.

@@ -115,26 +119,41 @@ The [style checker job](http://ci.mlpack.org/job/pull-requests%20mlpack%20style%

 ## Cross-compilation checks

-The [cross-compilation checks](http://ci.mlpack.org/job/CrossCompile-mlpack-for-embedded-aarch64/)
-run on Jenkins.
+The [cross-compilation checks](http://ci.mlpack.org/job/mlpack%20cross-compile%20tests/)
+run on Jenkins and test cross-compilation of mlpack to a number of low-resource
+and embedded devices.  The job is defined in
+`.jenkins/cross-compilation/Jenkinsfile`.

 * The job builds mlpack in a
-   [cross-compilation environment](../embedded/supported_boards.md).
+   [cross-compilation environment](../embedded/supported_boards.md), targeting a
+   number of architectures, and then running tests on actual embedded hardware.

 * Any failures seen here *that are not seen in other jobs* will probably be
   failures specific to the cross-compilation environment.

-## Static code analysis checks
+ * For the list of targeted devices, see the
+   [list of CI infrastructure](#list-of-ci-infrastructure).

-The [static code analysis checks](http://ci.mlpack.org/job/pull-requests-mlpack-static-code-analysis/)
-use a few C++ code analysis tools to try and report issues with the codebase.
+## Memory checks

-Currently, most of the output by this job is not actionable---there are too many
-false positives or spurious issues---and therefore should be used only as
-informational output.
+The [memory checks](http://ci.mlpack.org/job/mlpack%20memory%20checks/) run
+valgrind on any tests that were detected to be changed.  This detection is
+performed via a heuristic and may not always be correct.  The job is defined in
+`.jenkins/memory-checks/Jenkinsfile`.

-Configuration can be found in the
-[`jenkins-conf` repository](https://github.com/mlpack/jenkins-conf).
+ * The [`parse-test.py` script](https://github.com/mlpack/jenkins-conf/blob/master/memory/parse-tests.py)
+   is used to find tests that are affected by the changes.
+
+ * The [`memory-check.sh` script](https://github.com/mlpack/jenkins-conf/blob/master/memory/memory-check.sh)
+   is used to actually run the tests.
+
+If there are any memory issues with the code, this should be reported by a
+failed memory check job.  If you encounter one of these, try compiling with
+debugging symbols and running valgrind on the affected test, like this:
+
+```
+valgrind --leak-check=full --track-origins=yes bin/mlpack_test "TestName"
+```

 ## List of CI infrastructure

@@ -155,3 +174,9 @@ Link: [***Jenkins (`ci.mlpack.org`)***](http://ci.mlpack.org)
   maintainer to make changes, or if you are on the Contributors team but still
   don't have access, ask somewhere and someone will give you access.  (Probably
   `#mlpack:matrix.org` is the best bet!)
+
+ * A number of embedded devices are available to Jenkins and are used in the
+   cross-compilation job.  Each system is named after a main ingredient in a
+   good meal eaten just before receiving the embedded device.
+    - `couscous.ratml.org`: [Raspberry Pi 5](https://datasheets.raspberrypi.com/rpi5/raspberry-pi-5-product-brief.pdf),
+      4GB RAM, 4-core ARM Cortex-A76
@@ -132,8 +132,8 @@ the new architecture added to this table.
 ### CORTEXA76

 ```
-DTOOLCHAIN_PREFIX=/path/to/bootlin/toolchain/aarch64--glibc--stable-2024.02-1/bin/aarch64-buildroot-linux-gnueabihf-
-DCMAKE_SYSROOT=/path/to/bootlin/toolchain/aarch64--glibc--stable-2024.02-1/aarch64-buildroot-linux-gnueabihf/sysroot
+-DTOOLCHAIN_PREFIX=/path/to/bootlin/toolchain/aarch64--glibc--stable-2024.02-1/bin/aarch64-buildroot-linux-gnu-
+-DCMAKE_SYSROOT=/path/to/bootlin/toolchain/aarch64--glibc--stable-2024.02-1/aarch64-buildroot-linux-gnu/sysroot
 ```

 ### C906
@@ -560,7 +560,7 @@ For implementation details, see

 The `XTreeSplit` class implements the improved splitting strategy for the
 [`XTree`](x_tree.md) as described in the
-[X-tree paper (pdf)](http://www.vldb.org/conf/1996/P028.PDF).  This strategy is
+[X-tree paper (pdf)](https://www.vldb.org/conf/1996/P028.PDF).  This strategy is
 an improved version of the standard [`RTreeSplit`](#rtreesplit), where the
 overlap of sibling nodes is minimized.

@@ -72,6 +72,8 @@ TEST_CASE("PaddingTest", "[ConvolutionalNetworktest]")
 {
  arma::mat X;
  X.load("mnist_first250_training_4s_and_9s.csv");
+  // Make sure the data loaded okay.
+  REQUIRE(!X.is_empty());

  // Create the network.
  FFN<NegativeLogLikelihood, RandomInitialization> model;
@@ -149,6 +151,8 @@ TEST_CASE("VanillaNetworkTest", "[ConvolutionalNetworkTest]")
 {
  arma::mat X;
  X.load("mnist_first250_training_4s_and_9s.csv");
+  // Make sure the data loaded okay.
+  REQUIRE(!X.is_empty());

  // Normalize each point since these are images.
  arma::uword nPoints = X.n_cols;
@@ -265,6 +269,8 @@ TEST_CASE("VanillaNetworkBatchSizeTest", "[ConvolutionalNetworkTest]")

  arma::mat X;
  X.load("mnist_first250_training_4s_and_9s.csv");
+  // Make sure the data loaded okay.
+  REQUIRE(!X.is_empty());

  // Normalize each point since these are images.
  arma::uword nPoints = X.n_cols;
@@ -348,6 +354,8 @@ TEST_CASE("CheckCopyVanillaNetworkTest", "[ConvolutionalNetworkTest]")
 {
  arma::mat X;
  X.load("mnist_first250_training_4s_and_9s.csv");
+  // Make sure the data loaded okay.
+  REQUIRE(!X.is_empty());

  // Normalize each point since these are images.
  arma::uword nPoints = X.n_cols;
@@ -401,6 +401,8 @@ TEST_CASE("FFVanillaNetworkTest", "[FeedForwardNetworkTest]")

  arma::mat dataset;
  dataset.load("mnist_first250_training_4s_and_9s.csv");
+  // Make sure the data loaded okay.
+  REQUIRE(!dataset.is_empty());

  // Normalize each point since these are images.
  for (size_t i = 0; i < dataset.n_cols; ++i)
@@ -422,6 +424,8 @@ TEST_CASE("ForwardBackwardTest", "[FeedForwardNetworkTest]")
 {
  arma::mat dataset;
  dataset.load("mnist_first250_training_4s_and_9s.csv");
+  // Make sure the data loaded okay.
+  REQUIRE(!dataset.is_empty());

  // Normalize each point since these are images.
  for (size_t i = 0; i < dataset.n_cols; ++i)
@@ -549,6 +553,8 @@ TEST_CASE("DropoutNetworkTest", "[FeedForwardNetworkTest]")
  TestNetwork<>(model, trainData, trainLabels, testData, testLabels, 10, 0.1);
  arma::mat dataset;
  dataset.load("mnist_first250_training_4s_and_9s.csv");
+  // Make sure the data loaded okay.
+  REQUIRE(!dataset.is_empty());

  // Normalize each point since these are images.
  for (size_t i = 0; i < dataset.n_cols; ++i)
@@ -628,6 +634,8 @@ TEST_CASE("DropConnectNetworkTest", "[FeedForwardNetworkTest]")

  arma::mat dataset;
  dataset.load("mnist_first250_training_4s_and_9s.csv");
+  // Make sure the data loaded okay.
+  REQUIRE(!dataset.is_empty());

  // Normalize each point since these are images.
  for (size_t i = 0; i < dataset.n_cols; ++i)
@@ -953,6 +961,8 @@ TEST_CASE("RBFNetworkTest", "[FeedForwardNetworkTest]")

  arma::mat dataset;
  dataset.load("mnist_first250_training_4s_and_9s.csv");
+  // Make sure the data loaded okay.
+  REQUIRE(!dataset.is_empty());

  // Normalize each point since these are images.
  for (size_t i = 0; i < dataset.n_cols; ++i)
@@ -16862,7 +16862,7 @@ namespace Catch {
            xml( _config.stream() )
        {
            m_reporterPrefs.shouldRedirectStdOut = true;
-            m_reporterPrefs.shouldReportAllAssertions = true;
+            m_reporterPrefs.shouldReportAllAssertions = false;
        }

    JunitReporter::~JunitReporter() {}