apaam/mlpack
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

First attempt at GHA CI instead of Azure Pipelines. (#3881)

Browse Source 
* First attempt at GHA CI instead of Azure Pipelines.

* Could I have the matrix config wrong?

* Oh, oops, I was just looking in the wrong place for the GHA log.

* Disable Azure Pipelines Linux job.

* Fix some additional small syntax errors.

* Minor updates.

* Try to figure out what is happening with this test!

* Ha, actually remember the argument.

* Try to parse the CTest results.

* Don't build CLI programs for the debug-only mlpack_test build.

* Try to figure out why we don't have the CTest output file.

* Seems like maybe the test just needs more iterations.

* Fix directory for CTest output.

* Try running mlpack_test manually to see if we get better test output.

* Remove version numbers and other non-XML from XML output test file.

* Try to fix debug build.

* Filtering regex.

* Try to fail the build when the tests fail.

* Minor fixes and cleanups to run Python binding tests.

* Print test failure information in the actual build step.

* Update Julia testing strategy.

* Try to also get outputs from other test bindings.

* Fix Julia and Go testing blocks.

* Avoid random collisions that cause test failures.

* Try moving splitting into the trial loop.

* Try a different test reporter action.

* Install pytest for Python tests.

* Fix working directory for Julia test.

* Add test case that will fail to see what it looks like.

* Back to the old action, the new one I tried didn't do what I needed.

* Try a better way to print the test output.

* I still don't know how to mark the job failed yet.

* Maybe this stdout character works better.

* Try a forked test-summary action.

* Try to use distribution version of the modified action.

* Try to see if ccache is doing anything.

* Always run ccache stats step.

* Turn off PCH for ccache.

* Let's just see what it looks like with more error output.

* Clean up ccache configuration.

* Remove fake test because it seems like now I have the output working right.

* Try to reduce file size by not printing debug output into it.

* Try and refactor binding environment seutp into a composite local action.

* Try to fix some syntactical errors.

* Refactor test running into a separate composite action.

* Start trying to make the R build part of the main CI build.

* Oops, the include directory was in the wrong place.

* Try actually running R tests in the workflow too.

* Try to fix a few action issues.

* Try to work around libicu issue.

* Remove unnecessary option.

* Try to run OS X builds too.

* Try to fix matrix configuration.

* Revert "Try to fix matrix configuration."

This reverts commit 22a08d6dcd0d39de0a27f2df16e885bd68e18fda.

* Oops, maybe it was just an extra comma.

* Fix naming, and maybe we have to remove stringi first.

* Try to set Julia location correctly.

* Some fixes to the pipelines.

* Oops, I need root for that.

* Some additional fixes for jobs.

* Hopefully fix a few more builds.

* What, how did that get there?

* Don't use non-portable -i option with sed.

* Run R tests properly to get junit output.

* Oops, I have to redirect the output.

* Where is the output file?

* Try to fix macOS Python location.

* Oops, ROOTDIR is just not set.

* Try to store the name of the R package correctly.

* Oops, we need to specify pip.

* Try to get the right directory for the R package.

* Try to fix another round of issues.  I'm getting closer, at least.

* Okay, fine, let's do it the brutalistic way.

* What is actually happening with the Go binding tests?

* Try a different strategy for printing.

* Make some updates to prepare for Windows builds.

* What's the output of go test?

* Remove -Wall as per golang/go#6883.

* Try to fix duplicate library warning.

* Maybe I just specified the variable in an invalid way that didn't stick?

* This should fix the Go build.

* How did that cd go missing?

* Try to remove DEBUG output from the debug tests.

* Try to be more specific about the ccache key.

* Oops, incorrect variable name.

* Try to speed up the testing step.

* Try to run the Windows build through GHA.

* Try to install locally to deps/.

* Try to install and set up Windows dependencies correctly.

* Try to debug Windows build.

* Use bash for CMake configuration.

* Some additional path fixes.

* A couple fixes, but I don't have the Armadillo path right.

* Fix curl command.

* Maybe this is closer to the right path...

* Okay, look one directory deeper...

* Temporarily print STB compilation failure.

* Try to use absolute path for STB inclusion test.

* Oops, fix syntax.

* Try using REALPATH instead.

* Try just skipping the check...

* Try specifying BLAS and LAPACK locations directly.

* Try to figure out what is going on.

* Are we even using the correct FindArmadillo script?

* Try specifying the libraries manually.

* Try disabling the wrapper library.

* What version of CMake is this?

* Print the configuration.

* What if we specify these library locations?

* Double-check: this should fail.

* Okay, that actually surprisingly did not fail.

* Fix style issues.

* Compress into only one GHA file so all the jobs show up in the same place.

* Try to use absolute path to OpenBLAS.

* Try to fix path mangling on Windows.

* Fix STB path and no PS please.

* Try to fix output on Windows tests.

* Call test from the right place.

* Remove some unnecessary output.

* Try just running the test.

* What is going on, why doesn't the test run?

* Try just running the test with PowerShell.

* Maybe I have the filename wrong?

* PowerShell makes me angry...

* Right, .exe is the suffix...

* I love PowerShell!

* Okay, forget PowerShell.

* Can we even run the test?

* Is it a DLL search path issue?

* Okay, I think this might work for Windows.

* Fix porting of new r2u/p3m workflow (hopefully).

* Fix possibly incorrect variable name.

* Reduce the number of macOS jobs.

* Fix name of job.

* Remove potentially unnecessary step.

* Document CI changes and reorganize jobs for better viewing.

* Remove invalid link.  (Awesome that the link check build picked this up.)
This commit is contained in:
Ryan Curtin
2025-02-14 20:22:13 -05:00
committed by GitHub
parent e30f83c028
commit e7f79637f6
22 changed files with 571 additions and 718 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/ci.yml
+279
View File
@@ -0,0 +1,279 @@
# ci.yml: defines all the CI jobs run for each commit and pull request. For
# Linux and macOS, we run a matrix job that tests all the bindings. For
# Windows, we just run one build (see the later part of the file).
on:
workflow_dispatch:
push:
branches:
- master
- main
pull_request:
branches:
- master
- main
release:
types: [published, created, edited]
permissions:
contents: read
name: mlpack.mlpack
jobs:
###
### Linux and macOS test matrix for all binding types.
###
ci_matrix:
strategy:
fail-fast: false # Run all configurations even if one fails.
matrix:
config: [
{
runner: ubuntu-latest,
os: 'Linux',
name: 'Debug',
# Disable [DEBUG] output from the tests because it's WAY too verbose.
cmakeVars: '-DCMAKE_BUILD_TYPE=Debug -DBUILD_CLI_EXECUTABLES=OFF -DCMAKE_CXX_FLAGS="-DMLPACK_NO_PRINT_DEBUG"'
},
{
runner: ubuntu-latest,
os: 'Linux',
name: 'CLI',
cmakeVars: '-DBUILD_CLI_EXECUTABLES=ON'
},
{
runner: ubuntu-latest,
os: 'Linux',
name: 'Python',
cmakeVars: '-DBUILD_CLI_EXECUTABLES=OFF -DBUILD_PYTHON_BINDINGS=ON'
},
{
runner: ubuntu-latest,
os: 'Linux',
name: 'Julia',
cmakeVars: '-DBUILD_CLI_EXECUTABLES=OFF -DBUILD_JULIA_BINDINGS=ON'
},
{
runner: ubuntu-latest,
os: 'Linux',
name: 'R',
cmakeVars: '-DBUILD_CLI_EXECUTABLES=OFF -DBUILD_R_BINDINGS=ON -DCEREAL_INCLUDE_DIR=../cereal-1.3.2/include/'
},
{
runner: ubuntu-latest,
os: 'Linux',
name: 'Go',
cmakeVars: '-DBUILD_CLI_EXECUTABLES=OFF -DBUILD_GO_BINDINGS=ON'
},
{
runner: ubuntu-latest,
os: 'Linux',
name: 'Markdown',
cmakeVars: '-DBUILD_CLI_EXECUTABLES=OFF -DBUILD_MARKDOWN_BINDINGS=ON'
},
{
runner: macOS-latest,
os: 'macOS',
name: 'CLI',
cmakeVars: '-DBUILD_CLI_EXECUTABLES=ON'
},
{
runner: macOS-latest,
os: 'macOS',
name: 'Python',
cmakeVars: '-DBUILD_CLI_EXECUTABLES=OFF -DBUILD_PYTHON_BINDINGS=ON'
}
]
name: '${{ matrix.config.os }} (${{ matrix.config.name }})'
if: ${{ github.repository == 'mlpack/mlpack' }}
runs-on: ${{ matrix.config.runner }}
steps:
- uses: actions/checkout@v3
# Set up ccache.
- name: Get ccache
uses: hendrikmuhs/ccache-action@v1.2
with:
key: ${{ matrix.config.os }}-${{ matrix.config.name }}
variant: ccache
max-size: 1G
# Set up the build environment for any bindings, if needed.
- name: Set up binding dependencies
uses: ./.github/actions/binding_setup
with:
lang: ${{ matrix.config.name }}
# Install build dependencies.
- name: Install build dependencies (Linux)
if: matrix.config.os == 'Linux'
run: |
git clone --depth 1 https://github.com/mlpack/jenkins-conf.git conf
sudo apt-get update
sudo apt-get install --yes --allow-unauthenticated \
libopenblas-dev libstb-dev libcereal-dev xz-utils
# Install the oldest Armadillo version that we support.
curl -k -L https://sourceforge.net/projects/arma/files/armadillo-10.8.2.tar.xz | tar -xvJ && \
cd armadillo* && \
cmake . && \
make && \
sudo make install && \
cd ..
# Install the latest ensmallen version.
wget https://ensmallen.org/files/ensmallen-latest.tar.gz
tar -xvzpf ensmallen-latest.tar.gz # Unpack into ensmallen-*/.
cd ensmallen-*/ && \
sudo cp -vr include/* /usr/include/ && \
cd ..
- name: Install build dependencies (macOS)
if: matrix.config.os == 'macOS'
run: |
sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer
brew install libomp openblas armadillo cereal ensmallen ccache
# Because mlpack's precompiled headers basically include everything
# (mlpack.hpp, core.hpp, etc.), changes to a single file will invalidate
# the ccache entry---so it is better to leave it turned off, so that
# ccache entries will be at a more granular level and are more likely to
# be reused across runs.
- name: Configure mlpack with CMake
run: |
mkdir build && cd build
cmake ${{ matrix.config.cmakeVars }} $CMAKE_BINDING_ARGS -DUSE_PRECOMPILED_HEADERS=OFF -DBUILD_TESTS=ON ../
- name: Build mlpack
run: cd build && make -j4
# Run the tests manually so that we can get JUnit output.
- name: Run mlpack_test
run: |
cd build
# The use of only one thread is to prevent thrashing on older versions
# of OpenBLAS (0.3.26 and older) where OpenMP and pthreads aren't
# playing together well.
OMP_NUM_THREADS=1 bin/mlpack_test -r junit | tee /dev/stderr > mlpack_test.junit.xml
# Remove version numbers and other non-xml from the output.
cat mlpack_test.junit.xml | sed '/<?xml/,$!d' > mlpack_test.junit.xml.tmp
mv mlpack_test.junit.xml.tmp mlpack_test.junit.xml
# Run binding tests for each binding type.
- name: Run binding tests
uses: ./.github/actions/binding_run_tests
with:
lang: ${{ matrix.config.name }}
- name: Parse test output
uses: rcurtin/test-summary-action@dist
if: success() || failure()
with:
paths: "build/*.junit.xml"
show: "fail, skip"
fail_job: true
print_output: true
###
### Windows CI testing.
###
ci_windows:
name: 'Windows build: VS2022'
if: ${{ github.repository == 'mlpack/mlpack' }}
runs-on: windows-2022
steps:
- uses: actions/checkout@v3
# Set up ccache.
- name: Get ccache
uses: hendrikmuhs/ccache-action@v1.2
with:
key: ${{ runner.os }}
variant: ccache
max-size: 5G
# Set up Visual Studio.
- name: Set up Visual Studio
uses: microsoft/setup-msbuild@v2
# Install build dependencies.
- name: Install build dependencies (Windows)
shell: powershell
run: |
nuget install OpenBLAS -o deps
nuget install ensmallen -o deps -Version 2.17.0
# Delete all ensmallen dependencies including armadillo headers, we do not need them here.
Remove-Item deps\ensmallen.2.17.0\installed\x64-linux\share -Force -Recurse
Remove-Item deps\ensmallen.2.17.0\installed\x64-linux\include\armadillo_bits -Force -Recurse
Remove-Item deps\ensmallen.2.17.0\installed\x64-linux\include\armadillo -Force
# Install the oldest supported version of Armadillo and install cereal.
- name: Install some dependencies manually
shell: bash
run: |
cd deps/
curl -L https://sourceforge.net/projects/arma/files/armadillo-10.8.2.tar.xz -o armadillo-10.8.2.tar.xz
tar -xf armadillo-10.8.2.tar.xz
cd armadillo-10.8.2/
cmake -G "Visual Studio 17 2022" \
-DBLAS_LIBRARY:FILEPATH=${{ github.workspace }}/deps/OpenBLAS.0.2.14.1/lib/native/lib/x64/libopenblas.dll.a \
-DLAPACK_LIBRARY:FILEPATH=${{ github.workspace }}/deps/OpenBLAS.0.2.14.1/lib/native/lib/x64/libopenblas.dll.a \
-DBUILD_SHARED_LIBS=OFF .
# Disable the wrapper because transitive linking doesn't work on
# Windows anyway.
cp tmp/include/armadillo_bits/config.hpp tmp/include/armadillo_bits/config.hpp.tmp
cat tmp/include/armadillo_bits/config.hpp.tmp | sed 's|#define ARMA_USE_WRAPPER|// #define ARMA_USE_WRAPPER|' \
> tmp/include/armadillo_bits/config.hpp
# Now download and unpack cereal.
cd ../
curl -L https://github.com/USCiLab/cereal/archive/refs/tags/v1.3.2.tar.gz -o cereal-1.3.2.tar.gz
tar -xzf cereal-1.3.2.tar.gz
# Download and unpack STB.
curl -O https://www.mlpack.org/files/stb.tar.gz
tar -xzf stb.tar.gz
# Because mlpack's precompiled headers basically include everything
# (mlpack.hpp, core.hpp, etc.), changes to a single file will invalidate
# the ccache entry---so it is better to leave it turned off, so that
# ccache entries will be at a more granular level and are more likely to
# be reused across runs.
- name: Configure mlpack with CMake
shell: bash
run: |
mkdir build && cd build
cmake -DUSE_PRECOMPILED_HEADERS=OFF -DBUILD_TESTS=ON \
-DARMADILLO_INCLUDE_DIR=../deps/armadillo-10.8.2/tmp/include/ \
-DBLAS_LIBRARIES='${{ github.workspace }}'/deps/OpenBLAS.0.2.14.1/lib/native/lib/x64/libopenblas.dll.a \
-DLAPACK_LIBRARIES='${{ github.workspace }}'/deps/OpenBLAS.0.2.14.1/lib/native/lib/x64/libopenblas.dll.a \
-DENSMALLEN_INCLUDE_DIR=../deps/ensmallen.2.17.0/installed/x64-linux/include/ \
-DCEREAL_INCLUDE_DIR=../deps/cereal-1.3.2/include/ \
-DSTB_IMAGE_INCLUDE_DIR='${{ github.workspace }}'/deps/stb/include/ \
-DCMAKE_HAS_WORKING_STATIC_STB=1 \
../
- name: Build mlpack
run: cd build && cmake --build . --config Release
# Run the tests manually so that we can get JUnit output.
- name: Run mlpack_test
shell: bash
run: |
# The .dlls are stored in the bin/ directory, and those are the ones
# we need to run with.
cp -v '${{ github.workspace }}'/deps/OpenBLAS.0.2.14.1/lib/native/bin/x64/* build/Release
cd build/
./Release/mlpack_test.exe -r junit > mlpack_test.junit.xml
# Remove version numbers and other non-xml from the output.
cat mlpack_test.junit.xml | sed '/<?xml/,$!d' > mlpack_test.junit.xml.tmp
rm mlpack_test.junit.xml
mv mlpack_test.junit.xml.tmp mlpack_test.junit.xml
- name: Parse test output
uses: rcurtin/test-summary-action@dist
if: success() || failure()
with:
paths: "build/*.junit.xml"
show: "fail, skip"
fail_job: true
print_output: true
.github/workflows/main.yml
-181
View File
@@ -1,181 +0,0 @@
on:
workflow_dispatch:
push:
branches:
- master
- main
pull_request:
branches:
- master
- main
release:
types: [published, created, edited]
permissions:
contents: read
name: mlpack.mlpack
jobs:
jobR:
name: mlpack R tarball
if: ${{ github.repository == 'mlpack/mlpack' }}
runs-on: ubuntu-latest
outputs:
r_bindings: ${{ steps.mlpack_version.outputs.mlpack_r_package }}
steps:
- uses: actions/checkout@v3
- name: Extract mlpack version
id: mlpack_version
run: |
MLPACK_VERSION_MAJOR=$(grep -i ".*#define MLPACK_VERSION_MAJOR.*" src/mlpack/core/util/version.hpp | grep -o "[0-9]*")
MLPACK_VERSION_MINOR=$(grep -i ".*#define MLPACK_VERSION_MINOR.*" src/mlpack/core/util/version.hpp | grep -o "[0-9]*")
MLPACK_VERSION_PATCH=$(grep -i ".*#define MLPACK_VERSION_PATCH.*" src/mlpack/core/util/version.hpp | grep -o "[0-9]*")
MLPACK_VERSION_VALUE=${MLPACK_VERSION_MAJOR}.${MLPACK_VERSION_MINOR}.${MLPACK_VERSION_PATCH}
echo "mlpack_r_package=$(echo mlpack_"$MLPACK_VERSION_VALUE".tar.gz)" >> $GITHUB_OUTPUT
# Setup Pandoc
- name: Setup pandoc
uses: r-lib/actions/setup-pandoc@v2
# Setup r2u (linux)
- name: Setup r2u
uses: eddelbuettel/github-actions/r2u-setup@master
# Get and set up ccache
- name: Get CCache
uses: hendrikmuhs/ccache-action@v1.2
with:
# can also be eg ${{ matrix.os }} but just ubuntu-latest here
key: ${{ runner.os }}
# this is the default but 'sccache' can be selected here
variant: ccache
- name: Prepare R for Ccache
run: |
mkdir -p ~/.R
cp -vax .github/etc/R_Makevars_${{ runner.os }} ~/.R/Makevars
- name: Configure Ccache for R
run: |
ccache --set-config "sloppiness=include_file_ctime"
ccache --set-config "hash_dir=false"
ccache --show-config
ccache --zero-stats
- name: Query dependencies
run: |
cp src/mlpack/bindings/R/mlpack/DESCRIPTION.in DESCRIPTION
Rscript -e 'install.packages(c("remotes", "roxygen2", "pkgbuild"))'
Rscript -e 'remotes::install_deps(".", dependencies=TRUE)'
- name: Install Additional Build Dependencies
run: |
sudo apt-get update
# We don't install cereal via apt, because the Debian packagers
# split the rapidjson dependency into a separate package. We will
# bundle the cereal sources with the R package, so we want them to
# be exactly the upstream sources (with rapidjson included).
sudo apt-get install -y --allow-unauthenticated libopenblas-dev liblapack-dev g++ libensmallen-dev libhdf5-dev libarmadillo-dev libcurl4-openssl-dev
wget https://github.com/USCiLab/cereal/archive/refs/tags/v1.3.2.tar.gz
tar -xvzpf v1.3.2.tar.gz
#
# These directives cause warnings on CRAN:
# https://github.com/USCiLab/cereal/blob/master/include/cereal/external/base64.hpp#L28-L31
# The command below comments them out.
sed -i 's|#pragma|// #pragma|' cereal-1.3.2/include/cereal/external/base64.hpp
#
# Fix cereal compilation on clang 19+; see
# https://github.com/USCiLab/cereal/pull/835
sed -i 's|::template apply|::apply|' cereal-1.3.2/include/cereal/types/tuple.hpp
- name: CMake
run: |
mkdir build
cd build && cmake -DDEBUG=OFF -DPROFILE=OFF -DBUILD_CLI_EXECUTABLES=OFF -DBUILD_PYTHON_BINDINGS=OFF -DBUILD_JULIA_BINDINGS=OFF -DBUILD_GO_BINDINGS=OFF -DBUILD_R_BINDINGS=ON -DDOWNLOAD_DEPENDENCIES=ON -DBUILD_TESTS=ON -DCEREAL_INCLUDE_DIR=../cereal-1.3.2/include/ ..
- name: Build
run: |
cd build && make -j4
- name: Run tests via ctest
run: |
# The OMP_NUM_THREADS is to prevent thrashing on older versions of
# OpenBLAS (0.3.26 and older) where OpenMP and pthreads aren't playing
# together well.
cd build && OMP_NUM_THREADS=2 CTEST_OUTPUT_ON_FAILURE=1 ctest -T Test .
- name: Upload R packages
uses: actions/upload-artifact@v4.4.0
with:
name: mlpack_r_tarball
path: build/src/mlpack/bindings/R/${{ steps.mlpack_version.outputs.mlpack_r_package }}
R-CMD-check:
needs: jobR
runs-on: ${{ matrix.config.os }}
name: ${{ matrix.config.name }}
if: ${{ github.repository == 'mlpack/mlpack' }}
strategy:
fail-fast: false
matrix:
config:
#- { os: windows-latest, r: "release", name: "Windows R" }
- { os: macOS-latest, r: "release", name: "macOS R" }
- { os: ubuntu-latest, r: "release", name: "Linux R" }
env:
MAKEFLAGS: "-j 4"
R_BUILD_ARGS: "--no-build-vignettes"
R_CHECK_ARGS: "--no-build-vignettes"
_R_CHECK_FORCE_SUGGESTS: 0
R_REMOTES_NO_ERRORS_FROM_WARNINGS: true
GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
R_KEEP_PKG_SOURCE: yes
steps:
- uses: actions/download-artifact@v4.1.8
with:
name: mlpack_r_tarball
- name: Setup pandoc
uses: r-lib/actions/setup-pandoc@v2
- name: Setup p3m.dev (macos)
uses: r-lib/actions/setup-r@v2
if: runner.os == 'Windows' || runner.os == 'macOS'
with:
r-version: ${{ matrix.config.r }}
use-public-rspm: true
- name: Setup r2u (linux)
if: runner.os != 'Windows' && runner.os != 'macOS'
uses: eddelbuettel/github-actions/r2u-setup@master
- name: Add system packages (linux)
if: runner.os != 'Windows' && runner.os != 'macOS'
run: |
sudo apt update -qq
sudo apt install -y --allow-unauthenticated libopenblas-dev liblapack-dev libensmallen-dev libarmadillo-dev
- name: Install dependencies
run: |
Rscript -e "install.packages(c('remotes', 'rcmdcheck'))"
Rscript -e "remotes::install_deps('${{ needs.jobR.outputs.r_bindings }}', dependencies=TRUE)"
- name: Run the package check
# TODO: revert to error_on = 'warning'
run: |
Rscript -e "rcmdcheck::rcmdcheck('${{ needs.jobR.outputs.r_bindings }}', args = c('--no-manual','--as-cran'), error_on = 'error', check_dir = 'check')"
- name: Upload check results
if: failure()
uses: actions/upload-artifact@v4.4.0
with:
name: ${{ runner.os }}-r${{ matrix.config.r }}-results
path: |
check/mlpack.Rcheck/00check.log
check/mlpack.Rcheck/00install.out