Skip to content

CI: update toolchains to use latest GitHub runner images and rewrite set_matrix.py without polly #780

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 23 commits into from
May 20, 2025

Conversation

NeroBurner
Copy link

@NeroBurner NeroBurner commented Apr 29, 2025

Update all used GitHub runner images (Linux, Windows and Mac) and update the set_matrix.py script to work without the polly toolchains.


  • Update from the deprecated ubuntu-20.04 images to the currently latest ubuntu-24.04 image. In the process update the gcc version from 7, 8, 9 to 12, 13, 14.
  • Update from the deprecated windows-2019 image with vs-16-2019 toolchain to windows-2022 with vs-17-2022 toolchain
  • Update from deprecated macos-12 image to macos-15
  • rewrite set_matrix.py to generate the CMAKE_TOOLCHAIN_FILE from the toolchain matrix name
  • update CI and remove the dependency on the polly toolchain project
  • remove analyze and sanitize jobs, no-one looks at them and they are a hassle to implement, PR welcome, but for now removed
  • remove the build.sh and build.cmd default build scripts and replace them with a build.py python script
  • remove build_hunter_tests.sh build script and use build.py there as well
  • remove unused jenkins.py file
  • update custom matrix.json and build.sh for the following projects:
    • filament
    • pthreads-win32
    • occt
    • freetype-gl
    • Boost

@NeroBurner NeroBurner marked this pull request as draft April 29, 2025 15:24
@NeroBurner NeroBurner linked an issue Apr 29, 2025 that may be closed by this pull request
@NeroBurner NeroBurner force-pushed the ci_update_ubuntu_image_20.04 branch 2 times, most recently from b23ca88 to c3e7d55 Compare May 5, 2025 05:34
@NeroBurner
Copy link
Author

NeroBurner commented May 5, 2025

from #781

windows-2019 image with vs-16-2019 toolchain will be deprecated next month. Will remove it from build matrix as well

The Windows 2019 Actions runner image will begin deprecation on 2025-06-01 and will be fully unsupported by 2025-06-30

actions/runner-images#12045

@NeroBurner
Copy link
Author

need to update macos-12 as well: #782

Update from the deprecated ubuntu-20.04 images to the currently latest
ubuntu-24.04 image. In the process update the `gcc` version from 7, 8, 9
to 12, 13, 14.
@NeroBurner NeroBurner force-pushed the ci_update_ubuntu_image_20.04 branch from f00adc4 to edbac40 Compare May 19, 2025 11:37
@NeroBurner
Copy link
Author

basic functionality restored, set_matrix.py and build.py should now work.

Now need to update custom matrix.json and build.sh scripts

found the following files:

$ find cmake/projects/ -name "*.sh" -o -name "*.json" -o -name "*.cmd"
cmake/projects/filament/ci/build.sh
cmake/projects/filament/ci/matrix.json
cmake/projects/Qt/generate.sh
cmake/projects/pthreads-win32/ci/matrix.json
cmake/projects/occt/ci/build.sh
cmake/projects/freetype-gl/ci/build.sh
cmake/projects/freetype-gl/ci/matrix.json
cmake/projects/Boost/scripts/create-boost-1-66-ios-fatlib.sh
cmake/projects/Boost/scripts/create-boost-ios-fatlib.sh
cmake/projects/Boost/ci/matrix.json
cmake/projects/Boost/generate.sh

so will have to fix the following projects:

  • filament
  • pthreads-win32
  • occt
  • freetype-gl
  • Boost

To test I'll locally try to successfully run the following command:

python .github/workflows/set_matrix.py Eigen filament pthreads-win32 occt freetype-gl Boost

@NeroBurner NeroBurner force-pushed the ci_update_ubuntu_image_20.04 branch from 3dc0793 to 536cffc Compare May 19, 2025 12:38
@NeroBurner NeroBurner marked this pull request as ready for review May 19, 2025 14:06
@NeroBurner NeroBurner changed the title CI: update Linux toolchains to use Ubuntu 24.04 images CI: update toolchains to use latest GitHub runner images and rewrite set_matrix.py without polly May 19, 2025
@NeroBurner NeroBurner added the infrastructure Problems with the CI infrastructure or CI jobs label May 19, 2025
@NeroBurner NeroBurner self-assigned this May 19, 2025
Noone looks at them and it is a hassle to implement in `set_matrix.py`
- set_matrix: add --projects override for local debugging
- set_matrix: add optional --output parameter
- set_matrix: default output to stdout without indent
- set_matrix: create toolchain files first steps non-complete
- set_matrix: handle specific android toolchains
- set_matrix: handle CMAKE_C_STANDARD and set undetectable
- set_matrix: toolchain creation for vs-xx-xxxx, no generator yet
- set_matrix: toolchain creation for mingw and msys
- set_matrix: newlines and c11 undetectable
- set_matrix: comment about generator fixed and libc++ closing ) fixed
- set_matrix: use android-ndk-api
- set_matrix: set VCVARSALL and generator
- set_matrix: add project_name to error message
- set_matrix: update path to vcvarsall.bat for vs-17-2022
- set_matrix: info about deprecation of vs-16-2019
- set_matrix: specified scripts are mandatory, no default build scripts
- set_matrix: no script for hunter_tests, but VERBOSE=True
- set_matrix: write out project name for CI job name
- set_matrix: set default python version for all jobs
- build.py: copy from jenkins.py and black format
- build.py: call cmake in script
- build.py: sys.exit not just exit
- build.py: fix cmake calling
- build.py: as posixpath for args
- build.py: use full path for TOOLCHAIN
- build.py: toolchain filepath as posix path
- build.py: all paths as posix
- build.py: alsways build with --config Release
- build.py: hunter_sha1 is str, not path
- build.py: only build Release config to save time
- build.py: CMP0069 is on per default since 3.9
- build.py: no chdir, remove all_release flag
- build.py: use cdir instead of recreate
- build.py: try without archive, maybe fixes Windows Store
- build.py: disable zip/archive creation, messes with Windows 10 Store
- build.py: fix release type, delete build dir, no archive
- build.py: handle optional SCRIPT environment variable
- build.py: allow hunter_tests as toolchain, special handling
- build.py: on main and master and GITHUB_USER_PASSWORD upload cache
- build.py: allow toolchain name and path and find file if needed
@NeroBurner NeroBurner force-pushed the ci_update_ubuntu_image_20.04 branch from b278c94 to 28c5ac7 Compare May 19, 2025 14:27
@NeroBurner
Copy link
Author

removed the Eigen and hunter_tests [WIP] commits to be ready for squash-merging

- ci: matrix: remove ndk from toolchain, depends on github runner image
- ci: matrix: use android-ndk-api-xxx
- ci: matrix: macos: move to macos-15
- ci: matrix: windows: move to vs-17-2022 on windows-2022 image
- ci: matrix: use Windows SDK 22000, as it is one of the available
- ci: matrix: remove python and script from default matrix, add comments
- ci: upload generated toolchains folder
- ci: on windows run VCVARSALL if needed
- ci: download artifacts for .github/toolchains folder
- ci: windows: download artifacts downloads folder directly, forgets full path
- ci: linux: download artifacts downloads folder directly, forgets full path
- ci: linux: use build.py script
- ci: run build.py for windows build as well
- ci: use build.py for hunter_tests as well
- ci: use matrix.project
- ci: use TOOLCHAIN with just name, remove hunter_tests job
- ci: on error show output of set_matrix.py
- ci: actually pass matrix.script as SCRIPT env-variable
also update docs such that custom build scripts are called by build.py
@NeroBurner NeroBurner force-pushed the ci_update_ubuntu_image_20.04 branch from 28c5ac7 to 93a5d9c Compare May 19, 2025 14:43
@NeroBurner
Copy link
Author

something is not yet working with the SCRIPT variable/file

 FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/work/hunter/hunter/cmake/projects/filament/ci/build.sh'
PermissionError: [Errno 13] Permission denied: '/home/runner/work/hunter/hunter/cmake/projects/freetype-gl/ci/build.sh'

@NeroBurner
Copy link
Author

made the SCRIPT variable to be relative to repo-root and added executable bit for build.sh files. Now dependencies are installed

@NeroBurner NeroBurner merged commit 0ddc5ce into master May 20, 2025
151 of 209 checks passed
@NeroBurner NeroBurner deleted the ci_update_ubuntu_image_20.04 branch May 20, 2025 09:15
@NeroBurner
Copy link
Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infrastructure Problems with the CI infrastructure or CI jobs
Projects
None yet
1 participant