Overview¶
We provide support for automated Continuous Integration (CI), Continuous Testing (CT) and Continuous Deployment (CD) workflows on the Tier-2 and Tier-3 HPC systems (HoreKa, bwUniCluster 2.0) and other systems within the HPC realm, e.g. the Future Technologies Partition (FTP).
At the moment our focus is on the Gitlab stack. gitlab-runner is supported on all systems, support for other stacks will be added later. There are no limits as to where the necessary Gitlab instances can be located and who controls them. The only limitation is that the HPC systems at SCC must be able to reach the Gitlab instance via an internal KIT network or the public internet. Using a VPN connection is not possible.
CI integration levels overview¶
There are currently three different CI integration levels available, depending on your use case and the amount of resources needed.
| CI Level | Number of CI jobs per day this level is well-suited for | Node the CI runner is executed on | Self-management of CI runner possible | CI runner executes jobs permanently | Access to special hardware (GPUs etc.) possible | Usage of container technologies possible | Suited for CI jobs with timing constraints |
|---|---|---|---|---|---|---|---|
| 1 | Low | Compute Node | |||||
| 2 | Medium | Dedicated HPC Login node | For medium-size workloads | ||||
| 3 | High | Dedicated CI hardware |
CI Level 1¶
In this level the CI runner binary is executed by submitting it as a batch job to one of the HPC or FTP systema. When the batch job is started, the CI runner binary inside the job contacts the CI server instance, pulls all waiting CI jobs, executes all stages, downloads/uploads all the artifacts and then quits.
There are no strict guarantees as to how much time may pass between the time a CI job is triggered by the CI pipeline and the time it is actually executed by the CI runner. It is possible to make the latency more predictable by requesting special reservations and increased batch system priorities, and it is possible to automate the submission of the batch job using tools like scrontab.
Conclusion
This CI level is therefore aimed at projects that generate a low number of CI jobs per day and don't have strict timing constraints. Nightly builds and nightly integration tests and are a good use case.
More details on this level can be found here.
CI Level 2¶
In this level the CI runner binary is executed as a permanent service on a dedicated login node of one of the HPC/FTP systems. These CI runners can be self-managed by the users.
Since all users share the same dedicated login node and the hardware is only equipped with CPUs, there are limitations as to which CI jobs can run on the same node the CI runner is running on and which ones have to run somewhere else.
Short CI jobs like e.g. building, testing, packaging and deploying a medium-size software project can be executed directly on the dedicated login node. There will be runtime variations, but usually the jobs start instantly.
CI jobs that require more or special resources, e.g. building and testing a large-scale project, running performance-sensitive regression benchmarks, or running code on hardware like ARM CPUs or GPUs, cannot be run on the dedicated login nodes. These workloads have to be submitted as batch jobs from within the CI scripts.
It is possible and encouraged to handle different CI jobs within the same project differently. For example it is a good idea to trigger a build job on every source code commit and execute it on the dedicated login node to guarantee quick feedback, while the nightly performance regression tests that require a GPU and are not as time-sensitive are submitted as a normal batch job.
Conclusion
This CI level is therefore aimed at projects that generate a medium number of CI jobs per day and need predictable runtimes for medium-size workloads.
More details on this level can be found here.
CI Level 3¶
We provide dedicated CI worker nodes for projects that generate many CI jobs per day and need predictable runtimes and performance as well as privileged access to special resources like e.g. ARM processors and GPUs.
Access to these dedicated nodes and the configuration of the CI runners on these nodes has to be discussed with the CI Operations Team.
More details on this level can be found here.
Support¶
Please contact the CI Team by sending an e-mail to hpc-ci-hotline@lists.kit.edu if you have questions regarding the CI infrastructure, especially if you are not sure how the provided CI levels and services match your project requirements.