Container Compute Service:Use GitLab Runner to build a CI/CD environment in an ACS cluster

Background information

GitLab Runner is an open source project for running jobs in GitLab CI/CD pipelines. GitLab Runner is an external job execution framework that provides various executors, such as Shell and Kubernetes, to allow you to run CI jobs in data centers or in the cloud. After a job is complete, GitLab Runner returns the result to GitLab.

The key configurations of GitLab Runner include the configurations and initialization settings of the runner manager pod and the configurations of the Kubernetes executor. For more information, see Configure GitLab Runner.

Procedure

The following figure shows the procedures described in this topic.

Prerequisites

A kubectl client is connected to the cluster. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

Installation procedure

In this example, gitlab-runner 17.3.1 is installed. The chart version used to install gitlab-runner 17.3.1 is 0.68.1. For more information about how to install gitlab-runner, see GitLab Runner Helm chart.

1. Obtain the GitLab Runner chart.

   1. Add the GitLab Helm repository.

      helm repo add gitlab https://charts.gitlab.io

   2. Update the chart.

      helm repo update gitlab

   3. Obtain the GitLab runner chart.

      helm pull gitlab/gitlab-runner --version 0.68.1 && tar zvxf gitlab-runner-0.68.1.tgz

2. Register a runner.

   You need to register a token and record the runner token in the GitLab console. For more information, see Registering runners.

3. Create a file named values.yaml to configure initialization settings.

   Show the YAML file content

   ## GitLab Runner Image
   ##
   ## By default it's using registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v{VERSION}
   ## where {VERSION} is taken from Chart.yaml from appVersion field
   ##
   ## ref: https://gitlab.com/gitlab-org/gitlab-runner/container_registry/29383?orderBy=NAME&sort=asc&search[]=alpine-v&search[]=
   ##
   ## Note: If you change the image to the ubuntu release
   ##       don't forget to change the securityContext;
   ##       these images run on different user IDs.
   ##
   ...
   
   ## The GitLab Server URL (with protocol) that want to register the runner against
   ## ref: https://docs.gitlab.com/runner/commands/index.html#gitlab-runner-register
   ##
   gitlabUrl: https://jihulab.com/
   
   ## DEPRECATED: The Registration Token for adding new Runners to the GitLab Server.
   ##
   ## ref: https://docs.gitlab.com/ee/ci/runners/new_creation_workflow.html
   ##
   # runnerRegistrationToken: ""
   
   ## The Runner Token for adding new Runners to the GitLab Server. This must
   ## be retrieved from your GitLab instance. It is the token of an already registered runner.
   ## ref: (we don't have docs for that yet, but we want to use an existing token)
   ##
   runnerToken: "glrt-t3_sz6xxxxxxxxxDsWF77"
   #
   
   ## Unregister all runners before termination
   ##
   ## Updating the runner's chart version or configuration will cause the runner container
   ## to be terminated and created again. This may cause your Gitlab instance to reference
   ## non-existant runners. Un-registering the runner before termination mitigates this issue.
   ## ref: https://docs.gitlab.com/runner/commands/index.html#gitlab-runner-unregister
   ##
   unregisterRunners: true
   
   ## When stopping the runner, give it time to wait for its jobs to terminate.
   ##
   ## Updating the runner's chart version or configuration will cause the runner container
   ## to be terminated with a graceful stop request. terminationGracePeriodSeconds
   ## instructs Kubernetes to wait long enough for the runner pod to terminate gracefully.
   ## ref: https://docs.gitlab.com/runner/commands/#signals
   terminationGracePeriodSeconds: 3600
   
   ## Set the certsSecretName in order to pass custom certficates for GitLab Runner to use.
   ## Provide resource name for a Kubernetes Secret Object in the same namespace,
   ## this is used to populate the /home/gitlab-runner/.gitlab-runner/certs/ directory
   ## ref: https://docs.gitlab.com/runner/configuration/tls-self-signed.html#supported-options-for-self-signed-certificates-targeting-the-gitlab-server
   ##
   # certsSecretName:
   
   ## Configure the maximum number of concurrent jobs
   ## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
   ##
   concurrent: 10
   
   ...
   
   ## For RBAC support:
   rbac:
     ## Specifies whether a Role and RoleBinding should be created
     ## If this value is set to `true`, `serviceAccount.create` should also be set to either `true` or `false`
     ##
     create: true
     ## Define the generated serviceAccountName when create is set to true
     ## It defaults to "gitlab-runner.fullname" if not provided
     ## DEPRECATED: Please use `serviceAccount.name` instead
     generatedServiceAccountName: ""
   ...
   
   ## Configuration for the Pods that the runner launches for each new job
   ##
   runners:
     # runner configuration, where the multi line string is evaluated as a
     # template so you can specify helm values inside of it.
     #
     # tpl: https://helm.sh/docs/howto/charts_tips_and_tricks/#using-the-tpl-function
     # runner configuration: https://docs.gitlab.com/runner/configuration/advanced-configuration.html
     config: |
       [[runners]]
         [runners.kubernetes]
           namespace = "{{.Release.Namespace}}"
           image = "alpine"
   
     ## Absolute path for an existing runner configuration file
     ## Can be used alongside "volumes" and "volumeMounts" to use an external config file
     ## Active if runners.config is empty or null
     configPath: ""
   ...

   Parameter	Description
   gitlabUrl	The URL of the GitLab server on which you want to register a runner. Example: https://gitlab.example.com.
   runnerToken	Specifies the runner token you obtained in the previous step. The token of a runner is the identifier of the runner. The runner manager associates the pods and Secrets created by the manager with a token based on the token.
   rbac	Specifies whether to enable role-based access control (RBAC). After you enable RBAC, a service account is automatically created. rbac: create: true
   concurrent	The concurrency of the job. Default value: 10. The manager pod consumes a specific amount of memory when managing jobs. If your job has high concurrency, we recommend that you increase the resource configurations of the manager pod.
   unregisterRunners	To terminate the manager pod, run the unregister command. When you use runner registration tokens to register runners, jobs may be disassociated each time a new token is generated. You can add this configuration to resolve this issue. For more information, see FAQ.
   runners.config	The configurations of the runner. The configurations are specified in multi-line strings. You can modify the strings to modify the runner configurations.

4. Install GitLab Runner in your ACS cluster.

   helm install --namespace default gitlab-runner -f values.yaml --version 0.68.1 gitlab/gitlab-runner

5. Run the following command to check the status of GitLab Runner:

   kubectl get pod | grep gitlab

   Expected output:

   gitlab-runner-7c5b4xxxxx-xxxxx   1/1     Running     0          5m17s

   The output shows that GitLab Runner is installed.

Build an image

This section describes how to build an image in Docker-in-Docker mode and by using kaniko. In this section, the features of ACS are used in the build procedure. For more information about the sample project used in this section, see Java demo.

Reduce CI resource costs based on ACS scaling policies

You can select one of the following methods to use the BestEffort pods provided by ACS to reduce the resource costs of job execution in the CI/CD pipeline.

1. Configure the BestEffort QoS class for all pods in the cluster or the pods that belong to the project.

   We recommend that you configure the BestEffort QoS class for all pods in the cluster. You can also select other QoS classes based on your business requirements.

   1. Configure the runners parameter to specify the BestEffort QoS class for all pods in the cluster.

      ...
      runners:
        config: |
          [[runners]]
            [runners.kubernetes]
              ...
              pod_labels_overwrite_allowed = ".*" # Allow the system to overwrite the labels of project pods based on variables.
            [[runners.kubernetes.pod_labels]]
            "app" = "acs-gitlab-runner"
            "alibabacloud.com/compute-class" = "general-purpose"
            "alibabacloud.com/compute-qos" = "best-effort" # Configure the BestEffort QoS class for pods.
      ...      

   2. You can use the gitlab-ci.yml file to configure pod QoS classes for each repository based on your business requirements. To do this, add the variables parameter to the gitlab-ci.yml file. For more information, see the Use kaniko to build an image in non-privileged mode section in this topic.

2. Configure a ResourcePolicy to ensure elasticity.

   You can use a ResourcePolicy to configure a custom scheduling policy to reduce resource costs for your CI jobs. This way, your CI jobs are preferentially scheduled to BestEffort jobs, which are cost-effective. When BestEffort pods are exhausted in a region, the system automatically creates Default pods to ensure the continuity and availability of your CI jobs.

   apiVersion: scheduling.alibabacloud.com/v1alpha1
   kind: ResourcePolicy
   metadata:
     name: rp-demo
     namespace: default
   spec:
     selector: # Specify the label selector used to select pods. In this example, the ResourcePolicy is applied to pods that have the app=stress label.
       app: acs-gitlab-runner
     units: # Specify the priorities of different types of nodes for pod scheduling.
     - resource: acs # Prioritize BestEffort resources.
       podLabels:
         alibabacloud.com/compute-class: general-purpose
         alibabacloud.com/compute-qos: best-effort
     - resource: acs # Apply for Default resources when the preceding resources are out of stock.
       podLabels:
         alibabacloud.com/compute-class: general-purpose
         alibabacloud.com/compute-qos: default

FAQ

How do I delete residual pods after the manager pod is recreated or restarted?

Cause

The system used a different runner token to recreate or restart the manager pod. The token of a runner is the identifier of the runner. In this case, the manager pod cannot manage the original job pods.

Solution

• Go to the GitLab console and recreate the runner. Specify the runner token in the runner configuration file.

• If a registration token is used for runner registration during the startup process, we recommend that you mount a Secret to the manager pod and specify the token generated during initial runner registration in the Secret. This ensures that the initial token is used each time the manager pod is restarted.

• We recommend that you enable the FF_USE_POD_ACTIVE_DEADLINE_SECONDS feature gate. This allows you to specify the time to live (TTL) of each runner worker, which is similar to a resource reclaim policy.