README

AWS CDK GitLab Runner autoscaling on EC2

This project provides a CDK construct to execute jobs on auto-scaled EC2 instances using the Docker Machine executor.

Running out of Runner minutes, using Docker-in-Docker (dind), speed up jobs with shared S3 Cache, cross compiling/building environment multiarch, cost effective autoscaling on EC2, deploy directly from AWS accounts (without AWS Access Key), running on Spot instances, having a bigger build log size

Install

TypeScript

npm install @pepperize/cdk-autoscaling-gitlab-runner

yarn add @pepperize/cdk-autoscaling-gitlab-runner

Python

pip install pepperize.cdk-autoscaling-gitlab-runner

C# / .Net

dotnet add package Pepperize.CDK.AutoscalingGitlabRunner

Quickstart

Create a new AWS CDK App in TypeScript with projen

mkdir gitlab-runner
cd gitlab-runner
git init
npx projen new awscdk-app-ts

Configure your project in .projenrc.js
- Add deps: ["@pepperize/cdk-autoscaling-gitlab-runner"],
Update project files and install dependencies
```
npx projen
```
Register a new runner

Registering runners:
- For a shared runner, go to the GitLab Admin Area and click Overview > Runners
- For a group runner, go to Settings > CI/CD and expand the Runners section
- For a project runner, go to Settings > CI/CD and expand the Runners section
Optionally enable: Run untagged jobs [x] Indicates whether this runner can pick jobs without tags

See also Registration token vs. Authentication token

Retrieve a new runner authentication token

curl --request POST "https://gitlab.com/api/v4/runners" --form "token=<your register token>" --form "description=gitlab-runner" --form "tag_list=pepperize,docker,production"

Store runner authentication token in SSM ParameterStore

Create a String parameter

aws ssm put-parameter --name "/gitlab-runner/token" --value "<your runner authentication token>" --type "String"

Add to your main.ts

import { Vpc } from "@aws-cdk/aws-ec2";
import { App, Stack } from "@aws-cdk/core";
import { GitlabRunnerAutoscaling } from "@pepperize/cdk-autoscaling-gitlab-runner";

const app = new App();
const stack = new Stack(app, "GitLabRunnerStack");
const vpc = Vpc.fromLookup(app, "ExistingVpc", {
  vpcId: "<your vpc id>",
});
const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
  parameterName: "/gitlab-runner/token",
});
new GitlabRunnerAutoscaling(stack, "GitlabRunner", {
  network: {
    vpc: vpc,
  },
  runners: [
    {
      token: token,
      configuration: {
        // optionally configure your runner
      },
    },
  ],
});

Create service linked role

(If requesting spot instances, default: true)

aws iam create-service-linked-role --aws-service-name spot.amazonaws.com

Configure the AWS CLI
Deploy the GitLab Runner
```
npm run deploy
```

Example

Custom cache bucket

By default, an AWS S3 Bucket is created as GitLab Runner's distributed cache. It's encrypted and public access is blocked. A custom S3 Bucket can be configured:

const cache = new Bucket(this, "Cache", {
  // Your custom bucket
});
const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
  parameterName: "/gitlab-runner/token",
});

new GitlabRunnerAutoscaling(this, "Runner", {
  runners: [
    {
      token: token,
    },
  ],
  cache: { bucket: cache },
});

See example, GitlabRunnerAutoscalingCacheProps

Configure Docker Machine

By default, docker machine is configured to run privileged with CAP_SYS_ADMIN to support Docker-in-Docker using the OverlayFS driver and cross compiling/building with multiarch.

See runners.docker section in Advanced configuration

import { GitlabRunnerAutoscaling } from "@pepperize/cdk-autoscaling-gitlab-runner";
import { StringParameter } from "aws-cdk-lib/aws-ssm";

const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
  parameterName: "/gitlab-runner/token",
});

new GitlabRunnerAutoscaling(this, "Runner", {
  runners: [
    {
      token: token,
      configuration: {
        environment: [], // Reset the OverlayFS driver for every project
        docker: {
          capAdd: [], // Remove the CAP_SYS_ADMIN
          privileged: false, // Run unprivileged
        },
        machine: {
          idleCount: 2, // Number of idle machine
          idleTime: 3000, // Waiting time in idle state
          maxBuilds: 1, // Max builds before instance is removed
        },
      },
    },
  ],
});

See example, DockerConfiguration

Bigger instance type

By default, t3.nano is used for the manager/coordinator and t3.micro instances will be spawned. For bigger projects, for example with webpack, this won't be enough memory.

const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
  parameterName: "/gitlab-runner/token",
});

new GitlabRunnerAutoscaling(this, "Runner", {
  manager: {
    instanceType: InstanceType.of(InstanceClass.T3, InstanceSize.SMALL),
  },
  runners: [
    {
      instanceType: InstanceType.of(InstanceClass.T3, InstanceSize.LARGE),
      token: token,
      configuration: {
        // optionally configure your runner
      },
    },
  ],
});

See example, GitlabRunnerAutoscalingManagerProps, GitlabRunnerAutoscalingJobRunnerProps

Different machine image

By default, the latest Amazon 2 Linux will be used for the manager/coordinator. The manager/coordinator instance's cloud init scripts requires yum is installed, any RHEL flavor should work. The requested runner instances by default using Ubuntu 20.04, any OS implemented by the Docker Machine provisioner should work.

const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
  parameterName: "/gitlab-runner/token",
});

new GitlabRunnerAutoscaling(this, "Runner", {
  manager: {
    machineImage: MachineImage.genericLinux(managerAmiMap),
  },
  runners: [
    {
      machineImage: MachineImage.genericLinux(runnerAmiMap),
      token: token,
      configuration: {
        // optionally configure your runner
      },
    },
  ],
});

See example, GitlabRunnerAutoscalingManagerProps, GitlabRunnerAutoscalingJobRunnerProps

Multiple runners configuration

Each runner defines one [[runners]] section in the configuration file. Use Specific runners when you want to use runners for specific projects.

const privilegedRole = new Role(this, "PrivilegedRunnersRole", {
  // role 1
});

const restrictedRole = new Role(this, "RestrictedRunnersRole", {
  // role 2
});

const token1 = StringParameter.fromStringParameterAttributes(stack, "Token", {
  parameterName: "/gitlab-runner/token1",
});

const token2 = StringParameter.fromStringParameterAttributes(stack, "Token", {
  parameterName: "/gitlab-runner/token2",
});

new GitlabRunnerAutoscaling(this, "Runner", {
  runners: [
    {
      token: token1,
      configuration: {
        name: "privileged-runner",
      },
      role: privilegedRole,
    },
    {
      token: token2,
      configuration: {
        name: "restricted-runner",
        docker: {
          privileged: false, // Run unprivileged
        },
      },
      role: restrictedRole,
    },
  ],
});

See example, GitlabRunnerAutoscalingProps

Spot instances

By default, EC2 Spot Instances are requested.

const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
  parameterName: "/gitlab-runner/token",
});

new GitlabRunnerAutoscaling(this, "Runner", {
  runners: [
    {
      token: token,
      configuration: {
        machine: {
          machineOptions: {
            requestSpotInstance: false,
            spotPrice: 0.5,
          },
        },
      },
    },
  ],
});

See example, MachineConfiguration, MachineOptions

Custom runner's role

To deploy from within your GitLab Runner Instances, you may pass a Role with the IAM Policies attached.

const role = new Role(this, "RunnersRole", {
  assumedBy: new ServicePrincipal("ec2.amazonaws.com", {}),
  inlinePolicies: {},
});
const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
  parameterName: "/gitlab-runner/token",
});

new GitlabRunnerAutoscaling(this, "Runner", {
  runners: [
    {
      role: role,
      token: token,
      configuration: {
        // optionally configure your runner
      },
    },
  ],
});

See example, GitlabRunnerAutoscalingProps

Vpc

If no existing Vpc is passed, a VPC that spans a whole region on will be created. This can become costly, because AWS CDK configured also the routing for the private subnets and creates NAT Gateways (one per AZ).

const vpc = new Vpc(this, "Vpc", {
  // Your custom vpc
});
const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
  parameterName: "/gitlab-runner/token",
});

new GitlabRunnerAutoscaling(this, "Runner", {
  runners: [
    {
      token: token,
      configuration: {
        // optionally configure your runner
      },
    },
  ],
  network: { vpc: vpc },
});

See example, GitlabRunnerAutoscalingProps

Zero config

Deploys the Autoscaling GitLab Runner on AWS EC2 with the default settings mentioned above.

Happy with the presets?

const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
  parameterName: "/gitlab-runner/token",
});

new GitlabRunnerAutoscaling(this, "Runner", {
  runners: [
    {
      token: token,
      configuration: {
        // optionally configure your runner
      },
    },
  ],
});

See example, GitlabRunnerAutoscalingProps

Projen

This project uses projen to maintain project configuration through code. Thus, the synthesized files with projen should never be manually edited (in fact, projen enforces that).

To modify the project setup, you should interact with rich strongly-typed class AwsCdkTypeScriptApp and execute npx projen to update project configuration files.

In simple words, developers can only modify .projenrc.js file for configuration/maintenance and files under /src directory for development.

Usage no npm install needed!

README

AWS CDK GitLab Runner autoscaling on EC2

Install

TypeScript

Python

C# / .Net

Quickstart

Example

Custom cache bucket

Configure Docker Machine

Bigger instance type

Different machine image

Multiple runners configuration

Spot instances

Custom runner's role

Vpc

Zero config

Projen