Lab: Setting up a Runner in the Cloud (AWS)

Steven J Zeil

Last modified: Dec 29, 2020
Contents:

Abstract

In an upcoming section, we will take the next step in project automation by introducing continuous integration, in which the more time-consuming build activities are run in the background on a remote server.

Running continuous integration on gitlab-ci requires

  1. a gitlab project with an automated build,

  2. a designated runner machine/process to execute that build, and

  3. a configuration script, .gitlab-ci.yml that tells the runner what to do to run your build.

You should already have the first of these, courtesy of this previous assignment.

In this lab, you will tackle the second of these, setting up a GitLab CI runner on an AWS virtual machine “in the cloud.”

1 Access to AWS

1.1 AWS Educate

It’s possible that parts of this will be different for me as an instructor than for you with student access. Let me know

  1. Log in to AWS Educate.

  2. Go “Classrooms & Credits” on the tool bar.

  3. Select “Request or go to an AWS Educate Classroom”

  4. Select “Go to my classrooms”

    You should see this course listed under “Classrooms where I am a Student”.

1.2 AWS Education Starter Account

  1. Still on the My Classrooms page, select “AWS Account” from the toolbar.

  2. You should see something like

     

    possibly with a link to set up your account.

  3. Set your account up, if necessary.

  4. Click the orange button to see your account status:

     

  5. Click the AWS Console button to actually access AWS.

2 Creating a Linux Server

You should be looking at something like:

 

We’re going to set up a basic Linux server that we can use to do test and CI for your Gitlab1 Java project.

2.1 Select an OS

We are limited to services that are handled by the AWS Educate program. This does not include LightBeam, the newer, simplified approach to launching Linux/Windows servers.

Fortunately, the older interface is not terribly complicated.

  1. Select EC2. (EC stands for “Elastic Computing”.)

  2. Select “Launch Instance” from the orange menu titled “Launch Instance” in the box titled “Launch Instance”.

  3. You’ll be presented with a list of OS versions and machine types that are available.

    • Select “Ubuntu Server 20.04 LTS”, 64-bit (x86).

2.2 Choose an Instance Type

Next, you will be asked to select the simulated machine “size”.

 

Our educational accounts are limited to “nano” and “micro” sizes.

  1. Pick General purpose, t2.micro
  2. Click Next: Configure Instance Details
    • I selected “Request Spot instances”. Purportedly, this saves money at the cost of occasional service unavailibility. For a CI runner, I don’t think that would be a major concern.
    • I left everything else on defaults, and clicked “Next: Add Storage”
  3. I raised the drive size to 30Gb, the largest available without extra charge.

  4. Click “Next: Add Tags”

  5. Adding tags is optional. It can help to identify your machines if you have a lot of them.

    • Click “Next: Configure Security Group”

  6. This lets you set up firewall rules that restrict access to your machine.

    • I used my home IP address and the address of my CS office PC.
    • /32 on the end of the IP addresses allows the last 5 bits to vary.
  7. Click “Review and Launch”, then “Launch”.

2.3 SSH Keys

Next you will be given the option of supplying an SSH key pair or creating a pair for access to the machine.

Follow the instructions to create a new pair and to download a copy of the private key (.pem format).

Note:

  1. Follow through the instructions to actually launch your machine.

2.4 Checking on Your Instance

Return to the AWS Console and click on “Running instances” to see the details of your server.

 

  1. Take note of the “Public IPv4 DNS” – that’s the name of your new server. You can use ordinary ssh with the private key you downloaded to access the server.
    • User names are OS defaults, in this case, ubuntu.
    • That’s more secure than it might look, because the ssh server is configured to only connect by SSH keys, not by username/password.
  2. Alternatively, select the row with your server, click the Connect button above, then go to the “SSD client” tab for detailed instructions.

3 Installing and Setting Up a CI Runner

3.1 Install gitlab-ci-runner

Open an ssh connection to your new AWS server.

gitlab-ci-runner is distributed as a standard Ubuntu install package.

sudo apt-get update
curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
sudo apt-get install gitlab-runner
sudo gitlab-ci-multi-runner status

The last command should report that the service is “running”.

Note: this program can be invoked either as gitlab-ci-multi-runner or as gitlab-runner.

3.2 Create a shell Runner

Still in that SSH session:

  1. sudo gitlab-runner register

  2. You will be asked for the gitlab-ci coordinator URL (e.g. https://forge350.cs.odu.edu/ or https://git-community.cs.odu.edu/

  3. You will be asked for the gitlab-ci token for this runner.

     

    You get that by using your web browser to visit your GitLab1 project, look under Settings for CI/CD, click Expand in the “Runners” section and look for something like the text to the right. Copy the registration token, and paste it into your SSH session.

  4. Enter a short description and tag (e.g., my-aws-shell)

  5. When asked for the executor, answer shell.

    • You can explore other possibilities later, but shell is the easiest to get started with.

  6. When gitlab-runner is finished, give the command

    sudo gitlab-runner list

    You should see your new runner listed there.

  7. Finally, the default account setup in Ubuntu clashes with the gitab runner:

    sudo rm /home/gitlab-runner/.bash_logout

  8. Return to your web browser and refresh the Runners page. You should see your new runner listed.

4 Install Other Required Software

If your runner will need additional software, now is the time to install it in your SSH. In this case, you will need

  1. Java, both to compile the code and simply to run gradle,
  2. rsync, to transfer results back to your project website.
ubuntu@ip-xxx-xx-xx-xxx:~$ sudo apt-get install rsync
    ⋮
ubuntu@ip-xxx-xx-xx-xxx:~$ javac -version

Command 'javac' not found, but can be installed with:

sudo apt install default-jdk              # version 2:1.11-72, or
sudo apt install openjdk-11-jdk-headless  # version 11.0.8+10-0ubuntu1~20.04
sudo apt install openjdk-8-jdk-headless   # version 8u265-b01-0ubuntu2~20.04
sudo apt install openjdk-13-jdk-headless  # version 13.0.3+3-1ubuntu2
sudo apt install openjdk-14-jdk-headless  # version 14.0.1+7-1ubuntu1
sudo apt install ecj                      # version 3.16.0-1

ubuntu@ip-xxx-xx-xx-xxx:~$ sudo apt install openjdk-11-jdk-headless
    ⋮
ubuntu@ip-xxx-xx-xx-xxx:~$ javac -version
javac 11.0.8

5 Set up CI in the Project

All that remains is to set up the .gitlab-ci.yml file in the GitLab project.

You’ll tackle that in the next assignment.