Continuous Integration

Steven J Zeil

Last modified: Dec 21, 2019
Contents:
1 Big Builds
2 Continuous Integration
2.1 Key Ideas
2.2 Continuous Integration Systems
3 Case study: Jenkins
3.1 Projects on Jenkins
4 Case study: gitlab-ci
4.1 gitlab-ci setup
5 gitlab-ci vs Jenkins
6 Case Study: Enhanced reporting
6.1 Generating Graphs
6.2 Generating the Data
6.3 Report Accumulator
7 Related ideas

Abstract

In continuous integration, the practices of version control, automated building, automated configuration, and automated testing are combined so that, as changes are checked in to the version control repository, the system is automatically rebuilt, tested, reports generated, and the results posted to a project website.

1 Big Builds

Think of everything we have started to put into our automated builds:

and, coming up, we will want to expand our testing to include

There’s a danger of the builds becoming so unwieldy and slow that programmers will start to look for ways to circumvent steps,

Do We Need to do All of Those Steps, All of the Time?

One possible breakdown:

Every build Occasional
fetching and setup of 3rd party libraries documentation generation
static analysis static analysis reports
compilation deployment/publication of artifacts
unit testing updating of project website
packaging of artifacts integration testing
test coverage reporting
system testing

This should provide someone actively working on a specific module/story the info they need, deferring some of the more time-consuming build activities.

How do we divide these steps in the build?

2 Continuous Integration

When we combine

we can rebuild and retest automatically as developers check in changes.

2.1 Key Ideas

Our project should have the characteristics:

2.1.1 Advantages

2.1.2 Disadvantages

2.2 Continuous Integration Systems

A CI system consists of a server/manager and one or more runners…

2.2.1 The Continuous Integration Server

A continuous integration server is a network-accessible machine that

2.2.2 Continuous Integration Runners

A CI runner (a.k.a., nodes or slave processors) is a process on a machine that

When notified by the server, the runner

  1. Checks out a designated branch of a project from its version control system.

  2. Runs the build.

  3. Publishes reports on the results of the build(s).

3 Case study: Jenkins

Jenkins is a popular CI server.

The CS Dept runs its own Jenkins server

3.1 Projects on Jenkins

When you set up a project on Jenkins you must supply:

3.1.1 Jenkins and Project Reports

4 Case study: gitlab-ci

gitlab-ci is a CI server integrated into Gitlab.

4.1 gitlab-ci setup

  1. Projects must activate gitlab-ci by designating a runner, generally on a remote machine under the developer’s control.

    • You can have multiple runners. For example, you could have a Linux runner, a Windows 10 runner, and a MacOS runner.

  2. Then add a file .gitlab-ci.yml to the repository root directory.

    This is a YAML script that gets run after each new commit.

    Script can limit which branches it applies to

4.1.1 An example of .gitlab-ci.yml

    stages:
      - build
      - test
      - deploy


    build-job:
      tags:
       - e-3208
      stage: build
      script:
       - eval $(ssh-agent -s -t 600)
       - ssh-add <(echo "$REPORTS_SSH_KEY")
       - cd codeCompCommon
       - ./gradlew build deployReports
      only:
       - master
      artifacts:
       paths:
         - codeCompCommon/build/libs/codeCompCommon-1.3.jar
         - README.md

4.1.2 The script dissected

      script:
       - eval $(ssh-agent -s -t 600)                         ➀
       - ssh-add <(echo "$REPORTS_SSH_KEY")                  ➁
       - cd codeCompCommon                                   ➂
       - ./gradlew build deployReports                       ➃
  1. This line launches an ssh key agent. The -t 600 option limits this agent to a maximum of 600 seconds before it shuts itself down.
  2. We know the ssh-add command as a way to add private ssh keys to the agent.

    In this case, the text of the (passphrase-free) private key is being supplied by GitLab as a secret project variable REPORTS_SSH_KEY.

    • These are created as part of the project settings and visible only to project “masters”.

    • They provide a useful way to add private keys and other secret credentials without putting them into the repository files where anyone might be able to download them.

  3. cd into the subproject directory containing the code and the Gradle files

  4. Run the build with targets build and deployReports.

    • We’ll look at deployReports shortly.

5 gitlab-ci vs Jenkins

6 Case Study: Enhanced reporting

Goal: add project website pages with reports from analysis tools and trend information (historical graphs) similar to those offered by Jenkins.

6.1 Generating Graphs

Highcharts is a Javascript package that can generate plots from data captured in CSV (Comma-Separated-Values) format.

Highcharts requires a bit of Javascript to inject a chart into an HTML div element.

6.2 Generating the Data

Where does the data for the plots come from?

6.3 Report Accumulator

Back to build.gradle to load another plugin:

build.gradle.listing 
buildscript {
	repositories {
		⋮
        ivy { // for report-accumulator
            url 'https://secweb.cs.odu.edu/~zeil/ivyrepo'     ➀
        }
	}
	
	dependencies {
		⋮
	    classpath 'edu.odu.cs.zeil:report_accumulator:1.2'
		⋮
    }
}

⋮


// Reporting


import edu.odu.cs.zeil.report_accumulator.ReportStats     ➁
import edu.odu.cs.zeil.report_accumulator.ReportsDeploy


task collectStats (type: ReportStats, dependsOn: ['build','reports']) {  ➂
    description "Collect statistics from various reports & analysis tools"
    reportsURL = 'https://www.cs.odu.edu/~zeil/gitlab/' + project.name + '/reports'
}


task site (dependsOn: ['copyBake', 'copyJDocs', 'collectStats']){ ➃
    description "Build the project website (in build/reports)"
    group "reporting"
}



task deployReports (type: ReportsDeploy, dependsOn: 'site') {  ➄
    description 'Deploy website to remote server'
    group 'reporting'
    deployDestination = 'rsync://zeil@atria.cs.odu.edu:codeCompCommon/reports/'
}
  1. The usual steps to include a plugin.
  2. Import the new task types created by the plugin
  3. A ReportStats task
    • scans the build/reports directory for reports from various tools,
    • extracts a new data point where it can,
    • downloads an existing CSV file for that report from the reportsURL
    • adds the new data point to the end of the CSV file

  4. A slight tweak to our earlier site target to make sure that the ReportStats task is performed

  5. A ReportsDeploy task uploads the build/reports directory to the deployDestination URL.
    • Assumes that any required ssh keys are already in an agent.

7 Related ideas

© 2015-2019, Old Dominion Univ.