# Managing Third-Party Libraries

Abstract

Modern build managers provide support for

• importing 3rd-party libraries,
• managing dependencies among those libraries, and
• publishing our own project deliverables.

so that all of these can be handled quickly and efficiently as part of the normal build.

# 1 Issues

Baselines may include multiple 3rd-party libraries.

• We may rely on specific version of our chosen libaries.

• Older versions may not provide the functions we need, or may have bugs that would affect our project.
• Newer versions may alter or remove functions we have been using, or introduce new bugs that affect our project.
• Those libraries may require other libraries.

• The library versions that we require might onloy be compatible with specific versions of those other libraries.

Example: Report Accumulator

A project of mine, this page lists the 3rd party libraries required by the project.

• The page lists various stages of the build.

• Expand each stage to see the libraries that I requested for that stage
• E.g., for testCompile, I requested JUnit 4.12 8 Which in turned needed hamcrest-core 1.3

• For the findbugs stage, I requested findbugs 3.0.1

• And that required multiple libraries, including asm-commons 5.0.2
• asm-commons 5.0.2 required asm-tree 5.0.2
• asm-tree 4.0.2 asm-asm 5.0.2

# 2 Basic Concepts

## 2.1 Repositories

A repository is a collection of packages (usually libraries) and metadata about packages

• The metadata identifies
• the package,
• the versions available,
• their dependencies on other packages, and
• the location from which the package may be downloaded.
• In some cases, the metadata may point to other repositories as the actual download location.

### 2.1.1 Common Java Repositories

The Java world is dominated by

• Maven Central (a.k.a ibiblio)

• JCenter

• Try searching each of these for junit

• Notice range of versions available
• Select one (e.g., 4.12)
• Explore the snippets for the various build managers
• These are instructions on how to add the library to your build.

### 2.1.2 Local Repositories

It’s common for development organizations to host their own repository.

• A place to put their own deliverables
• A “cache” to speed access to libraries commonly used by many developers within the organization.
• preserve older versions needed for a project baseline

Local repositories can be

• These may support a “pass-through” to Maven Central and/or JCenter for libraries that have not yet been locally cached.
• or little more than a file drop

## 2.2 Identifying Packages

Packages are typically identified by

• The group or organization

• Generally given in the Java style of a reversed URL, e.g. org.apache.ant, edu.odu.cs.extract
• Like some Hollywood celebrities, some organizations become big enough stars to be simply known by a single name, e.g., junit
• The artifact name

• E.g., junit, hamcrest-core, apache-commons
• Version number

• Single versions are identified exactly: 4.12
• Depending on the build manger being used, you may be able to specify things like
• 4.11+: 4.11 or later
• latest.integration: latest official release
• 1.2-SNAPSHOT: a version under development. The stapshot modifier serves as a notice that the actual package may be updated without a change in version number.

## 2.3 Scopes

Not every library that your project needs is needed all of the time.

• If your “main code” uses a library, you would need that during compilation, testing, and possibly packaging.
• Libraries like junit would only be needed when compiling and running the unit tests.
• Libraries like findbugs, checkstyle (discussed later in the semester) are only used during report generation.
• A build manager may support “plug-ins” that extend or modify the build process. These plug-ins are typically packaged much like other 3rd-party libraries, but are needed much earlier in the build process.

So build managers allow you to import libraries in selected scopes which indicate when the library is actually needed.

This affects

• Whether a library is downloaded or updated during a given build.
• Java CLASSPATH and other settings made available to the build steps.

# 3 Adding Libraries to a Java Project in Different Build Managers

## 3.1 Maven

### 3.1.1 Specifying a Dependency

• Examine the pom.xml file and look for the dependencies section.

• You’ll see something like

  <dependencies>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.10</version>
<scope>test</scope>
</dependencies>


• Could also say [4.12,] to get versions 4.12 or greater

### 3.1.2 Fetching Dependencies

When the build is run (e.g. mvn package):

• Maven does a transitive search over the dependencies for a project

• Tries to find a mutually compatible set of versions
• Helps if you give it some flexibility
• Maven then downloads the required libraries automatically

• Downloaded libraries are cached (e.g., ~/.m2)

### 3.1.3 Transitive Dependencies

How does Maven know whether junit itself depends on other libraries?

• Here is the actual published content for Junit 4.12.

The .pom file is the metadata.

• Look inside. Search for <dependencies>. You’ll find

  <dependencies>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-core</artifactId>
<version>1.3</version>
<scope>compile</scope>
</dependencies>


• This is the same kind of info that we put into our own pom.xml file

• And is, presumably, taken from the pom.xml that the JUnit team used to maintain their builds.
• Publishing the dependency information along with the libraries leads to an accumulated base of information on library dependencies.

### 3.1.4 Choosing Repositories

• Technically, our project has two repositories

• a remote repository, at ibiblio
• a local repository holding my cache
• Team projects will often employ an intermediate shared repository

• reduce cache duplication
• provide a mechanism for managing subproject modules
• provide a common storage area for libraries not available on ibiblio
• preserve older versions needed for a project baseline

### 3.1.5 Example: specifying a repository

In pom.xml:

  <repositories>
<repository>
<snapshots>
<enabled>false</enabled>
</snapshots>
<id>central</id>
<name>libs-release</name>
<url>http://archive350.cs.odu.edu:8080/artifactory/libs-release</url>
</repository>
</repositories>


• We could then, for example, import early versions of zipdiff and trilead-ssh that were part of the baseline for the Extract project
• At the time they were added to the project baseline, they were not in the remote Maven repository
• As time has passed, those libraries were added, but have started with versions later than the project baseline

## 3.2 Ivy (ant)

• Technically can run as a standalone application, but really the ant integration is key.

• Use is similar to the dependencies/repositories portion of Maven POMs

• Ivy can retrieve from and publish to the established Maven repositories

### 3.2.1 Getting Started with Ivy

The Ivy project pages include some “magic” ant code to * download the latest version of Ivy * add a resolve-ivy task to be invoked before you compile your code * This task will download any 3rd-party dependencies and add them to your compilation classpaths.

### 3.2.2 Specifying our Desired Libraries

We list the libraries our project needs in a separate ivy.xml file

<ivy-module version="2.0">
<dependencies>
<dependency org="de.jflex" name="jflex" rev="1.4.3"/>
<dependency org="junit" name="junit" rev="4.10"/>
</dependencies>
</ivy-module>



This should look familiar to the section from the Maven pom:

  <dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.10</version>
<scope>test</scope>
</dependency>
</dependencies>



### 3.2.3 Invoking Ivy

Once Ivy is installed, we can invoke it:

build.xml2.listing
<project name="codeAnnotation" basedir="." default="build"
xmlns:ivy="antlib:org.apache.ivy.ant">
⋮

<target name="resolve-ivy" depends="install-ivy">
<ivy:resolve/>                         ➀
<ivy:cachepath pathid="ivy.path" />  ➁
</target>

<target name="generateSource" depends="resolve-ivy">
<mkdir dir="target/gen/java"/>
<jflex file="src/main/jflex/code2html.flex"    ➃
destdir="target/gen/java"/>
<jflex file="src/main/jflex/code2tex.flex"
destdir="target/gen/java"/>

⋮

• The ivy:resolve task fetches the libraries we need.

• They will be kept in a cache (~/.ivy).
• This creates an Ant path named ivy.path containing the list of all libraries fetched.

• Includes both the ones we explicitly named in our ivy.xml and ones that those depended upon.

• Here we can bind the task name for the JFlex library that we have loaded,

• enabling us to later use the <jflex ... > task
• Making the build depend on the Ivy sequence guarantees that the library availability will be managed at each build

### 3.2.4 Ivy & Repositories

• Ivy can use most maven repositories.

• Choice of repositories is controlled by an ivysettings.xml file.

• By default, uses the same ibiblio service that is Maven’s default.

An example of an ivysettings.xml file:

ivysettings.xml.listing
<?xml version="1.0" encoding="UTF-8"?>
<ivy-settings>
<settings defaultResolver="default" />
<resolvers>
<chain name="default">   ➀
<ibiblio name="central" m2compatible="true"/> ➁
</chain>
</resolvers>
</ivy-settings>

• A “chain” means that we want Ivy to search these repositories in sequence.

In this case, I only have one thing in the chain, but for other projects I may add additional repositories.

• Look in the Maven Central archive.

### 3.2.5 Ivy & Eclipse

• Allows you to add a “Library” to your Eclipse project’s build path
• This library holds everything that Ivy fetches based upon your ivy.xml file.

As is often the case, gradle tends to provide simpler syntax to do much the same things as Maven and Ant.

### 3.3.1 Selecting libraries in Gradle

In build.gradle:

apply plugin: 'java'

⋮

repositories {
jcenter()   ➀

// Use my own CS dept repo
ivy {      ➁
url 'https://secweb.cs.odu.edu/~zeil/ivyrepo'
}
}

dependencies { ➂
compile 'net.sf.saxon:saxon-dom:8.7+'
testCompile 'junit:junit:4.12'
testRuntime 'edu.odu.cs.zeil:reportAccumulator:1.1+'


}

Here you see both the choice of libraries and of repositories , .

The dependencies indicate

• The scope (e.g., testCompile, which actually includes compiling and running of the unit tests)

• The package, specified as organization:package:versionNumber

# 4 Eclipse and Library Management

Each of the build/library managers we have discussed have support in Eclipse

• May provide special editors for editing the build manager configuration files.

• Provide ways to launch the build manager from within Eclipse

• Sometimes it’s easier to launch separately in a separate window, e.g., from the command line from the Gradle GUI mode.

• Remember to “refresh” the project in Eclipse afterwards

• Most importantly, allows Eclipse Java projects to use any 3rd-party libraries fetched by the build/library manager.

• Otherwise the built-in Eclipse compilation and smart-editing features would mistakenly believe that your code was full of references to non-existent classes and funnctions.

• May allow some management of the cache of old libraries.

## 4.1 Eclipse and Maven

• Can also hope for convenience functions

• In particular, POM editors
• Two plugins currently

• m2eclipse, older plugin
• Eclipse IAM
• Allows you to create new Maven projects or to import existing ones into Eclipse.

## 4.3 Eclipse and Gradle

Eclipse has various Gradle plugins available.

The most popular seems to be the BuildShip package, available from the Gradle Marketplace (in the Eclipse Help menu).

• Allows import (only) of existing Gradle projects

• Allows launching Gradle from within Eclipse

• from a special Gradle Tasks panel, not from the package explorer.
• Allows Eclipse Java projects to find 3rd-party libraries loaded by Gradle.

# 5 Complicating Factors

## 5.1 Plug-ins

Plug-ins to build managers modify the build process itself

# 6 <<<<<<< HEAD

• unlike “normal” 3rd-party libraries that affect how the code is compiled and run, but not when or whether the compilation steps take place.

Consequently, plug-ins generally need to be fetched and processed at the very beginning of a build, much earlier than most 3rd-party libraries.

• Nonetheless, it is convenient to use the same dependency/repository scheme to publish and import plug-ins.

### 6.0.1 Separate but similar

Plug-ins to build managers are generally handled separately from other items, though the syntax is often similar.

refs/remotes/origin/f19kennedy

• unlike “normal” 3rd-party libraries that affect how the code is compiled and run, but not when or whether the compilation steps take place.

Consequently, plug-ins generally need to be fetched and processed at the very beginning of a build, much earlier than most 3rd-party libraries.

• Nonetheless, it is convenient to use the same dependency/repository scheme to publish and import plug-ins.

### 6.0.2 Separate but similar

Plug-ins to build managers are generally handled separately from other items, though the syntax is often similar.

• For example, in gradle, plug-ins are retrieved from respositories specified in a pluginManagement section in settings.gradle:

pluginManagement {
repositories {
jcenter()
ivy { // for
url 'https://secweb.cs.odu.edu/~zeil/ivyrepo'
}
}
}

• The syntax and options within the pluginManagement area are almost identical to the way we handle normal package repositories.
• Then we load the plugins from a plugins section in build.gradle:

plugins {
id 'java'
id 'edu.odu.cs.report_accumulator' version '1.3'
}

• Maven has a similar approach, isolating plug-ins in a <plugins> section.

## 6.1 Publishing

Publishing artifacts to a repository is also supported by maven, Ant/Ivy, & Gradle.

• Typically more complicated.
• mainly because most repositories are open to download, but require login credentials to upload.

Example: in Gradle build.xml

publishing {
publications {
ivyJava(IvyPublication) {
organisation 'edu.odu.cs.zeil'
module 'cowem-plugin'
revision project.version
descriptor.status = 'integration'   // milestone, release
descriptor.branch = 'v1'

from components.java
}
}

repositories {
ivy {
name 'ivyRepo'
url 'sftp://atria.cs.odu.edu:22/home/zeil/secure_html/ivyrepo'
// Readable via https://www.cs.odu.edu/~zeil/ivyrepo
credentials {
⋮
}
}
}
}


## 6.2 Login Credentials

Providing login credentials to a build is problematic.

• Can’t insert passwords into the build file, because the build file will be checked in to version control, so we would be publishing our credentials in plain view, for all time!

A common work-around is to store a few build “properties” in a file in the user’s home directory, not kept as part of the package version control.

Example: in Gradle build.xml

// Credentials are loaded from ~/.gradle/gradle.properties
if(project.hasProperty("ivyRepoUser")){        ➀
ext.ivyRepoUser = "$ivyRepoUser"; } else { ext.ivyRepoUser = "user"; } if(project.hasProperty("ivyRepoPass")){ ext.ivyRepoPass = "$ivyRepoPass";
} else {
}

⋮

publishing {
⋮
credentials {                    ➁
}
⋮
}

• looks to see if properties ivyRepoUser and ivyRepoPass were loaded from ~/.gradle/gradle.properties. If so, it adds them as “extension” properties of the current gradle project.

• passes those new extension properties as login credentials to a repository.

• If the person running the build di not have a ~/.gradle/gradle.properties or it did not contain the properties ivyRepoUser and ivyRepoPassproeprties, then attempted uploads to the repository would use a default username and password that would, presumably, fail.

# 7 What about C++?

Sadly, the situation for C++ is somewhat more complicated.

• Should C++ libraries be deployed as source or binaries?

• Binary library formats (e.g., DLL’s in windows, .a or .so in *nix) are OS-specific.
• Developers generally need header (.h) files as well as the compiled code if they are to compile their own applications.
• There are Operating System-dependent solutions for deploying binaries.

• Windows install files can check to see if required libraries for an application are already present and fetch and install them if necessary.
• Linux repositories provide a similar mechanism.
• Even more than OS-dependent, these are distribution dependent.
• Different variants of Linux use different repository formats
• These solutions work for deploying applications but not so well for developers.

## 7.1 What’s Holding Us Up?

We Don’t Have Anything Equivalent to Maven Central for C++

There’s no recognized central place to publish and find open source C++ libraries.

• No standard for what such a library would look like.

• Compiled binaries would have to be provided for multiple OS variants.
• Partly, that’s because we don’t have build managers that would take advantage of such libraries if they existed.

We Don’t Have Anything Equivalent to Maven/Ivy/Gradle Dependency Management for C++

There are no build managers capable of processing dependencies for C++ while simultaneously factoring in the platform-specific requirements.

• Who would write such a tool when there’s no central repositories where such libraries can be stored?

Which Came First, the Chicken or the Egg?

• Who wants to set up repositories without build managers that can use them?

• Who wants to write such build managers if there are no repositories that they could work with?