Managing Third-Party Libraries
Steven J Zeil
Abstract
Modern build managers provide support for
- importing 3rd-party libraries,
- managing dependencies among those libraries, and
- publishing our own project deliverables.
so that all of these can be handled quickly and efficiently as part of the normal build.
1 Issues
Baselines may include multiple 3rd-party libraries.
-
We may rely on specific version of our chosen libaries.
- Older versions may not provide the functions we need, or may have bugs that would affect our project.
- Newer versions may alter or remove functions we have been using, or introduce new bugs that affect our project.
-
Those libraries may require other libraries.
- The library versions that we require might onloy be compatible with specific versions of those other libraries.
Example: Report Accumulator
A project of mine, this page lists the 3rd party libraries required by the project.
-
The page lists various stages of the build.
- Expand each stage to see the libraries that I requested for that stage
-
E.g., for testCompile, I requested JUnit 4.12 8 Which in turned needed hamcrest-core 1.3
-
For the spotbugs stage, I requested spotbugs 3.1.12
- And that required multiple libraries, including asm-analysis 7.0
- asm-analysis 7.0 required asm-tree 7.0
- asm-tree 7.0 reuqired asm 7.0
- And that required multiple libraries, including asm-analysis 7.0
2 Basic Concepts
2.1 Repositories
A repository is a collection of packages (usually libraries) and metadata about packages
- The metadata identifies
- the package,
- the versions available,
- their dependencies on other packages, and
- the location from which the package may be downloaded.
- In some cases, the metadata may point to other repositories as the actual download location.
2.1.1 Common Java Repositories
The Java world is dominated by
-
Maven Central (a.k.a ibiblio)
-
Try searching each of these for junit
- Notice range of versions available
- Select one (e.g., 4.12)
- Explore the snippets for the various build managers
- These are instructions on how to add the library to your build.
- Notice range of versions available
2.1.2 Local Repositories
It’s common for development organizations to host their own repository.
- A place to put their own deliverables
- A “cache” to speed access to libraries commonly used by many developers within the organization.
- preserve older versions needed for a project baseline
Local repositories can be
-
elaborate (e.g., Artifactory )
- These may support a “pass-through” to Maven Central and/or JCenter for libraries that have not yet been locally cached.
-
or little more than a file drop
2.2 Identifying Packages
Packages are typically identified by
-
The group or organization
- Generally given in the Java style of a reversed URL, e.g.
org.apache.ant
,edu.odu.cs.extract
- Like some Hollywood celebrities, some organizations become big enough stars to be simply known by a single name, e.g.,
junit
- Generally given in the Java style of a reversed URL, e.g.
-
The artifact name
- E.g.,
junit
,hamcrest-core
,apache-commons
- E.g.,
-
Version number
- Single versions are identified exactly:
4.12
- Depending on the build manger being used, you may be able to specify things like
4.11+
: 4.11 or laterlatest.integration
: latest official release1.2-SNAPSHOT
: a version under development. The stapshot modifier serves as a notice that the actual package may be updated without a change in version number.
- Single versions are identified exactly:
2.3 Scopes
Not every library that your project needs is needed all of the time.
- If your “main code” uses a library, you would need that during compilation, testing, and possibly packaging.
- Libraries like
junit
would only be needed when compiling and running the unit tests. - Libraries like
spotbugs
,checkstyle
(discussed later in the semester) are only used during report generation. - A build manager may support “plug-ins” that extend or modify the build process. These plug-ins are typically packaged much like other 3rd-party libraries, but are needed much earlier in the build process.
So build managers allow you to import libraries in selected scopes which indicate when the library is actually needed.
This affects
- Whether a library is downloaded or updated during a given build.
- Java
CLASSPATH
and other settings made available to the build steps.
3 Adding Libraries to a Java Project in Different Build Managers
3.1 Maven
3.1.1 Specifying a Dependency
-
Examine the pom.xml file and look for the
dependencies
section. -
You’ll see something like
<dependencies> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.10</version> <scope>test</scope> </dependencies>
- Could also say
[4.12,]
to get versions 4.12 or greater
- Could also say
3.1.2 Fetching Dependencies
When the build is run (e.g. mvn package
):
-
Maven does a transitive search over the dependencies for a project
- Tries to find a mutually compatible set of versions
- Helps if you give it some flexibility
- Tries to find a mutually compatible set of versions
-
Maven then downloads the required libraries automatically
- Downloaded libraries are cached (e.g.,
~/.m2
)
- Downloaded libraries are cached (e.g.,
3.1.3 Transitive Dependencies
How does Maven know whether junit itself depends on other libraries?
-
Here is the actual published content for Junit 4.12.
The
.pom
file is the metadata. -
Look inside. Search for
<dependencies>
. You’ll find<dependencies> <groupId>org.hamcrest</groupId> <artifactId>hamcrest-core</artifactId> <version>1.3</version> <scope>compile</scope> </dependencies>
-
This is the same kind of info that we put into our own pom.xml file
- And is, presumably, taken from the pom.xml that the JUnit team used to maintain their builds.
- Publishing the dependency information along with the libraries leads to an accumulated base of information on library dependencies.
3.1.4 Choosing Repositories
-
Technically, our project has two repositories
- a remote repository, at ibiblio
- a local repository holding my cache
-
Team projects will often employ an intermediate shared repository
- reduce cache duplication
- provide a mechanism for managing subproject modules
- provide a common storage area for libraries not available on ibiblio
- preserve older versions needed for a project baseline
3.1.5 Example: specifying a repository
In pom.xml:
<repositories>
<repository>
<snapshots>
<enabled>false</enabled>
</snapshots>
<id>central</id>
<name>libs-release</name>
<url>http://archive350.cs.odu.edu:8080/artifactory/libs-release</url>
</repository>
</repositories>
- We could then, for example, import early versions of zipdiff and trilead-ssh that were part of the baseline for the Extract project
- At the time they were added to the project baseline, they were not in the remote Maven repository
- As time has passed, those libraries were added, but have started with versions later than the project baseline
3.2 Ivy (ant)
YAAP1 , Ivy adds the dependency management features pioneered by Maven to ant
-
Technically can run as a standalone application, but really the ant integration is key.
-
Use is similar to the dependencies/repositories portion of Maven POMs
-
Ivy can retrieve from and publish to the established Maven repositories
3.2.1 Getting Started with Ivy
The Ivy project pages include some “magic” ant code to
- download the latest version of Ivy
- add a
resolve-ivy
task to be invoked before you compile your code- This task will download any 3rd-party dependencies and add them to your compilation classpaths.
3.2.2 Specifying our Desired Libraries
We list the libraries our project needs in a separate ivy.xml file
<ivy-module version="2.0">
<dependencies>
<dependency org="de.jflex" name="jflex" rev="1.4.3"/>
<dependency org="junit" name="junit" rev="4.10"/>
</dependencies>
</ivy-module>
This should look familiar to the section from the Maven pom:
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.10</version>
<scope>test</scope>
</dependency>
</dependencies>
3.2.3 Invoking Ivy
Once Ivy is installed, we can invoke it:
<project name="codeAnnotation" basedir="." default="build"
xmlns:ivy="antlib:org.apache.ivy.ant">
⋮
<target name="resolve-ivy" depends="install-ivy">
<ivy:resolve/> ➀
<ivy:cachepath pathid="ivy.path" /> ➁
<taskdef classpath="JFlex.jar"
classname="JFlex.anttask.JFlexTask" name="jflex" /> ➂
</target>
<target name="generateSource" depends="resolve-ivy">
<mkdir dir="target/gen/java"/>
<jflex file="src/main/jflex/code2html.flex" ➃
destdir="target/gen/java"/>
<jflex file="src/main/jflex/code2tex.flex"
destdir="target/gen/java"/>
⋮
-
➀ The
ivy:resolve
task fetches the libraries we need.- They will be kept in a cache (
~/.ivy
).
- They will be kept in a cache (
-
➁ This creates an Ant path named
ivy.path
containing the list of all libraries fetched.-
Includes both the ones we explicitly named in our
ivy.xml
and ones that those depended upon.
-
-
➂ Here we can bind the task name for the JFlex library that we have loaded,
- ➃ enabling us to later use the
<jflex ... >
task
- ➃ enabling us to later use the
-
Making the build depend on the Ivy sequence guarantees that the library availability will be managed at each build
3.2.4 Ivy & Repositories
-
Ivy can use most maven repositories.
-
Choice of repositories is controlled by an
ivysettings.xml
file.- By default, uses the same
ibiblio
service that is Maven’s default.
- By default, uses the same
An example of an ivysettings.xml
file:
<?xml version="1.0" encoding="UTF-8"?>
<ivy-settings>
<settings defaultResolver="default" />
<resolvers>
<chain name="default"> ➀
<ibiblio name="central" m2compatible="true"/> ➁
</chain>
</resolvers>
</ivy-settings>
-
➀ A “chain” means that we want Ivy to search these repositories in sequence.
In this case, I only have one thing in the chain, but for other projects I may add additional repositories.
-
➁ Look in the Maven Central archive.
3.2.5 Ivy & Eclipse
The developers of Ivy provide an Eclipse plugin, IvyDE.
- Allows you to add a “Library” to your Eclipse project’s build path
- This library holds everything that Ivy fetches based upon your
ivy.xml
file.
- This library holds everything that Ivy fetches based upon your
3.3 Gradle
As is often the case, gradle
tends to provide simpler syntax to do much the same things as Maven and Ant.
3.3.1 Selecting libraries in Gradle
In build.gradle
:
apply plugin: 'java'
⋮
repositories {
jcenter() ➀
// Use my own CS dept repo
ivy { ➁
url 'https://www.cs.odu.edu/~zeil/ivyrepo'
}
}
dependencies { ➂
implementation 'net.sf.saxon:saxon-dom:8.7+'
testImplementation 'junit:junit:4.12'
testRuntimeOnly 'edu.odu.cs.zeil:reportAccumulator:1.1+'
}
Here you see both the choice of libraries ➂ and of repositories ➀, ➁.
The dependencies indicate
-
The scope (e.g.,
testCompile
, which actually includes compiling and running of the unit tests) -
The package, specified as organization:package:versionNumber
Scopes in Gradle
dependencies {
implementation 'net.sf.saxon:saxon-dom:8.7+'
testImplementation 'junit:junit:4.12'
testRuntimeOnly 'edu.odu.cs.zeil:reportAccumulator:1.1+'
}
If project P depends on library L, then the most common scopes recognized by Gradle are:
- api
- P requires L to compile and to run. If P is itself a library, anyone who uses P must have L available to compile and run as well.
Generally this means that some class declared in L is used as a parameter or a return type by some public function in P.
- implementation
- P requires L to compile and run the code, but this is hidden from users of P.
- testImplementation
- P requires L to compile and run unit tests.
- classpath
- P uses L as a plugin.
There are other variations possible: “Implementation” can be replaced by “CompileOnly” and “RuntomeOnly”, but those will be fairly uncommon.
In older Gradle examples (including a lot of mine in this course) you will see the scopes * compile * runTime * testCompile
* testRuntime
These are now deprecated.
4 Eclipse and Library Management
Each of the build/library managers we have discussed have support in Eclipse
-
May provide special editors for editing the build manager configuration files.
-
Provide ways to launch the build manager from within Eclipse
-
Sometimes it’s easier to launch separately in a separate window, e.g., from the command line from the Gradle GUI mode.
-
Remember to “refresh” the project in Eclipse afterwards
-
-
-
Most importantly, allows Eclipse Java projects to use any 3rd-party libraries fetched by the build/library manager.
-
Otherwise the built-in Eclipse compilation and smart-editing features would mistakenly believe that your code was full of references to non-existent classes and funnctions.
-
-
May allow some management of the cache of old libraries.
4.1 Eclipse and Maven
-
Can also hope for convenience functions
- In particular, POM editors
-
Two plugins currently
- m2eclipse, older plugin
- Eclipse IAM
-
Allows you to create new Maven projects or to import existing ones into Eclipse.
4.2 Eclipse and Ant/Ivy
-
Support for Ant is built-in to Eclipse
-
The developers of Ivy provide an Eclipse plugin, IvyDE.
-
Allows you to add a “Library” to your Eclipse project’s build path
-
This library holds everything that Ivy fetches based upon your
ivy.xml
file.
-
-
4.3 Eclipse and Gradle
Eclipse has various Gradle plugins available.
The most popular seems to be the BuildShip package, now standard in most Eclipse distributions.
-
Allows import (only) of existing Gradle projects
-
Allows launching Gradle from within Eclipse
- from a special Gradle Tasks panel, not from the package explorer.
-
Allows Eclipse Java projects to find 3rd-party libraries loaded by Gradle.
After any change to the dependencies in
build.gradle
, you must right-click onbuild.gradle
and selectGradle -> refresh Gradle project
.- Eclipse re-reads your
build.gradle
file and obtains the desired libraries.
- Eclipse re-reads your
5 Complicating Factors
5.1 Plug-ins
Plug-ins to build managers modify the build process itself
- unlike “normal” 3rd-party libraries that affect how the code is compiled and run, but not when or whether the compilation steps take place.
Consequently, plug-ins generally need to be fetched and processed at the very beginning of a build, much earlier than most 3rd-party libraries.
- Nonetheless, it is convenient to use the same dependency/repository scheme to publish and import plug-ins.
5.1.1 Separate but similar
Plug-ins to build managers are generally handled separately from other items, though the syntax is often similar.
- unlike “normal” 3rd-party libraries that affect how the code is compiled and run, but not when or whether the compilation steps take place.
Consequently, plug-ins generally need to be fetched and processed at the very beginning of a build, much earlier than most 3rd-party libraries.
- Nonetheless, it is convenient to use the same dependency/repository scheme to publish and import plug-ins.
5.1.2 Separate but similar
Plug-ins to build managers are generally handled separately from other items, though the syntax is often similar.
-
For example, in gradle, plug-ins are retrieved from repositories specified in a
pluginManagement
section insettings.gradle
:pluginManagement { repositories { jcenter() ivy { // for url 'https://www.cs.odu.edu/~zeil/ivyrepo' } } }
- The syntax and options within the
pluginManagement
area are almost identical to the way we handle normal package repositories.
- The syntax and options within the
-
Then we load the plugins from a
plugins
section inbuild.gradle
:plugins { id 'java' id 'edu.odu.cs.report_accumulator' version '1.3' }
-
Maven has a similar approach, isolating plug-ins in a
<plugins>
section.
5.2 Publishing
Publishing artifacts to a repository is also supported by maven, Ant/Ivy, & Gradle.
- Typically more complicated.
-
mainly because most repositories are open to download, but require login credentials to upload.
-
Example: in Gradle build.xml
publishing {
publications {
ivyJava(IvyPublication) {
organisation 'edu.odu.cs.zeil'
module 'cowem-plugin'
revision project.version
descriptor.status = 'integration' // milestone, release
descriptor.branch = 'v1'
from components.java
}
}
repositories {
ivy {
name 'ivyRepo'
url 'sftp://linux.cs.odu.edu:22/home/zeil/secure_html/ivyrepo'
// Readable via https://www.cs.odu.edu/~zeil/ivyrepo
credentials {
⋮
}
}
}
}
5.3 Login Credentials
Providing login credentials to a build is problematic.
- Can’t insert passwords into the build file, because the build file will be checked in to version control, so we would be publishing our credentials in plain view, for all time!
A common work-around is to store a few build “properties” in a file in the user’s home directory, not kept as part of the package version control.
Example: in Gradle build.xml
// Credentials are loaded from ~/.gradle/gradle.properties
if(project.hasProperty("ivyRepoUser")){ ➀
ext.ivyRepoUser = "$ivyRepoUser";
} else {
ext.ivyRepoUser = "user";
}
if(project.hasProperty("ivyRepoPass")){
ext.ivyRepoPass = "$ivyRepoPass";
} else {
ext.ivyRepoPass = "password";
}
⋮
publishing {
⋮
credentials { ➁
username project.ivyRepoUser
password project.ivyRepoPass
}
⋮
}
-
➀ looks to see if properties
ivyRepoUser
andivyRepoPass
were loaded from~/.gradle/gradle.properties
. If so, it adds them as “extension” properties of the current gradle project. -
➁ passes those new extension properties as login credentials to a repository.
-
If the person running the build di not have a
~/.gradle/gradle.properties
or it did not contain the propertiesivyRepoUser
andivyRepoPass
proeprties, then attempted uploads to the repository would use a default username and password that would, presumably, fail.
-
6 What about C++?
Sadly, the situation for C++ is somewhat more complicated.
-
Should C++ libraries be deployed as source or binaries?
- Binary library formats (e.g., DLL’s in windows,
.a
or.so
in *nix) are OS-specific. - Developers generally need header (
.h
) files as well as the compiled code if they are to compile their own applications.
- Binary library formats (e.g., DLL’s in windows,
-
There are Operating System-dependent solutions for deploying binaries.
- Windows install files can check to see if required libraries for an application are already present and fetch and install them if necessary.
- Linux repositories provide a similar mechanism.
- Even more than OS-dependent, these are distribution dependent.
- Different variants of Linux use different repository formats
-
These solutions work for deploying applications but not so well for developers.
6.1 What’s Holding Us Up?
We Don’t Have Anything Equivalent to Maven Central for C++
There’s no recognized central place to publish and find open source C++ libraries.
-
No standard for what such a library would look like.
- Compiled binaries would have to be provided for multiple OS variants.
-
Partly, that’s because we don’t have build managers that would take advantage of such libraries if they existed.
We Don’t Have Anything Equivalent to Maven/Ivy/Gradle Dependency Management for C++
There are no build managers capable of processing dependencies for C++ while simultaneously factoring in the platform-specific requirements.
- Who would write such a tool when there’s no central repositories where such libraries can be stored?
Which Came First, the Chicken or the Egg?
-
Who wants to set up repositories without build managers that can use them?
-
Who wants to write such build managers if there are no repositories that they could work with?
6.2 On the Horizon
-
Gradle seems to be paying serious attention to C/C++ build support.
- Gradle’s existing Java package management might be leveraged into C++ support.
-
Conan is an attempt to provide a packaging standard for C++ library repositories.
- Good news: the same company that hosts
JCenter
is also hostingConan
artifacts. - Bad news: Only a handful of rather specialized projects have been contributed so far (as of Jan 2018).
- Encouraging news: the number is growing steadily (as of Oct 2019)
- Gradle support for this is still in the works.
- Good news: the same company that hosts
-
In mid-summer of 2019, the Gradle project released a heavily revised set of C++ plugins, which greatly simplify building C++ with Gradle.
-
These include provisions for downloading both binary and source code libraries.
-
Not clear how stable these are, but it’s a promising sign.
-
1: Yet Another Apache Project