2 Basic Concepts

2.1 Repositories

A repository is a collection of packages (usually libraries) and metadata about packages

• The metadata identifies
  • the package,
  • the versions available,
  • their dependencies on other packages, and
  • the location from which the package may be downloaded.
• In some cases, the metadata may point to other repositories as the actual download location.

2.1.1 Common Java Repositories

The Java world is dominated by

2.1.2 Local Repositories

It’s common for development organizations to host their own repository.

• A place to put their own deliverables
• A “cache” to speed access to libraries commonly used by many developers within the organization.
  • preserve older versions needed for a project baseline

Local repositories can be

2.2 Identifying Packages

Packages are typically identified by

• The group or organization

  • Generally given in the Java style of a reversed URL, e.g. org.apache.ant, edu.odu.cs.extract
  • Like some Hollywood celebrities, some organizations become big enough stars to be simply known by a single name, e.g., junit

• The artifact name

  • E.g., junit, hamcrest-core, apache-commons

• Version number

  • Single versions are identified exactly: 4.12
  • Depending on the build manger being used, you may be able to specify things like
    • 4.11+: 4.11 or later
    • latest.integration: latest official release
    • 1.2-SNAPSHOT: a version under development. The stapshot modifier serves as a notice that the actual package may be updated without a change in version number.

• If your “main code” uses a library, you would need that during compilation, testing, and possibly packaging.
• Libraries like junit would only be needed when compiling and running the unit tests.
• Libraries like spotbugs, checkstyle (discussed later in the semester) are only used during report generation.
• A build manager may support “plug-ins” that extend or modify the build process. These plug-ins are typically packaged much like other 3rd-party libraries, but are needed much earlier in the build process.

So build managers allow you to import libraries in selected scopes which indicate when the library is actually needed.

This affects

• Whether a library is downloaded or updated during a given build.
• Java CLASSPATH and other settings made available to the build steps.

3 Adding Libraries to a Java Project in Different Build Managers

3.1.1 Specifying a Dependency

• Examine the pom.xml file and look for the dependencies section.

• You’ll see something like

    <dependencies>
  	<groupId>junit</groupId>
  	<artifactId>junit</artifactId>
  	<version>4.10</version>
  	<scope>test</scope>
    </dependencies>
  
  

  • Could also say [4.12,] to get versions 4.12 or greater

3.1.2 Fetching Dependencies

When the build is run (e.g. mvn package):

• Maven does a transitive search over the dependencies for a project

  • Tries to find a mutually compatible set of versions
    • Helps if you give it some flexibility

• Maven then downloads the required libraries automatically

  • Downloaded libraries are cached (e.g., ~/.m2)

3.1.3 Transitive Dependencies

How does Maven know whether junit itself depends on other libraries?

3.1.4 Choosing Repositories

• Technically, our project has two repositories

  • a remote repository, at ibiblio
  • a local repository holding my cache

• Team projects will often employ an intermediate shared repository

  • reduce cache duplication
  • provide a mechanism for managing subproject modules
  • provide a common storage area for libraries not available on ibiblio
  • preserve older versions needed for a project baseline

3.1.5 Example: specifying a repository

In pom.xml:

  <repositories>
     <repository>
        <snapshots>
           <enabled>false</enabled>
        </snapshots>
        <id>central</id>
           <name>libs-release</name>
           <url>http://archive350.cs.odu.edu:8080/artifactory/libs-release</url>
    </repository>
  </repositories>

• We could then, for example, import early versions of zipdiff and trilead-ssh that were part of the baseline for the Extract project
  • At the time they were added to the project baseline, they were not in the remote Maven repository
  • As time has passed, those libraries were added, but have started with versions later than the project baseline

3.2 Ivy (ant)

• Technically can run as a standalone application, but really the ant integration is key.

• Use is similar to the dependencies/repositories portion of Maven POMs

• Ivy can retrieve from and publish to the established Maven repositories

3.3.1 Selecting libraries in Gradle

In build.gradle:

apply plugin: 'java'


⋮

repositories {
    jcenter()   ➀

    // Use my own CS dept repo
    ivy {      ➁
        url 'https://www.cs.odu.edu/~zeil/ivyrepo'
    }
}



dependencies { ➂
    implementation 'net.sf.saxon:saxon-dom:8.7+'
    testImplementation 'junit:junit:4.12'
    testRuntimeOnly 'edu.odu.cs.zeil:reportAccumulator:1.1+'

}

Here you see both the choice of libraries ➂ and of repositories ➀, ➁.

The dependencies indicate

• The scope (e.g., testCompile, which actually includes compiling and running of the unit tests)

• The package, specified as organization:package:versionNumber

dependencies {
    implementation 'net.sf.saxon:saxon-dom:8.7+'
    testImplementation 'junit:junit:4.12'
    testRuntimeOnly 'edu.odu.cs.zeil:reportAccumulator:1.1+'

}

If project P depends on library L, then the most common scopes recognized by Gradle are:

api

P requires L to compile and to run. If P is itself a library, anyone who uses P must have L available to compile and run as well.

Generally this means that some class declared in L is used as a parameter or a return type by some public function in P.

implementation

P requires L to compile and run the code, but this is hidden from users of P.

testImplementation

P requires L to compile and run unit tests.

classpath

P uses L as a plugin.

• May provide special editors for editing the build manager configuration files.

• May provide special editors for editing the build manager configuration files.

• Provide ways to launch the build manager from within Eclipse

• May provide special editors for editing the build manager configuration files.

• Provide ways to launch the build manager from within Eclipse

  • Sometimes it’s easier to launch separately in a separate window, e.g., from the command line from the Gradle GUI mode.

• May provide special editors for editing the build manager configuration files.

• Provide ways to launch the build manager from within Eclipse

  • Sometimes it’s easier to launch separately in a separate window, e.g., from the command line from the Gradle GUI mode.

    • Remember to “refresh” the project in Eclipse afterwards

• May provide special editors for editing the build manager configuration files.

• Provide ways to launch the build manager from within Eclipse

  • Sometimes it’s easier to launch separately in a separate window, e.g., from the command line from the Gradle GUI mode.

    • Remember to “refresh” the project in Eclipse afterwards

• Most importantly, allows Eclipse Java projects to use any 3rd-party libraries fetched by the build/library manager.

• May provide special editors for editing the build manager configuration files.

• Provide ways to launch the build manager from within Eclipse

  • Sometimes it’s easier to launch separately in a separate window, e.g., from the command line from the Gradle GUI mode.

    • Remember to “refresh” the project in Eclipse afterwards

• Most importantly, allows Eclipse Java projects to use any 3rd-party libraries fetched by the build/library manager.

  • Otherwise the built-in Eclipse compilation and smart-editing features would mistakenly believe that your code was full of references to non-existent classes and funnctions.

4 Eclipse and Library Management

Each of the build/library managers we have discussed have support in Eclipse

• May provide special editors for editing the build manager configuration files.

• Provide ways to launch the build manager from within Eclipse

  • Sometimes it’s easier to launch separately in a separate window, e.g., from the command line from the Gradle GUI mode.

    • Remember to “refresh” the project in Eclipse afterwards

• Most importantly, allows Eclipse Java projects to use any 3rd-party libraries fetched by the build/library manager.

  • Otherwise the built-in Eclipse compilation and smart-editing features would mistakenly believe that your code was full of references to non-existent classes and funnctions.

• May allow some management of the cache of old libraries.

4.1 Eclipse and Gradle

Eclipse has various Gradle plugins available.

The most popular seems to be the BuildShip package, now standard in most Eclipse distributions.

• Allows import (only) of existing Gradle projects

• Allows launching Gradle from within Eclipse

  • from a special Gradle Tasks panel, not from the package explorer.

• Allows Eclipse Java projects to find 3rd-party libraries loaded by Gradle.

  After any change to the dependencies in build.gradle, you must right-click on build.gradle and select Gradle -> refresh Gradle project.

  • Eclipse re-reads your build.gradle file and obtains the desired libraries.

5 Complicating Factors

5.1 Plug-ins

Plug-ins to build managers modify the build process itself

• unlike “normal” 3rd-party libraries that affect how the code is compiled and run, but not when or whether the compilation steps take place.

Consequently, plug-ins generally need to be fetched and processed at the very beginning of a build, much earlier than most 3rd-party libraries.

• Nonetheless, it is convenient to use the same dependency/repository scheme to publish and import plug-ins.

5.1.1 Separate but similar

Plug-ins to build managers are generally handled separately from other items, though the syntax is often similar.

• unlike “normal” 3rd-party libraries that affect how the code is compiled and run, but not when or whether the compilation steps take place.

Consequently, plug-ins generally need to be fetched and processed at the very beginning of a build, much earlier than most 3rd-party libraries.

• Nonetheless, it is convenient to use the same dependency/repository scheme to publish and import plug-ins.

5.1.2 Separate but similar

Plug-ins to build managers are generally handled separately from other items, though the syntax is often similar.

• For example, in gradle, plug-ins are retrieved from repositories specified in a pluginManagement section in settings.gradle:

  pluginManagement {
      repositories {
          jcenter()
          ivy { // for
              url 'https://www.cs.odu.edu/~zeil/ivyrepo'
          }
      }
  }
  

  • The syntax and options within the pluginManagement area are almost identical to the way we handle normal package repositories.

• Then we load the plugins from a plugins section in build.gradle:

  plugins {
      id 'java'
      id 'edu.odu.cs.report_accumulator' version '1.3'
  }
  

• Maven has a similar approach, isolating plug-ins in a <plugins> section.

• Typically more complicated.

  • mainly because most repositories are open to download, but require login credentials to upload.

publishing {
    publications {
        ivyJava(IvyPublication) {
            organisation 'edu.odu.cs.zeil'
            module 'cowem-plugin'
            revision project.version
            descriptor.status = 'integration'   // milestone, release
            descriptor.branch = 'v1'

            from components.java
        }
    }

    repositories {
        ivy {
            name 'ivyRepo'
            url 'sftp://linux.cs.odu.edu:22/home/zeil/secure_html/ivyrepo'
            // Readable via https://www.cs.odu.edu/~zeil/ivyrepo
            credentials {
                ⋮
            }
        }
    }
}

5.3 Login Credentials

Providing login credentials to a build is problematic.

• Can’t insert passwords into the build file, because the build file will be checked in to version control, so we would be publishing our credentials in plain view, for all time!

A common work-around is to store a few build “properties” in a file in the user’s home directory, not kept as part of the package version control.

// Credentials are loaded from ~/.gradle/gradle.properties
if(project.hasProperty("ivyRepoUser")){        ➀
    ext.ivyRepoUser = "$ivyRepoUser";
} else {
    ext.ivyRepoUser = "user";
}
if(project.hasProperty("ivyRepoPass")){
    ext.ivyRepoPass = "$ivyRepoPass";
} else {
    ext.ivyRepoPass = "password";
}

⋮

publishing {
    ⋮
            credentials {                    ➁
                username project.ivyRepoUser
                password project.ivyRepoPass
            }
    ⋮
}

• ➀ looks to see if properties ivyRepoUser and ivyRepoPass were loaded from ~/.gradle/gradle.properties. If so, it adds them as “extension” properties of the current gradle project.

• ➁ passes those new extension properties as login credentials to a repository.

  • If the person running the build di not have a ~/.gradle/gradle.properties or it did not contain the properties ivyRepoUser and ivyRepoPassproeprties, then attempted uploads to the repository would use a default username and password that would, presumably, fail.

6 What about C++?

Sadly, the situation for C++ is somewhat more complicated.

• Should C++ libraries be deployed as source or binaries?

  • Binary library formats (e.g., DLL’s in windows, .a or .so in *nix) are OS-specific.
  • Developers generally need header (.h) files as well as the compiled code if they are to compile their own applications.

• There are Operating System-dependent solutions for deploying binaries.

  • Windows install files can check to see if required libraries for an application are already present and fetch and install them if necessary.
  • Linux repositories provide a similar mechanism.
    • Even more than OS-dependent, these are distribution dependent.
    • Different variants of Linux use different repository formats

• These solutions work for deploying applications but not so well for developers.

6.1 What’s Holding Us Up?

We Don’t Have Anything Equivalent to Maven Central for C++

There’s no recognized central place to publish and find open source C++ libraries.

• No standard for what such a library would look like.

  • Compiled binaries would have to be provided for multiple OS variants.

• Partly, that’s because we don’t have build managers that would take advantage of such libraries if they existed.

There are no build managers capable of processing dependencies for C++ while simultaneously factoring in the platform-specific requirements.

• Who would write such a tool when there’s no central repositories where such libraries can be stored?

• Who wants to set up repositories without build managers that can use them?

• Who wants to write such build managers if there are no repositories that they could work with?

6.2 On the Horizon