File Dependencies: make

Steven J Zeil

Last modified: Sep 14, 2017
Contents:

Abstract

make is perhaps the oldest build management tool. It introduced the idea of dependency-based builds.

In this lesson we will look at the build model implemented by make and at how to describe projects to the make tool.

We will look at how make could be applied to some of our sample projects from the prior lesson.


make

make is a command/program that enacts builds according to a dependency graph expressed in a makefile.

1 The make Command

 


make Options

Some useful options:

-n
Print the commands that make would issue to rebuild the target, but don’t actually perform the commands.
-k
“Keep going.” Don’t stop the build at the first failue, but continue building any required targets that do not depend on the one whose construction has failed.
-f filename
Use filename instead of the default makefile or Makefile

2 makefiles

At its heart, a makefile is a collection of rules.

2.1 Rules


The Components of a Rule

where


Rule Examples

 

codeAnnotation.jar: code2HTML.class CppJavaScanner.class
        jar tvf codeAnnotation.jar code2HTML.class CppJavaScanner.class

CppJavaScanner.class: CppJavaScanner.java
        javac CppJavaScanner.java

code2HTML.class: code2HTML.java CppJavaScanner.java
        javac code2HTML.java

CppJavaScanner.java: code2html.flex
        java -cp JFlex.jar JFlex.Main code2html.flex


Why is This Better than Scripting?

 
Suppose that we edit code2html.java and then invoke make


How make Works

2.2 Variables

A makefile can use variables to simplify the rules or to add flexibility in configuring the makefile.


Referencing Variables

CppJavaScanner.class: CppJavaScanner.java
        javac $(JOPTIONS) CppJavaScanner.java

code2HTML.class: code2HTML.java CppJavaScanner.java
        javac $(JOPTIONS) code2HTML.java


Adding Power to Variables

GNU make adds some special extensions useful in setting up variables.


Example: Using variables

This allows us to write a “generic” rule for compiling C++ programs:

PROGRAM=myProgramName
SOURCEFILES=$(wildcard src/*.cpp)
OBJFILES=$(SOURCEFILES:%.cpp=%.o)

$(PROGRAM): $(OBJFILES)
        g++ -o $(PROGRAM) $(OBJFILES)

2.3 Implicit Rules and Patterns


Implicit Rules

An implicit rule looks like

.ext1.ext2:
        commands

where ext1 and ext2 are file extensions, and commands are the commands used to convert a file with the first extension into a file wit hthe second.

Example:

.cpp.o:
        g++ -g -c $<


Using Implicit Rules

The extensions used in implicit rules must be declared:

.SUFFIXES: .cpp .o

An implicit rule will be used when a target ends in one of these suffixes and


Implicit Rule Example

PROGRAM=myProgramName
SOURCEFILES=src/main.cpp src/adt.cpp
OBJFILES=$(SOURCEFILES:%.cpp=%.o)
.SUFFIXES: .cpp .o

.cpp.o:
        g++ -g -c $<

$(PROGRAM): $(OBJFILES)
        g++ -o $(PROGRAM) $(OBJFILES)

src/adt.o: adt.cpp adt.h


Pattern Rules

A pattern rule looks like a regular rule, but uses ‘%’ as a wildcard in the target and one of their dependencies:

src/test/java/%.class: src/test/java/%.java junit4.jar
        javac -cp junit4.jar -g src/test/java/$*.java

3 Working with Make

3.1 Touching Files

Modification Dates

Although this is fairly robust, there are ways to fool make


Touching a File

 

Question: What would happen if we touched code2html.flex?

Answer

Sometimes this is a useful thing to do on purpose.


Inadvertant Touches

Suppose we had our code annotation project in a directory project1 and did the following:

> cd project1
> make
> cd ..
> cp -rf project1 project2
> cd project2
> make

What would be re-built by the second make?


Inadvertant Touches


Created != Success

3.2 Artificial Targets

Fooling make Again

A creative way to fool make:

What happens if we give a rule whose commands never actually create the target?

target: dependency1 dependency2
        echo Nope. Not going make that target!


Artificial Targets

We can take advantage of this trick by adding artificial targets that serve as the names for tasks to be performed.

build: codeAnnotation.jar

install: build
        cp codeAnnotation.jar $(INSTALLDIR)

clean:
        rm *.class CppJavaScanner.java

codeAnnotation.jar: code2HTML.class CppJavaScanner.class
        jar tvf codeAnnotation.jar code2HTML.class CppJavaScanner.class

CppJavaScanner.class: CppJavaScanner.java
        javac CppJavaScanner.java

code2HTML.class: code2HTML.java CppJavaScanner.java
        javac code2HTML.java

CppJavaScanner.java: code2html.flex
        java -cp JFlex.jar JFlex.Main code2html.flex


Common Artificial Targets

all
Often made the first rule in the makefile so that it becomes the default. Builds everything. May also run tests.
build
Build everything.
install
Build, then install
test
Build, then run tests
clean
Delete everything that would have been produced by the makefile in a build or test run.

3.3 Dependency Analysis

Coming up with a list of dependencies (and keeping it current) can be troublesome.


Self-Building Makefile

selfBuilding.listing
MAINPROG=testpicture
CPPS:=$(wildcard *.cpp)

CPPFLAGS=-g -D$(DISTR)
CPP=g++

OBJS=$(CPPS:%.cpp=%.o)
DEPENDENCIES = $(CPPS:%.cpp=%.d)

%.d: %.cpp
	touch $@

%.o: %.cpp
	$(CPP) $(CPPFLAGS) -MMD -o $@ -c $*.cpp

make.dep: $(DEPENDENCIES)
	-cat $(DEPENDENCIES) > $@

include make.dep

3.4 Managing Subproject Builds

Subprojects are generally handled by giving each subproject its own makefile and using a master makefile to invoke the artificial targets:

all:
        cd model; make
        cd vcncurses; make
        cd vcjava; make

clean:
        cd model; make clean
        cd vcncurses; make clean
        cd vcjava; make clean

4 Case Studies – Introduction

For make and for the other buiild managers we will be discussing, we will use 3 case studies.

4.1 Simple Java Build

The simple Java build features


Layout

We will follow the Apache/Android guidelines for source code layout.

project root/
|--src/
|--|--main/
|--|--|--java/
|--|--|--|-- ...java packages...
|--|--test/
|--|--|--java/
|--|--|--|-- ...junit packages...
|--|--|--data/
|--|--|--|-- ...test data ...
|--lib/
|--|-- ...3rd party libraries (temporary hack)...

We will look at better ways to supply 3rd party libraries in later lessons on configuration management.


Layout

For outputs from the build, we will follow either the Apache

project root/
|--target/
|--|--classes/
|--|--|-- ...compiled code from src/main
|--|--test-classes/
|--|--|-- ...compiled code from src/test
|--|--reports/

or the Android standard

project root/
|--build/
|--|--classes/
|--|--|--main/
|--|--|--|-- ...compiled code from src/main
|--|--|--test/
|--|--|--|-- ...compiled code from src/test
|--|--lib/
|--|--reports/

whichever is more “natural” for the build manager under discussion.

4.2 Java Build with Code Generation

A slightly less conventional build:


Layout

Project layout is the same as the simple Java build, except for an input directory for JFlex specifications and a working directory for the generated Java source code:

project root/
|--src/
|--|--main/
|--|--|--java/
|--|--|--|-- ...java packages...
|--|--|--jflex/
|--|--|--|-- ...JFlex input files...
|--|--test/
|--|--|--java/
|--|--|--|-- ...junit packages...
|--|--|--data/
|--|--|--|-- ...test data ...
|--lib/
|--|-- ...3rd party libraries (temporary hack)...
|--target/   (or build/)
|--|--generated-src/
|--|--|-- ...Java code generated by JFlex
|--|...

4.3 C++ Multi-project Build

C and C++ pose a challenge for the build systems that want to provide defaults for simple projects.


Unit testing and C/C++ Builds

Even simple projects with unit tests wind up being divided into multiple sub-projects.

4.3.1 Source Layout

project root/
|--lib/       (ADTs with unit tests)
|--|--src/
|--|--|--main/
|--|--|--|--cpp/
|--|--|--|--|-- .cpp and local .h files for ADTs
|--|--|--|--headers/
|--|--|--|--|-- Header files "exported" to other subprojects
|--|--|--mainTest/
|--|--|--|--cpp/
|--|--|--|--|-- .cpp unit tests (GTest)
|--exe/     (may be named for executable, particularly if more than one)
|--|--src/
|--|--|--main/
|--|--|--|--cpp/
|--|--|--|--|-- .cpp main function for one executable
|--gtest/       (Google Test framework)
|--|--include/
|--|--|-- Header files exported to unit tests
|--|--|--src/
|--|--|--|-- .cpp and local .h files for framewwork

The gtest/ subproject has a slightly different layout because it’s a third party project and it’s not worth trying to repackage their sourcecode.

4.3.2 Output layout

5 Case Studies – Makefiles

5.1 Simple Java Build – Makefiles

TARGET=codeAnnotation.jar

SRC=src/main/java                                 ➀
CLASSDEST=build/classes
JARDEST=build
TESTSRC=src/test/java
TESTCLASSDEST=build/testclasses

JAVA=$(shell find $(SRC) -type f -name '*.java')  ➁
TESTJAVA=$(shell find $(TESTSRC) -type f -name '*.java')
CLASSES=$(patsubst $(SRC)/%, $(CLASSDEST)/%, $(JAVA:%.java=%.class))
TESTCLASSES=$(patsubst $(TESTSRC)/%, $(TESTCLASSDEST)/%, $(TESTJAVA:%.java=%.class))
TESTCLASSNAMES=$(subst /,.,$(subst $(TESTSRC)/, ,$(TESTJAVA:%.java=%)))



.SUFFIXES:


# 
# Targets:
# 
all: test $(JARDEST)/$(TARGET)

build/setup:                                  ➂
    -mkdir -p $(JARDEST)
    -mkdir -p $(CLASSDEST)
    -mkdir -p $(TESTCLASSDEST)
    date > $@


$(JARDEST)/$(TARGET): $(CLASSES)               ➃
    cd $(CLASSDEST); jar cf temp.jar *
    mv $(CLASSDEST)/temp.jar $@




test: $(TESTCLASSES) $(CLASSES)                ➄
    java -cp lib/junit-4.10.jar:lib/junit/hamcrest-core-1.1.jar:$(CLASSDEST):$(TESTCLASSDEST) org.junit.runner.JUnitCore $(TESTCLASSNAMES)



$(CLASSDEST)/%.class: $(SRC)/%.java build/setup    ➅
    javac -g -cp $(CLASSDEST) -d $(CLASSDEST) -sourcepath $(SRC) $(SRC)/$*.java

$(TESTCLASSDEST)/%.class: $(TESTSRC)/%.java build/setup $(CLASSES)
    javac -g -cp $(CLASSDEST):lib/junit-4.10.jar:lib/junit/hamcrest-core-1.1.jar -d $(TESTCLASSDEST)  -sourcepath $(TESTSRC) $(TESTSRC)/$*.java



clean:
    -rm -f build

5.2 Java Build with Code Generation – Makefile

This project adds a stage before compilation to generate some of the source code that then needs to be compiled.

TARGET=codeAnnotation.jar

SRC=src/main/java
CLASSDEST=build/classes
JARDEST=build
TESTSRC=src/test/java
TESTCLASSDEST=build/testclasses
GENSRC=build/gen/java
GENDEST=build/classes
FLEXSRC=src/main/jflex

JAVA=$(shell find $(SRC) -type f -name '*.java')
TESTJAVA=$(shell find $(TESTSRC) -type f -name '*.java')
CLASSES=$(patsubst $(SRC)/%, $(CLASSDEST)/%, $(JAVA:%.java=%.class))
TESTCLASSES=$(patsubst $(TESTSRC)/%, $(TESTCLASSDEST)/%, $(TESTJAVA:%.java=%.class))
TESTCLASSNAMES=$(subst /,.,$(subst $(TESTSRC)/, ,$(TESTJAVA:%.java=%)))

GENJAVA=$(GENSRC)/CppJavaScanner.java $(GENSRC)/CppJavaTeXScanner.java \
   $(GENSRC)/ListingScanner.java $(GENSRC)/ListingTeXScanner.java
GENCLASSES=$(GENDEST)/CppJavaScanner.class $(GENDEST)/CppJavaTeXScanner.class \
   $(GENDEST)/ListingScanner.class $(GENDEST)/ListingTeXScanner.class


.SUFFIXES:


# 
# Targets:
# 
all: test $(JARDEST)/$(TARGET)

build/setup:
    -mkdir -p $(JARDEST)
    -mkdir -p $(GENSRC)
    -mkdir -p $(CLASSDEST)
    -mkdir -p $(TESTCLASSDEST)
    date > $@


$(JARDEST)/$(TARGET): $(CLASSES)
    cd $(CLASSDEST); jar cf temp.jar *
    mv $(CLASSDEST)/temp.jar $@




test: $(TESTCLASSES) $(CLASSES)
    java -cp lib/junit-4.10.jar:lib/junit/hamcrest-core-1.1.jar:$(CLASSDEST):$(TESTCLASSDEST) org.junit.runner.JUnitCore $(TESTCLASSNAMES)


$(GENSRC)/CppJavaScanner.java: $(FLEXSRC)/code2html.flex build/setup
    java -jar lib/jflex-1.4.3.jar -d $(GENSRC) $<

$(GENSRC)/CppJavaTeXScanner.java: $(FLEXSRC)/code2tex.flex build/setup
    java -jar lib/jflex-1.4.3.jar -d $(GENSRC) $<


$(GENSRC)/ListingScanner.java: $(FLEXSRC)/list2html.flex build/setup
    java -jar lib/jflex-1.4.3.jar -d $(GENSRC) $<


$(GENSRC)/ListingTeXScanner.java: $(FLEXSRC)/list2tex.flex build/setup
    java -jar lib/jflex-1.4.3.jar -d $(GENSRC) $<


$(GENDEST)/%.class: $(GENSRC)/%.java build/setup
    javac -g -d $(GENDEST) -sourcepath $(GENSRC)  $(GENSRC)/$*.java

$(CLASSDEST)/%.class: $(SRC)/%.java build/setup $(GENCLASSES)
    javac -g -cp $(CLASSDEST) -d $(CLASSDEST) -sourcepath $(SRC) $(SRC)/$*.java

$(TESTCLASSDEST)/%.class: $(TESTSRC)/%.java build/setup
    javac -g -cp $(CLASSDEST):lib/junit-4.10.jar:lib/junit/hamcrest-core-1.1.jar -d $(TESTCLASSDEST)  -sourcepath $(TESTSRC) $(TESTSRC)/$*.java



clean:
    -rm -f build

5.3 C++ Multi-project Build – Makefiles

Because this project is divided into multiple subprojects, we will have multiple makefiles:

5.3.1 The master file

In the project root directory:

all:
    $(MAKE) -C gtest
    $(MAKE) -C lib
    $(MAKE) -C exe

clean:
    $(MAKE) -C gtest
    $(MAKE) -C lib
    $(MAKE) -C exe

5.3.2 The lib subproject

TARGET=libmain.a   ➀
CXX=g++
CXXFLAGS=-g -pthread -I ../gtest/include -I src/main/headers -std=c++11
LINK=$(CXX)
LFLAGS=../gtest/build/libs/libgtest.a -lpthread

SRC=src/main/cpp
OBJDEST=build/objs/main
DEST=build/libs
EXEDEST=build/exe

TESTSRC=src/mainTest/cpp
TESTOBJDEST=build/objs/test

CPP=$(wildcard $(SRC)/*.cpp)
OBJS=$(patsubst $(SRC)/%, $(OBJDEST)/%, $(CPP:%.cpp=%.o))
TESTCPP=$(wildcard $(TESTSRC)/*.cpp)
TESTOBJS=$(patsubst $(TESTSRC)/%, $(TESTOBJDEST)/%, $(TESTCPP:%.cpp=%.o))


.SUFFIXES:


# 
# Targets:
# 
all: $(DEST)/$(TARGET) test


build/setup:                          ➁
    -mkdir -p $(DEST)
    -mkdir -p $(EXEDEST)
    -mkdir -p $(OBJDEST)
    -mkdir -p $(TESTOBJDEST)
    date > $@


test: $(EXEDEST)/runtest              ➂
    $(EXEDEST)/runtest 


$(DEST)/$(TARGET): $(OBJS) test build/setup   ➃
    ar rcs $@ $(OBJS)
    ranlib $@

$(EXEDEST)/runtest: $(TESTOBJS) $(OBJS)       ➄
    $(LINK) -o $@ $(CPPFLAGS) $(TESTOBJS) $(OBJS) $(LFLAGS)


$(OBJDEST)/%.o: $(SRC)/%.cpp build/setup        ➅
    $(CXX) $(CXXFLAGS) -o $@ -c $<

$(TESTOBJDEST)/%.o: $(TESTSRC)/%.cpp build/setup    ➆
    $(CXX) $(CXXFLAGS) -o $@ -c $<


clean:
    -rm -f build