Abstract
make is perhaps the oldest build management tool. It introduced the idea of dependency-based builds.
In this lesson we will look at the build model implemented by make and at how to describe projects to the make tool.
We will look at how make could be applied to some of our sample projects from the prior lesson.
make
make is a command/program that enacts builds according to a dependency graph expressed in a makefile.
make devised by Dr. Stuart Feldman of Bel Labs in 1977
It has long been a standard component of *nix systems
make looks for its instructions in a file named, by default, makefile or Makefile
The make command can name any file in the graph as the target to be built, e.g.,
make CppJavaScanner
If no target is given, make builds the first file described in the makefile
make Options
Some useful options:
At its heart, a makefile is a collection of rules.
A rule describes how to build a single file of the project.
Each rule indicates
Rules may appear in any order
The Components of a Rule
target: dependencies
commands
where
target is the target file,
dependencies is a space-separated list of files on which the target is dependent
commands is a set of zero or more commands, one per line, each preceded by a Tab character.
Rule Examples
codeAnnotation.jar: code2HTML.class CppJavaScanner.class
jar tvf codeAnnotation.jar code2HTML.class CppJavaScanner.class
CppJavaScanner.class: CppJavaScanner.java
javac CppJavaScanner.java
code2HTML.class: code2HTML.java CppJavaScanner.java
javac code2HTML.java
CppJavaScanner.java: code2html.flex
java -cp JFlex.jar JFlex.Main code2html.flex
Why is This Better than Scripting?
Only one javac will be issued, after which the jar command is run.
make has determined the minimum number of steps required to rebuild after a change.
How make Works
Construct the dependency graph from the target and dependency entries in the makefile
Do a topological sort to determine an order in which to construct targets.
For each target visited, invoke the commands if the target file does not exist or if any dependency file is newer
A makefile can use variables to simplify the rules or to add flexibility in configuring the makefile.
All variables hold strings.
Variables are initialized by a simple assignment
variable =
value
Variables are immutable (constants)
Assignments may appear within the makefile or in the command line, e.g.:
make JOPTIONS=-g codeAnnotation.jar
Referencing Variables
$(variable)
or ${variable}
, e.g.,CppJavaScanner.class: CppJavaScanner.java
javac $(JOPTIONS) CppJavaScanner.java
code2HTML.class: code2HTML.java CppJavaScanner.java
javac $(JOPTIONS) code2HTML.java
Adding Power to Variables
GNU make adds some special extensions useful in setting up variables.
Globbing:
SOURCEFILES=$(wildcard src/*.cpp)
collects a list of all C++ compilation units in the filename{src} directory
Substitutions:
OBJFILES=$(SOURCEFILES:%.cpp=%.o)
collects a list of all object code files expected by compiling those compilation units.
Example: Using variables
This allows us to write a “generic” rule for compiling C++ programs:
PROGRAM=myProgramName
SOURCEFILES=$(wildcard src/*.cpp)
OBJFILES=$(SOURCEFILES:%.cpp=%.o)
$(PROGRAM): $(OBJFILES)
g++ -o $(PROGRAM) $(OBJFILES)
This is technically, incomplete.
Nonetheless, it would work on some systems for the initial build, because they have an “implicit” rule for working with C++
Implicit rules describe how to produce a single “kind” (extension) of file from another.
Pattern rules are a GNU extension for writing “generic” rules
Implicit Rules
An implicit rule looks like
.ext1.ext2:
commands
where ext1 and ext2 are file extensions, and commands are the commands used to convert a file with the first extension into a file wit hthe second.
Example:
.cpp.o:
g++ -g -c $<
the implicit variable $<
holds the dependency file
Also commonly used, $@
denotes the target file.
Using Implicit Rules
The extensions used in implicit rules must be declared:
.SUFFIXES: .cpp .o
An implicit rule will be used when a target ends in one of these suffixes and
there is no rule listing that file as a target, or
the rule listing that file as a target has no commands
Implicit Rule Example
PROGRAM=myProgramName
SOURCEFILES=src/main.cpp src/adt.cpp
OBJFILES=$(SOURCEFILES:%.cpp=%.o)
.SUFFIXES: .cpp .o
.cpp.o:
g++ -g -c $<
$(PROGRAM): $(OBJFILES)
g++ -o $(PROGRAM) $(OBJFILES)
src/adt.o: adt.cpp adt.h
Both main.cpp and adt.cpp will be compiled on the initial build.
If adt.h is subsequently modified, then adt.cpp would be re-compiled.
Pattern Rules
A pattern rule looks like a regular rule, but uses ‘%’ as a wildcard in the target and one of their dependencies:
src/test/java/%.class: src/test/java/%.java junit4.jar
javac -cp junit4.jar -g src/test/java/$*.java
Another implicit variable, $*
contains the string matched by the %
wildcard.
One advantage of pattern rules, is that we can add dependencies on other files e.g., junit.jar
Modification Dates
make compares the modification dates of targets and dependencies to determine if the target is out of date.
It uses the success/fail status value returned by commands to determine if construction of a target was successful.
Although this is fairly robust, there are ways to fool make
Touching a File
Question: What would happen if we touched code2html.flex?
Sometimes this is a useful thing to do on purpose.
Inadvertant Touches
Suppose we had our code annotation project in a directory project1 and did the following:
> cd project1
> make
> cd ..
> cp -rf project1 project2
> cd project2
> make
What would be re-built by the second make?
(better to have done cp -arf project1 project2
)
Inadvertant Touches
Similarly, successive calls to make can sometimes be confused if the time between creation of some intermediate targets is within a single clock “tick”.
Clock drift between different machines can be particularly troublesome.
make
command and a file server responsible for storing the files).Created != Success
Some commands we might give to create a target will create no file if the command fails.
Others will create some kind of file anyway.
E.g., any command that is invoked with output redirection,
command > target
which could cause make to assume that the target need not be re-constructed the next time around.
Fooling make Again
A creative way to fool make:
What happens if we give a rule whose commands never actually create the target?
target: dependency1 dependency2
echo Nope. Not going make that target!
The first time we run make, the dependencies will be created and the echo performed.
Each subsequent time we run make, the dependencies will be re-created if necessary and the echo performed.
Artificial Targets
We can take advantage of this trick by adding artificial targets that serve as the names for tasks to be performed.
build: codeAnnotation.jar
install: build
cp codeAnnotation.jar $(INSTALLDIR)
clean:
rm *.class CppJavaScanner.java
codeAnnotation.jar: code2HTML.class CppJavaScanner.class
jar tvf codeAnnotation.jar code2HTML.class CppJavaScanner.class
CppJavaScanner.class: CppJavaScanner.java
javac CppJavaScanner.java
code2HTML.class: code2HTML.java CppJavaScanner.java
javac code2HTML.java
CppJavaScanner.java: code2html.flex
java -cp JFlex.jar JFlex.Main code2html.flex
Common Artificial Targets
build
or test
run.Coming up with a list of dependencies (and keeping it current) can be troublesome.
Various tools exist for this purpose for programming languages
The gcc and g++ compilers have a compile-time option, -MMd
, which emits a .d file containing a target and dependency line.
Self-Building Makefile
MAINPROG=testpicture
CPPS:=$(wildcard *.cpp)
CPPFLAGS=-g -D$(DISTR)
CPP=g++
OBJS=$(CPPS:%.cpp=%.o)
DEPENDENCIES = $(CPPS:%.cpp=%.d)
%.d: %.cpp
touch $@
%.o: %.cpp
$(CPP) $(CPPFLAGS) -MMD -o $@ -c $*.cpp
make.dep: $(DEPENDENCIES)
-cat $(DEPENDENCIES) > $@
include make.dep
On the first make,
On subsequent make runs,
Subprojects are generally handled by giving each subproject its own makefile and using a master makefile to invoke the artificial targets:
all:
cd model; make
cd vcncurses; make
cd vcjava; make
clean:
cd model; make clean
cd vcncurses; make clean
cd vcjava; make clean
For make
and for the other buiild managers we will be discussing, we will use 3 case studies.
The simple Java build features
Java code to be compiled and built into a jar
.
JUnit tests to be compiled and run.
Layout
We will follow the Apache/Android guidelines for source code layout.
project root/
|--src/
|--|--main/
|--|--|--java/
|--|--|--|-- ...java packages...
|--|--test/
|--|--|--java/
|--|--|--|-- ...junit packages...
|--|--|--data/
|--|--|--|-- ...test data ...
|--lib/
|--|-- ...3rd party libraries (temporary hack)...
We will look at better ways to supply 3rd party libraries in later lessons on configuration management.
Layout
For outputs from the build, we will follow either the Apache
project root/
|--target/
|--|--classes/
|--|--|-- ...compiled code from src/main
|--|--test-classes/
|--|--|-- ...compiled code from src/test
|--|--reports/
or the Android standard
project root/
|--build/
|--|--classes/
|--|--|--main/
|--|--|--|-- ...compiled code from src/main
|--|--|--test/
|--|--|--|-- ...compiled code from src/test
|--|--lib/
|--|--reports/
whichever is more “natural” for the build manager under discussion.
A slightly less conventional build:
Java code to be compiled and built into a jar
.
Some of the Java code is generated automatically
JFlex program is used to produce lexical analyzers from collections of regular expressions.
JUnit tests to be compiled and run.
Layout
Project layout is the same as the simple Java build, except for an input directory for JFlex specifications and a working directory for the generated Java source code:
project root/
|--src/
|--|--main/
|--|--|--java/
|--|--|--|-- ...java packages...
|--|--|--jflex/
|--|--|--|-- ...JFlex input files...
|--|--test/
|--|--|--java/
|--|--|--|-- ...junit packages...
|--|--|--data/
|--|--|--|-- ...test data ...
|--lib/
|--|-- ...3rd party libraries (temporary hack)...
|--target/ (or build/)
|--|--generated-src/
|--|--|-- ...Java code generated by JFlex
|--|...
C and C++ pose a challenge for the build systems that want to provide defaults for simple projects.
A collection of .o
files can only be linked together if they have no more than one main()
function.
If you want a project to produce more than one executable, you have to tell the build manager how to divide up the .o
files for each one.
Most IDE build managers and most default rules for non-IDE build managers don’t allow that.
One project – one executable
But unit testing adds additional executables.
Unit testing and C/C++ Builds
Even simple projects with unit tests wind up being divided into multiple sub-projects.
project root/
|--lib/ (ADTs with unit tests)
|--|--src/
|--|--|--main/
|--|--|--|--cpp/
|--|--|--|--|-- .cpp and local .h files for ADTs
|--|--|--|--headers/
|--|--|--|--|-- Header files "exported" to other subprojects
|--|--|--mainTest/
|--|--|--|--cpp/
|--|--|--|--|-- .cpp unit tests (GTest)
|--exe/ (may be named for executable, particularly if more than one)
|--|--src/
|--|--|--main/
|--|--|--|--cpp/
|--|--|--|--|-- .cpp main function for one executable
|--gtest/ (Google Test framework)
|--|--include/
|--|--|-- Header files exported to unit tests
|--|--|--src/
|--|--|--|-- .cpp and local .h files for framewwork
The gtest/
subproject has a slightly different layout because it’s a third party project and it’s not worth trying to repackage their sourcecode.
lib/
and gtest/
subprojects each produce a static library.exe
subproject produces an executable
project root/ |–lib/ (ADTs with unit tests) |–|–build/ |–|–|–objs/ holding area for .o files |–|–|–libs/ generated library goes here |–|–|–test-results/ reports from unit test |–exe/ (may be named for executable, particularly if more than one) |–|–build/ |–|–|–objs/ holding area for .o files |–|–|–exe/ generated executables go here |–gtest/ (Google Test framework) |–|–build/ |–|–|–objs/ holding area for .o files |–|–|–libs/ generated library goes here
TARGET=codeAnnotation.jar
SRC=src/main/java ➀
CLASSDEST=build/classes
JARDEST=build
TESTSRC=src/test/java
TESTCLASSDEST=build/testclasses
JAVA=$(shell find $(SRC) -type f -name '*.java') ➁
TESTJAVA=$(shell find $(TESTSRC) -type f -name '*.java')
CLASSES=$(patsubst $(SRC)/%, $(CLASSDEST)/%, $(JAVA:%.java=%.class))
TESTCLASSES=$(patsubst $(TESTSRC)/%, $(TESTCLASSDEST)/%, $(TESTJAVA:%.java=%.class))
TESTCLASSNAMES=$(subst /,.,$(subst $(TESTSRC)/, ,$(TESTJAVA:%.java=%)))
.SUFFIXES:
#
# Targets:
#
all: test $(JARDEST)/$(TARGET)
build/setup: ➂
-mkdir -p $(JARDEST)
-mkdir -p $(CLASSDEST)
-mkdir -p $(TESTCLASSDEST)
date > $@
$(JARDEST)/$(TARGET): $(CLASSES) ➃
cd $(CLASSDEST); jar cf temp.jar *
mv $(CLASSDEST)/temp.jar $@
test: $(TESTCLASSES) $(CLASSES) ➄
java -cp lib/junit-4.10.jar:lib/junit/hamcrest-core-1.1.jar:$(CLASSDEST):$(TESTCLASSDEST) org.junit.runner.JUnitCore $(TESTCLASSNAMES)
$(CLASSDEST)/%.class: $(SRC)/%.java build/setup ➅
javac -g -cp $(CLASSDEST) -d $(CLASSDEST) -sourcepath $(SRC) $(SRC)/$*.java
$(TESTCLASSDEST)/%.class: $(TESTSRC)/%.java build/setup $(CLASSES)
javac -g -cp $(CLASSDEST):lib/junit-4.10.jar:lib/junit/hamcrest-core-1.1.jar -d $(TESTCLASSDEST) -sourcepath $(TESTSRC) $(TESTSRC)/$*.java
clean:
-rm -f build
➀ Symbolic names for directories make it easier to rearrange layout.
➁ Various functions for manipulating and rewriting lists of files
➂ Create output directories. Note that this appears as a dependency in most of the later compilation rules (to guarantee it’s done before compiling).
➃ Build a Jar file
➄ Run the unit tests
➅ Compile .cpp
files.
This project adds a stage before compilation to generate some of the source code that then needs to be compiled.
TARGET=codeAnnotation.jar
SRC=src/main/java
CLASSDEST=build/classes
JARDEST=build
TESTSRC=src/test/java
TESTCLASSDEST=build/testclasses
GENSRC=build/gen/java
GENDEST=build/classes
FLEXSRC=src/main/jflex
JAVA=$(shell find $(SRC) -type f -name '*.java')
TESTJAVA=$(shell find $(TESTSRC) -type f -name '*.java')
CLASSES=$(patsubst $(SRC)/%, $(CLASSDEST)/%, $(JAVA:%.java=%.class))
TESTCLASSES=$(patsubst $(TESTSRC)/%, $(TESTCLASSDEST)/%, $(TESTJAVA:%.java=%.class))
TESTCLASSNAMES=$(subst /,.,$(subst $(TESTSRC)/, ,$(TESTJAVA:%.java=%)))
GENJAVA=$(GENSRC)/CppJavaScanner.java $(GENSRC)/CppJavaTeXScanner.java \
$(GENSRC)/ListingScanner.java $(GENSRC)/ListingTeXScanner.java
GENCLASSES=$(GENDEST)/CppJavaScanner.class $(GENDEST)/CppJavaTeXScanner.class \
$(GENDEST)/ListingScanner.class $(GENDEST)/ListingTeXScanner.class
.SUFFIXES:
#
# Targets:
#
all: test $(JARDEST)/$(TARGET)
build/setup:
-mkdir -p $(JARDEST)
-mkdir -p $(GENSRC)
-mkdir -p $(CLASSDEST)
-mkdir -p $(TESTCLASSDEST)
date > $@
$(JARDEST)/$(TARGET): $(CLASSES)
cd $(CLASSDEST); jar cf temp.jar *
mv $(CLASSDEST)/temp.jar $@
test: $(TESTCLASSES) $(CLASSES)
java -cp lib/junit-4.10.jar:lib/junit/hamcrest-core-1.1.jar:$(CLASSDEST):$(TESTCLASSDEST) org.junit.runner.JUnitCore $(TESTCLASSNAMES)
$(GENSRC)/CppJavaScanner.java: $(FLEXSRC)/code2html.flex build/setup
java -jar lib/jflex-1.4.3.jar -d $(GENSRC) $<
$(GENSRC)/CppJavaTeXScanner.java: $(FLEXSRC)/code2tex.flex build/setup
java -jar lib/jflex-1.4.3.jar -d $(GENSRC) $<
$(GENSRC)/ListingScanner.java: $(FLEXSRC)/list2html.flex build/setup
java -jar lib/jflex-1.4.3.jar -d $(GENSRC) $<
$(GENSRC)/ListingTeXScanner.java: $(FLEXSRC)/list2tex.flex build/setup
java -jar lib/jflex-1.4.3.jar -d $(GENSRC) $<
$(GENDEST)/%.class: $(GENSRC)/%.java build/setup
javac -g -d $(GENDEST) -sourcepath $(GENSRC) $(GENSRC)/$*.java
$(CLASSDEST)/%.class: $(SRC)/%.java build/setup $(GENCLASSES)
javac -g -cp $(CLASSDEST) -d $(CLASSDEST) -sourcepath $(SRC) $(SRC)/$*.java
$(TESTCLASSDEST)/%.class: $(TESTSRC)/%.java build/setup
javac -g -cp $(CLASSDEST):lib/junit-4.10.jar:lib/junit/hamcrest-core-1.1.jar -d $(TESTCLASSDEST) -sourcepath $(TESTSRC) $(TESTSRC)/$*.java
clean:
-rm -f build
Because this project is divided into multiple subprojects, we will have multiple makefiles:
In the project root directory:
all:
$(MAKE) -C gtest
$(MAKE) -C lib
$(MAKE) -C exe
clean:
$(MAKE) -C gtest
$(MAKE) -C lib
$(MAKE) -C exe
TARGET=libmain.a ➀
CXX=g++
CXXFLAGS=-g -pthread -I ../gtest/include -I src/main/headers -std=c++11
LINK=$(CXX)
LFLAGS=../gtest/build/libs/libgtest.a -lpthread
SRC=src/main/cpp
OBJDEST=build/objs/main
DEST=build/libs
EXEDEST=build/exe
TESTSRC=src/mainTest/cpp
TESTOBJDEST=build/objs/test
CPP=$(wildcard $(SRC)/*.cpp)
OBJS=$(patsubst $(SRC)/%, $(OBJDEST)/%, $(CPP:%.cpp=%.o))
TESTCPP=$(wildcard $(TESTSRC)/*.cpp)
TESTOBJS=$(patsubst $(TESTSRC)/%, $(TESTOBJDEST)/%, $(TESTCPP:%.cpp=%.o))
.SUFFIXES:
#
# Targets:
#
all: $(DEST)/$(TARGET) test
build/setup: ➁
-mkdir -p $(DEST)
-mkdir -p $(EXEDEST)
-mkdir -p $(OBJDEST)
-mkdir -p $(TESTOBJDEST)
date > $@
test: $(EXEDEST)/runtest ➂
$(EXEDEST)/runtest
$(DEST)/$(TARGET): $(OBJS) test build/setup ➃
ar rcs $@ $(OBJS)
ranlib $@
$(EXEDEST)/runtest: $(TESTOBJS) $(OBJS) ➄
$(LINK) -o $@ $(CPPFLAGS) $(TESTOBJS) $(OBJS) $(LFLAGS)
$(OBJDEST)/%.o: $(SRC)/%.cpp build/setup ➅
$(CXX) $(CXXFLAGS) -o $@ -c $<
$(TESTOBJDEST)/%.o: $(TESTSRC)/%.cpp build/setup ➆
$(CXX) $(CXXFLAGS) -o $@ -c $<
clean:
-rm -f build
➀ Symbolic names for directories, programs, and lists of files.
➁ Set up output directories
➂ Run the unit tests.
➃ Create the library from the .o
files generated by compilation
➄ Generate the executable for running unit tests.
➅ Compile the ADTs source code
➆ Comile the unit test source code.