Build a common set of images from Dockerfiles.
Core functionality is in place. See the TODO section below for open thoughts/items.
Automate the process of building structured images from Dockerfiles that are tagged/linked together.
See Dotfiles for bringing these into your shell environment.
If you're an end user, you should consume the images from the configured image registry that are uploaded there as part of the CI/CD process.
By default, these are located here on Dockerhub and are deployed automatically via Travis CI.
If you're a developer working on this repository, here's some hints to hopefully guide you through the process.
Usage
All Makefile should support make help to display their usage and have help set as their default goal.
$ make
all Run recursive 'make all' to build an deploy all images.
build Run recursive 'make build' to build all images.
clean Run recursive 'make clean' to clean all images.
deploy Run recursive 'make deploy' to deploy all images.
test Run recursive 'make test' to test all images.
help Print Makefile usage.While not visible in the help output, the main Makefile supports recursive target generation. This enables invoking targets on the Makefile defined in subdirectories.
For example, let's access the alpine image.
$ make alpine
all Run recursive 'make all' to build an deploy all images.
build Run recursive 'make build' to build all images.
clean Run recursive 'make clean' to clean all images.
deploy Run recursive 'make deploy' to deploy all images.
test Run recursive 'make test' to test all images.
help Print Makefile usage.Hmm, this looks the same. Well that's because alpine is also a recursive Makefile for all of it's subdirectories. Let's drill into a subdirectory at the bottom of the tree to see what it looks like for building an explicit version of an alpine image.
$ make alpine/3.7
all Build and deploy image created from Dockerfile.
build Build image from Dockerfile.
clean Clean state generated by previous images built from Dockerfile.
deploy Deploy image built from Dockerfile.
test Test and validate files necessary to build image.
help Print Makefile usage.That's just the default target, help. Let's specify one by invoking the build target for this alpine/3.7 image.
$ make alpine/3.7-build
[TRACE] - [dockerfiles] - Running 'build' for child image 'alpine/3.7'
[TRACE] - [alpine-3.7] - Running 'build'
Sending build context to Docker daemon 9.216kB
...
...
Successfully built 57642cb07bbe
Successfully tagged ahawker/alpine:6cdaa04-1531262409-2704231498
Successfully tagged ahawker/alpine:latest
[TRACE] - [alpine-3.7] - Completed 'build'
[TRACE] - [dockerfiles] - Completed 'build' for child image 'alpine/3.7'Based on how the make help usage output is created, the recursive targets automatically created from subdirectories will not appear. However, they all follow the same simple pattern of <directory-path>-<makefile-target-name>. Let's walk through a few examples:
make alpine/3.7-build- Build thealpine-3.7image.make alpine/3.7-deploy- Deploy thealpine-3.7image.make python/3/3.6.4-clean- Clean artifacts for thepython3.6.4image.
As you can see, the python example includes subdirectories at depth+1 beyond that of the alpine images. The pattern will hold for invoking targets of subdirectories to any depth.
However let's say you don't want to build just alpine-3.7 but all alpine images within the repository. You can simply invoke the same targets on the parent directory and it will recursively invoke them for all subdirectories. For example:
make alpine-build- Recursively builds all availablealpineimages.make python/2-build- Recursively builds all avialablepython2images.make python-build- Recursively builds all availablepython2andpython3images.
That's about it in terms of "usage". The directory structure in conjunction with the Makefile define the targets that can be run against the directories.
If you're looking to add new images to this repository, check out the Contributing section below.
This repository uses make as the build system. If you're not familiar with it, debugging what's happening behind the scenes can sometimes feel like dark magic. Without opening a large can of worms, let's walk through some of the common issues one might encounter while developing within this repository.
Example:
$ make ruby/2.4-build
[TRACE] - [dockerfiles] - Running 'build' for child image 'ruby/2.4'
[TRACE] - [ruby-2.4] - Running 'build' for base image 'alpine'
[TRACE] - [ruby-2.4] - Completed 'build' for base image 'alpine'
[TRACE] - [ruby-2.4] - Running 'build'
Sending build context to Docker daemon 11.78kB
Step 1/21 : ARG REPO
Step 2/21 : ARG BASE_IMAGE
Step 3/21 : ARG TAG=latest
Step 4/21 : FROM $REPO/$BASE_IMAGE:$TAG
manifest for ahawker/alpine:6cdaa04-1531262592-835582456 not found
make[1]: *** [build] Error 1
make: *** [ruby/2.4-build] Error 2Context:
When any make command is invoked, a number of runtime defined parameters are loaded by the root Makefile from it's local .env. The two major ones to consider are BUILD_ID and BUILD_TS. If not specified by the build environment, these will be set to a random identifier and the current timestamp epoch respectively. These values, along with git metadata, will be used to tag every image build in the current process execution.
Issue:
This happens when you build a base image e.g. alpine-3.7 in one process execution and then attempt to build a image that depends on alpine-3.7 in a new process execution.
Solution:
Run a clean command against the image you're currently trying to build e.g. make terraform/0.11.7-clean. If you're trying to rebuild all images, run make clean.
Example:
$ make build
[TRACE] - [ruby] - Running 'build' for child image '2.4'
Required variable "REPO" not set
make[1]: *** [requires-REPO] Error 1
make: *** [2.4-build] Error 2Context:
Each Makefile performs validation on the build context to make sure all necessary values are set (or loaded from .env files) into the make context before executing the corresponding docker [build|deploy] command.
Issue:
One of the variables defined in a requires-XYZ target dependency is not set/empty.
Solution:
A few possibilities:
- You've defined a
requires-XYZin yourMakefilebut it is not yet defined in your.envfile. Define it! - You're running the
Makefiledirectly in a subdirectory. This is not currently supported and all images can/should be possible to build from the rootMakefile.
This section describes some of the more technical details that may be helpful to the curious minds out there or those contributing/debugging.
While you may see many Makefile within this repository, they're all symlinks back to two files: Base.make and Recursive.make.
Base.make- Defines how to build a docker image from a set of.envfiles &Dockerfile.Recursive.make- Defines how to recursively call knownmaketargets on subdirectories.
An example of the symlinking nature can be seen by examining the hierarchy for the alpine image.
⇒ tree -a alpine .common
alpine
├── .common
│ ├── .env
│ ├── Dockerfile
│ └── Makefile -> ../../.common/Base.make
├── 3.3
│ ├── .env
│ ├── Dockerfile -> ../.common/Dockerfile
│ └── Makefile -> ../.common/Makefile
├── 3.7
│ ├── .env
│ ├── Dockerfile -> ../.common/Dockerfile
│ └── Makefile -> ../.common/Makefile
├── 3.8
│ ├── .env
│ ├── Dockerfile -> ../.common/Dockerfile
│ └── Makefile -> ../.common/Makefile
└── Makefile -> ../.common/Recursive.make
.common
├── .env
├── Base.make
├── Recursive.makeThe version-specific Makefile symlinks back to the .common copy which is a symlink to the root directory .common/Base.make while the Makefile in the alpine directory is a symlink to the root directory .common/Recursive.make.
There is a directory named partials within the root .common directory that contains files that are automatically loaded by both the Base.make and Recursive.make files. Files within the partials directory are small snippets of Makefile code that are included into the others as a means of writing re-usable code between the two files.
Optional .env file(s) can be placed in stratgic locations within the directory structure to define values that are passed to the Makefile and potentially Dockerfile. The structure of a .env file is just a Makefile partial and commonly appears as:
# Name of the image created from the Dockerfile.
IMAGE = foobar-1.2.3
# Name of the base image to build on.
BASE_IMAGE = foobar-base
# Version of some package.
PACKAGE_VERSION = 1.2.3There are five types of .env files: default, root, parent, local, and custom.
default- Located within a.commonor local directory in.defaultfiles.root- Located in the repository root at.common/.env.parent- Located within a.common/.envdirectory that is a sibling to aRecursive.makesymlink.local- Located within the same directory as aDockerfileandBase.makesymlink.custom- Located within a.commonor local directory in.customfiles.
The values of variables within .env files can be overwritten by other .env files based on the resolution order, which is default, root, parent, local, and custom. This means that more generic variables defined within the parent directory can be overwritten by image/version specific values within their local copy.
Within .env files, there's some known/required variables that should be defined. These are classified into three categories: meta, build, and custom.
The meta variables are those that are used by the dockerfiles repo code itself. Some examples of this are:
BUILD_ID- Identifier (random or CI environment specified) for the build.BUILD_TS- Unix epoch of when the build process started.TAG- Identifier string containing multiple pieces of metadata that is applied as a tag to every image created.ARGS- List ofbuildarguments to pass todocker buildas--build-argvalues.LABELS- List ofbuildarguments to pass todocker buildas--labelvalues.TAGS- List ofbuildarguments to pass todocker buildas--tagsvalues.
For a comprehensive list of meta variables, see the Variables page.
The build variables are those that configure the build and deploy environment of the images themselves. These are commonly seen as ARG entries in the Dockerfile themselves. Some examples of these are:
# Version of system 'gem' to install.
GEMS_VERSION = 2.7.7
# Version of the 'bundler' gem to install.
BUNDLER_VERSION = 1.16.2These values should also be added to the ARGS meta variable if they need to be exposed to the docker build process.
Custom variables are no different than any other .env variable/file beyond the fact that they are last to load, thus giving them the highest priority.
These files should be used for your specific build environments committed within your fork of this repository but not pushed back to the upstream repository.
Base.make is responsible for validation of build arguments before invoking the build and deploy targets which call docker build and docker push respectively. The logic has been generalized so .env must just specify some variables that contain a list of variable names if they need to be validated prior to execution. This commonly appears as:
# List of additional variable names that are required by the 'build' target.
BUILD_REQUIREMENTS_VARIABLES = RUBY_VERSION \
RUBY_GEM_VERSION \
GEMS_VERSION \
BUNDLER_VERSION \
ALPINE_RUBY_PACKAGE_NAME \
ALPINE_RUBY_PACKAGE_VERSION \
ALPINE_RUBY_JSON_PACKAGE_NAME \
ALPINE_RUBY_IRB_PACKAGE_NAME
# List of additional variable names that are required by the 'clean' target.
CLEAN_REQUIREMENTS_VARIABLES =
# List of additional variable names that are required by the 'deploy' target.
DEPLOY_REQUIREMENTS_VARIABLES = RUBY_VERSION
# List of additional variable names that are required by the 'test' target.
TEST_REQUIREMENTS_VARIABLES =If new variables come into existence that need to be validated for all images within the repository, these should be added to the root .common/.env file using the GLOBAL_ prefix variables.
The Recursive.make file will automatically detect dependencies and add them as prerequisites for the build target.
- If your image specifies a
BASE_IMAGEthat is one built by this repository, the$(BASE_IMAGE)-buildprerequisite is added. - All children/sibling files of the
Recursive.makesymlink that aren't empty target files are added as prequisites.
The Recursive.make file will use Empty Targets to track the execution times of the build, deploy, and test targets. With these empty targets, it can then determine if/when targets should be re-run based on changes to the target prerequsities.
Read this section carefully if you're looking to contribute to this repository.
If you're contributing a new version to an existing image, here's a rough checklist of things you should be looking at. For this example, let's say the image is alpine and we're adding version 4.2.
- Create the
4.2directory under thealpinedirectory.mkdir -p alpine/4.2 - Create
Makefilesymlink within the4.2directory.ln -s alpine/.common/Makefile alpine/4.2/Makefile - Create
Dockerfilesymlink within the4.2directory.ln -s alpine/.common/Dockerfile alpine/4.2/Dockerfile - Create
.envwithin the4.2directory and specify version-specific variables.
If you're adding a new image type to the repository, I first suggest that you examine an existing image, e.g. alpine to get a better understanding of the directory/file structure. Once you've done that, continue on with the notes below about the build process.
This is just a standard Dockerfile. At a minimum, this should be parameterized with ARG instructions for specifying the base image repository, image name and tag. This commonly appears as:
ARG REPO
ARG BASE_IMAGE
ARG TAG=latest
FROM $REPO/$BASE_IMAGE:$TAG
This allows the Dockerfile to build off the most recent version of the BASE_IMAGE that was created as part of the current build process. If this does not happen or works incorrect, the image will not be properly linked to the others of the same build.
It is highly recommended to use ARG instructions to parameterize as much of the Dockerfile as possible. These values can be injected during the docker build process and can be defined in a set of .env files. This allows for potential reuse of the Dockerfile combination with any number of .env files.
It is highly recommend that you follow the existing patterns of this repository and just use symlinks of the Recurisve.make and Base.make files found in the root .common directory instead of trying to create your own. For example, let's take a look at how the alpine directory is structured:
$ tree -a alpine
alpine
├── .common
│ ├── .env
│ ├── Dockerfile
│ └── Makefile -> ../../.common/Base.make
├── 3.3
│ ├── .env
│ ├── Dockerfile -> ../.common/Dockerfile
│ └── Makefile -> ../.common/Makefile
├── 3.7
│ ├── .env
│ ├── Dockerfile -> ../.common/Dockerfile
│ └── Makefile -> ../.common/Makefile
├── 3.8
│ ├── .env
│ ├── Dockerfile -> ../.common/Dockerfile
│ └── Makefile -> ../.common/Makefile
└── Makefile -> ../.common/Recursive.makeThe alpine directory contains it's own .common directory, a number of version-specific directories and a Makefile. The key items to note are:
- The
Makefileis a relative path symlink to the.common/Recursive.makefile in the root of the repository. - The
.common/Makefileis a relative path symlink to the.common/Base.makefile in the root of the repository. - The
MakefileandDockerfilewithin each version-specific directory are relative path symlinks into the local.commondirectory. - The
.envfiles are all real and not symlinks.
Within the .common directory we have three files:
Makefileis just a symlink toBase.makethat provides targets for building.Dockerfileis a standardDockerfilethat is parameterized withARGand covered in the README section above..envis a Makefile partial file that gets automaticalled loaded by theMakefile.
Within the version-specific directories, we have the same three files as .common. The Dockerfile and Makefile are just symlinks into .common. The .env file contains variables used by the ARG parameterized Dockerfile that are specific to this version of the image.
This is just a standard Makefile for GNU Make. At a minimum, it needs to define the following targets:
all- Invokes thebuildand then subsequentdeploytarget.build- Builds a image by injecting build arguments intodocker buildfor the localDockerfile.clean- Cleans up any state created by previous builds, including images stored by theDockerdaemon.deploy- Pushes tagged versions of the locally built image to a remote repository.test- Runs a test/lint process against files required to build the image.
If you wish to customize the dockerfiles repository to your own liking, I recommend forking the repository and optionally submitting pull-requests upstream to me if it's a fix or helpful addition I might enjoy.
$ git remote add upstream git@github.com:ahawker/dockerfiles.git
$ git fetch upstream
$ git rebase upstream/master
$ git push origin masterIf you wish to override variables in your local setup, define them within .custom files and they should be given the highest priority based on the variable resolution order. These files should not be pushed back in pull-requests to the upstream repository.