Working as a web agency (or more specifically at marmelab, as an innovation workshop), we have to deal with several different customers and projects. Each of these projects has its own set of technologies, and sometimes, their own version requirements. Installing concurrently all these heterogeneous components would be a nightmare.
Fortunately, Docker exists. Docker containers are a kind of very light virtual machines (let be simple for this post). They have a lot of advantages, the best one being probably the Docker repository. It provides a wide range of ready-to-use components. Either you need a WordPress or a Golang server, we can simply download the related Docker image and we are ready to go.
Setting Up a Node Server with Docker
For instance, if we want to develop with Node.js, we can simply download the official Node.js image and use it as following:
This command instantiates a container using the
node:7 already configured image. It maps current working directory (
/app container folder, and launch the command
node index.js. The
--rm option means Docker should delete the container once its command is executed. And the
-it flag aims to map standard container input and output with the host ones.
We got a single container here, so this method may be just fine. Yet, if we deal with several containers (a PostGreSQL database for instance), it would be easier to use
docker-compose to specify how they link to each other. Docker-compose configuration is stored in YAML format:
If we run
docker-compose up, it would then start two containers, one for PostGreSQL and one for Node. Your architecture is now done, and every new developer on our project would then be able to bootstrap their whole environment in a couple of minutes, with exact required versions. Really exciting isn’t it?
Fixing Container Created Files Permissions Issues
Now, let’s suppose we don’t have
npm installed on our host machine. Bootstrapping our project requires to install all Node dependencies we declared in our
package.json file. So, we would need to execute a command on our
node container, using the
run command provided by
--no-deps argument, which prevents to start
db service in this case.
This command would work fine. Yet, if we check
node_modules file permissions, we would get an unpleasantly surprise:
As you can notice, all files created from Docker container are created as
root user. These permissions issues are not really terrible on local environment (we can just
sudo after all). But on a CI server, when a PR merge triggers the removal of the branch folder, we would face some troubles. CI low-privileged agent would never be able to remove
There is no easy solution to this problem. One way is to specify the
user option on our
node service with host user and group id:
docker-compose would replace automatically variables from its configuration file using environment ones. Hence, we would just need to export both
GID variables before calling our
docker-compose run command. To ease our future dependencies installation, we can use a
id -u (or
-g) retrieves current user (and group respectively) ids. We
export them to put them in current environment, and so to transmit them to the docker container.
If you check file permissions, they all should be fine now.
Installing Dependencies From Private GitHub Repository
Everything worked fine… until we wanted to use a private GitHub repository. In this case,
npm would simply fail to authenticate on GitHub, providing us a 401 error message.
Sharing Local SSH Key with Docker Container
So, we would need to share our SSH credentials with our container. The simplest solution would be to map our
~/.ssh folder within the container. Yet, I am not really confortable to share my own personal key, even within a private container. Fortunately, a better solution exists: SSH agent forwarding. It allows to use local SSH keys even while being connected on other machines.
To enable SSH forwarding within our container, we just need to map SSH authentication socket path (
SSH_AUTH_SOCK environment variable) to
/ssh_agent, and to specify this folder as our container
Before trying to install our private repository, we are going to test our SSH connection to GitHub:
Our container still refuses our request, with the following error message:
No user exists for uid 1000
Creating a Dummy User for SSH Connection
Indeed, we mapped our container user to our host UUID (1000), yet our container has no user with such an id. A solution is to instantiate a fresh new container from main
node image as
root. It would then allow us to create a dummy user with expected UUID and to execute our
ssh command correctly:
This command looks intimidating, but we simply create a user with our host UUID, create a folder required by
ssh, set correct permissions on it, and execute our test
ssh command. When ran, this command displays another error:
Host key verification failed.
Disabling Host Key Verification
Not an issue: just disable host key verification for GitHub, adding a
.ssh/config file as following:
Node Yarn Docker Patterns
And the new
If we run another time the above command, we should then get a success message:
Hi jpetitcolas! You’ve successfully authenticated, but GitHub does not provide shell access.
Npm Install a Private Repository From Docker Container
Finally! We are now almost done. If we try to replace the
ssh -T command by
npm install, we would get an error because the
~/.npm folder doesn’t exist. Just add its creation to previous command, which would become finally:
And now, your private dependency should also be installed correctly.
What About Yarn?
yarn is often promoted as a fully compatible
npm clone on steroids. Our own experience proved this is not exactly the case. And here again, there are some subtle differences. First, we need to specify the
user to our
ssh_config file. It then becomes:
Moreover, we had to change the way we declared our dependency in our
Indeed, without the
yarn doesn’t seem to understand it should clone it over SSH.
The solution proposed here is far from trivial, yet we were not able to find another solution. If a developer reading this post has an easier way, please, tell us! :)-->
Did you know that you can look at what makes up an image? Using the
docker image history command, you can see the command that was used to create each layer within an image.
docker image historycommand to see the layers in the
getting-startedimage you created earlier in the tutorial.
You should get output that looks something like this (dates/IDs may be different).
Each of the lines represents a layer in the image. The display here shows the base at the bottom with the newest layer at the top. Using this, you can also quickly see the size of each layer, helping diagnose large images.
You'll notice that several of the lines are truncated. If you add the
--no-truncflag, you'll get the full output (yes, you use a truncated flag to get untruncated output).
Now that you've seen the layering in action, there's an important lesson to learn to help decrease build times for your container images.
Docker Node Yarn Version
Once a layer changes, all downstream layers have to be recreated as well
Let's look at the Dockerfile you were using one more time...
Going back to the image history output, you see that each command in the Dockerfile becomes a new layer in the image. You might remember that when you made a change to the image, the yarn dependencies had to be reinstalled. Is there a way to fix this? It doesn't make much sense to ship around the same dependencies every time you build, right?
To fix this, you can restructure your Dockerfile to help support the caching of the dependencies. For Node-based applications, those dependencies are defined in the
package.json file. So, what if you copied only that file in first, install the dependencies, and then copy in everything else? Then, you only recreate the yarn dependencies if there was a change to the
package.json. Make sense?
Update the Dockerfile to copy in the
package.jsonfirst, install dependencies, and then copy everything else in.
Build a new image using
You should see output like this...
You'll see that all layers were rebuilt. Perfectly fine, since you changed the Dockerfile quite a bit.
Now, make a change to the
src/static/index.htmlfile (like change the
<title>to say 'The Awesome Todo App').
Build the Docker image now using
docker buildagain. This time, your output should look a little different.
First off, you should notice that the build was MUCH faster! And, you'll see that steps 1-4 all have
Using cache. So, hooray! You're using the build cache. Pushing and pulling this image and updates to itwill be much faster as well. Hooray!
While we're not going to dive into it too much in this tutorial, multi-stage builds are an incredibly powerfultool to help use multiple stages to create an image. There are several advantages for them:
- Separate build-time dependencies from runtime dependencies
- Reduce overall image size by shipping only what your app needs to run
When building Java-based applications, a JDK is needed to compile the source code to Java bytecode. However,that JDK isn't needed in production. Also, you might be using tools like Maven or Gradle to help build the app.Those also aren't needed in your final image. Multi-stage builds help.
This example uses one stage (called
build) to perform the actual Java build using Maven. The second stage (starting at
FROM tomcat) copies in files from the
build stage. The final image is only the last stage being created (which can be overridden using the
When building React applications, you need a Node environment to compile the JS code (typically JSX), SASS stylesheets, and more into static HTML, JS, and CSS. If you aren't doing server-side rendering, you don't even need a Node environment for the production build. Why not ship the static resources in a static nginx container?
Here, you're using a
node:12 image to perform the build (maximizing layer caching) and then copying the output into an nginx container. Cool, huh?
Dockerfile Node Yarn Install
Yarn Docker Compose
By understanding a little bit about how images are structured, you can build images faster and ship fewer changes. Multi-stage builds also help reduce overall image size and increase final container security by separating build-time dependencies from runtime dependencies.
Continue with the tutorial!