Node Yarn Docker

Node.js is a software platform for scalable server-side and networking applications. Node.js applications are written in JavaScript and can be run within the Node.js runtime on Mac OS X, Windows, and Linux without changes. Node.js applications are designed to maximize throughput and efficiency, using non-blocking I/O and asynchronous events. Like the repo says. I wanted an example that had real code in it and that had the tools I normally use but dockerised. A boilerplate/prototype docker yarn REACT pipeline. Requirements Docker. If your running on Linux/Ubuntu you probably have this all already, if not, i'm sure you can Google that.

Working as a web agency (or more specifically at marmelab, as an innovation workshop), we have to deal with several different customers and projects. Each of these projects has its own set of technologies, and sometimes, their own version requirements. Installing concurrently all these heterogeneous components would be a nightmare.

Fortunately, Docker exists. Docker containers are a kind of very light virtual machines (let be simple for this post). They have a lot of advantages, the best one being probably the Docker repository. It provides a wide range of ready-to-use components. Either you need a WordPress or a Golang server, we can simply download the related Docker image and we are ready to go.

Setting Up a Node Server with Docker

Docker

For instance, if we want to develop with Node.js, we can simply download the official Node.js image and use it as following:

This command instantiates a container using the node:7 already configured image. It maps current working directory ($PWD) to /app container folder, and launch the command node index.js. The --rm option means Docker should delete the container once its command is executed. And the -it flag aims to map standard container input and output with the host ones.

We got a single container here, so this method may be just fine. Yet, if we deal with several containers (a PostGreSQL database for instance), it would be easier to use docker-compose to specify how they link to each other. Docker-compose configuration is stored in YAML format:

If we run docker-compose up, it would then start two containers, one for PostGreSQL and one for Node. Your architecture is now done, and every new developer on our project would then be able to bootstrap their whole environment in a couple of minutes, with exact required versions. Really exciting isn’t it?

Fixing Container Created Files Permissions Issues

Now, let’s suppose we don’t have npm installed on our host machine. Bootstrapping our project requires to install all Node dependencies we declared in our package.json file. So, we would need to execute a command on our node container, using the run command provided by docker-compose.

Note the --no-deps argument, which prevents to start db service in this case.

This command would work fine. Yet, if we check node_modules file permissions, we would get an unpleasantly surprise:

As you can notice, all files created from Docker container are created as root user. These permissions issues are not really terrible on local environment (we can just sudo after all). But on a CI server, when a PR merge triggers the removal of the branch folder, we would face some troubles. CI low-privileged agent would never be able to remove root folders.

There is no easy solution to this problem. One way is to specify the user option on our node service with host user and group id:

docker-compose would replace automatically variables from its configuration file using environment ones. Hence, we would just need to export both UID and GID variables before calling our docker-compose run command. To ease our future dependencies installation, we can use a Makefile task:

Yarn

The id -u (or -g) retrieves current user (and group respectively) ids. We export them to put them in current environment, and so to transmit them to the docker container.

Now, if we want to install our JavaScript dependencies with correct file permissions, we would just need to launch following command:

If you check file permissions, they all should be fine now.

Installing Dependencies From Private GitHub Repository

Everything worked fine… until we wanted to use a private GitHub repository. In this case, npm would simply fail to authenticate on GitHub, providing us a 401 error message.

Sharing Local SSH Key with Docker Container

So, we would need to share our SSH credentials with our container. The simplest solution would be to map our ~/.ssh folder within the container. Yet, I am not really confortable to share my own personal key, even within a private container. Fortunately, a better solution exists: SSH agent forwarding. It allows to use local SSH keys even while being connected on other machines.

To enable SSH forwarding within our container, we just need to map SSH authentication socket path (SSH_AUTH_SOCK environment variable) to /ssh_agent, and to specify this folder as our container SSH_AUTH_SOCK path:

Before trying to install our private repository, we are going to test our SSH connection to GitHub:

Our container still refuses our request, with the following error message:

No user exists for uid 1000

Creating a Dummy User for SSH Connection

Indeed, we mapped our container user to our host UUID (1000), yet our container has no user with such an id. A solution is to instantiate a fresh new container from main node image as root. It would then allow us to create a dummy user with expected UUID and to execute our ssh command correctly:

This command looks intimidating, but we simply create a user with our host UUID, create a folder required by ssh, set correct permissions on it, and execute our test ssh command. When ran, this command displays another error:

Host key verification failed.

Disabling Host Key Verification

Not an issue: just disable host key verification for GitHub, adding a .ssh/config file as following:

Node Yarn Docker Patterns

And the new ssh_config file:

If we run another time the above command, we should then get a success message:

Hi jpetitcolas! You’ve successfully authenticated, but GitHub does not provide shell access.

Npm Install a Private Repository From Docker Container

Finally! We are now almost done. If we try to replace the ssh -T command by npm install, we would get an error because the ~/.npm folder doesn’t exist. Just add its creation to previous command, which would become finally:

And now, your private dependency should also be installed correctly.

What About Yarn?

yarn is often promoted as a fully compatible npm clone on steroids. Our own experience proved this is not exactly the case. And here again, there are some subtle differences. First, we need to specify the user to our ssh_config file. It then becomes:

Dockerfile

Moreover, we had to change the way we declared our dependency in our package.json file:

Indeed, without the ssh:// protocol, yarn doesn’t seem to understand it should clone it over SSH.

The solution proposed here is far from trivial, yet we were not able to find another solution. If a developer reading this post has an easier way, please, tell us! :)

-->

Did you know that you can look at what makes up an image? Using the docker image history command, you can see the command that was used to create each layer within an image.

  1. Use the docker image history command to see the layers in the getting-started image you created earlier in the tutorial.

    You should get output that looks something like this (dates/IDs may be different).

    Each of the lines represents a layer in the image. The display here shows the base at the bottom with the newest layer at the top. Using this, you can also quickly see the size of each layer, helping diagnose large images.

  2. You'll notice that several of the lines are truncated. If you add the --no-trunc flag, you'll get the full output (yes, you use a truncated flag to get untruncated output).

Layer caching

Now that you've seen the layering in action, there's an important lesson to learn to help decrease build times for your container images.

Docker Node Yarn Version

Once a layer changes, all downstream layers have to be recreated as well

Let's look at the Dockerfile you were using one more time...

Going back to the image history output, you see that each command in the Dockerfile becomes a new layer in the image. You might remember that when you made a change to the image, the yarn dependencies had to be reinstalled. Is there a way to fix this? It doesn't make much sense to ship around the same dependencies every time you build, right?

To fix this, you can restructure your Dockerfile to help support the caching of the dependencies. For Node-based applications, those dependencies are defined in the package.json file. So, what if you copied only that file in first, install the dependencies, and then copy in everything else? Then, you only recreate the yarn dependencies if there was a change to the package.json. Make sense?

  1. Update the Dockerfile to copy in the package.json first, install dependencies, and then copy everything else in.

  2. Build a new image using docker build.

    You should see output like this...

    You'll see that all layers were rebuilt. Perfectly fine, since you changed the Dockerfile quite a bit.

  3. Now, make a change to the src/static/index.html file (like change the <title> to say 'The Awesome Todo App').

  4. Build the Docker image now using docker build again. This time, your output should look a little different.

    First off, you should notice that the build was MUCH faster! And, you'll see that steps 1-4 all haveUsing cache. So, hooray! You're using the build cache. Pushing and pulling this image and updates to itwill be much faster as well. Hooray!

Multi-stage builds

While we're not going to dive into it too much in this tutorial, multi-stage builds are an incredibly powerfultool to help use multiple stages to create an image. There are several advantages for them:

  • Separate build-time dependencies from runtime dependencies
  • Reduce overall image size by shipping only what your app needs to run

Maven/Tomcat example

When building Java-based applications, a JDK is needed to compile the source code to Java bytecode. However,that JDK isn't needed in production. Also, you might be using tools like Maven or Gradle to help build the app.Those also aren't needed in your final image. Multi-stage builds help.

This example uses one stage (called build) to perform the actual Java build using Maven. The second stage (starting at FROM tomcat) copies in files from the build stage. The final image is only the last stage being created (which can be overridden using the --target flag).

Node alpine docker yarn

React example

When building React applications, you need a Node environment to compile the JS code (typically JSX), SASS stylesheets, and more into static HTML, JS, and CSS. If you aren't doing server-side rendering, you don't even need a Node environment for the production build. Why not ship the static resources in a static nginx container?

Here, you're using a node:12 image to perform the build (maximizing layer caching) and then copying the output into an nginx container. Cool, huh?

Dockerfile Node Yarn Install

Recap

Yarn Docker Compose

By understanding a little bit about how images are structured, you can build images faster and ship fewer changes. Multi-stage builds also help reduce overall image size and increase final container security by separating build-time dependencies from runtime dependencies.

Next steps

Continue with the tutorial!