In Oct of 2019, I was attending an online training and watched the instructor giving a demo using a Docker container. Then I saw a friend’s profile picture at LinkedIn wearing a Docker t-shirt. These are all signs. The days of manually installing software and getting our hands dirty are over. The days of post uninstall residue is over (something that fills my drive due to unclean uninstalls). Finally, you know that it came when you see the big dinosaurs like companies are using it or experimenting with it. These companies are skeptical and normally stands at the end of the adoption line. I have nothing against them, it’s a strategy they use. With technology landscape changing so frequently it’s how they play safe.
I am creating this know-how on Docker for those of you, who still did not get a chance to use it. It should help one get quickly hooked to Docker. It is as easy as Git. If you are using Git you will be able to spot the similarities. All my references are based out of my Docker experience on Ubuntu. If you have LINUX based environment it will be easy to follow.
Who’s there? It’s Docker!
Before jumping in lets see what is Docker in short:
- Docker is a technology to build, ship and deploy software
- Docker packages all the software dependencies in one image
- Docker isolates processes and resources in a container
So what are containers? Containers are completely isolated environments having their own processes/services, network interfaces, their own mounts like Virtual Machines (VM) except they all share the same OS Kernel. Docker provides a high-level tool that makes setting up a container very simple. There are different Linux OS, but they all share one underlying kernel and have different software on top of it which differentiate OSs from each other. Docker containers share the underlying Kernel. Let’s say, we have a Ubuntu machine with Docker installed on it. Docker can run any flavor of OS on top of it as long as they share the same Kernel (Linux). If the underlying OS is Ubuntu Docker can run a container based on another distribution like CentOS, Debian, Fedora, SuSe, etc. Each Docker contained only have the additional software that makes these OS different. In Windows, we have a completely different kernel, but we can still run a docker container based out of Linux. That is done by making Windows run a Linux container on a Linux Virtual Machine (VM). This enables us to run software anywhere.
What is the difference between VM and a docker container? In case of VMs there is underlying hardware on which runs a hypervisor and on top of it runs multiple VMs. Each VMs have OS, where we run our application(s). On this OS we have the application and its dependencies (MQ client, Oracle client, etc.) installed. On the other hand, for a container, we have the underlying hardware on which runs the OS. On top of OS, we have Docker and above it, we have multiple docker containers where we have the application and their dependencies. Thus, Dockers are more lightweight. Additionally, VMs needs more resource like CPU and disk space as they are running different OSs. Note that, Docker is not against VMs, instead you will see containers provisioned on VMs (Virtual Docker hosts). This enables us to enjoy the benefits of both worlds.
There are many containerized versions of applications readily available for use. Many organizations are having their product containerized and made their images available at a public docker repository like Docker hub (https://hub.docker.com/). Once you identify the images you need and have docker installed on your machine bringing up an application/container is very easy. It is as simple as executing a docker run command with the name of an image.
$ docker run redis <output truncated> $ docker run jenkins/jenkins <output truncated> $ docker run mongodb <output truncated>
What is an image? An image is a package or template that is used to create a container.
Opening the door for Docker!
If you have a Mac or a Windows machine you can use the link and follow along to get Docker desktop installed. If you have an older version of Windows or Mac, then you may need to work with Docker toolbox.
Now those on Linux machine, check out the steps at https://docs.docker.com/engine/install/ubuntu/. It may seem a lengthy process but in the middle of the page there is an option to “Install using the convenience script“. This is the simplest way to get Docker installed.
$ curl -fsSL https://get.docker.com -o get-docker.sh$ sudo sh get-docker.sh<output truncated>
You will have to use Docker as a root user, meaning you will have to use sudo before every docker command or run a shell using sudo (i.e. you go in as root). Writing sudo before every command painful. I prefer to execute a sudo bash and then run all Docker command from this shell.
$ sudo bash
But there is another option: consider adding your user to the “docker” group with below command:
$ sudo usermod -aG docker <your-user>
Remember to log out and log back in for this to take effect! Now that you have let docker in, let’s say “Beginnen sie” (that’s what my German teacher used to say at the beginning of every lesson).
Coffee with Docker!
Lets say hi to docker with a “hello world!!”
$ docker run hello-world Unable to find image 'hello-world:latest' locally latest: Pulling from library/hello-world 0e03bdcc26d7: Pull complete Digest: sha256:d58e752213a51785838f9eed2b7a498ffa1cb3aa7f946dda11af39286c3db9a9 Status: Downloaded newer image for hello-world:latest Hello from Docker! This message shows that your installation appears to be working correctly. ... $
When you see the text “Hello from Docker!”, it means your docker installation is successful. The command you ran did the following:
- The Docker client contacted the Docker daemon
- Docker daemon could not find the image called “hello-world” in local repository
- Docker daemon pulled the image ‘hello-world:latest’ from Docker Hub. Here ‘latest’ is a tag pointing to the most recent image of hello-world
- The Docker daemon created a new container from that image which runs the executable that produces the output you see in your screen
- The Docker daemon streamed that output to the Docker client, which sent it to your terminal
Those familiar with Git must have spotted the familiarity docker share with Git. This really helps to flatten the learning curve.
The hello-world program is an executable that just prints to the console and exits. We can check all the docker image that were running in the past and the ones that are currently running. We do this using:
$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES edbf0ab590df hello-world "/hello" 11 minutes ago Exited (0) 11 minutes ago interesting_kapitsa
If you just execute ‘docker ps’ without the ‘- a’ option, it only shows you the images that are currently running.
$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
Aah!! Nothing is running at this moment. If you work on Linux, you would feel at home with the ps command, which shows running processes in Unix. The ‘-a’ options tell docker to show all docker processes both running and not running ones.
In the output, the CONTAINER ID is like a unique hash key (similar to Git’s SHA-1s) for every image executed through docker. The IMAGE shows the image name. The COMMAND column shows the starting command that got executed from the image. CREATED, tell you when was the command started or invoked. STATUS, tells you the current state of the Docker process with its exit status. In the above example, it says that the docker process has existed 11 minutes ago with exit status 0. PORTS are used to show if the docker process is listening to any port. Most of the time we will be run a database or a server through docker which listens on one or more ports. Lastly, the NAME column, tells the name of a docker process. You can explicitly name it or docker will assign a unique name to it. Docker generated names are a combination of two words. It is always fun to see what name it generates.
Now lets run few more docker images. But this time lets pull the image separately and then start the container separately.
$ docker pull docker/whalesay Using default tag: latest latest: Pulling from docker/whalesay Image docker.io/docker/whalesay:latest uses outdated schema1 manifest format. Please upgrade to a schema2 image for better future compatibility. More information at https://docs.docker.com/registry/spec/deprecated-schema-v1/ e190868d63f8: Pull complete 909cd34c6fd7: Pull complete 0b9bfabab7c1: Pull complete a3ed95caeb02: Pull complete 00bf65475aba: Pull complete c57b6bcc83e3: Pull complete 8978f6879e2f: Pull complete 8eed3712d2cf: Pull complete Digest: sha256:178598e51a26abbc958b8a2e48825c90bc22e641de3d31e18aaf55f3258ba93b Status: Downloaded newer image for docker/whalesay:latest docker.io/docker/whalesay:latest
By default “docker run” command would pull an image if it is not in the local repository and then execute it. However this time we just pulled the image explicitly into our local repository but did not execute it.
This docker/whalesay image contains an adaption of the Linux cowsay game. You can see all the images in your local repository by executing:
$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE hello-world latest bf756fb1ae65 6 months ago 13.3kB docker/whalesay latest 6b362a9f73eb 5 years ago 247MB $ docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE hello-world latest bf756fb1ae65 6 months ago 13.3kB docker/whalesay latest 6b362a9f73eb 5 years ago 247MB
Both commands give the same output. Notice one has ‘s’ after image. I failed to notice that when I tried it for the first time. The size at the end is the amount of space used on your disk to store these images. Now, let’s run this new container:
$ docker run docker/whalesay cowsay Sidd _____ < Sidd > ------\\\ ## ## . ## ## ## == ## ## ## ## ===/""""""""""""""""___/ ===~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~ \______ o __/ \ \ __/ \____\______/
Here “cowsay Sidd” is the argument passed to this program. It shows a docker whale image saying ‘Sidd’. Wow, so far we have executed 2 simple containers. Now let’s learn some housekeeping. Previously showed you how “docker ps -a” command can show you all previously executed containers. We need to manually remove them from the ps list else they will clutter your system.
$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6d9b5286be64 docker/whalesay "cowsay sidd" 20 minutes ago Exited (0) 20 minutes ago beautiful_noether 7b67e865dad3 hello-world "/hello" About an hour ago Exited (0) 11 minutes ago mystifying_bohr edbf0ab590df hello-world "/hello" 17 hours ago Exited (0) 11 minutes ago interesting_kapitsa $ docker rm 7b67e865dad3 7b67e865dad3 $ docker rm interesting_kapitsa interesting_kapitsa $ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6d9b5286be64 docker/whalesay "cowsay sidd" 20 minutes ago Exited (0) 20 minutes ago beautiful_noether
So, “docker rm” removes the data from the process list. To this command, we are either passing the CONTAINER ID or the NAME as they uniquely identify the process. Successful execution of this command returns the input parameter in the output. Now let’s remove the hello-world image from our local image repository.
$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE hello-world latest bf756fb1ae65 6 months ago 13.3kB docker/whalesay latest 6b362a9f73eb 5 years ago 247MB $ docker rmi bf756fb1ae65 Untagged: hello-world:latest Untagged: hello-world@sha256:d58e752213a51785838f9eed2b7a498ffa1cb3aa7f946dda11af39286c3db9a9 Deleted: sha256:bf756fb1ae65adf866bd8c456593cd24beb6a0a061dedf42b26a993176745f6b Deleted: sha256:9c27e219663c25e0f28493790cc0b88bc973ba3b1686355f221c38a36978ac63 $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE docker/whalesay latest 6b362a9f73eb 5 years ago 247MB
Here rmi stands for remove image; Then we pass the image id. Now let’s see what happens when we try to delete the image, docker/whalesay.
$ docker rmi 6b362a9f73eb Error response from daemon: conflict: unable to delete 6b362a9f73eb (must be forced) - image is being used by stopped container 6d9b5286be64
Why we got an error? Remember we did not remove this containers reference from docker’s process list. You can check it by executing “docker ps -a”. To remove this image, you need to remove its reference from the process list first and then try to remove the image.
Before we move forward let me stress on a couple of things: Containers are meant to run a specific process/task like host an application server instance, or a webserver/database instance. Or a container can simply carry out some specific task (like a batch process, or some analytics task, etc. Once the task is over the container exits. A container only lives as long as the process inside is alive.
Now that we have some experience with two containers. Lets explore some more.
$ docker run ubuntu Unable to find image 'ubuntu:latest' locally latest: Pulling from library/ubuntu a4a2a29f9ba4: Pull complete 127c9761dcba: Pull complete d13bf203e905: Pull complete 4039240d2e0b: Pull complete Digest: sha256:35c4a2c15539c6c1e4e5fa4e554dac323ad0107d8eb5c582d6ff386b383b7dce Status: Downloaded newer image for ubuntu:latest $ docker ps -all CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES bea18f99dc3d ubuntu "/bin/bash" About a minute ago Exited (0) About a minute ago mystifying_haslett
It just downloaded the OS and did nothing. We did not ask the OS to do anything. We need to ask it to do something.
$ docker run ubuntu sleep 200 $
Once you run the above command, there will not be any output and you won’t get back the prompt till 200 seconds are over. In the meantime, you can open another terminal and check “docker ps”. If you can run this before 200 seconds are over then you will see a running container for the 1st time.
$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 20e65c3a9256 ubuntu "sleep 200" 2 minutes ago Up 2 minutes affectionate_bhaskara
Lets forcefully stop the container.
$ docker stop 20e65c3a9256 20e65c3a9256 $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
This kills the container.
Now lets run bash shell in Ubuntu container.
$ docker run -it ubuntu bash root@86f4afc361a6:/# ls bin boot dev etc home lib lib32 lib64 libx32 media mnt opt proc root run sbin srv sys tmp usr var root@86f4afc361a6:/# exit exit
This gives us a new command prompt. This is the root prompt of our Ubuntu container. Meaning you are logged in as root in your Ubuntu container. You can run normal commands in it; Here I tried “ls”. Finally, to come out you need to type “exit”.
Here our docker run command executed with -it option. -i is equivalent to –interactive and -t is equivalent to –tty. It means that you will be interacting with the container through your terminal.
In this Ubuntu example when we pull ubuntu or run Ubuntu, by default docker gets the latest version which has the tag ‘latest’. What if you need to get a specific version? Then you need to mention a different tag name under which you have the required version. You can information about all available tags or versions from https://hub.docker.com/. To download a container image from hub.docker.com you don’t need to login, just search for the image by name and then go to the dedicated page for that image. There you will see a tab called Tags (Here is the link to tags section in ubuntu’s page under hub.docker.com: https://hub.docker.com/_/ubuntu?tab=tags). There I found a tag called “20.04”, let us install it.
$ docker pull ubuntu:20.04 20.04: Pulling from library/ubuntu Digest: sha256:52259450119427dab05c0c455121c48d7b04cee2d61b5dbdde1219b2163af572 Status: Downloaded newer image for ubuntu:20.04 docker.io/library/ubuntu:20.04 $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE ubuntu 20.04 74435f89ab78 2 weeks ago 73.9MB ubuntu latest 74435f89ab78 2 weeks ago 73.9MB $ docker run ubuntu:20.04 $
As our next step, let us run Jenkins using docker. Jenkins is an automation server that primarily helps to automate the parts of software development related to building, testing, and deploying, facilitating continuous integration and continuous delivery.
$ docker pull jenkins/jenkins<output truncated>$ docker run jenkins/jenkins Running from: /usr/share/jenkins/jenkins.war webroot: EnvVars.masterEnvVars.get("JENKINS_HOME") 2020-07-05 15:51:52.310+0000 [id=1] INFO org.eclipse.jetty.util.log.Log#initialized: Logging initialized @4690ms to org.eclipse.jetty.util.log.JavaUtilLog<output truncated>Jenkins initial setup is required. An admin user has been created and a password generated. Please use the following password to proceed to installation: 952de945c78e4a59a702e108886433c2 This may also be found at: /var/jenkins_home/secrets/initialAdminPassword ************************************************************* ************************************************************* ************************************************************* 2020-07-05 15:52:47.380+0000 [id=33] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization 2020-07-05 15:52:47.517+0000 [id=20] INFO hudson.WebAppMain$3#run: Jenkins is fully up and running 2020-07-05 15:52:48.350+0000 [id=46] INFO h.m.DownloadService$Downloadable#load: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller 2020-07-05 15:52:48.351+0000 [id=46] INFO hudson.util.Retrier#start: Performed the action check updates server successfully at the attempt #1 2020-07-05 15:52:48.385+0000 [id=46] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Download metadata. 24,859 ms
Great Jenkins is running, but we have multiple problems. Firstly, this container took hold of the console and is printing logs on the console. If you don’t want that to happen, let’s stop the container and start it again with a different option. Using a different terminal stop the container.
$ docker stop 41b92eab4e19 41b92eab4e19 $ docker run -d jenkins/jenkins b01c65b1f5d7720fcabb99819490ff8da2ce6000b738eb108eb5b343037bb58d $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b01c65b1f5d7 jenkins/jenkins "/sbin/tini -- /usr/…" 3 minutes ago Up 2 minutes 8080/tcp, 50000/tcp objective_gagarin
The “-d” option with docker run makes the container run in background and prints the container id. Here d stands for detach. So that takes care of our first problem. Here comes the second problem: Jenkins is a web server and it needs to listen to some port for incoming connections. It used port 8080 for HTTP traffic and port 50000 is used by JNLP-based Jenkins agents to communicate with the Jenkins master. That’s why in the output of “docker ps” command you can see “8080/tcp, 50000/tcp” under PORT. These ports are only available from inside docker, we cannot access it from outside. Without making them available outside docker we cannot establish communication with this Jenkins server. Below is how we solve this problem.
$ docker stop b01c65b1f5d7 b01c65b1f5d7 $ docker run -d -p 8181:8080 jenkins/jenkins 2a8f7ae552deca1b2114d047938a7446149a8d66917aeb49875581efbbc27d56 $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 2a8f7ae552de jenkins/jenkins "/sbin/tini -- /usr/…" 3 minutes ago Up 2 minutes 50000/tcp, 0.0.0.0:8181->8080/tcp zen_ptolemy
The the “-p” option in “docker run” command does a port mapping, it publishes the container’s port(s) to the host. You can see how 0.0.0.0:8181 on host/your machine is mapped with containers 8080 port. Even though port 50000 used by Jenkins agents are not that important at this moment but if you want to expose them to your host you can use an additional “-p 50000:50000” in the above docker run command. Now, to prove that it is working, open a new tab in your browser, paste 0.0.0.0:8181 in the address bar, and hit enter. If you see a login form that says “Unlock Jenkins” then we have successfully established a connection with our Jenkins container.
But wait, the page says I need to enter “Administrator password” and it is available in file /var/jenkins_home/secrets/initialAdminPassword. If you try to access the file you get an error.
$ cat /var/jenkins_home/secrets/initialAdminPassword cat: /var/jenkins_home/secrets/initialAdminPassword: No such file or directory
This is because the file is not in our system, but inside the container. How do we see its content?
$ docker exec zen_ptolemy cat /var/jenkins_home/secrets/initialAdminPassword 99976c28397e4d9888d24fc20c01587b
“docker exec” command can execute a command inside the container. As a parameter, it took the name of the container, and then the command it needs to execute. Okay, now that we have the password lets enter it in the login page. Once logged in, you will be given the option of selecting the plugins or go for the recommended plugins. Say you install plugins by selecting whichever option, where did these plugins got installed? Hmm, they are inside the container. If we stop the container all these plugins will be lost. Not only the plugins but any change you did to the system will be lost. This is our next problem.
The solution to this problem is to give a folder from the host machine to the container; meaning we will create a folder of our choice and tell the container that it can use the folder to store any change it is doing while running the container. For example, if you are running a database using docker, all the data that you stored and used will be lost once the container is stopped. To persist the data we share our disk with the container and tells it to store all changes there. So that when the container is started again after a system shutdown, stop or crash it will not lose any data. We do this using -v option of docker run. Here v stands for volume (it bind mount a volume). Additionally, we will have to let it know the user with which it should access the filesystem.
$ docker stop zen_ptolemy zen_ptolemy $ mkdir /home/sidd/dev/docker/jenkinsData $ docker run -d -p 8181:8080 -v /home/sidd/dev/docker/jenkinsData:/var/jenkins-home -u root jenkins/jenkins dcbc64645344567b29597cd59bc6bf63f8d1988564f1a044856b08bd305d4474 $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES dcbc64645344 jenkins/jenkins "/sbin/tini -- /usr/…" 3 minutes ago Up 2 minutes 50000/tcp, 0.0.0.0:8181->8080/tcp interesting_clarke $ docker exec dcbc64645344 cat /var/jenkins_home/secrets/initialAdminPassword 271c2bb32cce4c38b03ab72fb39e5caf
Again log into http://0.0.0.0:8181 use administrator password (note it changes with every container run) and install the recommended plugins. Now you will see files getting created in folder /home/sidd/dev/docker/jenkinsData
Now you have a Jenkins server running on your system. Congratulations!! Also we have covered the basics of using a docker container.