Docker overview

Docker is becoming a very popular in virtualization world.
What is Docker? It is an open-source project that automates the deployment of applications inside linux container.
What is Linux container? It provides “operation-system level virtualization” on Linux, i.e. it provides a way for user to create a userspace enviroment(root file system) on top of underlying host linux kernel.
What is needed from Linux kernel? In general, it requires kernel 3.10+, with namespace, cgroup, plus libcontainer.

The following diagram shows the overall picture of Docker.

Docker Overview](/uploads/docker/docker-overview.jpg)

In this post, I will share what I learned about Docker.

Docker main components

  • Host: the machine that is running the containers.
  • Image: a hierarchy of files, with meta-data for how to run a container.
  • Container: a contained running process, started from an image.
  • Registry: a repository of images.
  • Volume: storage outside the container.
  • Dockerfile: a script for creating images.

Docker installation

There a a few Linux distributions supports docker, such as ubuntu, centos, Redhat, Fedora, coreos.

Let’s take ubuntu 14.04 desktop system which runs kernel 3.16 as example.

root@weng-VirtualBox:/home/weng# uname -a
Linux weng-VirtualBox 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 17:43:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
root@weng-VirtualBox:/home/weng# cat /etc/os-release 
NAME="Ubuntu"
VERSION="14.04.2 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.2 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
root@weng-VirtualBox:/home/weng# 

In ubuntu, typically we can use apt-get to install application. However for Docker, the situation is a bit different. Because Docker is under actively development, we want to install pretty new version, as we can see below, the Docker version in ubuntu 14.04 repository, it is a bit too old: version 1.0.1

root@weng-VirtualBox:/home/weng# apt-cache search docker
pidgin - graphical multi-protocol instant messaging client for X
docker - System tray for KDE3/GNOME2 docklet applications
docker.io - Linux container runtime
karbon - vector graphics application for the Calligra Suite
kdocker - lets you dock any application into the system tray
vim-syntax-docker - Docker container engine - Vim highlighting syntax files
root@weng-VirtualBox:/home/weng# apt-cache policy docker.io
docker.io:
  Installed: (none)
  Candidate: 1.0.1~dfsg1-0ubuntu1~ubuntu0.14.04.1
  Version table:
     1.0.1~dfsg1-0ubuntu1~ubuntu0.14.04.1 0
        500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe amd64 Packages
     0.9.1~dfsg1-2 0
        500 http://us.archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages
root@weng-VirtualBox:/home/weng# 

The following shows how to configure to install the latest available Docker:

root@weng-VirtualBox:/home/weng# apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.XDMDaqn5G2 --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.
gpg --primary-keyring /etc/apt/trusted.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9 
gpg: requesting key A88D21E9 from hkp server keyserver.ubuntu.com gpg: key A88D21E9: public key "Docker Release Tool (releasedocker) 
<docker@dotcloud.com>" imported 
gpg: Total number processed: 1 
gpg: imported: 1 (RSA: 1)
root@weng-VirtualBox:/home/weng# sh -c "echo deb https://get.docker.com/ubuntu docker main > /etc/apt/sources.list.d/docker.list"
root@weng-VirtualBox:/home/weng# apt-get update
....
root@weng-VirtualBox:/home/weng# apt-get install lxc-docker

After installation, we can verify the status of Docker:

root@weng-VirtualBox:/home/weng# docker --version
Docker version 1.6.2, build 7c8fca2
root@weng-VirtualBox:/home/weng# docker info
Containers: 17
Images: 39
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 73
 Dirperm1 Supported: true
Execution Driver: native-0.2
Kernel Version: 3.16.0-30-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 1
Total Memory: 1.955 GiB
Name: weng-VirtualBox
ID: 2GAF:7I7A:E467:UWN2:6GDS:EXNL:QDOX:ZORL:I6RR:JSXV:2Q3K:54QR
WARNING: No swap limit support
root@weng-VirtualBox:/home/weng# 

For system like Centos, to install latest, the following instructions are to be run to install binary directly:

# To install, run the following command as root:
curl -sSL -O https://get.docker.com/builds/Linux/x86_64/docker-1.6.2 && chmod +x docker-1.6.2 && sudo mv docker-1.6.2 /usr/local/bin/docker
# Then start docker in daemon mode:
sudo /usr/local/bin/docker -d

For more information about installation of docker, check [here] (https://docs.docker.com/installation/)

How Does Docker Work?

Docker is implemented as a client-server mode; The Docker daemon runs on the Host and it is accessed via a socket connection from the client. The client may, but does not have to, be on the same machine as the daemon. The Docker CLI client works the same way as any other client but it is usually connected through a Unix domain socket instead of a TCP socket. The daemon receives commands from the client and manages the containers on the Host where it is running.

root@weng-VirtualBox:/home/weng# ps -ef | grep docker
root      1311     1  0 22:48 ?        00:00:00 /usr/bin/docker -d
root      2773  2311  0 23:04 pts/1    00:00:00 grep --color=auto docker
root@weng-VirtualBox:/home/weng# which docker
/usr/bin/docker
root@weng-VirtualBox:/home/weng# 

In above, it show Docker daemon runs as regular linux host process 1311.
“/usr/bin/docker” is to be invoked as Docker client to send commands to Docker daemon.

When Docker daemon runs, it uses /etc/default/docker as a configuration file:

root@weng-VirtualBox:/home/weng# cat /etc/default/docker 
# Docker Upstart and SysVinit configuration file

# Customize location of Docker binary (especially for development testing).
#DOCKER="/usr/local/bin/docker"

# Use DOCKER_OPTS to modify the daemon startup options.
#DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"

# If you need Docker to use an HTTP proxy, it can also be specified here.
#export http_proxy="http://127.0.0.1:3128/"

# This is also a handy place to tweak where Docker's temporary files go.
#export TMPDIR="/mnt/bigdrive/docker-tmp"
root@weng-VirtualBox:/home/weng# 

Run Docker container

The following are commands to for client to interact with containers:

# Commands for interacting with containers
$ docker create  # creates a container but does not start it.
$ docker run     # creates and starts a container.
$ docker stop    # stops it.
$ docker start   # will start it again.
$ docker restart # restarts a container.
$ docker rm      # deletes a container.
$ docker kill    # sends a SIGKILL to a container.
$ docker attach  # will connect to a running container.
$ docker wait    # blocks until container stops.
$ docker exec    # executes a command in a running container.

For more details, use “docker help”.

Below is an example of starting a container using Fedora image:

root@weng-VirtualBox:/home/weng# docker run -it fedora /bin/bash
Unable to find image 'fedora:latest' locally
latest: Pulling from fedora
48ecf305d2cf: Pull complete 
ded7cd95e059: Already exists 
fedora:latest: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Digest: sha256:10ba981a70632d7764c21deae25c6521db6d39730e1dd8caff90719013858a7b
Status: Downloaded newer image for fedora:latest
[root@c3cfa2c8ba3f /]# cat /etc/os-release 
NAME=Fedora
VERSION="22 (Twenty Two)"
ID=fedora
VERSION_ID=22
PRETTY_NAME="Fedora 22 (Twenty Two)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:fedoraproject:fedora:22"
HOME_URL="https://fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=22
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=22
PRIVACY_POLICY_URL=https://fedoraproject.org/wiki/Legal:PrivacyPolicy
[root@c3cfa2c8ba3f /]# uname -a
Linux c3cfa2c8ba3f 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 17:43:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@c3cfa2c8ba3f /]# ls /dev/
console  full  kcore   null  pts     shm     stdin   tty      zero
fd       fuse  mqueue  ptmx  random  stderr  stdout  urandom
[root@c3cfa2c8ba3f /]# 

Here there are a few points to make:

  1. Command "docker run -it fedora /bin/bash" is invoked by user so that docker client send requests to docker daemon through UNIX socket to start a conatiner using fedora docker image, then run /bin/bash command.
  2. In the output of first line "Unable to find image 'fedora:latest' locally", it means no fedora image available, then it goes to default docker registry to fetch fedora docker image, nice!
  3. After fedora image is pulled and verified, container is started as instance ID c3cfa2c8ba3f, and runs command /bin/bash.
  4. cat /etc/os-release does tell that container is running Fedora image.
  5. "uname -a" tells it runs on top of kernel 3.16.0-30-generic from ubuntu build, which is host kernel.
  6. "ls /dev" shows very limted number of device files, because these are only devices available to container by default. Additional device can be exposed to container by using "--device" option.

From another xterm, we can verify the container running:

root@weng-VirtualBox:/home/weng# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
c3cfa2c8ba3f        fedora:latest       "/bin/bash"         23 minutes ago      Up 23 minutes                           stupefied_perlman   
root@weng-VirtualBox:/home/weng# docker images 
REPOSITORY                   TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
dockerlab/webserver          0.2.0               371419fd7fc5        9 hours ago         180.1 MB
localhost:5000/devenv-weng   latest              430f440ac0e0        12 hours ago        226 MB
wengcentos/webserver         version-0.1         5db16a40e56e        30 hours ago        295 MB
ubuntu                       14.04               fa81ed084842        9 days ago          188.3 MB
ubuntu                       latest              fa81ed084842        9 days ago          188.3 MB
ubuntu                       12.04               5c97af892079        9 days ago          133.4 MB
fedora                       latest              ded7cd95e059        13 days ago         186.5 MB
registry                     latest              d849e35be7b0        2 weeks ago         413.9 MB
cirros                       latest              d8de71c04044        7 weeks ago         7.698 MB
centos                       latest              fd44297e2ddb        7 weeks ago         215.7 MB
root@weng-VirtualBox:/home/weng# 

It is interesting to note that even after exiting from container, the container is still there, meaning the configuration/meta-data about the conatiner is still in the host, unless using “docker rm $containerID” to remove it. See example below “docker ps -a” shows all existing containers, even thoug only one container is active.

root@weng-VirtualBox:/home/weng# docker ps -a
CONTAINER ID        IMAGE                                                                     COMMAND                CREATED             STATUS                         PORTS                    NAMES
c3cfa2c8ba3f        fedora:latest                                                             "/bin/bash"            37 minutes ago      Up 37 minutes                                           stupefied_perlman   
cbf8cb80fb88        ubuntu:latest                                                             "/bin/bash"            39 minutes ago      Exited (0) 39 minutes ago                               stupefied_fermi     
0f28e4e66ffb        ubuntu:latest                                                             "/bin/bash"            7 hours ago         Exited (0) About an hour ago                            fervent_hypatia     
736872d3afef        ubuntu:latest                                                             "/bin/bash"            7 hours ago         Exited (0) 7 hours ago                                  angry_lalande       
125e90b41314        ubuntu:latest                                                             "/bin/bash"            7 hours ago                                                                 agitated_bell       
7a06a0f8d2cb        ubuntu:latest                                                             "/bin/bash"            7 hours ago         Exited (0) 7 hours ago                                  berserk_babbage     
6d1dfd58d2e7        dockerlab/webserver:0.2.0                                                 "/usr/sbin/apache2 -   9 hours ago         Exited (0) 8 hours ago         0.0.0.0:8989->80/tcp     hopeful_galileo     
17648f98b2eb        566a8b1441e5f5224476ee76d83e944b50ce714a246489e76a76ab851bc6f40f:latest   "/bin/sh -c 'apt-get   9 hours ago         Exited (1) 9 hours ago                                  serene_pike         
28769fdee95d        ubuntu:latest                                                             "/bin/bash"            10 hours ago        Exited (0) 9 hours ago                                  sharp_banach        
7c32935fbec1        ubuntu:latest                                                             "/bin/bash"            12 hours ago        Exited (0) 12 hours ago                                 reverent_jones      
0a27428cedda        registry:latest                                                           "docker-registry"      12 hours ago        Exited (0) 8 hours ago         0.0.0.0:5000->5000/tcp   adoring_einstein    
140c03c2aea3        wengcentos/webserver:version-0.1                                          "/bin/bash"            31 hours ago        Exited (0) 8 hours ago                                  modest_galileo      
58a5029f6c3b        ubuntu:latest                                                             "/bin/bash"            33 hours ago        Exited (0) 12 hours ago                                 reverent_stallman   
dd07b42d33ae        ubuntu:latest                                                             "/bin/bash"            33 hours ago        Exited (0) 33 hours ago                                 trusting_curie      
b22f1cd231f4        centos:latest                                                             "/bin/bash"            33 hours ago        Exited (1) 31 hours ago                                 modest_engelbart    
30b927bd5b2c        cirros:latest                                                             "/bin/sh"              33 hours ago        Exited (0) 8 hours ago                                  stupefied_morse     
cce87dd65fa2        cirros:latest                                                             "/bin/sh"              33 hours ago        Exited (0) 33 hours ago                                 boring_lovelace     
acf99358d8c6        cirros:latest                                                             "/bin/bash"            33 hours ago                                                                clever_babbage      
root@weng-VirtualBox:/home/weng# 
root@weng-VirtualBox:/home/weng#  docker rm acf99358d8c6
acf99358d8c6
root@weng-VirtualBox:/home/weng# docker ps -a
CONTAINER ID        IMAGE                                                                     COMMAND                CREATED             STATUS                         PORTS                    NAMES
c3cfa2c8ba3f        fedora:latest                                                             "/bin/bash"            38 minutes ago      Up 38 minutes                                           stupefied_perlman   
cbf8cb80fb88        ubuntu:latest                                                             "/bin/bash"            40 minutes ago      Exited (0) 40 minutes ago                               stupefied_fermi     
0f28e4e66ffb        ubuntu:latest                                                             "/bin/bash"            7 hours ago         Exited (0) About an hour ago                            fervent_hypatia     
736872d3afef        ubuntu:latest                                                             "/bin/bash"            7 hours ago         Exited (0) 7 hours ago                                  angry_lalande       
125e90b41314        ubuntu:latest                                                             "/bin/bash"            7 hours ago                                                                 agitated_bell       
7a06a0f8d2cb        ubuntu:latest                                                             "/bin/bash"            7 hours ago         Exited (0) 7 hours ago                                  berserk_babbage     
6d1dfd58d2e7        dockerlab/webserver:0.2.0                                                 "/usr/sbin/apache2 -   9 hours ago         Exited (0) 8 hours ago         0.0.0.0:8989->80/tcp     hopeful_galileo     
17648f98b2eb        566a8b1441e5f5224476ee76d83e944b50ce714a246489e76a76ab851bc6f40f:latest   "/bin/sh -c 'apt-get   9 hours ago         Exited (1) 9 hours ago                                  serene_pike         
28769fdee95d        ubuntu:latest                                                             "/bin/bash"            10 hours ago        Exited (0) 9 hours ago                                  sharp_banach        
7c32935fbec1        ubuntu:latest                                                             "/bin/bash"            12 hours ago        Exited (0) 12 hours ago                                 reverent_jones      
0a27428cedda        registry:latest                                                           "docker-registry"      12 hours ago        Exited (0) 8 hours ago         0.0.0.0:5000->5000/tcp   adoring_einstein    
140c03c2aea3        wengcentos/webserver:version-0.1                                          "/bin/bash"            31 hours ago        Exited (0) 8 hours ago                                  modest_galileo      
58a5029f6c3b        ubuntu:latest                                                             "/bin/bash"            33 hours ago        Exited (0) 12 hours ago                                 reverent_stallman   
dd07b42d33ae        ubuntu:latest                                                             "/bin/bash"            33 hours ago        Exited (0) 33 hours ago                                 trusting_curie      
b22f1cd231f4        centos:latest                                                             "/bin/bash"            33 hours ago        Exited (1) 31 hours ago                                 modest_engelbart    
30b927bd5b2c        cirros:latest                                                             "/bin/sh"              33 hours ago        Exited (0) 8 hours ago                                  stupefied_morse     
cce87dd65fa2        cirros:latest                                                             "/bin/sh"              33 hours ago        Exited (0) 33 hours ago                                 boring_lovelace     
root@weng-VirtualBox:/home/weng# 

Docker image

One of most amazing features of Docker to me is the Docker image management. It is done in an excellent fashion.
Docker image file system](/uploads/docker/docker-filesystems-multilayer.png)

  • Docker containers are constructed by sequentially mounting a set of file systems from one or more images.
  • A Docker image is a file system layer with an optional parent image reference.
    • Layered images tend to supply one specific feature on top of parent image
    • Upper layer files mask files at lower layers with the same pathname
  • An image with no parent image is called a Base image, e.g. ubuntu, centos, Debian, fedora, cirros
  • Image file systems are immutable within container.
    • Allow one image to support multiple container instances with repeatable results
    • Reduces the disk and memory footprint of a given set of containers which share the same read only images
  • Conatiners have a writable file system
    • The container file system is initially empty
    • All writes go to this file system and overlay any matching underlying image files
    • In this way, container file systems contain only the delta between their file system state and that of their underlying images.
    • The view from the top down, including all file systems in the stack, is called the union file system.
  • Docker bootfs is a special layer.
    • Supplies the in memory file system interface to the kernel
    • Supplies the kernel library interface

Docker filesystem need backend support.

  • Docker depends on the efficient use of layered images. Various file system features in kernel implement file system layering.
  • Each container has two layers: 1. init layer : child of specified image; 2. child of init layer: container specific data.
  • Committing a container creates a new image layer based on the image the container was created from.
  • Docker backends file system supported (in preference order)
    • aufs: Advanced multi layered Unification FileSystem.
    • btrfs
    • devicemapper: LVM
    • vfs: no shared storage
    • OverlayFS
  • Container independant Volumes:
    • write-heavy IO (database, logs)
    • host volumes can be accessed across multiple concurrent containers

Docker Repositories & Registries

Docker repo reg](/uploads/docker/docker-repo-reg.png)

  • A repository is a hosted collection of tagged images that together create the file system for a container.
  • A registry is a service that stores repositories and provides an HTTP API for managing the upload and downloading of repositories.
    • Default registry used by docker engine is Docker hub public registry: http://hub.docker.com
    • Companies can deploy private registries using open source solution like Docker registry from Docker Inc, Artifactory.

Dockerfile

Dockerfile is just like a shell script, and it is understood by docker, can build image by parsing Dockefile.

Dockerfile image](/uploads/docker/docker-use-dockerfile.jpg)

Below is an example of Dockerfile

Dockerfile example](/uploads/docker/dockerfile-example.jpg)