Five Proven Ways to Debug a Container

As you might already know, containers create an isolated and secure space where you can install applications and their dependencies. You can store the containers as image files either locally or in remote registries so that services can download and run them on demand. The primary benefits of containers are that they provide consistency, the ability to isolate applications, and improved operational agility.

However, you may encounter some significant issues when you try to package your application into containers. These issues usually stem from the fact that your application wasn’t originally designed for a container environment, or else Docker is not creating the right environment for your application.

If your application doesn’t work at all when you run it inside a container, you’ll need to dig deeper to find the root cause. This article will show you where to start digging and how to get to the bottom of it. To help you solve your containerization issues, we’ll explain five proven ways to debug a container.

1. Understand What’s Inside the Container

Regardless of whether you have access to your Dockerfiles or Buildah files, you need to understand what’s in them. This means that you need to understand the steps involved in building the container image, and you need to know how many layers it creates, which libraries it installs, and what the entrypoint is.

To help you get started, you can run the docker inspect command, which will return information about a container or an image. This can be useful for illuminating the current state of the container and seeing which arguments have been passed to a command.

To get a better understanding, you can use a tool called DockerSlim, which enables you to analyze an existing image and print a detailed summary of these steps. You can invoke it into any image you have, whether it’s stored locally or in a registry.

For example, this is what the analysis of the official PHP Composer image looks like:

docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock dslim/docker-slim xray composer > report

As you can see, we used the xray command and provided the name of the composer image. This will generate a report and store it in a file named report. If you review the analysis, you can examine the impact of each instruction in detail. For example, consider the following instruction obtained from the Dockerfile:

RUN printf "# composer php cli ini settings\n\
date.timezone=UTC\n\
memory_limit=-1\n\
" > $PHP_INI_DIR/php-cli.ini

ENV COMPOSER_ALLOW_SUPERUSER 1
ENV COMPOSER_HOME /tmp
ENV COMPOSER_VERSION 2.1.8

The above corresponds to Layer 12 in the XRay report:

cmd=xray info=layer.start
cmd=xray info=layer index='12' id='5da02dd8c16f242916bd010d5704fe9ee1b4067021b803f0fa6a952dbfe5f5d7' path='5da02dd8c16f242916bd010d5704fe9ee1b4067021b803f0fa6a952dbfe5f5d7/layer.tar'
cmd=xray info=change.instruction index='0:24' type='RUN' snippet='RUN printf "# composer php cli ini settings\\...' all='RUN printf "# composer php cli ini settings\\ndate.timezone=UTC\\nmemory_limit=-1\\n" > $PHP_INI_DIR/php-cli.ini'
cmd=xray info=other.instructions count='3'
cmd=xray info=other.instruction snippet='ENV COMPOSER_ALLOW_SUPERUSER=1' all='ENV COMPOSER_ALLOW_SUPERUSER=1' pos='0' index='0:25' type='ENV'
cmd=xray info=other.instruction pos='1' index='0:26' type='ENV' snippet='ENV COMPOSER_HOME=/tmp' all='ENV COMPOSER_HOME=/tmp'
cmd=xray info=other.instruction index='0:27' type='ENV' snippet='ENV COMPOSER_VERSION=2.1.6' all='ENV COMPOSER_VERSION=2.1.6' pos='2'
cmd=xray info=layer.stats all_size.human='66 B' all_size.bytes='66'
cmd=xray info=layer.stats object_count='5'
cmd=xray info=layer.stats dir_count='4'
cmd=xray info=layer.stats file_count='1'
cmd=xray info=layer.stats max_file_size.human='66 B' max_file_size.bytes='66'
cmd=xray info=layer.stats added_size.human='66 B' added_size.bytes='66'
cmd=xray info=layer.change.summary added='1' all='5' deleted='0' modified='4'
cmd=xray info=layer.objects.count value='5'
cmd=xray info=layer.objects.top.start
A: mode=-rw-r--r-- size.human='66 B' size.bytes=66 uid=0 gid=0 mtime='2021-08-28T02:27:44Z' H=\[A:12\] '/usr/local/etc/php/php-cli.ini'
M: mode=drwxr-xr-x size.human='0 B' size.bytes=0 uid=0 gid=0 mtime='2021-08-27T21:33:29Z' H=\[A:3/M:6,8,11,12\] '/usr/local/etc'
M: mode=drwxr-xr-x size.human='0 B' size.bytes=0 uid=0 gid=0 mtime='2021-08-28T02:27:44Z' H=\[A:3/M:6,8,11,12\] '/usr/local/etc/php'
M: mode=drwxr-xr-x size.human='0 B' size.bytes=0 uid=0 gid=0 mtime='2021-08-27T21:33:25Z' H=\[A:0/M:1,3,5,6,7,8,10,11,12\] '/usr/local'
M: mode=drwxr-xr-x size.human='0 B' size.bytes=0 uid=0 gid=0 mtime='2021-08-28T02:27:43Z' H=\[A:0/M:1,3,4,5,6,7,8,9,10,11,12,13\] '/usr'
cmd=xray info=layer.objects.top.end
cmd=xray info=layer.end

As you can see, this shows the impact of each command at image build time as well as the environment variables that were provided. This is the first crucial step toward understanding how to debug the application that the container facilitates. You can experiment by running the xray command for your own images.

2. Fixing Application Start and Stop Failures

Now, let’s say that you have a clear picture of what’s inside a container image. You start your container, but sadly, you get an error. You might get an error when starting the application or find that your application’s health is not very good. For example, you might start your container only to see it stop with the status “exited.”

There are several possible reasons why your application won’t start inside the container. One of the most obvious is that it’s using the wrong ENTRYPOINT and CMD instructions.

The ENTRYPOINT step will be the starting command of the container image when it’s run by the daemon. It configures a container that will run as an executable. You can pass a list of parameters after the executable in the shell form like this:

["executable", "param1", "param2", ...]

For example, this Dockerfile

# Dockerfile
FROM ubuntu:latest
ENTRYPOINT [ "echo", "hello"]

would yield the following:

$ docker run -it --rm test world
hello world

The CMD instruction allows for a default command that will be executed only when you run a container without specifying anything else. CMD is ignored when you provide a different command. If ENTRYPOINT and CMD are both included, the ENTRYPOINT command will always be used by default, but you will have the option to override the CMD parameters.

If you try to substitute env variables in ENTRYPOINT’s exec form, it will pass them to the application without changing them:

# Dockerfile
FROM ubuntu:latest
ENTRYPOINT [ "echo", "$HOME" ]

$ docker run -it --rm test
$HOME

If you want to replace them with actual variables, you’ll need to either use the shell form to properly expand the variables or invoke the shell directly:

# Dockerfile
ENTRYPOINT [ "sh", "-c", "echo $HOME" ]

$ docker run -it --rm test
/root

You can also override the entrypoint of the container at this time and try to simulate the initial steps one by one:

$ docker run -d -it --rm --entrypoint /bin/sh test
44aa9eb3ad57994f5f1606f82e83dc64b6299f2a7fb755ec422c09dfc44b124a

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
44aa9eb3ad57 test "/bin/sh" 36 seconds ago Up 35 seconds serene_colden

$ docker attach 44aa9eb3ad57
echo "hello"
hello

Some of the tools that package applications inside Docker images only use the exec form, and you may find that they don’t always expand those arguments properly. If you aren’t sure about the arguments that your application is receiving, you can use the above Dockerfile to quickly debug it.

Naturally, the application might not start correctly if you misspell something or provide the wrong arguments. When this happens, the application usually logs an error in the console, which you will be able to inspect. We’ll show you some ways to do that next.

3. Debugging Container Logs and Events

The standard way to debug containers is to inspect their logs. If you know the container id or name, you can follow the log trail by using the following command:

docker logs -f <container_id>

For example, we can simulate a long running process like this:

$ docker run -d --name=test-long-running ubuntu /bin/sh -c "while true; do sleep 2;date; done"
dc9bbba1936e03f2496b1ed354bc9a3132de3a6c0ade109570ac631dccc7d177

Then, we inspect the logs:

$ docker logs -f dc9bbba1936e03f2496b1ed354bc9a3132de3a6c0ade109570ac631dccc7d177
Fri Sep 23 09:27:51 UTC 2021
Fri Sep 23 09:27:53 UTC 2021
…

Sometimes, however, the container will stop so quickly that you don’t have time to inspect these logs. In that case, you can find the list of “exited” containers and use the id to inspect their logs:

$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
894a1e790efc ubuntu "/bin/sh -c 'while t…" 2 minutes ago Exited (137) 37 seconds ago test-long-running

$ docker logs 894a1e790efc

In addition, you can use Docker events to inspect all of the events that happened inside the Docker server in parallel:

$ docker events &

2021-09-24T10:30:08.665267774+01:00 **container kill** dc9bbba1936e03f2496b1ed354bc9a3132de3a6c0ade109570ac631dccc7d177 (image=ubuntu, name=test-long-running, signal=9)
2021-09-24T10:30:08.686442560+01:00 **container die** dc9bbba1936e03f2496b1ed354bc9a3132de3a6c0ade109570ac631dccc7d177 (exitCode=137, image=ubuntu, name=test-long-running)
2021-09-24T10:30:08.732703578+01:00 **network disconnect** 85c5dfd30ecfd9ecb9dc57da818aafef2a81b14c8211b9f2856dca6adf6fa2c6 (container=dc9bbba1936e03f2496b1ed354bc9a3132de3a6c0ade109570ac631dccc7d177, name=bridge, type=bridge)
2021-09-24T10:30:08.740324632+01:00 **container stop** dc9bbba1936e03f2496b1ed354bc9a3132de3a6c0ade109570ac631dccc7d177 (image=ubuntu, name=test-long-running)
2021-09-24T10:30:23.292551625+01:00 **container destroy** dc9bbba1936e03f2496b1ed354bc9a3132de3a6c0ade109570ac631dccc7d177 (image=ubuntu, name=test-long-running)

According to the official docs, containers emit a variety of events, so you’ll be able to get an idea of what kind of event stopped the container.

4. Fixing Permission Errors

Many times, a container will start correctly but fail because the application triggers a file permission error. This means that either something is configured incorrectly or the permissions are wrong. This happens when you share volumes between a user and a host with different permissions.

When you get file permission errors, you should follow the standard safe approach to building Dockerfiles by adding a user and a group to the image. For example, you could add these steps to an Ubuntu image:

FROM ubuntu:latest
ARG USER_ID
ARG GROUP_ID
RUN bash -c 'if [ ${ostype} == Linux ]; then groupadd -r --gid $GROUP_ID user; fi'
RUN adduser --disabled-password --gecos '' --uid $USER_ID --gid $GROUP_ID user
USER user
ENTRYPOINT [ "echo" ]

Then, you will need to build the image by providing the host’s USER_ID and GROUP_ID arguments:

docker build -t test --build-arg USER_ID=$(id -u) --build-arg GROUP_ID=$(id -g) .

That will take care of most of the file permission errors you might encounter.

5. Attaching to Paused Container

Containers can also be paused by using the docker pause command. This will allow you to attach into the container without worrying about adjourning your debugging sessions, since all of the processes inside the container will be paused. Then, you can unpause the container and observe the STDOUT.

You can simulate the creation of a long running container by running the sleep command inside a container every now and then, like this:

$ docker run -d --name=test-long-running ubuntu /bin/sh -c "while true; do sleep 2;date; done"
71b606c5cc3d112647c9d19136dbd0dc7267d21fea4c6c370dd02d1fdfc32acc

You need to attach before you pause the container, because you cannot do it after it’s paused:

❯ docker attach 71b606c5cc3d112647c9d19136dbd0dc7267d21fea4c6c370dd02d1fdfc32acc

You can pause the container on a different tab like this:

❯ docker pause 71b606c5cc3d112647c9d19136dbd0dc7267d21fea4c6c370dd02d1fdfc32acc

Then, verify that it’s paused:

❯ docker inspect test-long-running
    [{
     "Id": "71b606c5cc3d112647c9d19136dbd0dc7267d21fea4c6c370dd02d1fdfc32acc",
     "Created": "2021-09-23T12:57:40.5913689Z",
     "Path": "/bin/sh",
     "Args": [
      "-c",
      "while true; do sleep 2; done"
         ],
     "State": {
      "Status": "paused",

Now, you can unpause the container to resume the process:

docker unpause 71b606c5cc3d112647c9d19136dbd0dc7267d21fea4c6c370dd02d1fdfc32acc

From here, you could review the output that the application sends to the STDIN/OUT/ERR, and you could also send OS signals to simulate a crash.

Next Steps with Debugging and Optimizing Containers

In this article, we explained several ways to debug containers. It’s a good starting point, but you will probably need something more detailed if your container still doesn’t work as expected. In that case, you’ll likely want a complete, developer-friendly solution for analyzing, visualizing, and optimizing containers.

This is where Slim.AI can help you. They offer purpose-built developer tools that help you create production-ready containers and optimized images.