Five Proven Ways to Debug a Container

When Things Just Are Not Working
Theofanis Despoudis
Oct 21, 2021

Photo by Steve Johnson on Unsplash

As you might already know, containers create an isolated and secure space where you can install applications and their dependencies. You can store the containers as image files either locally or in remote registries so that services can download and run them on demand. The primary benefits of containers are that they provide consistency, the ability to isolate applications, and improved operational agility.

However, you may encounter some significant issues when you try to package your application into containers. These issues usually stem from the fact that your application wasn’t originally designed for a container environment, or else Docker is not creating the right environment for your application.

If your application doesn’t work at all when you run it inside a container, you’ll need to dig deeper to find the root cause. This article will show you where to start digging and how to get to the bottom of it. To help you solve your containerization issues, we’ll explain five proven ways to debug a container.

1. Understand What’s Inside the Container

Regardless of whether you have access to your Dockerfiles or Buildah files, you need to understand what’s in them. This means that you need to understand the steps involved in building the container image, and you need to know how many layers it creates, which libraries it installs, and what the entrypoint is.

To help you get started, you can run the docker inspect command, which will return information about a container or an image. This can be useful for illuminating the current state of the container and seeing which arguments have been passed to a command.

To get a better understanding, you can use a tool called DockerSlim, which enables you to analyze an existing image and print a detailed summary of these steps. You can invoke it into any image you have, whether it’s stored locally or in a registry.

For example, this is what the analysis of the official PHP Composer image looks like:

$ docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock dslim/docker-slim xray composer > report

As you can see, we used the xray command and provided the name of the composer image. This will generate a report and store it in a file named report. If you review the analysis, you can examine the impact of each instruction in detail. For example, consider the following instruction obtained from the Dockerfile:

RUN printf "# composer php cli ini settings\n\
date.timezone=UTC\n\
memory_limit=-1\n\
" > $PHP_INI_DIR/php-cli.ini

ENV COMPOSER_ALLOW_SUPERUSER 1
ENV COMPOSER_HOME /tmp
ENV COMPOSER_VERSION 2.1.8

The above corresponds to Layer 12 in the XRay report:

cmd=xray info=layer.start
cmd=xray info=layer index='12' id='5da02dd8c16f242916bd010d5704fe9ee1b4067021b803f0fa6a952dbfe5f5d7' path='5da02dd8c16f242916bd010d5704fe9ee1b4067021b803f0fa6a952dbfe5f5d7/layer.tar'
cmd=xray info=change.instruction index='0:24' type='RUN' snippet='RUN printf "# composer php cli ini settings\\...' all='RUN printf "# composer php cli ini settings\\ndate.timezone=UTC\\nmemory_limit=-1\\n" > $PHP_INI_DIR/php-cli.ini'
cmd=xray info=other.instructions count='3'
cmd=xray info=other.instruction snippet='ENV COMPOSER_ALLOW_SUPERUSER=1' all='ENV COMPOSER_ALLOW_SUPERUSER=1' pos='0' index='0:25' type='ENV'
cmd=xray info=other.instruction pos='1' index='0:26' type='ENV' snippet='ENV COMPOSER_HOME=/tmp' all='ENV COMPOSER_HOME=/tmp'
cmd=xray info=other.instruction index='0:27' type='ENV' snippet='ENV COMPOSER_VERSION=2.1.6' all='ENV COMPOSER_VERSION=2.1.6' pos='2'
cmd=xray info=layer.stats all_size.human='66 B' all_size.bytes='66'
cmd=xray info=layer.stats object_count='5'
cmd=xray info=layer.stats dir_count='4'
cmd=xray info=layer.stats file_count='1'
cmd=xray info=layer.stats max_file_size.human='66 B' max_file_size.bytes='66'
cmd=xray info=layer.stats added_size.human='66 B' added_size.bytes='66'
cmd=xray info=layer.change.summary added='1' all='5' deleted='0' modified='4'
cmd=xray info=layer.objects.count value='5'
cmd=xray info=layer.objects.top.start
A: mode=-rw-r--r-- size.human='66 B' size.bytes=66 uid=0 gid=0 mtime='2021-08-28T02:27:44Z' H=\[A:12\] '/usr/local/etc/php/php-cli.ini'
M: mode=drwxr-xr-x size.human='0 B' size.bytes=0 uid=0 gid=0 mtime='2021-08-27T21:33:29Z' H=\[A:3/M:6,8,11,12\] '/usr/local/etc'
M: mode=drwxr-xr-x size.human='0 B' size.bytes=0 uid=0 gid=0 mtime='2021-08-28T02:27:44Z' H=\[A:3/M:6,8,11,12\] '/usr/local/etc/php'
M: mode=drwxr-xr-x size.human='0 B' size.bytes=0 uid=0 gid=0 mtime='2021-08-27T21:33:25Z' H=\[A:0/M:1,3,5,6,7,8,10,11,12\] '/usr/local'
M: mode=drwxr-xr-x size.human='0 B' size.bytes=0 uid=0 gid=0 mtime='2021-08-28T02:27:43Z' H=\[A:0/M:1,3,4,5,6,7,8,9,10,11,12,13\] '/usr'
cmd=xray info=layer.objects.top.end
cmd=xray info=layer.end

As you can see, this shows the impact of each command at image build time as well as the environment variables that were provided. This is the first crucial step toward understanding how to debug the application that the container facilitates. You can experiment by running the xray command for your own images.

2. Fixing Application Start and Stop Failures

Now, let’s say that you have a clear picture of what’s inside a container image. You start your container, but sadly, you get an error. You might get an error when starting the application or find that your application’s health is not very good. For example, you might start your container only to see it stop with the status “exited.

There are several possible reasons why your application won’t start inside the container. One of the most obvious is that it’s using the wrong ENTRYPOINT and CMD instructions.

The ENTRYPOINT step will be the starting command of the container image when it’s run by the daemon. It configures a container that will run as an executable. You can pass a list of parameters after the executable in the shell form like this:

["executable", "param1", "param2", ...]

For example, this Dockerfile

# Dockerfile
FROM ubuntu:latest
ENTRYPOINT [ "echo", "hello"]

would yield the following:

$ docker run -it --rm test world
hello world

The CMD instruction allows for a default command that will be executed only when you run a container without specifying anything else. CMD is ignored when you provide a different command. If ENTRYPOINT and CMD are both included, the ENTRYPOINT command will always be used by default, but you will have the option to override the CMD parameters.

If you try to substitute env variables in ENTRYPOINT’s exec form, it will pass them to the application without changing them:

# Dockerfile
FROM ubuntu:latest
ENTRYPOINT [ "echo", "$HOME" ]
$ docker run -it --rm test
$HOME

If you want to replace them with actual variables, you’ll need to either use the shell form to properly expand the variables or invoke the shell directly:

# Dockerfile
ENTRYPOINT [ "sh", "-c", "echo $HOME" ]
$ docker run -it --rm test
/root

You can also override the entrypoint of the container at this time and try to simulate the initial steps one by one:

$ docker run -d -it --rm --entrypoint /bin/sh test
44aa9eb3ad57994f5f1606f82e83dc64b6299f2a7fb755ec422c09dfc44b124a

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
44aa9eb3ad57 test "/bin/sh" 36 seconds ago Up 35 seconds serene_colden

$ docker attach 44aa9eb3ad57
echo "hello"
hello

Some of the tools that package applications inside Docker images only use the exec form, and you may find that they don’t always expand those arguments properly. If you aren’t sure about the arguments that your application is receiving, you can use the above Dockerfile to quickly debug it.

Naturally, the application might not start correctly if you misspell something or provide the wrong arguments. When this happens, the application usually logs an error in the console, which you will be able to inspect. We’ll show you some ways to do that next.

3. Debugging Container Logs and Events

The standard way to debug containers is to inspect their logs. If you know the container id or name, you can follow the log trail by using the following command:

$ docker logs -f <container_id>

For example, we can simulate a long running process like this:

$ docker run -d --name=test-long-running ubuntu /bin/sh -c "while true; do sleep 2;date; done"
dc9bbba1936e03f2496b1ed354bc9a3132de3a6c0ade109570ac631dccc7d177

Then, we inspect the logs:

$ docker logs -f dc9bbba1936e03f2496b1ed354bc9a3132de3a6c0ade109570ac631dccc7d177
Fri Sep 23 09:27:51 UTC 2021
Fri Sep 23 09:27:53 UTC 2021

Sometimes, however, the container will stop so quickly that you don’t have time to inspect these logs. In that case, you can find the list of “exited” containers and use the id to inspect their logs:

$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
894a1e790efc ubuntu "/bin/sh -c 'while t…" 2 minutes ago Exited (137) 37 seconds ago test-long-running

$ docker logs 894a1e790efc

In addition, you can use Docker events to inspect all of the events that happened inside the Docker server in parallel:

$ docker events &

2021-09-24T10:30:08.665267774+01:00 **container kill** dc9bbba1936e03f2496b1ed354bc9a3132de3a6c0ade109570ac631dccc7d177 (image=ubuntu, name=test-long-running, signal=9)
2021-09-24T10:30:08.686442560+01:00 **container die** dc9bbba1936e03f2496b1ed354bc9a3132de3a6c0ade109570ac631dccc7d177 (exitCode=137, image=ubuntu, name=test-long-running)
2021-09-24T10:30:08.732703578+01:00 **network disconnect** 85c5dfd30ecfd9ecb9dc57da818aafef2a81b14c8211b9f2856dca6adf6fa2c6 (container=dc9bbba1936e03f2496b1ed354bc9a3132de3a6c0ade109570ac631dccc7d177, name=bridge, type=bridge)
2021-09-24T10:30:08.740324632+01:00 **container stop** dc9bbba1936e03f2496b1ed354bc9a3132de3a6c0ade109570ac631dccc7d177 (image=ubuntu, name=test-long-running)
2021-09-24T10:30:23.292551625+01:00 **container destroy** dc9bbba1936e03f2496b1ed354bc9a3132de3a6c0ade109570ac631dccc7d177 (image=ubuntu, name=test-long-running)

According to the official docs, containers emit a variety of events, so you’ll be able to get an idea of what kind of event stopped the container.

4. Fixing Permission Errors

Many times, a container will start correctly but fail because the application triggers a file permission error. This means that either something is configured incorrectly or the permissions are wrong. This happens when you share volumes between a user and a host with different permissions.

When you get file permission errors, you should follow the standard safe approach to building Dockerfiles by adding a user and a group to the image. For example, you could add these steps to an Ubuntu image:

FROM ubuntu:latest
ARG USER_ID
ARG GROUP_ID
RUN bash -c 'if [ ${ostype} == Linux ]; then groupadd -r --gid $GROUP_ID user; fi'
RUN adduser --disabled-password --gecos '' --uid $USER_ID --gid $GROUP_ID user
USER user
ENTRYPOINT [ "echo" ]

Then, you will need to build the image by providing the host’s USER_ID and GROUP_ID arguments:

$ docker build -t test --build-arg USER_ID=$(id -u) --build-arg GROUP_ID=$(id -g) .

That will take care of most of the file permission errors you might encounter.

5. Attaching to Paused Container

Containers can also be paused by using the docker pause command. This will allow you to attach into the container without worrying about adjourning your debugging sessions, since all of the processes inside the container will be paused. Then, you can unpause the container and observe the STDOUT.

You can simulate the creation of a long running container by running the sleep command inside a container every now and then, like this:

$ docker run -d --name=test-long-running ubuntu /bin/sh -c "while true; do sleep 2;date; done"
71b606c5cc3d112647c9d19136dbd0dc7267d21fea4c6c370dd02d1fdfc32acc

You need to attach before you pause the container, because you cannot do it after it’s paused:

docker attach 71b606c5cc3d112647c9d19136dbd0dc7267d21fea4c6c370dd02d1fdfc32acc

You can pause the container on a different tab like this:

docker pause 71b606c5cc3d112647c9d19136dbd0dc7267d21fea4c6c370dd02d1fdfc32acc

Then, verify that it’s paused:

docker inspect test-long-running
    [{
        "Id": "71b606c5cc3d112647c9d19136dbd0dc7267d21fea4c6c370dd02d1fdfc32acc",
        "Created": "2021-09-23T12:57:40.5913689Z",
        "Path": "/bin/sh",
        "Args": [
            "-c",
            "while true; do sleep 2; done"
         ],
        "State": {
            "Status": "paused",

Now, you can unpause the container to resume the process:

$ docker unpause 71b606c5cc3d112647c9d19136dbd0dc7267d21fea4c6c370dd02d1fdfc32acc

From here, you could review the output that the application sends to the STDIN/OUT/ERR, and you could also send OS signals to simulate a crash.

Next Steps with Debugging and Optimizing Containers

In this article, we explained several ways to debug containers. It’s a good starting point, but you will probably need something more detailed if your container still doesn’t work as expected. In that case, you’ll likely want a complete, developer-friendly solution for analyzing, visualizing, and optimizing containers.

This is where Slim.AI can help you. They offer purpose-built developer tools that help you create production-ready containers and optimized images. You can join the Slim Developer Platform’s Early Access here.

About the Author

Theo Despoudis is a Senior Software Engineer, a consultant and an experienced mentor. He has a keen interest in Open Source software Architectures, Cloud Computing, best practices and functional programming. He occasionally blogs on several publishing platforms and enjoys creating projects from inspiration. Follow him on Twitter @nerdokto.

Related Articles

5 Most Commonly Asked DockerSlim Questions

We enlisted DockerSlim expert and Slim.AI Developer Experience Engineer to dive into how container slimming works.

Primož Ajdišek

Technical Staff

5 Ways Slim Containers Save You Money

Do slim containers really save you money on your cloud bill? Are there cost advantages to smaller containers? Find out here.

Chris Tozzi

Automating DockerSlim in Your CICD Pipeline

Using GitHub Actions, you can refine container images automatically making them smaller, faster to load, and more secure by default – all without sacrificing any capabilities.

Nicolas Bohorquez

Contributor

Building Apps Using Cloud Native Buildpacks

Getting started with this innovative technique

Vince Power

Contributor

Building DockerSlim into a Jenkins Pipeline

A step by step tutorial on building DockerSlim into your CI/CD pipeline.

Clarifying the Complex: Meet Ivan Velichko, Container Dude at Slim.AI

Ivan recently joined the team at Slim.AI, and we sat down with him to learn more about the path that led him here.

Ivan Velichko

Container Dude

Container Insights: Dissecting the World's Most Popular Containers

Join Ayse Kaya in this series, as she creates her 2022 Container Report Chalk Full of Important Security Findings for Developers.

Ayse Kaya

Analytics & Strategy

Container of the Week: Python & Flask

Our weekly breakdown of a popular container

What We Discovered Analyzing the Top 100 Public Container Images

Complexity abounds in modern development

Ayse Kaya

Analytics & Strategy

2022 Public Container Report

Vulnerabilities continue to increase and developers are struggling to keep up.

Ayse Kaya

Analytics & Strategy

Containerizing Python Apps for Lambda

A tutorial on deploying AWS Lambda using containers, Python edition.

Docker Containers for Your Raspberry Pi

Compact PCs need compact apps

Martin Wimpress

Community

Explore and analyze a Docker container with DockerSlim X-Ray

Understanding container composition

Martin Wimpress

Community

Five Things You Should Never Ship to Production in a Container

Here is our take on five things to avoid when creating a container or shipping it to production.

Chris Tozzi

Increasing Your CI/CD Velocity with Slim Containers

We’ll explain what Slim Containers are, how they speed up the build process, and how they can improve the efficiency of your testing.

Mike Mackrory

Contributor

Integrate Testing into Your Container Pipeline

A closer look at testing within container pipelines, CI/CD, software delivery, and containerization.

Faith Kilonzi

Software Engineer

Reducing Docker Image Size - Slimming vs Compressing

Know the difference

Pieter van Noordennen

Growth

Serverless Applications and Docker

How to Scale the Latest Trend in Infrastructure

Pieter van Noordennen

Growth

Slim.AI Docker Extension for Docker Desktop

How to access our Docker Extension and try it for yourself.

Josh Viney

Product

Slimming a Rails Application with DockerSlim

Dissect a simple Rails application container using DockerSlim to analyze, optimize, and deploy your product more quickly.

Theofanis Despoudis

Contributor

Where Do You Store Your Container Images?

Container Registry Options are Growing in Number and Complexity

Pieter van Noordennen

Growth

Using AppArmor and SecComp Profiles for Security Audits

Conduct better container security audits using tools like SecComp, NGINX, and Docker.

What’s in your container?

Why Docker Layers matter for container optimization

Pieter van Noordennen

Growth

Why Developers Shouldn't Have to Be Infrastructure Experts, Too

Simplifying processes required to containerize and deploy cloud-native apps.

Chris Tozzi

A New Workflow for Cloud Development

Leverage the benefits of containerization without the headaches & hassle

John Amaral

CEO

Why Don’t We Practice Container Best Practices?

Container best practices are easy to understand, hard to do

John Amaral

CEO

Better Security Audits with AppArmor and SecComp via DockerSlim

Combine the power of tools like SecComp, NGINX, and Docker.

Automatically reduce Docker container size using DockerSlim

REST Web Service example using Python/Flask

John Amaral

CEO

Comparing Container Versions with DockerSlim and Slim.AI

See differences between your original and slimmed images

Pieter van Noordennen

Growth

Getting Started with Multi-Container Apps

Up your container game with Docker Compose

Nicholas Bohorquez

Contributor

Quick Start Guide

Slim Developer Platform Early Access

Meet DockerSlim's Compose Mode

Optimize a multi-tier app with a single command

Ian Juma

Technical Staff

Creating a Container Pipeline with GitLab CI

Shipping containers the easy way

Nicolas Bohorquez

Contributor