Managing Docker Restart Policy On a Set of Hosts

Then you have been moving containers back and forth through the hosts (no Kubernetes yet), and each host has some containers running and some stopped. When you need to restart the host (or even the docker service), all containers start running, even the ones you already moved to another host.

It's common to run a container and set the Restart Policy to Always, so that the container is started automatically once docker determines it is stopped (possibly after a restart). And its common to forget resseting this setting once the container is moved to another host.

So StackOverflow comes to the rescue (thanks OscarAkaElvis) and provides a nice script to loop through all containers in a host and shows Restart Policy for each:

#!/usr/bin/env bash
#Script to check the restart policy of the containers

readarray -t CONTAINERS < <(docker ps -a | grep -v NAMES | awk '{print $NF}')

for item in "${CONTAINERS[@]}"; do

    #Hard-Bash way
    #data=$(docker inspect "${item}" | grep -A 1 RestartPolicy | awk -F '"' '{print $4}' | tail -n 1)

    #Docker-pr0 way
    data=$(docker inspect -f "{{ .HostConfig.RestartPolicy.Name }}" "${item}")

    echo "Container: ${item} / RestartPolicy: ${data}"
done

The result is:

Container: wildflyjmxbridge / RestartPolicy: always
Container: lonely_lichterman / RestartPolicy: no
Container: desperate_joliot / RestartPolicy: no
Container: jolly_galileo / RestartPolicy: no
Container: hungry_yonath / RestartPolicy: no
Container: romantic_lovelace / RestartPolicy: no
Container: awesome_thompson / RestartPolicy: no
Container: nostalgic_fermat / RestartPolicy: no
Container: bikibana / RestartPolicy: no
Container: bies / RestartPolicy: no
Container: webform / RestartPolicy: no
Container: licitacoes.cidadecompras-producao / RestartPolicy: always
Container: registroprecos-web / RestartPolicy: always
Container: legacy-api / RestartPolicy: no
Container: registroprecos-api / RestartPolicy: always
Container: gmat-api / RestartPolicy: no
Container: shipyard-controller / RestartPolicy: always
Container: shipyard-swarm-agent / RestartPolicy: always
Container: shipyard-swarm-manager / RestartPolicy: always
Container: shipyard-proxy / RestartPolicy: always
Container: shipyard-certs / RestartPolicy: always
Container: shipyard-discovery / RestartPolicy: always
Container: shipyard-rethinkdb / RestartPolicy: always

Next, we have a dozen servers. I wanted to loop through them and execute this script. Used Bash and Git, so that the bash loop script would download the above script from Git and execute in each host using ssh.

After adding the script to Git, I'd need to obtain it using curl and execute in bash. The beauty ends up being:

bash $(curl -s https://git.procempa.com.br/roger.krolow/docker_cmd/raw/master/docker-restart-policy.sh?private_token=PRIVATETOKEN) | grep always

PRIVATETOKEN must be replaced by the token got from Gitlab.

Notice the "| grep always" in the end, since I'm only interested in the ones with always.

As for scripting, ended up with two flavors: a bash script for standalone use, and a shell function, so it can be executed anytime and anywhere.

Bash Script

#Runs the script on each host
truncate -s0 drp.txt
while read i; do echo $i; echo $i >>dup.txt; ssh -n $i  "bash <(curl -s https://git.procempa.com.br/roger.krolow/docker_cmd/raw/master/docker-restart-policy.sh?private_token=PRIVATETOKEN) | grep always" >>drp.txt; done <hosts.txt
less -S drp.txt

The file drp.txt will contain the output. The file hosts.txt contains a list of docker host names, one per line.

Shell Function

Edit .bashrc and insert:

declare -a arr_prod=("docker_host1" "docker_host2" "docker_host3")

declare -a arr_stag=("docker_host4" "docker_host5" "docker_host6")

#Lists containers Restart Policy
function drp { if [ -z "$1" ]; then arr=("${arr_prod[@]}"); else n=arr_$1[@] && arr=("${!n}") ; fi
               truncate -s0 /tmp/drp.txt;
               for i in "${arr[@]}"; do echo $i; echo $i >>/tmp/drp.txt; ssh -n $i 'bash <(curl -s https://git.procempa.com.br/roger.krolow/docker_cmd/raw/master/docker-restart-policy.sh?private_token=-eokt7BYK8LV5qxiFNEy) | grep always' >>/tmp/drp.txt; echo "------------" >>/tmp/drp.txt; done;
               less -S /tmp/drp.txt ;}

The arrays arr_prod and arr_stag contain the docker hosts for each environment, production and staging. The environment can be chosen by passing "prod" or "stag" as parameter. The default is "prod".

File /tmp/drp.txt contains the output.

The image below shows the result. Docker host names are omitted.