[FIXED] How to deploy docker-compose file on Linux deployment server from Jenkins build job?

Issue

My Jenkins build job compiles the code, then creates two Docker images (each a separate piece of the application) and publishes them.

Then it comes time to deploying them to on-premise Linux servers. I have a docker-compose.yaml file that supposed to get everything up and running.

My problem is, I am looking for a way to do three things:

1. I need to place the compose file in the deployment server(s) which are not under Jenkins control, allowing perhaps some daemon to process it, as Docker images are already uploaded to Docker repo. I have root access to the deployment servers, ssh credentials are in Jenkins, and I am able to install things on the deployment servers. So, I tried to do that with scp'ing using a special service user, but it does not have permissions to create a directory (eg: /app) - I tried working around by adding that user to the root group (usermod -aG root username) - still same error.
2. Once the compose file is in, I assume I will be able to deploy (as that same user is in docker group too) with daemon flag, but what about scenario where the app is already running, and I want to just deploy an update (essentially pull newer version of the images), and how to make sure docker will always keep the deployed stack always running, even if something crashes?
3. If the running container(s) crashes persistently, how to avoid infinite loop, and get notified when say, it tried to restart it 5 times already?

Solution

I would copy an script in order to stop and start the containers when a new image is available:

1. Scp should work as it is described in Use ssh credentials in jenkins pipeline with ssh, scp or sftp.
2. You should copy an script stopping (docker-compose down) and starting the containers (docker-compose up -d) together with the docker-compose file. Then the daemon must execute the script. EDIT: it seems one can use docker stack deploy -c /path/to/docker-compose.yml stack_name --with-registry-auth without needing to stop/start each time!

3. You can use curl to check that the services are running (as a healthcheck). Retry a number of times returning error if one of the services is not available as It is explained here:

     #!/bin/bash
     url='http://website-to-test'
     attempts=5
     timeout=5
     online=false
   
     echo "Checking status of $url."
   
     for (( i=1; i<=$attempts; i++ ))
     do
         code=`curl -sL --connect-timeout 20 --max-time 30 -w "%{http_code}\\n" "$url" -o /dev/null`
   
         echo "Found code $code for $url."
   
         if [ "$code" = "200" ]; then
             echo "Website $url is online."
             online=true
             break
        else
             echo "Website $url seems to be offline. Waiting $timeout seconds."
             sleep $timeout
        fi
    done
   
    if $online; then
      echo "Monitor finished, website is online."
      exit 0
    else
      echo "Monitor failed, website seems to be down."
      exit 1
    fi
   

UPDATE: Modify the response taking into account your restrictions

Answered By - Carlos Cavero

Answer Checked By - Katrina (PHPFixing Volunteer)