[FIXED] What are the best gitlab-ci.yml CI/CD practices and runners configs?

Issue

This is a bit theoretical, but I'll try to explain my setup as much as I can:

• 1 server (instance) with a self-hosted gitlab
• 1 server (instance) for development
• 1 server (instance) for production

Let's say in my gitlab I have a ReactJs project and I configured my gitlab-ci.yml as following:

• job deploy_dev Upon pushing to dev branch, the updates will be copied with rsync to /var/www/html/${CI_PROJECT_NAME} (As a deployment to dev server)

• The runner that picks up the job deploy_dev is a shared runner installed on that same dev server that I deploy to and it picks up jobs with the tag reactjs

The question is: If I want to deploy to production what is the best practice should I follow?

I managed to come up with a couple of options that I thought of but I don't know which one is the best practice (if any). Here is what I came up with:

1. Modify gitlab-ci.yml adding a job deploy_prod with the same tag reactjs but the script should rsync with the production server's /var/www/html/${CI_PROJECT_NAME} using SSH?

2. Set up another runner on production server and let it pick up the jobs with tags reactjs-prod and modify gitlab-ci.yml to have deploy_prod with the tag reactjs-prod?

3. You have a better way other than the 2 mentioned above?

Last question (related): Where is the best place to install my runners? Is what I'm doing (Having my runners on my dev server) actually ok?

Please if you can explain to me the best way (that you would choose) with the reasons as in pros or cons I would be very grateful.

Solution

The best practice is to separate your CI/CD infrastructure from the infrastructure where you host your apps. This is done to minimize the number of variables which can lead to problems with either your applications or your runners.

Consider the following scenarios when you have a runner on the same machine where you host your application: (The below scenarios can happen even if the runner and app are running in separate Docker containers. The underlying machine is still a single point of failure.)

1. The runner executes a CPU/RAM heavy job and takes up most of the resources on the machine. Your application starts experiencing performance problems.

2. The Gitlab runner crashes and puts the host machine in an inoperable state. (docker panic or whatever). Your production app stops functioning.

3. Your app brakes the host machine (Doesn't matter how. It can happen), your CI/CD stops working and you can not deploy a fix to production.

Consider having a separate runner machine (or machines. Gitlab runner can scale horizontally), that is used to run your deployment jobs to both dev and production servers.

Answered By - cecunami

Answer Checked By - Mildred Charles (PHPFixing Admin)