[FIXED] How to cure 503 Service Unavailable in Apache running in Docker Container?

Issue

I am trying to create an Apache virtual host proxy in a Docker container (I am on Docker 1.6), so I did the following:

First set up two Docker containers, each running their own web application:

docker run -it -p 8001:80 -p 4431:443 --name build ubuntu:latest

apt-get update
apt-get install apache2 libapache2-mod-php -y
echo “<?php phpinfo(); ?>” >> /var/www/html/info.php
service apache2 restart

then

docker run -it -p 8002:80 -p 4432:443 --name cicd ubuntu:latest

apt-get update
apt-get install apache2 libapache2-mod-php -y
echo “<?php phpinfo(); ?>” >> /var/www/html/info.php
service apache2 restart

Each of these runs perfectly on their respective ports. So next I create a container running Apache:

docker run -it -p 8000:80 -p 4430:443 --name apache_proxy ubuntu:latest

apt-get update
apt-get install apache2 -y
a2enmod proxy
a2enmod proxy_http
service apache2 restart

This works perfectly on it's own at port 8000.

Then I created a virtual host file for each of the other Docker containers:

<VirtualHost *:80>
    ServerName build.example.com
    <Proxy *>
        Allow from localhost
    </Proxy>
    ProxyPass / http://localhost:8001/
</VirtualHost>

and

<VirtualHost *:80>
    ServerName cicd.example.com
    <Proxy *>
        Allow from localhost
    </Proxy>
    ProxyPass / http://localhost:8002/
</VirtualHost>

Then placed both in /etc/apache2/sites-available/ of the apache_proxy container.

Now, I went back into the apache_proxy container and performed the following:

a2ensite build.example.conf
a2ensite cicd.example.conf
service apache2 restart

Running apachectl -S from the command line of the apache_proxy container I can see the following is in effect:

VirtualHost configuration:
*:80 is a NameVirtualHost
     default server 172.17.0.17 (/etc/apache2/sites-enabled/000-default.conf:1)
     port 80 namevhost 172.17.0.17 (/etc/apache2/sites-enabled/000-default.conf:1)
     port 80 namevhost build.example.com (/etc/apache2/sites-enabled/build.example.conf:1)
     port 80 namevhost cicd.example.com (/etc/apache2/sites-enabled/cicd.example.conf:1)

Here is what the setup looks like:

I can reach each individual container via it's respective port and I should be able to go to the following URL's to get to the respective sites:

build.example.com:8000 should proxy to the container/website on port 8001 cicd.example.com:8000 should proxy to the container/website on port 8002

Instead I get the following error:

503 Service Unavailable

Checking the logs I get the following:

[Mon Oct 16 21:17:32.510127 2017] [proxy:error] [pid 165:tid 140552167175936] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8001 (localhost) failed
[Mon Oct 16 21:17:32.510278 2017] [proxy:error] [pid 165:tid 140552167175936] AH00959: ap_proxy_connect_backend disabling worker for (localhost) for 0s
[Mon Oct 16 21:17:32.510302 2017] [proxy_http:error] [pid 165:tid 140552167175936] [client 172.26.16.120:61391] AH01114: HTTP: failed to make connection to backend: localhost
[Mon Oct 16 21:17:32.799053 2017] [proxy:error] [pid 166:tid 140552217532160] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:8001 (localhost) failed
[Mon Oct 16 21:17:32.799232 2017] [proxy:error] [pid 166:tid 140552217532160] AH00959: ap_proxy_connect_backend disabling worker for (localhost) for 0s
[Mon Oct 16 21:17:32.799256 2017] [proxy_http:error] [pid 166:tid 140552217532160] [client 172.26.16.120:61392] AH01114: HTTP: failed to make connection to backend: localhost, referer: http://build.example.com:8000/info.php

I have been going down rabbit holes for the past several hours trying to get this to work and I am sure now I am missing something quite simple. Can anyone shed light on the error of my ways?

NOTE

I followed a huge rabbit hole concerning SELinux which is not enabled or even really installed in the Ubuntu/Apache proxy container.

I should also state that I am not a network guru or master web-server configurator. I only know enough to be dangerous.

EDIT

Based on suggestions I have tried the following:

ProxyPass / http://cicd:80/ causes a 502 error
ProxyPass / http://ip.address.of.server:8002/ times out
ProxyPass / http://ip.address.of.container:8002/ causes a 503 error
ProxyPass / http://127.0.0.1:8002/ retry=0 causes a 503 error (suggested in other answers)

Solution

What I really needed to remember is that we’re working within Docker’s network once we pass in the request, so using things like ProxyPass / http://localhost:8002/ will not work because ‘localhost’ belongs to the Docker container in which we’ve made the request.

So I started searching outside of the error box, so to speak, and came upon this answer From inside of a Docker container, how do I connect to the localhost of the machine?

What I determined is we need to pass the request to the Docker network. To get that information I ran sudo ip addr show docker0 from the server’s command line and it returns:

4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 56:84:7a:fe:97:99 brd ff:ff:ff:ff:ff:ff
    inet 172.17.42.1/16 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::5484:7aff:fefe:9799/64 scope link
       valid_lft forever preferred_lft forever

Showing Docker’s internal network to be on 172.17.42.1 Changing the pass to be ProxyPass / http://172.17.42.1:8002/ hands off the request to the Docker network and subsequently succeeds.

<VirtualHost *:80>
    ServerName cicd.example.com
    <Proxy *>
        #Allow from localhost
        Order deny,allow
        Allow from all
    </Proxy>
    ProxyPass / http://172.17.42.1:8002/ retry=0
</VirtualHost>

Answered By - Jay Blanchard

Answer Checked By - David Marino (PHPFixing Volunteer)