[FIXED] Why are some Nginx reverse proxied hosts trying to get files locally from /etc/nginx/html?

Issue

I've set up a simple Nginx reverse proxy using the "official image" Docker container. I've set it up to act as a front end for three containers, but only one is working correctly. The others load their index.html, but then try to pull ancillary files with URLs that make Nginx think they're local files.

The Docker host is called otto. I have Docker containers for Home Assistant listening on port 8123, Statping on port 8080, and Portainer on port 9000. Because I'm tired of remembering port numbers, my proxy_pass is set up so http://otto/homeassistant/ redirects to http://otto:8123/, http://otto/statping/ to http://otto:8080/ and http://otto/portainer/ to http://otto:9000

But, only Portainer works. The other two load their index.html just fine, but then start trying to pull javascript files from /js/somefilename.js. This of course fails, because there's no http://otto/homeassistant or http://otto/statping on the front of it. Nginx then tries to find the file locally in /etc/nginx/html, fails, and gives up.

My question is why does Portainer work so flawlessly when the other two can't even load a home page?

I've tried disabling local file serving, by commenting out location / { }, but it doesn't matter.

I've tried assigning HTTP headers that I've seen in other examples, like this:

    location /statping/ {
        proxy_pass http://otto:8080/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
    }

But it has no effect.

Running a curl command on each URL results in "301 Moved Permanently" for each new location. Could it be something simple like Portainer is better suited for reverse proxy and the others aren't?

I feel like I must be doing something right, because Portainer is working, but I fail to see what I need to do to make the other ones work. I'm also not entirely sure if the problem is with the Nginx config or something with the home assistant or statping containers.

Any ideas would be appreciated.

Here's my nginx configuration that replaces the delivered default.conf:

pi@otto:~/docker/nginx $ cat rproxy.conf
# Configuration to serve static files and do reverse proxy.
server {
    listen       80;
    listen  [::]:80;
    server_name  otto;

    #access_log  /var/log/nginx/host.access.log  main;

    # Reverse Proxy Configuration
    location /homeassistant/ {
        proxy_pass http://otto:8123/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
    }

    location /portainer/ {
        proxy_pass http://otto:9000/;
    }

    location /statping/ {
        proxy_pass http://otto:8080/;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
    }

    # Also serve up static HTML locally.
    location / {
        root /usr/share/nginx/html;
        index index.html index.htm;
    }
}

Here's the docker run command that starts the Nginx container.

pi@otto:~/docker/nginx $ cat run.sh
#!/bin/bash

if ! docker ps | grep nginx; then
  docker run -d \
    -p 80:80 \
    --hostname nginx \
    --name nginx \
    --restart unless-stopped \
    -v /home/pi/docker/nginx/rproxy.conf:/etc/nginx/conf.d/default.conf \
    -v /home/pi/docker/nginx/content:/usr/share/nginx/html:ro \
    nginx
fi

Here are the results of curl commands:

pi@otto:~/docker/nginx $ curl -I http://otto/homeassistant
HTTP/1.1 301 Moved Permanently
Server: nginx/1.19.7
Date: Tue, 16 Mar 2021 02:14:15 GMT
Content-Type: text/html
Content-Length: 169
Location: http://otto/homeassistant/
Connection: keep-alive

pi@otto:~/docker/nginx $ curl -I http://otto/statping
HTTP/1.1 301 Moved Permanently
Server: nginx/1.19.7
Date: Tue, 16 Mar 2021 02:14:40 GMT
Content-Type: text/html
Content-Length: 169
Location: http://otto/statping/
Connection: keep-alive

pi@otto:~/docker/nginx $ curl -I http://otto/portainer
HTTP/1.1 301 Moved Permanently
Server: nginx/1.19.7
Date: Tue, 16 Mar 2021 02:15:03 GMT
Content-Type: text/html
Content-Length: 169
Location: http://otto/portainer/
Connection: keep-alive

And here are the log entries that show where Home Assistant and Statping trying to find the files locally and failing, as well as the successful Portainer.

192.168.0.45 - - [16/Mar/2021:00:30:11 +0000] "GET /homeassistant/ HTTP/1.1" 200 3307 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Gecko/20100101 Firefox/86.0" "-",
192.168.0.45 - - [16/Mar/2021:00:30:11 +0000] "GET /frontend_latest/core.a3d9350b.js HTTP/1.1" 404 153 "http://otto/homeassistant/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Gecko/20100101 Firefox/86.0" "-",
03/16 00:30:11 [error] 24#24: *1 open() "/etc/nginx/html/frontend_latest/core.a3d9350b.js" failed (2: No such file or directory), client: 192.168.0.45, server: otto, request: "GET /frontend_latest/core.a3d9350b.js HTTP/1.1", host: "otto", referrer: "http://otto/homeassistant/",
...
192.168.0.45 - - [16/Mar/2021:00:31:12 +0000] "GET /statping/ HTTP/1.1" 200 3271 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Gecko/20100101 Firefox/86.0" "-",
2021/03/16 00:31:13 [error] 24#24: *3 open() "/etc/nginx/html/js/bundle.js" failed (2: No such file or directory), client: 192.168.0.45, server: otto, request: "GET /js/bundle.js HTTP/1.1", host: "otto", referrer: "http://otto/statping/",
...
192.168.0.45 - - [16/Mar/2021:00:31:26 +0000] "GET /portainer/ HTTP/1.1" 200 23180 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Gecko/20100101 Firefox/86.0" "-",
192.168.0.45 - - [16/Mar/2021:00:31:26 +0000] "GET /portainer/api/settings/public HTTP/1.1" 200 533 "http://otto/portainer/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Gecko/20100101 Firefox/86.0" "-",
192.168.0.45 - - [16/Mar/2021:00:31:26 +0000] "GET /portainer/api/status HTTP/1.1" 200 20 "http://otto/portainer/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Gecko/20100101 Firefox/86.0" "-",

Solution

The problem was exactly as @zigarn had pointed out in the comments. The apps being reverse proxied were relying on URLs with a base of / and not something like /statping or /homeassistant. Portainer worked because it apparently does not have that same dependency.

The solution is to move away from reverse proxying based on URL and instead use reverse proxy based on hostname. So now, rather than http://otto.local/statping/, I have http://statping.otto.local/ All reverse proxied apps load their home pages with this arrangement, so I feel like it's a win.

There area still some minor troubles with homeassistant and node-red, and I suspect this may be due to a need to reverse proxy web sockets. Still researching that, but the main objective of getting them all to pull up a home page or login page has been accomplished.

Now for the gory details...

Because this is a home setup, with the standard, basic functionality internet router, I had to put some work into setting up a DNS server before names like statping.otto.local or homeassistant.otto.local would resolve.

To do this, I installed bind 9 on otto, the same host running the Docker containers. It's installed from a package on Raspberry Pi OS rather than as a container. The trick was to use a wildcard CNAME in addition to the DNS entry for otto.local.

With *.otto.local pointing to otto.local, I am able to get any combination of {appname}.otto.local to resolve to otto's IP address. Now, typing http://portainer.otto.local et al. gets me to otto and Nginx.

In the Nginx config, it's a matter of creating several entries that look like this one:

server {
    server_name statping.otto.local;
    location / {
        proxy_pass http://192.168.0.23:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
    }
}

Each server { } block has a server_name with the {appname}.otto.local namin convention. The location is /, just like the apps expect, and that helps tremendously.

The proxy_pass points to otto's IP address, because even though otto will use its own DNS service, the Docker containers are pulling DHCP addresses from the internet router which hands out its own address for DNS. (I wasn't in the mood to build out a DHCP infrastructure as well, so I used the IP.)

The proxy_set_header variables are in there out of sheer cargo cult mentality. I have no idea if they make things better or not, but many configuration examples used them, so I figured I should too.

So, in summary, the use of server names is a more consistent way of doing reverse proxy for apps than using urls. But, it's not without its own drawbacks and not nearly as easy, because it requires more control over DNS than most internet routers give.

Yet another way of accomplishing my original goal of not having to remember port numbers, is to use Nginx redirects. For example, a block like this:

server {
    server_name portainer.otto.local;
    return 301 http://portainer.otto.local:9000;
}

Whenever a browser connects to portainer.otto.local on port 80, Nginx replies with 301 (moved permanently) and gives portainer.otto.local:9000 as the new destination.

This appears to work the best out of all the options I tried. It does require a DNS system that will do wildcards, so there is that bit of added work. It also does not hide the backend ports like a reverse proxy can. But for a home setup, it's the easiest method I've found.

Answered By - Dave H.

Answer Checked By - Clifford M. (PHPFixing Volunteer)