[FIXED] How to debug httputil.NewSingleHostReverseProxy

Issue

Problem:

1. I'm fowarding to a HTTPS address.
2. I want to see why removing req.Host = req.URL.Host causes it to fail. Instead of returning {"Code":"OBRI.FR.Request.Invalid","Id":"c37baec213dd1227","Message":"An error happened when parsing the request arguments","Errors":[{"ErrorCode":"UK.OBIE.Header.Missing","Message":"Missing request header 'x-fapi-financial-id' for method parameter of type String","Url":"https://docs.ob.forgerock.financial/errors#UK.OBIE.Header.Missing"}]} it returns a 404.
3. I want to trace the call the proxy returned from httputil. NewSingleHostReverseProxy is making when I uncommment the line req.Host = req.URL.Host.

Given a request as such:

$ curl http://localhost:8989/open-banking/v2.0/accounts

And the code below (main.go):

package main

import (
    "log"
    "net/http"
    "net/http/httputil"
    "net/url"
)

func main() {
    target, err := url.Parse("https://rs.aspsp.ob.forgerock.financial:443")
    log.Printf("forwarding to -> %s%s\n", target.Scheme, target.Host)

    if err != nil {
        log.Fatal(err)
    }
    proxy := httputil.NewSingleHostReverseProxy(target)

    http.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
        // https://stackoverflow.com/questions/38016477/reverse-proxy-does-not-work
        // https://forum.golangbridge.org/t/explain-how-reverse-proxy-work/6492/7
        // https://stackoverflow.com/questions/34745654/golang-reverseproxy-with-apache2-sni-hostname-error

        req.Host = req.URL.Host // if you remove this line the request will fail... I want to debug why.

        proxy.ServeHTTP(w, req)
    })

    err = http.ListenAndServe(":8989", nil)
    if err != nil {
        panic(err)
    }
}

Solution

Set the proxy.Transport field to an implementation that dumps the request before delegating to the default transport:

package main

import (
        "fmt"
        "log"
        "net/http"
        "net/http/httputil"
        "net/url"
)

type DebugTransport struct{}

func (DebugTransport) RoundTrip(r *http.Request) (*http.Response, error) {
        b, err := httputil.DumpRequestOut(r, false)
        if err != nil {
                return nil, err
        }
        fmt.Println(string(b))
        return http.DefaultTransport.RoundTrip(r)
}

func main() {
        target, _ := url.Parse("https://example.com:443")
        log.Printf("forwarding to -> %s\n", target)

        proxy := httputil.NewSingleHostReverseProxy(target)

        proxy.Transport = DebugTransport{}

        http.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
                req.Host = req.URL.Host

                proxy.ServeHTTP(w, req)
        })

        log.Fatal(http.ListenAndServe(":8989", nil))
}

The output from this program looks something like this:

2018/10/26 13:06:35 forwarding to -> https://example.com:443
GET / HTTP/1.1
Host: example.com:443
User-Agent: HTTPie/0.9.4
Accept: */*
Accept-Encoding: gzip, deflate
X-Forwarded-For: 127.0.0.1

Or, after removing the req.Host assignment:

2018/10/26 13:06:54 forwarding to -> https://example.com:443
GET / HTTP/1.1
Host: localhost:8989
User-Agent: HTTPie/0.9.4
Accept: */*
Accept-Encoding: gzip, deflate
X-Forwarded-For: 127.0.0.1

Since the Host header is very often used by webservers to route the request to the correct virtual host or backend server, it makes sense that an unexpected Host header ("localhost:8989" in the example above) causes the server to respond with 404.

Setting the Host header with httputil.ReverseProxy is typically done with the Director function:

    target, err := url.Parse("https://example.com:443")
    if err != nil {
            log.Fatal(err)
    }
    log.Printf("forwarding to -> %s\n", target)

    proxy := httputil.NewSingleHostReverseProxy(target)

    d := proxy.Director
    proxy.Director = func(r *http.Request) {
            d(r) // call default director

            r.Host = target.Host // set Host header as expected by target
    }

    log.Fatal(http.ListenAndServe(":8989", proxy))

Answered By - Peter

Answer Checked By - Terry (PHPFixing Volunteer)