[FIXED] How to set timeout in Kong v1.1.2

Issue

Problem

I am getting an error message in my Kong error log reporting that the upstream server has timed out. But I know that the upstream process was just taking over a minute, and when it completes (after Kong has logged the error) it logs a java error "Broken Pipe", implying that Kong was no longer listening for the response.

This is the behavior when the upstream process takes longer than 60 seconds. In some cases, it takes less than 60 seconds and everything works correctly.

How can I extend Kong's timeout?

Details

Kong Version

1.1.2

Kong's Error Message (slightly edited):

2019/12/06 09:57:10 [error] 1421#0: *1377 upstream timed out (110: Connection timed out) while reading response header from upstream, client: xyz.xyz.xyz.xyz, server: kong, request: "POST /api/...... HTTP/1.1", upstream: "http://127.0.0.1:8010/api/.....", host: "xyz.xyz.com"

Here is the error from the upstream server log (Java / Tomcat via SpringBoot)

Dec 06 09:57:23 gateway-gw001-99 java[319]: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
Dec 06 09:57:23 gateway-gw001-99 java[319]:         at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:364) ~[tomcat-embed-core-8.5.42.jar!/
Dec 06 09:57:23 gateway-gw001-99 java[319]:         at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:833) ~[tomcat-embed-core-8.5.42.jar!
...

My kong.conf (slightly edited)

trusted_ips = 0.0.0.0/0
admin_listen = 0.0.0.0:8001
proxy_listen = 0.0.0.0:8080 proxy_protocol,  0.0.0.0:8443 ssl proxy_protocol
database = postgres
pg_host = 127.0.0.1
pg_port = 5432
pg_user = kong
pg_password = xyzxyzxyzxyzxyz
pg_database = kong
plugins = bundled,session
real_ip_header = proxy_protocol

A little more Context

• Kong and the Upstream Server are hosted on the same Ubuntu VM
• The Ubuntu VM is hosted as a linux container (LXC) inside another Ubuntu VM
• The outer VM uses NGinX to receive public traffic and reverse proxies it to Kong. It does this using stream. This allows Kong to be my SSL demarcation point.

The Outer NGinX Stream Config:

stream {

    server {
        listen 80;
        proxy_pass xyz.xyz.xyz.xyz:8080;
        proxy_protocol on;
    }

    server {
        listen 443;
        proxy_pass xyz.xyz.xyz.xyz:8443;
        proxy_protocol on;
    }
}

What I've Tried

I've tried adding the following lines to kong.conf. In version 1.1.2 of Kong you basically alter the NGinX settings remotely by adding prefixes to NginX config and placing them in the kong.conf (https://docs.konghq.com/1.1.x/configuration/#injecting-individual-nginx-directives ). None of them seemed to do anything:

nginx_http_keepalive_timeout=300s
nginx_proxy_proxy_read_timeout=300s
nginx_http_proxy_read_timeout=300s
nginx_proxy_send_timeout=300s
nginx_http_send_timeout=300s

Solution

Per the documentation Kong Version 0.10 has three properties that you can set for managing proxy connections

1. upstream_connect_timeout: defines in milliseconds the timeout for establishing a connection to your upstream service.
2. upstream_send_timeout: defines in milliseconds a timeout between two successive write operations for transmitting a request to your upstream service.
3. upstream_read_timeout: defines in milliseconds a timeout between two successive read operations for receiving a request from your upstream service.

In this case, as Kong is timing out waiting for the response from the upstream you would need to add a property setting for upstream_read_timeout

In the Kong Version 1.1 documentation the Service object now includes these timeout attributes with slightly different names:

1. connect_timeout: The timeout in milliseconds for establishing a connection to the upstream server. Defaults to 60000.
2. write_timeout: The timeout in milliseconds between two successive write operations for transmitting a request to the upstream server. Defaults to 60000.
3. read_timeout: The timeout in milliseconds between two successive read operations for transmitting a request to the upstream server. Defaults to 60000.

Answered By - Shawn C.

Answer Checked By - Marilyn (PHPFixing Volunteer)

Read More

[FIXED] How does a double layer of reverse proxy pass through invalid SSL cert?

Issue

I have the following server setup:

                 +----------+        +-----------+
                 |          |        |           |
internet ----->  |   Kong   | -----> |  HAProxy  | -----> backend servers
                 |          |        |           |
                 +----------+        +-----------+

• Kong is set port 443 binding with a valid SSL cert
• HAProxy is set up with port 443 binding and a redirect from port 80 to 443

Now I just found that the SSL cert installed in the HAProxy expired. However, it's not visible through the internet because they access through Kong and Kong has a valid SSL cert. Accessing the HAProxy directly gives a SSL certificate problem: certificate has expired error. What is the explanation for this? How does the first layer of reverse proxy's SSL cert suppress the error on the second layer?

Solution

How does the first layer of reverse proxy's SSL cert suppress the error on the second layer?

The clients create a SSL connection to Kong and will only check the certificate from Kong. Kong creates an independent SSL connection to HAProxy and should check the certificate from HAProxy. Then only the application payload gets passed through between client and server via Kong and HAProxy, but not any SSL related information.

How does the first layer of reverse proxy's SSL cert suppress the error on the second layer?

It does not suppress anything. There is no mechanism to pass thru SSL errors at the application level. Proper validation of the certificate from HAProxy by Kong should lead to a connection close, i.e. not forwarding any of the application data between client and server. But if the certificate is not properly validated by Kong then certificate errors will not be noticed and the application data will be forwarded between client and server. The client will not notice anything since it sees only the certificate by Kong.

Answered By - Steffen Ullrich

Answer Checked By - Cary Denson (PHPFixing Admin)

Read More