Let nginx start if upstream host is unavailable or down

If you use proxy_pass or fastcgi_pass definitions in your nginx server config, then nginx checks the hostname during the startup phase. If one of these servers is not available, nginx will not start. This is not useful. If you use nginx as a gateway, why should all services unreachable if only one service is down due the nginx ramp time? This blog post shows a trick how to avoid such behaviour and exposes also the internal Docker DNS IP for the Docker DNS resolver.

Use nginx variables

The trick is to use variables for the upstream domain name. Maybe you even don't need an upstream definition like this GIT diff shows. Let's take a look at a common PHP nginx location example.

location ^~ /api/ {
    # other config entries omitted for breavity

    # nginx start will fail if host is not reachable
    fastcgi_pass    api.awesome.com:9000; 
    fastcgi_index   index.php;
}

The next example replaces the fastcgi_pass with a variable, so nginx will not check if the host is reachable on startup. This results in a 502 Bad Gateway message if the host is unavailable and that's fine. As soon as the service is back, everything works as expected.

server {
    location ^~ /api/ {
        # other config entries omitted for breavity
    
        set $upstream api.awesome.com:9000;

        # nginx will now start if host is not reachable
        fastcgi_pass    $upstream; 
        fastcgi_index   index.php;
    }
}

Internal Docker DNS resolver IP

If the definition above is used, a resovler definition is needed. Because we use Docker, we have to use the internal Docker DNS resolver IP which is 127.0.0.11. By the way, the internal AWS DNS resolver IP is your AWS VPC network range plus two. To further remove the downtime, reduce the resolve cache time to 30 seconds instead of the default 5 minutes. Let's at the nginx resolver definition to the config above.

server {
    # this is the internal Docker DNS, cache only for 30s
    resolver 127.0.0.11 valid=30s;
    
    location ^~ /api/ {
        # other config entries omitted for breavity
    
        set $upstream api.awesome.com:9000;
 
        # nginx will now start if host is not reachable
        fastcgi_pass    $upstream; 
        fastcgi_index   index.php;
    }
}

Conclusion

In this blog post you have seen a small but subtle difference in the nginx upstream host definition. You will only notice the change, if something is broken in your infrastructure and then it's to late. With this nginx config you will deliver more robust infrastructure. If you have some other handy tips, don't hesitate to leave a comment.