Situation: ELB configured with TCP listeners, passing traffic on to nginx
which then proxy_passes all traffic to the actual application server. Since we
want websockets to load balance properly, proxy protocol has been enabled on
both ELB and nginx. And since we want to get emails when stuff is not
working, there's an HTTP health check setup on the ELB which accesses the same
port on which nginx is listening.
Can you spot the mistake?
Enabling proxy protocol on the ELB does not automatically translate to health checks. That is, even though the normal requests would be getting the fancy proxy protocol line prepended in front of every request, it's going to be missing from the health check requests. Which means that soon, the load balancer is going to start taking instances out, thinking that they're no longer healthy. Of course, all the instances are still healthy. They just aren't receiving the extra proxy protocol info.
Solution?
Switch the health check to TCP. This only checks whether or not a TCP connection can be successfully established (which might be OK for some cases). But not so much in case you want the health check to check the health of your application.
Configuring the health check to directly access your application server over HTTP would also work.
Alternatively, just use ALB.