Unbounded connection acceptance in http4s-blaze-server

Impact

blaze-core, a library underlying http4s-blaze-server, accepts connections unboundedly on its selector pool. This has the net effect of amplifying degradation in services that are unable to handle their current request load, since incoming connections are still accepted and added to an unbounded queue. Each connection allocates a socket handle, which drains a scarce OS resource. This can also confound higher level circuit breakers which work based on detecting failed connections.

http4s provides a general MaxActiveRequests middleware mechanism for limiting open connections, but it is enforced inside the Blaze accept loop, after the connection is accepted and the socket opened. Thus, the limit only prevents the number of connections which can be simultaneously processed, not the number of connections which can be held open.

Patches

In 0.21.18, 0.22.0-M3, and 1.0.0-M16, a newmaxConnections property, with a default value of 1024, has been added to the BlazeServerBuilder. Setting the value to a negative number restores unbounded behavior, but is strongly disrecommended.

The NIO2 backend does not respect maxConnections. Its use is now deprecated in http4s-0.21, and the option is removed altogether starting in http4s-0.22.

The connections are bounded in 0.21.17, 0.22.0-M2, and 1.0.0-M14, but the maxConnections parameter was passed incorrectly, making it impossible to change the Blaze default of 512.

Workarounds

• An Nginx side-car acting as a reverse proxy for the local http4s-blaze-server instance would be able to apply a connection limiting semantic before the sockets reach blaze-core. Nginx’s connection bounding is both asynchronous and properly respects backpressure.
• http4s-ember-server is an alternative to http4s-blaze-server, but does not yet have HTTP/2 or web socket support. Its performance in terms of RPS is appreciably behind Blaze’s, and as the newest backend, has substantially less industrial uptake.
• http4s-jetty is an alternative to http4s-blaze-server, but does not yet have web socket support. Its performance in terms of requests per second is somewhat behind Blaze’s, and despite Jetty’s industrial adoption, the http4s integration has substantially less industrial uptake.
• http4s-tomcat is an alternative to http4s-blaze-server, but does not yet have HTTP/2 web socket support. Its performance in terms of requests per second is somewhat behind Blaze’s, and despite Jetty’s industrial adoption, the http4s integration has substantially less industrial uptake.

References

See the Blaze GHSA for more on the underlying issue.

For more information

If you have any questions or comments about this advisory:

• Open an issue in http4s/http4s
• Contact us according to the http4s security policy