A company had an issue with a website simply failing to perform. When load tested it could handle far fewer requests per second than were expected for the amount of hardware thrown at the problem.
The system had a front end web service for customers. It had a back end system that stored data the front end needed.
My investigations led me to more closely observe the interaction between the front and back ends. I found that every front end only had two TCP connections to each back end server – which was fewer than I expected for a fully loaded system.
I knew the front end was coded in Java – so started looking into what classes they were using to make web requests to the back end. And there I found the fault: the customer was using the Apache HTTP Components library and, in particular, the PoolingHttpClientConnectionManager class.
This class was clever in that it “pooled” HTTP connections to allow for more efficient use of existing connections. The problem was clear in the documentation that, by default:
Per default this implementation will create no more than than 2 concurrent connections per given route
I informed the development team and they expanded the maximum number of concurrent connections to the back end service.
Subsequent performance testing confirmed this was, indeed, the issue, and the front end began handling the expected load again.