Server optimization for high traffic web systems using multiple retry and learning timeout patterns

Excerpt

A webpage typically will be as slow as the slowest request in the page. So if for a high traffic website like Yahoo! frontpage has lots of such possibly slow external apis, it could hold webserver processes and also effect user experience. Multiple Retry is a feature meant to optimize server resource utilization and efficiently use webserver processes/threads.

Description

Server optimization for high traffic web systems using multiple retry and learning timeout patterns.
Multiple Retry is a feature meant to optimize system resource utilization while still allowing a webpage to accommodate requests that might take a long time for the responses to come back. It uses two open source technologies Squid and Traffic Server (recently open sourced by Yahoo! under apache license).
The main principle behind it is to delegate the request to an offline proxy system such as SQUID to perform the actual request. This allows the initial request to use a very small timeout but yet the proxy system will continue to wait for the request to complete and then cache the response once it comes back. A subsequent request or ‘retry request’ is made to check the data in the proxy system cache instead of the actual source. The timed out request sends the squid key back to the client to track the availability of the resource quickly without going through the normal web server route.

The traffic server which can handle thousands of concurrent connections has a plugin which can quickly check the status of the key from squid directly and if the data is indeed available, can forward the request to corresponding web servers.

With this approach, server resource does not need to be tied up for the entire duration for the response to come back. Instead, just use limited resource to fire the request and then tell the client(browser) to wait a period of time before firing another request(retry request) to check for response. As a result, transferring a significant amount of wait time from server resources to the client side and freeing up that server resource to process other requests. Each retry request will be optimized to firstly just check whether the data is available in cache using the traffic server plugin before executing the webpage or module which requires that data. This saves the need to process a webpage or a module only to find that the data is not yet available in cache and process the page only once the data is indeed available thus saving cpu cycles. Since the request reaches web server only when data is available and hence it is an efficient utilization of server processes.
Also an additional challenge the system need to address is determining what timeouts to use for each retry. This could be tracked by a very simple state at server and slowly adjusted and learned with requests to determine the optimal time interval to be given for retries and reducing failed attempts.
Thus for very high traffic systems like yahoo homepage , where we have data coming from different external sources with varying timeouts we can defer the request for multiple retry and load the page as soon as possible. Each retry request would be extremely fast to just check for the data availability in cache thus freeing server time and other processing resources. This solution does not require keep-alive connections to be maintained and hence has very little server setup overhead.