Contributed by dhartmei on from the bean-counter dept.
If you're using undeadly.org's RSS feed in any way, please read on.
In December 2005, about 17,000 unique IP addresses requested the RSS feed more than 1,000,000 times in total. The returned XML document is about 7kB in size, so that's roughly 7GB of traffic, almost half of the entire traffic generated.
Further investigation showed that about half of the clients do provide an If-Modified-Since: HTTP header, and the CGI is now using that to return a brief 304 Not Modified status code where appropriate (84 bytes vs. 7,000 bytes).
However, of the remaining 50 percent of RSS readers which don't provide this header, some are polling with a very high frequency, like once every minute. Such a client is adding up to 300MB/month, an unproportionally high number, which we couldn't afford to spend for a large number of clients.
I'd like to ask everyone who has an RSS reader/aggregator set up to please:
- Enable the use of the If-Modified-Since: header to prevent the client from fetching old redundant documents in full, or to switch to a client which supports this. With this enabled, you are free to poll as often as once every 5-15 minutes, as most replies will be small (Not Modified).
- If this is not possible, and your client fetches the full document on each request, limit polling to once every 15-60 minutes.
There were 214 stories in 2005, that means the average time between two stories posted was about 40 hours. It's a waste to fetch the very same document hundreds of times, just to get the rare changed ones with a latency of less than 15 minutes.
The CGI now (experimentally) keeps track of RSS requests per client IP address. If a client (who is not using If-Modified-Since:) fetches the full document more than 96 times within a 24h sliding window, it receives a 503 Service Unavailable status code instead. If this is hitting someone unfairly, please contact me.
The more bandwidth we can save, the more we can use for more valuable content, like stories with images.
Happy new year to everyone!
(Comments are closed)