The Slashdot Effect

The Slashdot Effect

An Analysis of Three Internet Publications

By Stephen Adler [ http://ssadler.phy.bnl.gov/adler/adler ]

Abstract
Apache httpd log files have been analyzed to show the Slashdot Effect. This effect is known as the spontaneous high hit rate upon a web server due to an announcement on a high volume news web site. Three papers were published on the Internet and announced on the high volume news web sites of slashdot.org, linuxtoday.org, and freshmeat.net. The hit rate plots as a function of time are correlated with the announcement times on these web sites. These correlations clearly demonstrate the magnitude and significance of the Slashdot Effect.
Author's note: An addendum to this paper has [ http://ssadler.phy.bnl.gov/adler/SDE/SlashDotEffectAddendum.html ] been published on the Internet which details the Slashdot effect on this paper.

Three papers were published on the Internet. One on October 20th, 1998 titled Fall Internet World 98, A View From The Show Floor [ http://ssadler.phy.bnl.gov/adler/fiw98/fiw98.html ], one on November 14th, 1998 titled Preserving the Information Ecosystem [ http://ssadler.phy.bnl.gov/adler/OSS/OSS.html ], and one on January 28th, 1999 titled An Ode to Richard Stallman [ http://ssadler.phy.bnl.gov/adler/Stallman/Stallman.html ]. Soon after these papers were published, Slashdot, Linux Today and Freshmeat ran announcements in one form or other regarding the availability of these papers. Within less than an hour, the hit rate to the system hosting these papers surged. This surge in hit rate on the hosting web server is known as the Slashdot Effect. The magnitude of the hit surge varied with each posting since not all three web sites posted the articles at the same time, if at all. What follows is a brief case study of the SlashDot Effect for each of these papers.

To start, a plot of the hit rate in hits/minute is shown above for the past 4 months during which the three papers were published on the Internet. The time on the X axis is that of the Unix system time in seconds since Jan 1st, 1970 scaled down by a factor of 1000. The width of one bin of the histogram is show in a text field on the bottom right of the plot. The first recorded hit was 27/Sep/1998:21:02:08, and the last recorded hit was 31/Jan/1999:03:59:43. These two dates set the two limits of the X axis time scale. The Y axis shows the number of raw hits per minute averaged over the time of one bin width. (In the case of the plot above, averaged over 902.39 minutes.) The data is shown in raw hits/minute, since for each paper referenced by a viewer on the Internet generated several hits. One needs to apply a scale factor, which is the number of hits generated per paper, in order to measure the number of readers of these publications. Since the purpose of this paper is to show the Slashdot Effect and not to try and measure the number of readers of the papers, this scaling is not performed.

Vertical lines are drawn on the above plot to indicate the time of the posting by the three web sites. The red lines indicate the announcements by Slashdot, the green lines by Linux Today, and the blue lines by Freshmeat. From this plot, one can see clear evidence of the Slashdot Effect. What follows is a closer examination of each one of these postings, since this full time view of the hit rate to the server hides some detail of the this effect.

This plot above is a zoom in around the time of the announcement of the Fall Internet World 98 paper. The paper was first announced around 8am on the 20th of October and about 30 minutes later, the hit rate to the server surged to well over a 100 hits/minute. This was when the author first experienced the Slashdot Effect. Soon after, around 1pm of the same day, Linux Today announced the article. One can see a slight resurgence in the hit rate about 30 minutes after the Linux Today announcement.

The plot above shows the hit rate to the same server during the time that "Preserving the Information Ecosystem" was published and announced on Linux Today and posted as a Freshmeat editorial. Linux Today announced it on November 14th, and the Freshmeat editorial was posted on Nov 21th. The announcement of this article on Linux Today was only visible for about 12 hours on its front web page, where as the article was visible on the front page of the Freshmeat web site for several days. This is an indication as to why the Freshmeat announcement generated more hits than that of the Linux Today announcement.

The final plot above shows the Slashdot Effect with respect to the last paper titled "An Ode to Richard Stallman". While the paper was being written, notification of its existence was sent to members of the LXNY group and to the general redhat mailing list. Therefore the activity to the server started to increase before the Linux news web sites announced this article. On Jan 28th, around 1pm, Linux Today announced the article and published a text only version on their web site. Slashdot followed with an announcement of the article and a hyper link to the article at around 4pm. One can see the very impressive surge in hits after the Slashdot announcement in which the hit rate went from about 30 hits/minute up to over 250 hits/minute in about a 15 minute period.

In conclusion, the term Slashdot Effect has been referenced many times on sites around the Internet. With the publication of articles related to Linux and the Open Source movement, and the announcement of these articles to Linux related news web sites, one has a chance of documenting, in a controlled environment, this effect. The plots of the hit rate received by the hosting server clearly shows the existence of the Slashdot Effect. This effect varies in magnitude for different reasons. One reason being the interest of the readership in the content of the article being announced. Another reason being the form in which the article is announced. For example, the article titled "an Ode to Richard Stallman" was announced by Slashdot and Linux Today. Linux Today published the text of the article on their own web site, thus keeping many of the hits, by readers interested in this article, on their server. This strongly dampens the surge in hit rate to the system hosting the full article. On the other hand, Slashdot announced the article via a hyper link to the server hosting the full article and thus the local host received all the hits from Slashdot readers. The purpose of this article is just to document the existence of the Slashdot Effect and not to try and gauge which news web site has the predominant reader-ship.