When did the Web surpass other TPs?

Rohit Khare (khare@w3.org)
Wed, 16 Apr 1997 14:41:05 -0400 (EDT)


So, I've been digging to recapture a certain chart Tim BL is very proud of:
the month that Web traffic surpassed gopher, email, etc on the internet
backbone traffic statistics.

[the above sentence is for the benefit of search engines, so others
will not wander through the wilderness of "no hits" or a "million
hits" like I did today]

The first really useful link, after 20 minutes futzing around
http://www.merit.edu/ and then giving up and using LiveTopics *once*
(thanks!):

http://www.fas.org/cp/netstats.htm

Which cites:

Matthew Gray's _Growth of the World Wide Web_ documented the rapid
growth of the Web from June 1993 through December 1994.

but that's an old link to a page at net.Genesis, where Matt does not appear
to work anymore. It can be found at:

http://www.mit.edu:8001/people/mkgray/net/

The hint he leaves is that between June and Dec 94, Web traffic leaped from
6% to 16% (ftp still way up there up to the demise of NSFnet, 5/95). Of
course, he links back to merit, where I thought the data was in the first
place!

and aha, we have it: in October 1994, the top dozen shows leading on bytes,
just behind on packets:

<NIC.MERIT.EDU> /nsfnet/statistics/1994/nsf-9410.highlights

NSFNET Traffic Distribution Highlights
October 1994
Packet Total: 100,133,948,600
Byte Total: 21,197,056,320,800

Service Name Port Rank Packet Count % Pkts Rank Byte Count % Byts
============ ==== ==== ============ ====== ==== ============= ======
ftp-data 20 1 19765276800 19.739 1 7030428058200 33.167
telnet 23 2 12649938000 12.633 6 972569572750 4.588
nntp 119 3 9857675150 9.844 2 2230098005750 10.521
smtp 25 4 8084691350 8.074 4 1410130927700 6.652
www 80 5 7009616350 7.000 3 2152956666650 10.157
domain 53 6 5270785150 5.264 8 518399058900 2.446
ip -4 7 3824054350 3.819 5 1302757353350 6.146
gopher 70 8 3064327550 3.060 7 864259375300 4.077
irc 6667 9 2774321000 2.771 9 304138351900 1.435
icmp -1 10 2327825100 2.325 10 240738260400 1.136
ftp 21 11 1766313150 1.764 13 134449178350 0.634
talk 517 12 1398484550 1.397 12 145212856650 0.685

But in November 1994, www trounces nntp, on the eve of an operational W3C/MIT:

<NIC.MERIT.EDU> /nsfnet/statistics/1994/nsf-9411.highlights

NSFNET Traffic Distribution Highlights
November 1994
Packet Total: 106,638,100,250
Byte Total: 22,462,439,417,900

Service Name Port Rank Packet Count % Pkts Rank Byte Count % Byts
============ ==== ==== ============ ====== ==== ============= ======
ftp-data 20 1 20460016300 19.186 1 7130364445000 31.743
telnet 23 2 12416140350 11.643 6 973506738300 4.334
www 80 3 10402528150 9.755 2 3126195255050 13.917
nntp 119 4 10110453700 9.481 3 2324060845300 10.346
smtp 25 5 8348142300 7.828 4 1380684299300 6.147
domain 53 6 5601761900 5.253 8 581492142200 2.589
ip -4 7 4265265100 4.000 5 1263445091900 5.625
gopher 70 8 3157795250 2.961 7 867043438250 3.860
irc 6667 9 2982802450 2.797 9 332665683200 1.481
icmp -1 10 2447652250 2.295 10 195121742600 0.869
ftp 21 11 1917557550 1.798 13 144001179500 0.641
talk 517 12 993553650 0.932 15 102742710650 0.457

Unfortunately, that's the end of the reliable NSFnet data, which trickles to
a halt in April 1995. At the end, www was king of the hill:

<NIC.MERIT.EDU> /nsfnet/statistics/1995/nsf-9504.highlights

NSFNET Traffic Distribution Highlights
April 1995
Packet Total: 59,201,257,750
Byte Total: 13,404,656,520,500

Service Name Port Rank Packet Count % Pkts Rank Byte Count % Byts
============ ==== ==== ============ ====== ==== ============= ======
www 80 1 12694282100 21.443 1 3518742182400 26.250
ftp-data 20 2 8301694400 14.023 2 2886742144950 21.535
nntp 119 3 4806774700 8.119 3 1160496874800 8.657
telnet 23 4 4446143400 7.510 6 339404180850 2.532
smtp 25 5 3777258050 6.380 5 620450282750 4.629
ip -4 6 3271274450 5.526 4 906851801150 6.765
domain 53 7 3176673700 5.366 7 329626925150 2.459
irc 6667 8 1315176000 2.222 9 164229990300 1.225
gopher 70 9 906084650 1.531 8 246343593050 1.838
ftp 21 10 863122100 1.458 12 60337308300 0.450

BUT, we still don't have our durned chart!!! Where is that GIF? Back to the
fas.org page we find:

* Internet Society data clearly demonstrates that the _advent of web
halted the growth of gopher_ traffic at the beginning of 1994.

but ftp://ftp.isoc.org/isoc/charts/traffic-gifs/www-byte.gif has
succumbed to link rot... but it's our most likely candidate.

So our final lead is:

* The Graphics, Visualization, & Usability [GVU] Center at the Georgia
Institute of Technology College of Computing _NSFNET Backbone
Statistics Page_ includes coverage up to April 1995, when the NSFNET
Backbone Service transitioned to a new Network Access Point
architecture, ending collection of this data.

1. World Wide Web traffic _surpassed gopher_ in volume of
traffic on the internet by the end of 1995

2. Based on these trends, _WWW was projected to surpass FTP_
as the dominant source of traffic on the internet by the end of 1995.

Inded, http://www.cc.gatech.edu/gvu/stats/NSF/Bytes.GIF shows the
falloff in Gopher growth and the 4/95 surpassing of gopher. Not the
smoking gun yet, though. Jim Pitkow did extrapolate, though, and did
predict the Nov '95 outcome correctly:
http://www.cc.gatech.edu/gvu/stats/NSF/Extrap.GIF

So, lo, we go back to the isoc, manually hunt around, and we have two
very flashy pictures of the results from Tony Rutkowski:

ftp://ftp.isoc.org/isoc/charts2/traffic/www-pcnt.gif
ftp://ftp.isoc.org/isoc/charts2/traffic/www-byte.gif

but that's still not it!!

back to Georgia Tech, whose colored plot most resemble our hazy memory of
Tim's slides...

http://www.cc.gatech.edu/gvu/stats/NSF/BothP.GIF

Well, that's an excellent "marketshare" summary of FTP, WWW, and gopher
by bytes and packets, but still not Tim's chart.

The irony is that it's somewhere within OUR OWN filespace at
/afs/w3.org, but we can't find it.

There's a moral in there somewhere...

Rohit Khare

APPENDIX: a warning from http://www.cc.gatech.edu/gvu/stats/NSF/Ports.html

Introduction

Since the statistics collected by MERIT NIC Services examine which
port the traffic is generated on, inferring the actual source of the
service can be problematic. That is, event hough Gopher servers by
default are to run on port 70, not all Gopher servers do. The same
applies for WWW and Z.39.50 servers. As a result, generalizing and
basing conclusions upon the numbers collected must take into account
the following information:

Gopher Servers:

As of April 1994, the total number of gopher servers accessible on the
Internet was 6958. However, only something like 2927 were found to be
running on port 70. These number were collected by the Veronica
harvester operated by Steven Foster and were presented at GopherCON
'94. As a result, maybe only about 42% the Gopher traffic is measured
on port 70.

Hypertext Transfer Protocol, or World-Wide Web Servers:

As of January 1994, approximately 78% of the servers being run were
being operated on port 80. These numbers were collected from _the
First WWW User Survey_(January 1994). However, based upon the port
numbers extracted from _Matthew Gray's Comprehensive HTTP Site
List_(July 1994), some 89% of the servers were operating on port
80. Additionally, _the Second WWW User Survey_ (November 1994) reports
that 87% of the participants replied that their server listened to
port 80.