Meltdown Or Monopolistic
Power Play?
Picture this scenario. You run a small ISP in a major urban
center. You
begin getting phone calls from disgruntled customers who say it's
taking them
over 30 hops to get from point to point. You get notification from
your
upstream connection provider that all of your assigned IP numbers
will soon be
changed, creating havoc for your customers. The costs of these
changes,
passed on to your customers, creates a second firestorm. Major
customers
begin to look for alternative ISPs. Sprint and other major
backbone carriers
begin advertising their "stable" services as an alternative. You
lose your largest
customer.
Your system continues to run at a snail's pace _ even though you
invested in the
best equipment you could afford. Losses mount _ and you begin to
worry
about how you're going to put food on the table. You call your
regional telco to
perform loopback testing to see where the problems are. Your telco
tells you
that your system is fine _ that the problems are all upstream _
perhaps even as
high as the Sprint network. People are abandoning your ISP
business to join
Sprint, AT&T and MCI networks _ when the actual slowdowns are
occurring
on their operations. You begin to wonder if something is going on.
THE NEXT INTERNET MELTDOWN
At least one backbone provider - Sprint _ is intentionally turning
away small
ISP traffic, and others are considering similar moves. Sprint
claims that it must
"dampen" some externally generated traffic during peak load
periods to avert an
Internet meltdown on its portion of the Internet. When traffic is
turned away, it's
like being turned off of the interstate highway _ and being forced
to take back
roads to your destination. When traffic from small ISPs is
selectively
dampened, it makes you wonder if there isn't some violation of the
law taking
place. The fact is that trade laws which might have protected you
under
traditional circumstances don't pertain to the Internet.
Even if they did, the fear of losing the stability of the Internet
would probably
make regulatory officials look the other way. The Internet is
facing a major
meltdown sometime by the end of 1996, according to some members of
the
Internet Engineering Task Force (IETF), the protocol engineering and
development arm of the Internet. The IETF warns that today's
Internet is on the
verge of collapse due to routing protocol implementation problems and
increasingly burdensome routing updates.
Essentially, the Internet is on the verge of failing because: 1)
The superhighways
of the Internet are being backed-up at the intersection traffic
lights (the routers);
2) Cisco routers, which dominate the Internet backbone, are
failing to properly
handle the peak loads because of inherent protocol handling
problems; 3) A lot
of needless traffic is duplicated daily on the Net _ IP address
changes, web site
announcements, and router updates; and 4) Many smaller ISPs clog the
backbone with announcements of dial-in user router updates.
LAST YEAR, INTERNET COLLAPSE
WAS ALSO IMMINENT
Last year, a similar collapse was imminent. That potential
collapse was caused
when the NSFNet turned the operation of the Internet over to the
private
sector, and major changes in routing information caused backbone
routers to
stop forwarding packets due to overloads in peak hours. Cisco
Systems came
to the rescue by issuing a temporary fix for their routers (which
hold over 85%
of the market on the Internet backbone), and by coming up with a new
technology called route dampening, to reduce the frequency of
router updates
and maintain router control over updates.
Since then, while no major collapse has materialized, periodic
router shutdowns
across multiple backbones have been occurring with greater
frequency _
slowing the Internet to a crawl at times. See:
(www.ietf.cnri.reston.va.),
(www.internic.com) and (www.saic.com).
CISCO'S DIRTY LITTLE SECRET
AND THE NANOG STUDY
While Cisco Systems is taking the lead in saving the Internet from
imminent
collapse, and in pushing for IP renumbering and fees that would
hurt the smaller
ISPs, Cisco seems to be hiding its own dirty laundry. According to an
engineering work group report of the North American Network Operators
Group (NANOG), which includes the Internet's largest commercial
network
operators, the problems that seem to be occurring on the Internet
only happen
where Cisco routers are used. NANOG suggests that a manufacturing
problem
inherent in Cisco routers may actually be at fault. The NANOG
traffic study
(see http://www.academ.com/nanog) found that:
Instability in Internet traffic is increasing despite use of
dampening.
There is a strong correlation between routing instability and
network
congestion. Daily incidents suggest that instability peaks occur
between 6:00 and 8:00 a.m., and there is an abrupt drop at
5:00 p.m.
An enormous number of routing update withdrawal announcements
are routed each morning. The study found that most such updates
are redundant or unnecessary, often done merely to advertise
the announced sites or to eliminate performance pressure
on specific routers;
all reported problems, so far, can be traced to Cisco routers
and configuration errors.
Use of non-Cisco routers is discouraged by the largest ISPs,
even though other routers do not appear to have the same kind
of routing problems.
A problem that Cisco routers have with the Internet Protocol
itself is causing router update oscillation, link/router
failures and
congestion.
Cisco routers cannot keep up with processing of the
announcements _ and rely on dampening technology to
continue operating during peak traffic periods.
SPRINT BLOCKS OUT SMALL ISPs
U.S. Sprint decided to intentionally block certain users'
communications that
originate from small ISPs as a result of router traffic problems.
Sprint says it is
being forced to block communications because of overcrowding on its
network. MCI and AT&T and the Regional Bells are weighing similar
moves.
For details see: http://ftp.sprintlink.net and
http://nic.merit.edu/mail.archives/html/nanog/threads/html.
The decision by Sprint to block out part of its routed traffic _
electronic mail,
file transfers, and visits to World Wide Web sites by users who
are passing
through Sprint from small ISPs _ in a medium where everybody is
supposed to
be linked to everyone else, is one possible alternative for the
future.
Sprint is being applauded by some members of the IETF and by Cisco
Systems
for being the first ISP to take action to begin to eliminate
address crowding. It
is estimated that over 10,000 transactions are blocked in the
national Sprint
network every day, and that in order to keep Sprint's Cisco
routers working
properly, the number of communications turned back is growing daily.
According to Sean Doran, a Sprint engineer who explained the
Sprint filtering
process to NANOG:
"Very roughly speaking, the goal (of Sprint's filtering) is that
if you have a very
long prefix and it flaps a couple times, you can go home for the
day...." (Your
messages are blocked.)
"At the moment, you are much more likely to be dampened if you are
NOT a
Sprint customer. That is, the normal values for 'bgp dampening'
are in place if
you have a customer connection to SprintLink or ICM. However, we
will be
evaluating the engineering and operational effects of a similar
policy with respect
to SprintLink customers, as an effort to encourage some of our
lazier customers
to begin doing their own (IP renumbering) aggregation wherever
possible."
According to Sprint, their problems stem from the recent
exponential growth of
the Internet and the growing number of companies that attempt to
do major
things with only temporary dial-in connections. Sprint explains
that the routing
tables of the many linked networks that are passing through Sprint
have
become too complex for most of its routers to handle. Their chief
concern,
according to a Sprint spokesman at the Dallas office, is that
things are just going
to continue to worsen. Sprint counters its critics by saying that
its first and
foremost responsibility is to preserve network stability for its
own customers.
Most IETF members tend to agree with the Sprint approach. If you
are a small
ISP, that means trouble.
PUNITIVE "SOLUTION" POSED FOR SMALL ISPs
Cisco Systems' Yakov Rekhter thinks he also has a solution to the
routing
collapse problem. He proposed a major overhaul of the routing
procedures at
the IETF meetings held in Dallas and Los Angeles this spring. The
proposal
called for IP renumbering according to a hierarchical aggregation
scheme in
accordance with an IETF proposal made in 1992.
Rekhter's approach is now close to adoption by the Internet Assigned
Numbers Authority (IANA), the central coordinator for the
assignment of
unique parameter values for Internet protocols. It would pose
little or no
hardship for the largest ISPs. In fact, it conserves their address
space, even if
they aren't using it.
But the proposal makes it harder for new ISPs to obtain address
blocks,
increasing the likelihood that small service providers will have
to face constant
IP renumbering. It also shores up the largest ISPs' ability to
block out IP
addresses. It calls for the "immediate renumbering" of all smaller
ISPs and their
customers _ and would pass on the costs of all renumbering to
those smaller
ISPs and their customers.
Rekhter also wants IANA to consider charging fees for any IP
address changes
_ and for advertising all router updates on the Internet. The
proposal does not
suggest who should receive the income from these fees.
Rekhter's proposal drew broad support from IETF membership, which is
dominated by router vendors and large ISPs. Pushpendra Mohta of
CERFnet,
for example, believes the best approach is to limit such charges
to those ISPs
who generate the most changes in router tables _ a process known
as flapping,
which tends to bring down routers. The largest backbone providers
tend to
generate the most flaps because of the extensive traffic they
carry. But small
ISPs also generate many flaps, because they often advertise the
status of
temporary dial-up links.
LARGE PROVIDERS BLAME SMALL ISPs
The large providers single out small inexperienced ISPs as the
major cause of
problems on the Internet. Some of the more charitable critics
suggest that
novice ISPs need to be educated on the damages they cause when they
advertise to the entire world that a dial-up connection is up or
down. Others
would like to see the total elimination of small ISPs _ or at
least their
aggregation into a larger ISP's camp.
The traffic problem is causing systemic changes in the Internet
architecture that
may bode very badly for the future of small ISPs. To eliminate
stress on their
own router platforms, many of the major backbone ISPs now peer
only with
providers of a similar size. These moves make it easier for the
backbone ISPs
to police their own downstream providers by aggregating their
traffic and
dampening, or delaying disproportionate updates. When dampening
occurs, it
slows downstream traffic to a crawl until the backlog of updates
have been
handled by the backbone router. These delays are immediately
apparent to
customers of the small ISPs _ who often fill the Internet with
flames and
complaints about the small ISP's inability to provide competent
services.
REKHTER'S PROPOSAL IS ANOTHER
INTERNIC FIASCO
As Rekhter put it in his proposal: "There are greedy ISPs and
incompetent
ISPs. Forcing these ISPs and their users to renumber every time
they change
service providers _ and charging them fees for all the IP changes
and router
updates will certainly motivate them and their users to lock their
organizations
into a single provider or face the expensive and complicated task
of total
network renumbering. The recent change at the INTERNIC on charging
for the
domain name seems to work. Should the route charging mechanism be
similarly
done?" Apparently, Rekhter isn't aware of all of the Domain Name
filing
problems InterNIC is facing.
The problem with Rekhter's notion is that the largest ISPs have
large blocks of
IP numbers allocated to them, while the small ISPs don't. Large
ISPs will often
be able to renumber within their network _ thus avoiding
additional fees.
Smaller ISPs will face higher operational costs and start-up
costs, creating
major barriers to market entry.
CONSUMERS WILL HAVE FEWER CHOICES
The proposed IP changes will have far-reaching implications.
According to
Jeffrey Schiller, MIT's network manager, "(When IP renumbering
schemes are
adopted), users will be locked into their service provider, who
can rake them
over the coals and jerk them around, and in order to change
providers they will
have to go through a painful and costly renumbering process." Larger
companies will definitely begin to shy away from small ISPs. This
will hurt most
ISPs' profits _ and perhaps run many out of business.
An AT&T study estimated that for an average company of 500
employees, the
cost of IP renumbering could exceed $100,000. Private networks
like Silicon
Graphics' 150 sites and 7,000 hosts will spend anywhere from
$700,000. to
over $2 million to renumber. Will this create an incentive for
users to go directly
to the large ISPs rather than risk facing major costs every time a
small ISP
shops for a better rate, or changes are made by providers upstream?
As if the costs of all these changes on the small ISP won't hurt
business enough,
the proposals also provide an exemption for non-profit providers
at the
expense of the for-profit providers. They call their solution
"address lending."
Certain addresses will be set aside for institutional non-profits
who cannot
afford to implement the renumbering process. These addresses, of
course, will
be maintained on the large ISPs, and part of the fees collected
from the
for-profit small ISPs will go toward maintaining addresses for the
non-profits.
What are the best solutions? IETF has been mulling over this for a
few years
now with no new alternatives imminent. In any case, they'd like to
hear from
you _ especially if you are an ISP. They are currently soliciting
comments on
the potential router collapse, and on the Rekhter proposals for
charging for IP
addressing and router updates. You can make your comments known at
http://www.ietf.cnri.reston.va.us.
If you are an ISP, immediately explain to your customers what is
happening,
and what might potentially happen. Ask them what they would
appreciate from
you to ensure their loyalty remains with you. Consider
establishing ISP co-ops
with other regional ISPs in your area _ and consider joining legal
groups and
associations that can help you negotiate better with the large
ISPs on the issue
of IP renumbering and who will pay.
What is for certain, is that the Internet customer understands
even less about
how the Internet works than does the small ISP. However, customers
are very
quick to observe sluggish operations (which as a result of reading
this article _
some may hopefully see _ is not necessarily caused by the small
ISP). And,
customers are fickle when it comes to having such sluggish
services _ or being
forced to increase renewal rates to offset new costs for
operations and fees. IP
renumbering and associated fees will provide disparate pricing
advantages to
the larger ISPs. Even the advantages of small ISP customer service
cannot
overcome a decision by backbone operators like Sprint to turn back
communications. The power is in owning the IP address blocks, and
unless you
act now to be heard by those who might do something to help you,
the fight to
build a business on the Internet could abruptly come to a painful
end.
Editor: Jack Rickard - Volume X: Issue 9 - ISSN:1054-2760 -
September 1996
Copyright 1996 Jack Rickard - ALL RIGHTS RESERVED
Fable Of Contents