What is Occam's Razor wrt Compression?

I Find Karma (adam@cs.caltech.edu)
Fri, 20 Feb 1998 06:37:21 -0800


I wonder if 50% of the bits stored in the entire universe are written as
0, 47% are 1, and 3% cannot be read. Makes you think about compression...
if not compassion...

If YML is the minimum useful markup language, where do you put the
compression in YML?

Philosophical problem: when do you stop compression?
First there was: compress a file. (e.g., gzip or jpeg)
Then: compress a disk. (e.g., Stac DoubleSpace)
Then: compress a wire. (e.g., compressing modems)
Then: compress in protocols. (e.g., HTTP gzip content encoding)
How about compression in RAM? (e.g., RAM bus optimizers)
Should internal computer bus use compression? (e.g., PCI)

Pretty much the only place not to use compression is sending
instructions to the microprocessor. But even HP Artist Graphics cards'
videoRAM uses pixel video compression... Occam's Razor applied here
says, where is the best place to put compression? It's more efficient
to put compression per file type than per disk, etc.

Occam. Ockham. I guess it can be spelled "Occam" or "Ockham" ...

http://hep3.phys.sinica.edu.tw/physics-faq/occam.html

[Physics FAQ] - [Copyright]
original by Phil Gibbs 17-September 1996

What is Occam's Razor?

Occam's (or Ockham's) razor is a principle attributed to the 14th
century logician and Franciscan monk William of Occam. Ockham was the
village in the English county of Surrey where he was born.

The principle states that "Entities should not be multiplied
unnecessarily." Sometimes it is quoted in one of it's original Latin
forms to give it authenticity.

"Pluralitas non est ponenda sine neccesitate"
"Frustra fit per plura quod potest fieri per pauciora"
"Entia non sunt multiplicanda praeter necessitatem"

In fact, only the first two of these forms appear in his surviving works
and the third was written by a later scholar. William used the principle
to justify many conclusions including the statement that "God's
existence can not be deduced by reason alone." That one didn't make him
very popular with the Pope.

The most useful statement of the principle for scientists is,

"When you have two competing theories which make exactly the same
predictions, the one that is simpler is the better."

In physics we use the razor to cut away metaphysical concepts. The
canonical example is Einstein's theory of special relativity compared
with Lorentz's theory that ruler's contract and clocks slow down when in
motion through the Ether. Einstein's equations for transforming
space-time are the same as Lorentz's equations for transforming rulers
and clocks, but Einstein recognised that the Ether could not be detected
according to the equations of Lorentz and Maxwell. By Occam's razor it
had to be eliminated.

The principle has also been used to justify uncertainty in quantum
mechanics. Heisenberg deduced his uncertainty principle from the quantum
nature of light and the effect of measurement.

Stephen Hawking explains in A Brief History of Time:

"We could still imagine that there is a set of laws that determines
events completely for some supernatural being, who could observe the
present state of the universe without disturbing it. However, such
models of the universe are not of much interest to us mortals. It seems
better to employ the principle known as Occam's razor and cut out all
the features of the theory which cannot be observed."

But uncertainty and the non-existence of the ether can not be deduced
from Occam's Razor alone. It can separate two theories which make the
same predictions but does not rule out other theories which might make a
different prediction. Empirical evidence is also required and Occam
himself argued for empiricism, not against it.

Ernst Mach advocated a version of Occam's razor which he called the
Principle of Economy, stating that "Scientists must use the simplest
means of arriving at their results and exclude everything not perceived
by the senses." Taken to its logical conclusion this philosophy becomes
positivism, the belief that what cannot be observed does not exist. Mach
influenced Einstein when he argued that space and time are not absolute
but he also applied positivism to molecules. Mach and his followers
claimed that molecules were metaphysical because they were too small to
detect directly. This was despite the success the molecular theory had
in explaining chemical reactions and thermodynamics. It is ironic that
while applying the principle of economy to throw out the concept of the
ether and an absolute rest frame, Einstein published almost
simultaneously a paper on Brownian motion which confirmed the reality of
molecules and thus dealt a blow against the use of positivism. The moral
of this story is that Occam's razor should not be wielded blindly.

Occam's razor, also known as the law of parsimony, or the law of
simplicity is often quoted in stronger forms as in the following
statements...

"If you have two theories which both explain the observed facts then you
should use the simplest until more evidence comes along"

"The simplest explanation for some phenomenon is more likely to be
accurate than more complicated explanations."

"If you have two equally likely solutions to a problem, pick the
simplest."

"The explanation requiring the fewest assumptions is most likely to be
correct."

... or in the only form which takes its own advice...
"Keep things simple!"

Notice how the principle has strengthened in these forms. To begin with
we used Occam's razor to separate theories which would predict the same
result for all experiments. Now we are using it to choose between
theories which make different predictions. Should we not test those
predictions instead? Obviously we should eventually, but suppose we are
at an early stage and are not yet ready to do the experiments. We are
just looking for guidance in developing a theory.

This principle goes back at least as far as Aristotle who wrote "Nature
operates in the shortest way possible." Aristotle went too far in
believing that experiment and observation were unnecessary. The
principle of simplicity works as a heuristic rule-of-thumb but some
people quote it as if it is an axiom of physics. It is not. It can work
well in philosophy or particle physics, but less often so in cosmology
or psychology, where things usually turn out to be more complicated than
you ever expected.

Simplicity is subjective and the universe does not always have the same
ideas about simplicity as we do. Successful theorists often speak of
symmetry and beauty as well as simplicity. in 1939 Paul Dirac wrote,

"The research worker, in his effort to express the fundamental laws of
Nature in mathematical form should strive mainly for mathematical
beauty. It often happens that the requirements of simplicity and beauty
are the same, but where they clash the latter must take precedence"

The law of parsimony is no substitute for insight, logic and the
scientific method. It should never be relied upon to make or defend a
conclusion. As arbiters of correctness only logical consistency and
empirical evidence are absolute. Dirac was very successful with his
method. He constructed the relativistic field equation for the electron
and used it to predict the positron. But he was not suggesting that
physics should be based on mathematical beauty alone. He fully
appreciated the need for experimental verification.

The final word falls to Einstein, himself a master of the quotable one
liner. He warned,

"Make your theory as simple as possible, but no simpler."

and,

"For every complex question there is a simple and wrong solution."

References:

W.M. Throburn. Occam's razor Mind 297-288, 1915

W.M. Throburn. The Myth of Occam's razor Mind 345-353, 1918

Stephen Hawking. A Brief History of Time.

----
adam@cs.caltech.edu

Bitmaps aren't scalable, right? You can't do anything with them.
They're nonsensical. Bitmaps are not graphics; they're the display
result of graphics. You can't express graphics in dots, and a bitmap
does not have a metric. It has no meaning.
-- Robert Cailliau