Jeff Bone wrote:
> BTW, determining (for reasonable "up to date" semantics) that something is "up to date" does
> not, in fact, require byzantine agreement / strict consistency. For almost all decentralized
Those gnomes from Zurch would beg to disagree. For some strange reasons, they want their
numbers consistent, always.
> apps I've looked at, significantly weak consistency semantics suffice. The trick is to make
> sure that *groups* of artifacts --- which are generally colocated anyway --- have internal
Would you say that CPUs in a shared-memory box are colocated? I guess you would,
since they're all sitting in the same rack. Nevertheless how do you know the
memory word sitting in the cache is up to date with the other copies in other
CPU's caches? Why, you have to reach out, and to check. If your native event rate
is GEvents/s, going into MEvents/s regime means to stall, and to stall _badly_.
If you think the problem set appears only on rack scale, nope. It appears on
die scale (clock skew), it appears on global scale, especially if you're going
over GEO (>80 Mmeters total trip, you'll hear the latency in audio, and it's
a question of life and death on a game server). What salvages you is that as
a biological being you can't react to stimuli below biological chronon (depending
upon stimulus complexity), but machines don't have that problem. As you
know, currently deployed photonic systems now live very comfortably in the TBps
regime. Here a bit is few 100 microns long, and we do have lasers already which
limit their pulse duration to a few wavelength cycles.
Let's face it, knowing what happens in a remote place faster than the photon
flies is plain impossible. And if you don't have centralism (one source,
many listeners) which broadcasts, you have to have point to point communcation.
And if many things tweak "shared" information simultaneously, and consistently,
you will get crazy drawing the lightcone diagrams (especially that you can't
send to multiple places at the same time, and that there's limitation to the
locally available bandwidth, and that you can get nonderterministic signalling
loss because somebody drops a bucket in the bucket brigade chain, requiring
ACKing), and all the things you're trying to synchronize will develop a severe
case of boredom, unless they can do doing other things meanwhile.
Notice that because of store-and-forward and byzantine logic decisions required
to route a packet (which makes cut-through currently plain impossible, until we
drop the current suite of protocols, and develop simpler one from scratch) the
real speed of information propagation is a great deal lower than c.
> consistency. This does NOT necessarily result in huge amounts of traffic. Regardless of the
> real world vs. theoretical argument, even if (in the real world) you have to resort to highly
> distributed byzantine agreement, it's provable that the problem (of determining consistency) can
> be reduced to agreement on one bit; Russell could weigh in at this point, as he coauthored the
> seminal paper proving it years and years ago... (wink wink nudge nudge...) Net-net,
> replication causes *eventual* significant traffic; detecting consistency doesn't. And in a
> frequently disconnected world, app builders should strive for *eventual consistency* of
> coarse-grained objects, rather than realtime / fine-grained / strict consistency. Happily, most
> apps are written to filesystem-like semantics for propagation of persistent data over time, and
> serendipitously those filesystem semantics are in a sense "normalizable" across various
> filesystems, and give the desired versioning semantics, iff you assume the artifacts written are
> immutable.
I think that relying on absolute consistency of cached data kills your performance so
badly,
so that everybody who doesn't have that strange craving, or whoever who doesn't coredump
all over the floor in case the consistency is sometimes violated will kick your ass in
terms
of long-term fitness.
Distributed memory is just plain unphysical. Until someone learns to make recursive
spacetime laces, and in a tiny volume to boot, distributed memory will always remain
a pipe dream.
This archive was generated by hypermail 2b29 : Sun May 06 2001 - 08:04:39 PDT