Re: 1teraflops cell processor possible?

> >As it is, I can think of no more eloquent and succinct statement than
> >the one that has already been made: bandwidth is expensive, arithmetic
> >is free.
> 
> But that's not true.
> 
> Bandwidth x distance is expensive.  Lots of bandwidth over
> short distance is not very expensive.  Lots of bandwidth across
> small distances on a modern chip is very very very cheap.

As a zeroth-order approximation, I would say it _is_ true. You added
the first-order and possibly second-order corrections.

I'm sure everybody has seen a cross-section of a mammalian - say, a human
- brain. You may remember that it consists of a darker material - called
the gray substance in anatomy - surrounded by a thin border of a lighter
material, which is called the white substance. This border is only about
2-3 millimeters thick. 

Now, _all_ of the grey substance is long-distance wiring - George's "going
off-chip to the neighbors on a circuit board". And this long-distance wiring
is a special invention of higher animals, basically a coax cable with
integrated interspersed amplification devices. This increases signal speed
from about 1 m/s to around 100 m/s, at the cost of losing density (I would
estimate about a factor of 30-100) and the cost of the amplifiers (energy,
homostasis etc). 

So that saying about thinking with grey cells is actually quite off the 
mark...the grey cells are in fact the dielectric of the coax cables!

The small border of white substance does contain the arithmetic devices -
the neuron cell bodies (yes, even the nucleus is involved in that, e.g., 
in memory, by controlling synthesis of proteins) and the dendritic trees.
However, a large part of even that volume is wiring - the axonal trees,
which correspond somewhat to George's "getting very good average neighbor 
to neighbor bandwidth". These are naked wires running at those slow 1 m/s,
but they are short, at most a few millimeters. (Remember this is an
asynchronous distributed machine running at an approximate cycle time of
the order of 100-200 Hz.) And because they are naked, you can - and the
brain does - pack a lot of them into a small volume. And it does so in 
three dimensions.

In fact, the human brain is always on the border of equilibrium, due to 
the dense packing in both white and grey substance, with regard to its
operating conditions (temperature, electrolytic balance, etc.), and can
destroy itself through too much activity (cf. epileptic seizures).

Physical damage to this system - e.g., through a stroke - occurs more to
the wiring than to the neurons. Due to the fact that the wiring is,
histologically, part of the neurons which don't like their axons being
cut, damage to the grey substance also leads to some consequential damage
(usually quite distributed) of the white substance.

So that impressive computing device, the mammalian brain, does consist
mainly of wiring, but the arithmetic is not quite for free - for one thing,
it must be carefully balanced locally and globally to avoid running out of
equilibrium. (There's a hypothesis that you can't have a brain much larger
than the human one because you cannot reliably cool it.)

        Jan