Switchless Opteron clusters

Well, I was at a talk yesterday, and a few things were clarified.

It seems that the limit of 8 CPUs is an addressing constraint in the
current design, HyperTransport pass-through  does not touch memory or 
the CPU (as one would hope) and that there is no reason that a HT
tunnel should not be a simple switch.

Therefore, if one had a very simple 1 to 3 tunnel/splitter, it would
be the ONLY special chip needed to build a 3-D torus.  If one was
attached to one HT for the negative directions and another to another
HT for the positive ones, that leaves the third HT and addressibility
for I/O etc.  I am pretty sure that this option was thought of, even
if it was very much a secondary objective.

Now, as the software and interlocking issues were studied and largely
resolved back in the transputer days, building a dedicated HPC system
along these lines would be pretty simple.  For a company with the
right skills, of course!  Obviously, due to the timing constraints,
this would have to fit in a rack and might even be limited to (say)
64 CPUs in a 'block'.

This would give an effective worst-case bandwidth for MPI_Alltoall
of 50 GB/sec, which is pretty impressive for what should be a cheap
and reliable design.

Does anyone know if any company is working on something like this?



Regards,
Nick Maclaren.