Re: Switchless Opteron clusters

Billy <[EMAIL PROTECTED]> wrote:
> I talked to them at SC2003 in Phoenix.  They are connecting their CPU

I was there as well.

> boards together using a Mellanox InfiniBand switch chip.  They built

I had not heard that it was Mellanox, just that it was a 12 port
HT, 12 port IB chip on the motherboard.  So one HT per CPU connected
to the switch.  I don't recall any mention of 12 inline HT<->IB
converters.

Octigabay is using L1/L2 of IB, allowing them to reuse switches,
cables, and connectors.

> their own HT->IB interface for the Opterons to get to the IB fabric. 
> I don't know if it's an FPGA or an ASIC.  However their HT->IB chip
> doesn't use IB verbs, so they wrote their own MPI stack.  They claim
> better latency numbers than solutions using InfiniBand HCAs (ie the
> Mellanox HCA).  However they won't have the full gamut of protocols

A big part of this is they avoid 2 trips (or more) across the PCI-X bus.
The booth demonstation showed an MPI ping pong test showing 1.60-1.68 us.
Certainly that is a very respectable low latency.  Alas I couldn't
change the packet size for a bandwidth test.  Octigabay claimed it was
a naive mpich implementation and they expected significant improvements
with tuning.

So the octigabay is about 1/2 what Myrinet is claiming with their new
unreleased drivers and hardware (gm-2.1 + mx).  The next lowest latency
that I know of is the Quadrics Elan4, I don't have exact numbers handy,
although I vaguely that 2/3rds of the latency was due to two PCI-X busses
(corrections welcome).

-- 
Bill Broadley
Computational Science and Engineering
UC Davis