Re: Newbie storage questions... (RAID5, SANs, SCSI)

    This is really great information. I apologize for the basic questions,
but I've only been examining this stuff for the better part of one day. :)
Let me ask you a few follow ups...

> Just to put a number on things, let's say you've got a RAID 5 array of
> five drives with 5ms typical access time (pretty optimistic).  So each
> drive can do about 200 I/Os per second.  So you could sustain
> something like 1000 random reads per second (with zero writes)....

    I don't quite understand this concept. You've got five drives, each of
which can handle 200 I/Os per second. You're multiplying 5*200 to get 1000
IOPs for the array. I understand your calculation but I'm not sure why it
works as you state. In a trivial example, let's say the RAID controller is
instrutcted to read 5 bytes of data. This is considered one IO by the RAID
controller, but doesn't the RAID controller then have to issue *5* read
commands, one to each disk? My understanding of RAID (as it applies to
reading data) is that the 5 disks would always be accessed simultaneously in
order to speed up the read process. So for each IO read-request that the
RAID controller receives, it has to issue 5 IO requests, one to each drive.
So it seems that the RAID controller would *still* be limited to 200 IOPs,
regardless of how many drives on are on the array. Why is it that you say
the reality of the situation is that the RAID controller can actually handle
1000 IOPs? I don't understand.

> With extensive caching at the RAID controller you can get higher
> numbers, but on modest sized systems like the one you're discussing,
> it's almost always a better idea to put extra cache into the server
> instead of the disk subsystem.

    When you say that the extra cache should be put in the server but not on
the RAID controller or disk subsystem, what do you mean exactly? Where in
the server would I want to increase the cache?

> FC HBAs are often attached to very large disk arrays (sometimes
> thousands of drives), where the number of I/Os per second the *HBA*
> can sustain can become a limiting factor.  It's rare to see a SCSI
> RAID subsystem that can support more than a couple of dozen drives
> which puts a pretty low upper limit on the number of random I/Os per
> second.  In any event, vendors quote raw I/Os per second for FC HBAs
> because that's all they can measure, as there's no disk involved.  The
> vendor of the disk subsystem that you attach to you FC HBA will have a
> set of performance numbers, those can be compared to the SCSI RAID
> controller performance figures.

    Ah, ok... This clarifies things a bit. I think I had a fundamental
understanding of what an HBA is. So an HBA is a "host bus adapter." It lives
in the server [in a PCI slot I assume]. This HBA has no idea how many drives
are in the array. I just passes I/O requests over a 2GB/s fibre cable. It
can pass up to 40,000 of these requests/second [using the example from the
Dell site in my previous post]. At the other end of the cable is that
rack-mountable box containing all of the drives. Are you saying that the
real brains of the RAID lies within that box instead of the HBA card? So I
really need to be asking myself "how many IOPs can those drives handle"
because it's the IOPs limitation of the DRIVES, not the HBA card, that is my
bottleneck. Is this correct?

> Loosely, an I/O is a single read or write operation.[...]

    I think I understand your explanation, but again, see my first question
above. In the simpler case of doing a single *read* operation, is the single
I/O request actually morphed into X number of requests where X is the number
of drives in the array since each drive will have to be touched in order to
perform the read?

David