Grid Benchmarking

It seems to me that one can show whatever results he/she wishes using
an "appropriate" benchmark. To say at least benchmarking is tricky.
Grid is a term that is quite overloaded and nowadays it seems to
include everything from globus to privately owned inexpensive arrays
running PVM/MPI...

My goal is to reason and to start developing a *useful* grid
benchmarking toolkit. Unfortunately not much seemed to be published.
What I could find as a major effort  are the NAS Grid Benchmarks
(http://www.ipg.nasa.gov/forum/benchmark.html) which to me make sense
besides what is written there doesn't address much of the problematic
of measuring the performance of heterogeneous distributed systems.

At this very preliminary point only several things seem clear to me:

-- A grid benchmarking can not be intrusive - i. e. we can not prevent
other users from using the grid while measuring its performance. To
solve that problem the benchmarking should be a process, the results
stored in a centralized archive and statistical methods should be used
for assessing the actual performance.

-- A useful grid benchmarking should account for the smaller relative
reliability of the grid infrastructure compared to a parallel machine.

-- A grid toolkit should include application and synthetic kernels as
well as methods for measuring the peak performance of the individual
components as well as the performance of the interconnection network.

What is *not* clear and what I hope you can help me is the following:

-- How to get comparative results for so heterogeneous environment.
Just to mention few - MPI, PVM, Fortran, Java, SOAP, multiple OSes and
hardware platforms;
-- How to overcome the security issues related to comprehensive
benchmarking? In fact we can measure only representative sample of our
grid but then we need a way to have an overview of the metrics of the
whole grid...

Your comments and ideas?

Regards:

-- Alex