Re: Grid Benchmarking

[EMAIL PROTECTED] (Alexander Feldman) wrote in message news:<[EMAIL PROTECTED]>...
> It seems to me that one can show whatever results he/she wishes using
> an "appropriate" benchmark. To say at least benchmarking is tricky.
> Grid is a term that is quite overloaded and nowadays it seems to
> include everything from globus to privately owned inexpensive arrays
> running PVM/MPI...
> 

I agree to that Alex, the difference in the Super Computer community
is that every system has a different goal and different applications
running in it, different architectures on the nodes, different
operating systems and even in the most widely used, Linux, different
distros with different kernels and patches, and they are all tweaked
for better performance to their applications.


> My goal is to reason and to start developing a *useful* grid
> benchmarking toolkit. Unfortunately not much seemed to be published.
> What I could find as a major effort  are the NAS Grid Benchmarks
> (http://www.ipg.nasa.gov/forum/benchmark.html) which to me make sense
> besides what is written there doesn't address much of the problematic
> of measuring the performance of heterogeneous distributed systems.
> 

any bench mark would be fine as long as you post the settings and
others can duplicate them with the same systems and settings across
the world. and there are a few more benchmarks than that. you could
use LAPACK and ATLAS for example, netpipe and netperf, can also be
used.

i imagine they will run on a grid environment just as well, plus if
more and more computers are going to be added every once in a while,
do you really need a benchmarck. i don't know, you might, but it will
be a never ending job to benchmark the system.

> At this very preliminary point only several things seem clear to me:
> 
> -- A grid benchmarking can not be intrusive - i. e. we can not prevent
> other users from using the grid while measuring its performance. To
> solve that problem the benchmarking should be a process, the results
> stored in a centralized archive and statistical methods should be used
> for assessing the actual performance.
> 

my oppinion is that you will be benchmarking the grid before entering
to production, even more, before buying the system, on the planning
stage of your project, you must at least benchmarck a couple of
different architectures with your applications to see which one will
give you the best performance.

> -- A useful grid benchmarking should account for the smaller relative
> reliability of the grid infrastructure compared to a parallel machine.
> 
> -- A grid toolkit should include application and synthetic kernels as
> well as methods for measuring the peak performance of the individual
> components as well as the performance of the interconnection network.
> 
> What is *not* clear and what I hope you can help me is the following:
> 
> -- How to get comparative results for so heterogeneous environment.
> Just to mention few - MPI, PVM, Fortran, Java, SOAP, multiple OSes and
> hardware platforms;

as far as OSes and software to use i am sure you will agree that using
fortran, as the language was created for matematical computations,
would be the fastest of all languages in the processing of
information, Linux can be used for the OS per the advantages of remote
boot the OS and the minimal ammount of daemons needed to run the
software, the rest depends on the computer architecture and the
communication speed, lets say 64bit vs 32bit and Gbit ethernet vs
Quadrics, or Myricom.

Now on the side of Grids opposed to Super Computers, the goal is to
have computers all over the world making computation for smaller
machines connected to the internet or one large ethernet, the
communication speed it slowly moving towards the Gbit but only the
large computational application will get anything out of it, there is
no sense on creating a program that will utilize a few threads and
share memory with a system in Germany to run the program or the
latency will be greater that if i run it in a slow computer localy. I
don't know if there is an upside just yet to utilize many computers to
the jobs of today but as the connection speeds go into the 10 Gbits it
will be worth it. And we are not that far away

> -- How to overcome the security issues related to comprehensive
> benchmarking? In fact we can measure only representative sample of our
> grid but then we need a way to have an overview of the metrics of the
> whole grid...
> 

What security issues? using Linux as your grid you can configure all
network communication among grid nodes to go through ssh and you won't
have to worry that much about security, besides the fact that only a
small part of your application and data is in one system at a time and
the results are sent to the main computer.

> Your comments and ideas?
> 
> Regards:
> 
> -- Alex

I do believe that only big corporations will take advantage of the
grid as it is envisioned today, and will only take place within their
organizations and not at the home user level.

--Lehi