DoD Skeptical About Supercomputer Benchmarks

PITTSBURGH, Pa — The Defense Department is not worshipping false gods as far as supercomputer metrics are concerned. Charles J. Holland Deputy Under Secretary of Defense for Science and Technology, said at SC2004 here that paying attention to the Top500 list of the world's fastest supercomputers is misleading for evaluating high-performance computing platforms for DoD.

"High-end computing enables the U.S. to solve 'unsolvable' problems first," said Holland. But 'faster' hardware, in isolation of the application and business context does not necessarily equate to added value".

The DoD applies the High Performance Linpack (HPL) metrics instead to make supercomputer buying decisions, said Holland. HPL not only measures speed but also accuracy of calculations within limited performance parameters.

Compounding the problem of choosing new machines is that progress in software has been moving at "glacial speed" compared to hardware in high performance computers (HPC), said Holland in an invited talk. "One challenge is end-to-end latency over time: it has not been so good for the last three years. Another is that much of written code cannot utilize an entire HPC effectively."

Holland noted also that code does not scale with processors in various architectures. In order to confront the software HPC problem Holland is applying a strategy he has used over 20 years at his earlier position at the National Security Agency: observe, orient, decide, and act mentality to find a faster path to solutions.