SPEC Forms Power-Performance Committee

SPEC Forms Power-Performance Committee

SPEC, the group responsible for the majority of industry standard computing benchmarks such as SPECcpu2000, SPECjbb2005 and SPECweb2005, has announced that they will begin work to help formalize power measurement. A new committee, headed by Larry D. Gray, will focus on creating a set of metrics for small to medium scale servers. Existing SPEC benchmarks (most likely jAppServer, JBB, web and perhaps CPU) will be used to model typical server behavior, and power metrics will be developed using those loads.

Currently, power and energy metrics are fairly inconsistent. Sun’s marketing and PR departments advocate the bizarre SWaP metric, which is performance/(watts * rack units). Unfortunately, while performance/watt and performance/rack unit both make sense and are important to customers, combining the two together in such a way seems unlikely to correlate with real user concerns. AMD and Intel marketing materials often cite performance/watt, but nobody is willing to say how watts are measured. Is it average power draw, TDP, max power, or something else entirely? Other metrics just as peculiar have been quoted in marketing material, such as performance per 8KW rack. This has created an industry-wide three ring circus, which does not help customers much.

In many respects, the current state of affairs for measuring power, energy or compute density is similar to the same sorry state that performance measurement was circa the 1980’s, when Dhrystone MIPS were the metric of choice. Ironically, it was this dismal situation that prompted the creation of SPEC itself in 1988 and the subsequent release of SPEC CPU 89.

SPEC is currently working with the Energy Star program in the EPA and is soliciting feedback from industry, customers, academia and the rest of the government. SPEC is targeting the first quarter of 2007 for introducing new metrics. We will be looking forward to more details from SPEC on > candidate workloads and metric definitions. I have my own thoughts on the matter, which hopefully I will be able to share in the near future.