Adding a Geographic Element to PUE Calculations

The PUE metric has become one of the most significant metrics for measuring the gross efficiency of a data center. As data center operators boast of PUE numbers that approach the optimal rating of 1.0, it's often difficult to separate out environmental or regional factors.

Is a PUE of 1.5 in Phoenix better or worse than a PUE of 1.4 in Seattle?

It depends. In absolute numbers, the lower PUE provides an indicator of the most efficient facility. However, achieving a PUE of 1.5 in Phoenix is much more difficult than an equivalent or slightly lower number in Seattle because Phoenix is so much hotter and requires more air conditioning. Moving data centers to cooler locations helps the PUE rating, but sometimes data centers need to be located in a specific city or region. How can you compare PUE values in regions with different environmental conditions?

One possible approach is to add a geographic compensating factor:

gPUE = G * PUE

The geographic compensating factor G would be determined by The Green Grid or other trusted body based on compiled weather data. Ideally, this could be calculated empirically through a formula using data maintained by the U.S. Department of Energy (refer to this blog link for information on that data and a free tool to visually represent that data).

This approach would allow somebody to measure the technical innovation of a given facility while providing an adjustment to account for geographic disparities in temperature, wind, solar loading, etc. It's not a perfect solution (since some cooling optimizations might not work in cooler or hotter climates), but it provides some measure of equalization to facilitate more equitable comparisons between PUE claims in different locations.

--kb

How a Good Metric Could Drive Bad Behaviors

The PUE (Power Usage Effectiveness) metric from The Green Grid has become a widely referenced benchmark in the data center community, and justifiably so. However, there can be a dark side to following this metric blindly.


Introduction

PUE is defined as follows:

PUE = Total Facility Power/IT Equipment Power

Using the PUE metric, a facility manager can judge what ratio of power is lost in "overhead" (infrastructure) to operate the facility. A PUE of 1.6 to 2.0 is typical, but facility managers are striving to approach a PUE of 1.0, the idealized state.

Companies willing to drive more sustainable practices may incent facility managers to improve facility PUE levels. However, if this is done without context towards the overall energy or other resource consumption, it could drive inefficient behaviors.



Issue #1: Dissimilar Infrastructure Power Scaling

If a facility manager tracks PUE over a variety of workloads, they will see how the data center's infrastructure power consumption tracks with the IT load. Ideally, the infrastructure overhead (HVAC system, UPS system, etc.) will match linearly with the consumption of the servers and other gear in the data center, but this is rarely the case.



In many cases, the fixed overhead for power and cooling systems will become a higher percentage of overall power consumption as the IT load diminished. In other cases, there will be significant step functions in overall power consumption as large infrastructure items such as chillers, CRACs, or other equipment is turned on or off (as depicted in the graph to the left).

In such situations, reducing the IT power consumption could increase the PUE even if it reduces the overall energy consumption of the data center. People will often act in the direction towards which they are incented (i.e., what improves their paycheck). Managers incented to improve PUE without any clear tie-in to overall energy consumption might be reluctant to shut off unused servers or aggressively implement power saving features on their IT infrastructure if it increased their PUE--even if doing so would reduce overall facility power consumption.

Ensuring overall energy consumption is part of the incentive package (not just PUE) is critical to driving the desired behaviors.

[Part of this needs to be linked with overall productivity of the data center so that increased use of the data centers is encouraged while still incenting improved efficiency. I'll write about this in an upcoming post.]



Issue #2: Shifting Infrastructure Loads to IT

Another issue to watch is a desire to classify some infrastructure-like services as IT loads in order to improve PUE efficiencies. Examples of this include moving UPS systems into IT racks or putting large air-mover devices into equipment cabinets and trying to classify them as IT loads. This is "gaming" the system and should be actively discouraged.

The Green Grid is aware of this issue and is adding more guidelines to help people improve the accuracy and consistency of their PUE reporting.



Issue #3: Improving Infrastructure Efficiency at the Expense of IT

The third issue to watch is a move towards facility or equipment practices that reduce the infrastructure power consumption but increase the IT power consumption. In particular, the adoption of higher operating temperatures for data centers warrants particular scrutiny.

I've noted previously that there are significant gains possible by raising data center temperatures and making greater use of dry-side or wet-side economizers. However, it's important to compare the energy savings on the infrastructure side with the energy costs on the IT side. At higher temperatures, leakage currents in silicon increase and fans inside servers need to run faster to move more air through each server.

Increase the IT consumption and lower the infrastructure consumption and you get a two-fer: the PUE numerator goes down and the PUE denominator goes up, lowering the overall PUE. However, if the net power consumption doesn't go down, it usually** doesn't make sense to increase the ambient temperature. Once again, looking at overall power consumption in addition to PUE is important in incenting the proper behaviors.

--kb


**Note: For greenfield (new) data centers or substantial datacenter retrofits, raising the allowed data center temperature may eliminate or substantially reduce the CapEx (capital expenditure) cost for that data center even if the direct energy costs are slightly higher. For example, if a data center doesn't need to purchase a chiller unit, that could shave millions of dollars off the construction cost for a facility. In such cases, more complicated parameters will be needed to evaluate the benefits of raising the ambient temperature in the facility; these likely will include a net present value analysis for the CapEx savings vs. OpEx (operating expense) costs, consideration of real estate savings, etc. The real win is when both CapEx costs are avoided AND OpEx costs are lower.

Building Codes and Roof Anchors

One of the biggest impediments to installing rooftop solar panels or wind energy devices such as helical windmills is the cost and risk associated with properly anchoring them to a building's roof. Extra effort is required to ensure retrofit roof penetrations do not leak; in many cases, installation of the roof anchors takes the majority of the installation time. The noise from cutting through the roof can significantly disrupt those on the floor or two below.

If "rooftop renewables" are designed into a building during initial construction, the cost is substantially lower. However, it may not be feasible to install these rooftop renewables when the building is first built (due to limited capital or other reasons).

A middle ground is to provide rooftop anchors during initial construction, regardless of whether or not rooftop renewables are installed with initial construction. That way, solar or wind devices could be installed at a later date much more easily and with no need to breach the roof seal.

All new data centers should be designed for the later installation of rooftop renewables, even if they aren't part of the initial build-out.

Taking this a step further, I advocate the following: building codes should be revised to **REQUIRE** all new commercial buildings with a roof area greater than 1000 square feet to install roof anchors every x (20?) feet, with a TBD load rating for each anchor. (These anchors should also be required when major roof renovations are initiated as well.) Requiring these anchors will facilitate the broader adoption of rooftop renewables in data centers and other commercial buildings.

I hope others will adopt this cause; together we can effect real changes,
--kb

Google's Server Power Supplies

This past Wednesday, Google finally provided a peek into their data centers. Green Data Center blog has a great roundoup of various articles related to this workshop, including pictures from Google's container data centers.

One of the more interesting aspects revealed Wednesday was the fact that Google has batteries attached to each of their servers.

At first, this seems rather odd. Google's explanation for this is that they use this arrangement as a 99.9% efficient replacement for UPS (Uninterruptible Power Supply) systems. Wow...99.9% efficient!

This is definitely a different approach from what most data centers do today, and it seems really far out there--until you break it down in its component parts. A simplified block diagram looks like the following:

Broken down this way, the arrangement really starts to look like a laptop. The Google server power system apparently operates just like a laptop:

• External power supply provides ~12Vdc
• Battery is included with every computer
• When the external power supply fails, the battery provides power until the generator starts or power is switched to a different source

Graceful shutdown in power outages may or may not be an issue for Google's applications (likely not an issue).

Google certainly thinks they've got a winner with this approach, and goodness knows they've had experience deploying this at scale. In a future posting, I'll look at some of the pro's and con's of this approach.

--kb

Eliminating the UPS Efficiency Penalty with -48Vdc: Part II

In Eliminating the UPS Efficiency Penalty with -48Vdc, there is a discussion of how a non-redundant AC and DC configuration can have nearly equivalent efficiency in facilities without a UPS. However, when redundancy is figured in, the advantages of DC power become more pronounced.

Let's start by looking at the power supply unit (PSU) component by itself. Based on the information in the quantitative analysis by The Green Grid, high-efficiency AC and DC power supplies look like this when compared to each other:

The graph shifts to the right when redundant power supplies are considered. Since there are numerous different voltage converters in a server (modern servers often have in excess of 25 voltage rails used internally), it's really impractical to try to duplicate every voltage converter in a server--at least if you want it for a reasonable price. However, servers with redundant power supplies provide three principal benefits:

1. Connectivity to separate primary power sources (i.e., different utility feeds)
2. Protection against failure in upstream power equipment (i.e., failure in a PDU)
3. Cabling problem or service failure (i.e., accidentally unplugging the wrong server)

In an AC system, separate power supplies are required to have redundant feeds, since each power feed might be slightly out of phase with the other feed by the time the power signal gets to the server (relative phasing can shift in different parts of the data center based on relative cable lengths). If a server has two power supplies equally sharing the load as is commonly done, then each power supply <50%>

In contrast, a DC system has no phasing issues to deal with. Therefore, DC-based equipment has two main options: full duplicate power supplies (like AC) or using a technique called diode OR'ing (or FET OR'ing) to safely combine power from two separate DC sources as inputs to a single power supply. [Since there are numerous downstream power converters that are not redundant, there's no need for the power supply itself to be redundant--it just needs to be fed from multiple inputs.] Many DC power supplies do this today, as this approach is commonly used in the highly-reliable telecommunications system with -48Vdc systems. The result is a wider gap between the net AC power supply efficiency and the DC power supply efficiency:

Taking this a step further, look at the typical operating point for servers vs. their power supply ratings. For example, look at the various published reports for SPECpower_ssj2008: you'll notice there are numerous cases where the power supply shipped with the system is 2-4 times the maximum power draw in the sytem. If the power supply in a system is 2x the necessary power, then the system would normally operate in the left half of the graph immediately above. If the average power is considerably less than the maximum power draw, then the system could spend the bulk of its time operating at the 25% load level or less in the graph above.

At these lower loads, the efficiency benefits of -48Vdc systems become more apparent, even when there's no UPS in the picture. If an installation uses UPSes, the efficiency gap widens further in favor of -48Vdc.

Eliminating the UPS Efficiency Penalty with -48Vdc

The Green Grid recently released Quantitative Efficiency Analysis Of Power Distribution Configurations For Data Centers, which shows how different power chains from 480Vac down to 12Vdc stack up in terms of efficiency. This showed -48Vdc to have the highest efficiency for systems at 60% of capacity and below--in an idealized world.

This is true when a UPS is required--but what happens if a UPS isn't needed?

Say what? Who would ever want to deploy servers without UPS
backup?

There are certain circumstances where a UPS is not needed:

• Services with sufficient geo-redundancy that a power failure at any one site doesn't have appreciable impact on the overall service availability


• Lower-priority services for which an infrequent service outage would be acceptable

In situations like this, how does a -48Vdc system stack up? Let's look at the data in the report from The Green Grid mentioned above:

• The best AC power supplies to go from 240Vac down to 12Vdc peak out at around 93% efficiency [Figure 31].


• The best DC rectifiers (with batteries) to go from 240Vdc down to -48Vdc peak out around 96.5% efficiency [Figure 29].


• The best DC power supplies to go from -48Vdc down to 12Vdc peak out at almost 95% efficiency [Figure 31].

Taken together, the 96.5% rectifier efficiency x the 95% power supply efficiency equate to ~91.7% efficiency, slightly less than the 93% efficiency of a pure AC to 12Vdc power supply solution.

However, this is using rectifiers with tightly regulated -48Vdc outputs designed to work with batteries along with wide-ranging inputs. This is a mis-match! It's understandable why this has traditionally been done (for applications needing battery backup), but it's overkill for applications not needing battery backup.

Since most -48Vdc power supplies can handle input voltages from -42Vdc to -56Vdc (or a wider range), think what could happen with a DC rectifier with a loosely regulated output well within this range. If a DC rectifier was allowed to vary its output voltage between -44Vdc and -54Vdc, the net efficiency of the -48Vdc system could meet or beat the approach with a straight AC power supply.

Without battery backup, a -48Vdc system could match an AC system; even with full-time battery backup, the -48Vdc system is within ~1.5% of the AC system without battery backup.

Next: the story gets even better when redundancy is considered...

Using Outside Air for Data Centers

Conventional wisdom holds that it takes one Watt of cooling to remove every Watt of ICT equipment inside the data center. Today's data centers can do a bit better than that, but cooling costs remain a considerable OpEx cost for facilities.

One approach that's generating increased interest is the use of dry-side economizers, which bring in outside air to cool the data center. Using outside air saves the power that is normally used by compressors and chiller plants to cool facilities; even bigger gains may be achieved by avoiding CapEx (Capital Expense) costs by eliminating the purchase of chiller plants entirely or at least reducing CapEx costs by installing smaller cooling plants.

At first blush, this approach may seem to only be of marginal value. However, higher density data centers (such as those with blade servers) may have a relatively large temperature increase between inlet and exhaust temperature. Even if the desired inlet temperature is only 75°F, a facility with a 50°F temperature rise would have an exhaust temperature of 125°F--most ambient temperatures are well below this temperature. Bringing in outside air could take less energy than cooling the recycled air--humidity considerations notwithstanding.

To look at the impact of using outside air to cool data center equipment, several data center operators have performed small-scale tests to see how data center equipment is impacted by outside air:

• Microsoft's test: http://blogs.msdn.com/the_power_of_software/archive/2008/09/19/intense-computing-or-in-tents-computing.aspx
• Intel's test: http://www.intel.com/it/pdf/Reducing_Data_Center_Cost_with_an_Air_Economizer.pdf and http://www.datacenterknowledge.com/archives/2008/09/18/video-intels-air-side-economization-test/

Air economizers look promising, based on these results.

--kb