Suggestion for Energy Star Measurement of Blade Power Consumption

The US EPA is developing an Energy Star for Servers specification. Based on information in the latest draft of the specification, it looks like the EPA may be backing away from including servers in the first release ("Tier 1") of the server Energy Star specification. Given the increasing prevalance of blade servers in data centers, this would be unfortunate.

Ideally, there would be a standardized benchmark like SPECpower_ssj2008 that would be able to measure power consumption on a per-blade basis, but the current benchmark doesn't have provisions to handle chassis.

As an alternative, here are suggestions for how the EPA could measure power consumption for Energy Star (until a chassis-friendly industry specification is developed by an industry group like SPEC):

• Apply Energy Star to blades, not to chassis. Chassis are ineligible to meet Energy Star, but the blades that go in them can be Energy Star certified.
• Configure a chassis with the minimal amount of chassis management modules and external modules required for operation, but include all supported power supplies for a given chassis and all the fan/cooling modules typically used (don't remove redundant fans or power supplies).
• Run a sample workload on all servers to keep them minimally active. Install the same server configuration in all server slots.

Measure total power consumption to all power feeds in the chassis under two conditions and with the following calculations:

1. Condition 1: Determine power consumption P1 with all N server blade slots installed.
2. Condition 2: Remove servers so that N/2 (round up) servers are evenly distributed in the chassis; call that number N'. Determine power consumption P2 at this level.
3. P3 = P1 / N. This is the weighted average power per server blade in a full chassis.
4. P4 = P2 / N'. This is the weighted average power per server blade in a half-full chassis.
5. P5 = (P3 + P4) / 2. This is the weighted average power per server blade.

Notes:

• This accounts for chassis overhead, including fans, power supplies, management modules, and network connectivity. There is a slight penalty to blades here since rack-mount servers don't include any allocation for network switch power, but represents the minimum configuration needed to use those blades. Additionally, many vendors have low-energy networking elements (i.e., passthrough blades) that minimize this impact.
• If the chassis contains power supplies to convert input voltages to a different voltage supplied on the backplane, the power supplies used in the chassis must meet the power supply qualification requirements outlined elsewhere in the Energy Star for Servers specification.
• If a chassis contains redundant power supplies, the server blades are eligible for an allowance of 20W per redundant power supply, divided by the number of servers. For example, if a chassis has 2+2 power supplies (2 redundant power supplies and 2 minimum power supplies for a fully loaded chassis) and 10 blades, then each server would get a 4W/server allowance (2 * 20W / 10 servers).

With all the notes above, this may look to be complicated, but it's actually a fairly simple configuration that provides a close analog to how standalone rack-mount servers are tested. This could be used in the initial version ("Tier 1") of the Energy Star for Servers specification if the EPA wanted to use it.

--kb

Eliminating the UPS Efficiency Penalty with -48Vdc: Part II

In Eliminating the UPS Efficiency Penalty with -48Vdc, there is a discussion of how a non-redundant AC and DC configuration can have nearly equivalent efficiency in facilities without a UPS. However, when redundancy is figured in, the advantages of DC power become more pronounced.

Let's start by looking at the power supply unit (PSU) component by itself. Based on the information in the quantitative analysis by The Green Grid, high-efficiency AC and DC power supplies look like this when compared to each other:

The graph shifts to the right when redundant power supplies are considered. Since there are numerous different voltage converters in a server (modern servers often have in excess of 25 voltage rails used internally), it's really impractical to try to duplicate every voltage converter in a server--at least if you want it for a reasonable price. However, servers with redundant power supplies provide three principal benefits:

1. Connectivity to separate primary power sources (i.e., different utility feeds)
2. Protection against failure in upstream power equipment (i.e., failure in a PDU)
3. Cabling problem or service failure (i.e., accidentally unplugging the wrong server)

In an AC system, separate power supplies are required to have redundant feeds, since each power feed might be slightly out of phase with the other feed by the time the power signal gets to the server (relative phasing can shift in different parts of the data center based on relative cable lengths). If a server has two power supplies equally sharing the load as is commonly done, then each power supply <50%>

In contrast, a DC system has no phasing issues to deal with. Therefore, DC-based equipment has two main options: full duplicate power supplies (like AC) or using a technique called diode OR'ing (or FET OR'ing) to safely combine power from two separate DC sources as inputs to a single power supply. [Since there are numerous downstream power converters that are not redundant, there's no need for the power supply itself to be redundant--it just needs to be fed from multiple inputs.] Many DC power supplies do this today, as this approach is commonly used in the highly-reliable telecommunications system with -48Vdc systems. The result is a wider gap between the net AC power supply efficiency and the DC power supply efficiency:

Taking this a step further, look at the typical operating point for servers vs. their power supply ratings. For example, look at the various published reports for SPECpower_ssj2008: you'll notice there are numerous cases where the power supply shipped with the system is 2-4 times the maximum power draw in the sytem. If the power supply in a system is 2x the necessary power, then the system would normally operate in the left half of the graph immediately above. If the average power is considerably less than the maximum power draw, then the system could spend the bulk of its time operating at the 25% load level or less in the graph above.

At these lower loads, the efficiency benefits of -48Vdc systems become more apparent, even when there's no UPS in the picture. If an installation uses UPSes, the efficiency gap widens further in favor of -48Vdc.

Eliminating the UPS Efficiency Penalty with -48Vdc

The Green Grid recently released Quantitative Efficiency Analysis Of Power Distribution Configurations For Data Centers, which shows how different power chains from 480Vac down to 12Vdc stack up in terms of efficiency. This showed -48Vdc to have the highest efficiency for systems at 60% of capacity and below--in an idealized world.

This is true when a UPS is required--but what happens if a UPS isn't needed?

Say what? Who would ever want to deploy servers without UPS
backup?

There are certain circumstances where a UPS is not needed:

• Services with sufficient geo-redundancy that a power failure at any one site doesn't have appreciable impact on the overall service availability


• Lower-priority services for which an infrequent service outage would be acceptable

In situations like this, how does a -48Vdc system stack up? Let's look at the data in the report from The Green Grid mentioned above:

• The best AC power supplies to go from 240Vac down to 12Vdc peak out at around 93% efficiency [Figure 31].


• The best DC rectifiers (with batteries) to go from 240Vdc down to -48Vdc peak out around 96.5% efficiency [Figure 29].


• The best DC power supplies to go from -48Vdc down to 12Vdc peak out at almost 95% efficiency [Figure 31].

Taken together, the 96.5% rectifier efficiency x the 95% power supply efficiency equate to ~91.7% efficiency, slightly less than the 93% efficiency of a pure AC to 12Vdc power supply solution.

However, this is using rectifiers with tightly regulated -48Vdc outputs designed to work with batteries along with wide-ranging inputs. This is a mis-match! It's understandable why this has traditionally been done (for applications needing battery backup), but it's overkill for applications not needing battery backup.

Since most -48Vdc power supplies can handle input voltages from -42Vdc to -56Vdc (or a wider range), think what could happen with a DC rectifier with a loosely regulated output well within this range. If a DC rectifier was allowed to vary its output voltage between -44Vdc and -54Vdc, the net efficiency of the -48Vdc system could meet or beat the approach with a straight AC power supply.

Without battery backup, a -48Vdc system could match an AC system; even with full-time battery backup, the -48Vdc system is within ~1.5% of the AC system without battery backup.

Next: the story gets even better when redundancy is considered...

Sealed Containers: Reality or Myth?

One of the interesting debates for those looking at containerized data centers is whether or not containerized data centers need to be serviceable in the field. Different products on the market today take different approaches:

• The Sun Modular Datacenter (nee "Blackbox") provides front and rear access to each rack by mounting the racks sideways and using a special tool to slide racks into the center aisle for servicing.
• The Rackable ICE Cube provides front access to servers, but the setup doesn't lend itself to rear access to the servers.
• HP's Performance-Optimized Datacenter (POD) takes an alternative approach: there's a wide service aisle on the front, but you need to go outside the container to get to the back side of the racks via external doors.

Some industry notables have advocated even more drastic service changes: James Hamilton (formerly with Microsoft, now with Amazon) was one of the early proponents of containerized data centers, and he has suggested that containerized data centers could be sealed, without the need for end-users to service the hardware. The theory is that it's cheaper to leave the failed servers in the rack, up until the point that so many servers have failed that the entire container is shipped back to the vendor for replacement.

How reasonable is this?

Prior to the advent of containers, fully-configured racks (cabinets) were the largest unit of integration typically used in data centers, and these remain the highest level of integrated product used in most data centers today. How many data centers seal these integrated cabinets and never open the door to the cabinet throughout the life of the equipment in that cabinet? This is perhaps the best indicator as to whether a sealed container really matches existing practices.

We had looked at the "fail in place" model in the company where I work, but it was difficult for managers to accept that it was okay for some number of servers to be failed in a rack. As long as the cost of fixing the hardware is cheaper than the cost of buying a new server (or the equipment is under warranty), most finance people and managers want to see the servers in a rack functional.

What do you think? Do you see people keeping cabinets sealed in data centers today? Does fail in place make sense to you?

Server Cost Adders for Higher-temp Operation

Numerous industry notables, including Microsoft's Christian Belady, have been advocating the operation of data centers with higher ambient temperatures. The cost savings by reducing or eliminating cooling plant costs could yield considerable savings. But what does it take to build servers designed to operate at these higher temperatures?

As mentioned in a previous post, telecommunications equipment is typically designed to meet the NEBS standards (55°C maximum ambient). Cost adders for NEBS equipment include the following:

• Higher temperature integrated circuits (ICs). Commercial-grade ICs are generally rated to 70°C; higher ambient temperatures could force the use of extended temp components.
• Heat sink costs. Higher temperatures often drive more expensive heat sink materials (i.e., copper rather than aluminum) and more use of heat sinks on components that don't need them at lower temperatures. For example, some servers need heat spreaders on DIMMs to be rated to operate at higher temperatures.
• Corrosive gases tolerance. Telecommunications equipment generally needs to pass tests to ensure reliability in the presence of corrosive gases, including high sulfur-content air. Before dismissing this requirement, consider the case of air-side economizers: if you're bringing in outside air, do you need to worry about contaminants in the air, such as diesel exhaust from nearby trucks or from diesel generators?
• Wider humidity range. Most NEBS equipment is designed for a wider range of allowable humidity exposure than most data center equipment. The broader use of economizers might make a wider humidity range desirable for data centers.
• Flame tests. NEBS flame tests may be overkill for most data center equipment, in part because most data centers have sprinklers or other fire suppression controls (unlike telecom central offices, which do not have sprinklers).
• Shake and vibe tests. NEBS equipment generally is tested to meet seismic Zone 4 earthquake tests. These tests could just as well apply to data center equipment, but it is something beyond what most data center equipment is validated against.
• Materials selection. The use of V0-rated plastics and HF-1 or better foams in data center equipment is not necessarily a cost adder if designed in up front, but it can add appreciable expense if retrofits have to be made after-the-fact.
• Air filters. Data center equipment generally doesn't need air filters, so these can be eliminated.
• Long life. This actually encompasses two aspects: extended availability of certain components and long-life reliability. Telecom products often require the availability of the same components for 5-7 years, much longer than typical data center products. Similarly, telecom products often are designed to meet usable lifetimes that are much longer than most data center refresh cycles.

Which of these attributes are needed for equipment in data centers with higher temperatures? What other attributes are needed for higher temps?

--kb

Containerized Data Centers in Buildings

Much of the focus with containerized data centers has been on mega-facilities that can house dozens of shipping containers.

Another use case where containerized data centers could make sense is in retrofitting buildings, though it may be somewhat counter-intuitive.

Building a state-of-the art facility can take a long time, but clearing out an open space and then lifting in a container could be a much faster approach to getting an optimized facility installed in a building than trying to get it built in place. Furthermore, it's possible to replace a set of equipment the same way.

This could be used with a single container, or there could be multiple containers placed together on the same floor of an office building.

--kb

Processors for Higher Temps

Higher Temperatures for Data Centers talks about emerging environmental standards that could well lead to increasing ambient temperatures in facilities. All other things being equal, higher ambient temperatures will lead to higher component temperatures.

In many cases, the maximum processor case temperature (Tcase) is the limiting factor for how high the ambient temperature can be raised. The Tcase limit is established by the semiconductor vendor as the maximum case temperature that the chip can experience and still meet the vendor's reliability goals.

This can put a crimp in plans to use outside air for cooling. In most likely data center locations, there are occasionally warm days that would increase the inlet temperature to the servers to the point that the processor Tcase would exceed the vendor's specified ratings.

The telecommunications market has had this issue for years. NEBS-rated equipment for central offices generally has to operate at 40°C ambient temperature, but they also need to operate at 55°C for short periods (up to 96 hours at a time and up to 360 hours per year).

To address the needs of the NEBS market, Intel provides some of their processors with dual Tcase ratings: one long-term T-case rating and a second short-term Tcase rating that is 15°C higher for up to 360 hours per year.

These processors with dual Tcase ratings may be a good fit for systems in data centers that use air-side economizers.

Higher Temperatures for Data Centers

As mentioned in Using Outside Air for Data Centers, there is increasing interest in using outside air to cool data centers. At times, this will mean higher ambient temperatures inside those data centers.

A couple years ago, the TEM-led SCOPE Alliance undertook the task of identifying the relevant environmental requirements for data centers and telecommunications from several different standards groups (Telcordia, ETSI, TIA, etc.) and came up with the Environmental Profile: Central Offices and Network Data Centers, eventually published last May. It was a lot of work, but I think we developed the most comprehensive overlay of those standards that has been published to date.

Last August, ASHRAE issued its 2008 ASHRAE Environmental Guidelines for Datacom Equipment, which changed the recommended setpoints in data centers from 20-25°C to 18-27°C. This may not seem like much, but it's a step in the right direction towards better allowances for dry-side (air) economizers.

Last November, Telcordia released GR-3160, NEBS(TM) Requirements for Telecommunications Data CenterEquipment and Spaces. [I had provided information for this to Telcordia through the two years this document was in process.] Though those outside the telecom world may not be familiar with Telcordia (or its predecessor, Bellcore), the Telcordia NEBS standards have become the de facto standards for how telecommunications facilties and equipment is designed in the U.S. and many places around the world. GR-3160 is effectively Telcordia's first major foray into data center standards, a clear recognition that telecommunications carriers like AT&T, Verizon, and Qwest have a large number of data centers and a growing need to ensure maximum availability in those facilities. Among the interesting elements of GR-3160 is Telcordia's expectation for allowable maximum ambient temperatures at the air inlet to the rack-mounted equipment:

• 30°C long-term max ambient
• 40°C short-term max ambient (up to 360 hours per year)

These two upper limits provide a reasonable target that equipment manufacturers can design to, while providing enough freedom that facility designers can make maximum use of economizers.

--kb

Using T-states with Phantom Servers

One of the more obvious ways to reduce power consumption in data centers is to shut off servers that are no longer needed (often referred to as phantom servers). Unfortunately, it's often difficult to identify what servers are no longer needed. Though most well-run organizations have well thought-out procedures for determining when and how to commission servers, few organizations have comparable policies for decommissioning servers.

• Who is responsible for older servers?
• What services are dependent on that server?
• What would happen if that server went away?

If your organization has a good way for determining what servers are no longer needed, congratulations--you're one of the few. For those less sure of what servers are needed, what options are available?

Unplugging Servers

Some consultants advocate simply unplugging servers whose provenance can't be determined; those whose services are dependent on those servers will soon complain, and you can then find out from the complaining party how long that server is needed.

Unfortunately, this approach induces a failure that can be rather drastic for your data center customers. In many cases, the service outage may not be acceptable; even worse, it may not be possible to properly recover from a server that is unexpectedly shut down.

Monitoring Activity

Another approach is to simply look at processor utilization and/or network utilization. This will tell you whether a server is busy--but a busy server is not necessarily a server performing a needed service. For example, a server could be disconnected from a company's search functions but still be actively crawling a company's intranet for faster internal searches. In other cases, a system could be furiously trying to connect to other services that have been decommissioned, so the system will look busy even while it is effectively unused.

Throttling Back Phantom Candidates

Processor Power Management in Linux discusses how T-states can be controlled from Linux. On most modern systems, T-states provide a way to throttle back performance in 12.5% increments by gating processor clocks between instructions.

• Rather than shutting a server down completely, throttling the server by 50% provides a "softer" way to test whether that server is needed. This way, the server stays up and continues to provide its services--albeit at a reduced performance level.
• If there are no complaints after a week or two at the lower performance level, drop the server performance by an additional 75% (i.e., to T7 state, 87.5% throttled).

If there are no complaints when the server is 87.5% throttled, there's a good chance the server is no longer needed. Shutting the server down at this stage can be done with higher confidence (and lower chance of reversal) than just unplugging the server without this analysis.

--kb

P.S. If there are still concerns about shutting down the server at this stage, one further approach is to migrate that server to a virtual machine. However, this last step may not be needed if the performance throttling provides high enough confidence that the throttled server is no longer needed.

Re-calibrating Efficiencies vs. Utilization

The Green Grid released a Quantitative Efficiency Analysis Of Power Distribution Configurations For Data Centers document late last year, outlining the best case conversion efficiencies of various power distribution options from 480Vac, 3-phase input down to a 12Vdc intermediate voltage rail. As a member of the power task force that created this document, I can attest to the many hours spent analyzing various data sources and finding a way to present the information as accurately as possible. What we ended up with was a document showing best-case efficiencies of all components in this conversion chain, using the best data we could publish from either vendors or third-party sources.

As good as this document is, there are some obvious next steps to be done to further refine the analysis and make it more relevant to real-world conditions. Probably the biggest area that needs to be addressed is better information on load levels. When we look at a graph of efficiency vs. percentage load, it's all too easy to look at the upper part of this graph and discard the lower loads as meaningless. Unfortunately, it's in these lower load levels that most data centers tend to live. Let's look at an example:

• Suppose a data center (or a portion of a data center) is anticipated to ultimately need 800kW of actual peak capacity of critical load backed up by a UPS (uninterruptible power supply).
• That same data center will be filled up incrementally over the next three years, with the electrical load doubling each year while it is being filled.
• For high reliability, the UPS system deployed in 1+1 configuration, with the outputs routed to a common bus bar that feeds redundant PDUs (Power Distribution Units); the PDUs provide redundant feeds to chassis power supplies, which provide redundant power to the blades in the chassis.
• To meet electrical code and company policies, 20% headroom is required in the power system to handle any unanticipated loads or planning/projection oversights.

The 20% headroom on an 800kW critical load drives the requirement for a 1MW UPS. [If the next largest size UPS is 1.2MW, the UPS would start out with an extra 200kW of unusable capacity; for the purposes of this comparison, however, let's assume that there's a perfect match available with a 1MW UPS.] The 1+1 redundancy requirement means that two 1MW UPS units will be required; under normal circumstances, these will share the load between them. Thus, an 800kW load will at most drive a 40% loading of the UPS systems.

But wait, there's more! That 40% load is when all the servers, network gear, and other critical loads are simultaneously operating at their maximum power draw. This rarely, if ever, happens (though special events can cause it to happen, so you have to be able to handle those events without losing power). Under normal circumstances, the daily peak load could be 70% of that value or less. If traffic in that data center follows significant daily cycles (i.e., high demand during the work day and minimal traffic at night), power consumption in the data center could be less than 50% of the potential peak.

Utilizing 50% of that 40% maximum UPS load puts overall load in the 20% of rated capacity range. Given the projected population of the data center over time, the data center manager could find themselves with power infrastructure operating 5, 10, or 20% of rated capacity on average. All of a sudden those esoteric numbers at the left edge of the efficiency vs. load graphs start to become much more important.

My advice to facility planners is to ignore the right-hand 2/3 of the efficiency graphs. Even in a well-designed operation with redundant systems, you're likely to spend the bulk of your time at load levels less than 30% of the rated infrastructure capacity. Focus on the numbers on the lower end of the load curve.

--kb