Airflow and Cooling in a Data Center

We live in a computer age. We search for information on the Internet. We reserve e-tickets for airline travel and get our boarding passes on a computer. Stocks are traded on computers and people use online banking. When we use a credit card, the transaction is instantaneously verified and approved. All major retail merchants offer online shopping. Large companies have their inventory, purchase orders, invoices, and all accounting computerized. Medical records are stored on computers. We use email and text messaging to communicate with others. The modern cell phone is a small computer that communicates with the rest of the world. The list goes on.

Whereas the visible transaction takes place at the “point of sale” on a personal computer or a small device, the whole mechanism can function only if all the relevant data is held at one place and processed at a very fast speed. Therefore, behind the small visible devices (such as a desktop computer, a laptop, or a cell phone) are large and powerful computer servers located in one place. For a credit-card or stock-trading company, it is common to have a large room (of the size $70×70m2$⁠) housing over 2000 server racks (each $1×1m2$ and 2 m tall). There are also colocation facilities, where different companies can place a few server racks each for their own use. Such a huge computer room is called a “data center.” It has become an essential part of any modern-day large business.

The most important requirement for a data center is its uninterrupted, zero-downtime operation. An interruption caused by equipment failure would entail costly repairs and replacement. But even more serious is the cost of business interruption; the business may lose thousands or even millions of dollars for every minute of downtime.

For uninterrupted operation, two things are crucial: power and cooling. Uninterrupted power is assured by having several backup sources of power that can be automatically brought on line as soon as a power failure is detected. Cooling is a more complex issue, which is discussed in this paper.

Each server rack in a data center consumes electrical energy and dissipates a large amount of heat in the range of 2–20 kW. For the electronics to function properly, it needs to be cooled and kept at an acceptable temperature level. Overheating may cause the equipment to malfunction, melt, or burn; but more commonly, safety devices on the server racks will detect high temperatures and shut down the equipment. It is this interruption that presents a serious problem for a data center and needs to be prevented.

Normally, cooling air enters a server rack through the front face and hot air exits from the rear face. In a large room, in which 2000 server racks may be spread all over the room, it is not easy to supply cooling air to each rack. This task is accomplished by a clever concept called the raised-floor data center. The server racks are installed on a tile floor that is raised 0.3–0.6 m (12–24 in.) above the real solid floor. Air-conditioners are used to pump cold air into the space below the raised floor. The floor tiles are removable and some of the solid tiles can be replaced by perforated tiles or grilles to permit the cold air to enter the above-floor space. By locating perforated tiles at the foot of the server racks, cooling air is delivered to them. The hot air then finds its way back to the air-conditioners.

The raised-floor arrangement gives unlimited flexibility. If the layout of the server racks is changed, all that is needed is to rearrange the perforated tiles so that cooling air is delivered at the new locations of the server racks. Since there is no permanent ducting, no elaborate dismantling or construction is necessary.

The raised-floor design makes it possible to create, but does not guarantee, proper cooling of the server racks. How much cold air do we need? How does it distribute in the whole data center? Are we meeting the individual demands of all server racks? Does the cold air introduced at the floor level reach to the top of the 2-m tall server rack? Can the hot air coming out of one rack enter the inlet of another rack and damage its cooling? These questions will be addressed in this paper.

Cooling a data center in its current configuration is a difficult challenge in itself. But data centers are dynamic; their equipment layout continually needs to change. Business conditions require that new server racks are installed and the old ones are removed. Typically, ten percent of the equipment in a data center is replaced each month. The cooling design has to keep pace with this frequent change.

The common practice to meet this challenge is trial-and-error and “overkill.” One detects hot spots and tries to force cold air there by placing more perforated tiles. If that fails, an extra air-conditioner is installed in the hope of removing the hot spots. It is estimated that the amount of cooling air used in most data centers is 2.5 times the required amount. So, there is a good opportunity for saving on air-conditioning equipment and energy. Whereas until now uninterrupted operation was the only focus (and data centers were ready to spend any amount of resources to achieve it), the current economic climate has shifted the focus to energy conservation. A “green data center” is a new goal in addition to the zero-downtime operation.

Manufacturers of computer servers design their equipment with a certain allowable maximum inlet temperature. This value is around $24°C(75°F)$⁠. The air-conditioners in a data center usually supply cold air at $13°C(55°F)$⁠. If the $13°C$ cold air enters the servers, there is no difficulty in satisfying the manufacturer’s criterion. As we will see later, the cold air does not always enter at all the inlet locations on the server rack. Often, the hot air exhausted by the rack finds its way to the inlet of the same rack or some other rack. This is how the cooling in a data center is compromised.

As in any complex engineering situation, the cooling in a data center has been conventionally handled by accumulated experience (including some rules of thumb), field measurements, and ad hoc design changes. A number of specialized cooling products have been introduced in recent years, but their evaluation for specific situations is costly and time-consuming. The need for a more scientific approach is quite obvious.

Cooling in a data center is an excellent application for Computational Fluid Dynamics (CFD). It offers a new paradigm for meeting the cooling challenge. One can create a computer model of the whole data center, complete with the raised floor, air-conditioning units, perforated tiles, and server racks. The CFD simulation then provides a detailed distribution of air velocity, pressure, and temperature throughout the room. The simulation can be used to analyze an existing data center, but more importantly, any proposed layout for a new or reconfigured data center. One can detect hot spots in a simulation (before they arise in reality) and explore ways of mitigating them. As already mentioned, data centers are dynamic environments; their equipment layout changes frequently. A CFD simulation provides invaluable help in planning the changes and ensuring proper cooling.

Archival literature on airflow management in data centers is rather scarce. Only in recent years, this topic has created strong scientific and practical interest. A brief overview of the available literature is given below.

Seymour (1) applied CFD analysis to show temperature and flow distributions in an atrium. Awbi and Gan (2) used CFD to predict airflow and temperature distributions within offices. Kiff (3) presented the results of a CFD analysis of rooms populated with telecommunications equipment. Bullock and Phillip (4) performed a CFD simulation for the Sistine Chapel renovation project. Schmidt (5) compared CFD results with measurements of temperature and velocity fields in an office-size data processing room. Cinato et al. (6) describe a tool to optimize the energy consumption of the environmental systems that provide cooling to telecommunication rooms. Quivey and Bailey (7) have presented results based on a CFD model for a Lawrence Livermore Data Center.

Kang et al. (8), on the basis of some CFD calculations, proposed a simplified model that assumes the whole volume under the raised floor to be at a uniform pressure. The entire flow system is then represented as a network of flow resistances. The results from this simplified model did agree well with CFD results for a particular small data center. However, as will be shown later in this paper, the assumption is not valid for most practical configurations.

In a subsequent study, Schmidt et al. (9) allowed the pressure variations in the under-floor space, but used a depth-averaged model to convert the three-dimensional problem into a two-dimensional one. For the cases they considered, a good agreement with measurements was demonstrated. However, later investigations have shown that the depth-averaged model is adequate when the height of the raised floor is small (normally less then 0.15 m or 6 in.). Practical data centers have floor heights in the range of 0.3–0.75 m. As a result, even this simplification is not acceptable and one must perform the full three-dimensional computation.

Schmidt (10) and Schmidt and Cruz (11,12) have studied the inlet temperatures to server racks in a data center. Patel et al. (13,14) describe application of CFD modeling to data centers and evaluate some specific overhead-cooling solutions. A detailed analysis of the flow in the under-floor space and the factors affecting it is presented by Karki et al. (15,16) and Patankar and Karki (17); in Ref. 16, computed results are compared with measurements of airflow through the perforated tiles. Guggari et al. (18) describe ways of optimizing data center layout. As the concerns about data center cooling became more important, Bash et al. (19) outlined the research needs in this area.

Temperature and flow rate measurements in an actual data center are reported by Schmidt (20). Later in this paper, this study will be used to provide a sample validation of CFD simulation. The issue of distributed leakage through the raised floor is addressed by Radmehr et al. (21) and Karki et al. (22); they report airflow measurements and corresponding CFD analysis. Van Gilder and Schmidt (23) consider the factors that govern the uniformity of airflow through the perforated tiles.

Whereas the full three-dimensional CFD analysis of the under-floor airflow yields useful results, a one-dimensional idealization of the flow can be used to get valuable insight. Karki and Patankar (24) have derived such a one-dimensional model and provided an analytical solution of the governing equations. The validity of the model is demonstrated with reference to a full three-dimensional CFD solution.

The interaction of the air stream emerging from a perforated tile with the internal fans in a front-to-rear server rack is analyzed by Radmehr et al. (25). They determine the conditions for which the inlet flow is essentially unaffected by the pressure variations in the inlet stream.

The purpose of this paper is to describe some interesting physical behavior in the airflow and cooling in a data center. Many physical effects are explained with the help of CFD simulation. Most of the material is taken from the papers mentioned above, where further details can be found.

A complete consideration of the flow and heat transfer in a data center would include both the under-floor and above-floor spaces. This paper initially focuses on the flow in the under-floor space and the resulting airflow rates through the perforated tiles. Although the under-floor space appears to be insignificant, it is here that the cooling battle is primarily won or lost. If we are able to deliver the required amount of airflow at the foot of each server rack, proper cooling is essentially assured. If we fail to satisfy this requirement, any attempted remedy in the above-floor space is usually ineffective.

The flow in the under-floor space is influenced by various factors such as the layout of perforated tiles, their open area, the height of the raised floor, and under-floor obstructions. A number of simple case studies are described to illustrate these effects.

The above-floor space brings its own unique behavior and surprises. These are explained through a number of examples. It is shown that the main challenge in the above-floor space is to ensure that the hot exhaust air from a server rack does not enter the inlet of the same rack or some other rack. Various strategies used for ensuring this are described.

The paper is written in an introductory style so that the readers who are not intimately familiar with data centers can get a good appreciation of the scientific and practical issues pertaining to cooling in a data center.

Figure1 shows an outline of a raised-floor data center. On the right, a down-flow air-conditioning unit is placed on the raised floor. It draws the hot air in the room into its top face and supplies cold air from its bottom into the under-floor space. Such an air-conditioner is called a computer room air-conditioner (CRAC) by the data center community. The cold under-floor air enters the above-floor space through perforated tiles that are placed at the foot of the server racks. The racks, in turn, draw in this air through their front face and exhaust hot air from the rear face. The hot air finally returns to the top of the air-conditioning unit (CRAC).

To eliminate the possibility that the hot air exhausted by one rack would enter the inlet of another rack, data centers are often laid out in the “hot aisle/cold aisle” arrangement, which is shown in Fig.2. This arrangement was suggested by Sullivan (26) and has become a standard practice in data centers. The so-called cold aisle has the perforated tiles. The server racks are placed on both sides of the cold aisle such that their inlets face the cold aisle. As a result, the exhausts from two neighboring rows of racks emerge into the hot aisle. Of course, there is no reason to place any perforated tiles in the hot aisle. The hot air collected there simply returns to the CRAC unit, without entering (hopefully) into the inlet of any server rack.

Figure3 shows a photograph of a cold aisle in a data center. The unit at the back is a CRAC. The inlets of the server racks are seen on both sides. The raised floor is made up of tiles. Many tiles in the cold aisle are perforated. It is through these tiles that cold air is supplied for the cooling of the racks.

A server rack has internal fans that draw a known amount of airflow rate. Also, as mentioned before, the cooling of the rack is designed on the basis of a maximum acceptable inlet temperature. If the actual inlet temperature exceeds this value, overheating occurs, which may lead to malfunction or automatic shutdown.

To ensure that cold air enters the rack, the airflow rate from the perforated tile at the foot of the rack has to be equal to (or greater than) the airflow rate demanded by the internal fans in the rack. Figures4,5 illustrate this concept via simplified examples. In both cases, each rack dissipates 2 kW of heat and requires an airflow rate of $0.15m3/s$⁠. In Fig. 4, the airflow supplied at the perforated tile is indeed $0.15m3/s$⁠. This meets the demand of the server rack. The cold air is supplied at $12.8°C$⁠, it heats up to $24°C$⁠, and this hot air returns to the CRAC unit. Since the temperature entering the rack is $12.8°C$⁠, which is well below the acceptable inlet temperature, proper cooling is assured.

As will be shown later, it is not always possible to control the amount of airflow emerging from each perforated tile. Figure 5 shows the case in which insufficient airflow is supplied from the perforated tile. Whereas the rack requires $0.15m3/s$⁠, the perforated tile supplies only half that amount. This reduced flow is sufficient to cool the bottom half of the rack. The cooling airflow needed by the top half is now taken from the “hot” air returning to the CRAC unit. This air has resulted from the mixing of the exhaust air at $24°C$ from the bottom half of the rack with the much hotter air emerging from the top half of the rack. The result is that the maximum inlet temperature to the rack is $35°C$⁠, which may be unacceptable for many electronics designs. (The idealized picture in Fig. 5 assumes that the exhausts of the top and bottom halves of the rack are well mixed to the temperature of $35°C$⁠. In reality, if such perfect mixing does not take place, even hotter air is likely to enter the inlet of the top half of the rack.)

These simple examples lead to a very important conclusion. The key to good cooling is to supply the required amount of cold airflow at the foot of each server rack. If this is done, satisfactory cooling is assured. If this cannot be done, cooling difficulties arise and then they are usually very hard to overcome.

Since the airflow rate emerging from each perforated tile holds the key to successful cooling, we turn our attention to what controls the distribution of airflow through the perforated tiles. Interestingly, it is not what happens above the raised floor but what happens below the raised floor that determines the flow through the perforated tiles. Thus, the fluid mechanics of the tiny (and usually invisible) space below the raised floor controls the success or failure of cooling in a data center. From a computational point of view, this is good news. If the CFD simulation is limited to the under-floor space, the calculation domain is small and a fast solution is possible. Yet, this small computational effort leads to the most valuable information needed for the cooling of the data center.

Figure6 shows the variation in the pressure drop $Δp$ with the volumetric airflow rate through the perforated tile for tiles of 6%, 11%, and 25% open area, as given by the above formula. For the case of 25% open tile, experimental data are also shown in the figure. The curve given by the formula is in good agreement with the data. (This figure assumes that the tiles are $2×2ft2$⁠, which is the common size used in the US. The size in other countries is $0.6×0.6m2$⁠, which is not very different.)

Table 1 gives, for the 25% open tile, the values of the airflow rates through the tile and the corresponding pressure drops. Practical flow rates through perforated tiles are of the order of $0.25m3/s$⁠, for which the pressure drop is about 12 Pa. That this value of the pressure drop is rather small is relevant for the discussion below.

The flow emerging from the perforated tiles originates at the CRAC unit. Usually, the flow is directed vertically downwards from one or more blowers in the CRAC. It impinges on the solid floor (usually called the subfloor), turns 90 deg, moves horizontally in the under-floor space, and emerges from the perforated tiles wherever they are placed. The CRAC blowers deliver a rated airflow; $5m3/s$ is a typical value.

Does the actual amount of airflow from a CRAC unit depend on the number of perforated tiles and their percent open area? Interestingly, for all practical purposes, a CRAC unit can be regarded as a constant-flow device that delivers that same amount of airflow rate for different numbers, layouts, and open areas of the perforated tiles.

To understand this conclusion, we need to examine the various pressure drops that the CRAC blower is required to overcome. These include the pressure drop in:

• The filter and cooling coils in the CRAC unit. (This is known as the internal static and is usually in the range 250–500 Pa.)

• The impingement and turning on the subfloor. (This is known as the external static and is usually in the range 100–200 Pa.)

• The perforated tiles (about 12 Pa, as seen above).

It is now obvious that the perforated tiles represent only a small fraction of the flow resistance experienced by the CRAC blower. Therefore, for all practical purposes, the CRAC unit gives nearly the same flow rate for different numbers, layouts, and open areas of the perforated tiles. (This is an important conclusion and it is often not understood even by experienced people in the data center field. They attempt to increase the flow rate from perforated tiles by using 40% open tiles instead of 25% open. As seen above, this action changes only the tiny pressure drop across the perforated tiles. Since the other two major pressure drops remain the same, the CRAC unit delivers essentially the same total flow rate.)

As seen in Fig. 3, different perforated tiles are at different distances from the CRAC unit. Does the airflow distribute uniformly through different perforated tiles? At first sight, we may conclude that, as we go way from the CRAC unit (which is the source of air), the airflow through the perforated tiles diminishes. Actually, the flow distributes in a surprising and counterintuitive manner. The perforated tiles that are farthest way from the CRAC unit get the largest flow. The airflow through the tiles close to the CRAC unit is quite small.

Figure7 shows the so-called maldistribution of airflow in a schematic manner. The reason for this maldistribution is also explained in the figure. The mechanism is similar to the maldistribution that occurs in manifolds (28 29). In Fig. 7, if we consider the velocity of the under-floor air in the horizontal direction, this velocity must decrease as air escapes through the perforated tiles. The Bernoulli equation would then imply that the pressure increases as we go away from the CRAC unit. The airflow rate through the perforated tiles depends on the pressure drop. Since the tiles on the right experience a greater pressure drop, they deliver more airflow. (This assumes that the pressure above the raised floor is nearly uniform. This is a valid assumption and has been verified by measurements and computation.)

The above discussion explains the maldistribution in a qualitative sense. To verify its quantitative accuracy, a simple experiment (9) was conducted at IBM Corporation in Poughkeepsie, NY. The experimental setup is shown in Fig.8. This includes a small data center with two CRACs and many perforated tiles. For the initial tests, the CRAC unit on the left was turned off. Then, only the unit on the right supplies cold air to all the perforated tiles. The airflow from each perforated tile was measured and compared with the airflow rate predicted by a CFD simulation of the under-floor space. The comparison is shown in Fig.9. The maldistribution is clearly visible. For tiles 14 and 15, which are closest to the CRAC unit, the flow is actually negative. The flow can be seen to increase rapidly for smaller tile numbers (which are further away from the CRAC unit). The agreement between the measurements and calculation is very satisfactory.

The prediction of airflow rates through the perforated tiles by CFD simulation has been validated by comparison with measurements in a number of data centers. A typical example is shown in Figs.10,11. Here Fig. 10 shows the layout of the data center, while Fig. 11 displays the comparison of calculated and measured airflow rates through the perforated tiles. These figures, which are directly taken from Ref. 16, contain “ft” as length units and “CFM” as the unit of volumetric flow rate through the perforated tiles. (If necessary, the unit conversion can be accomplished rather easily.) Once again, the computed results can be seen to agree quite well with the measured values.

As we have seen before, the flow rate through a perforated tile depends on the pressure drop across the tile, that is, the difference between the plenum pressure just below the tile and the ambient pressure above the raised floor. Pressure variations above the raised floor are generally small compared with the pressure drop across the perforated tiles. Thus, relative to the plenum, the pressure just above the perforated tiles can be assumed to be uniform. The flow rates, therefore, depend primarily on the pressure levels in the plenum, and the nonuniformity in the airflow distribution is caused by the pressure variations in the horizontal plane under the raised floor.

For the nonuniformity in the airflow distribution to be significant, the horizontal pressure variations (or change in velocity heads) must be comparable to the pressure drop across the perforated tiles. This condition is satisfied if the area available for horizontal flow in the plenum is comparable to or less than the total open area of the perforated tiles.

The key to controlling the airflow distribution is the ability to influence the pressure variation in the plenum. For specified (horizontal) floor dimensions and total flow rate, the effect of the following parameters is significant:

• plenum height

• open area of perforated tiles

Plenum height. The plenum height has a major influence on the horizontal velocity and pressure distributions in the plenum. As the plenum height increases, the velocities decrease and the pressure variations diminish, leading to a more uniform airflow distribution.

Open area of perforated tiles. As the open area of perforated tiles is reduced, the pressure drop across the tiles increases and, at some point, becomes much larger compared with the horizontal pressure differences under the raised floor. Under these conditions, all perforated tiles experience essentially the same pressure drop and the airflow distribution becomes nearly uniform.

The effect of these parameters on the airflow distribution will be illustrated with reference to the simple configuration shown in Fig.12. This layout uses the conventional hot aisle/cold aisle arrangement, with the perforated tiles placed in the cold aisles. The CRAC units are also located in the cold aisles. The server racks are arranged on both sides of the cold aisles, with their intake sides facing the cold aisles. The hot aisles are formed between the back ends of two rows of server racks. The cooling air exiting the perforated tiles is sucked in by the internal fans of the racks, heats up as it moves through the racks, and is exhausted from the back of the racks into the hot aisles. From the hot aisles, the heated air returns to the inlets of the CRAC units. Due to symmetry, only a portion of the data center around one CRAC unit needs to be considered; this portion is shown in Fig. 12.

Thus, the base configuration under consideration here consists of a CRAC unit and two rows of perforated tiles, each containing 15 tiles with 25% open area. The CRAC unit delivers $4.72m3/s$ (10,000 CFM) of cold air. The under-floor plenum height is 0.3048 m (12 in.). The tile size is $2×2ft2$⁠. The overall dimensions of this part of the data center are $36×14ft2$$(10.97×4.27m2)$⁠.

The distribution of airflow rates for this configuration is shown in Fig.13. The flow rates are smaller near the CRAC unit and increase toward the opposite wall. There is actually a reverse flow through the perforated tiles next to the CRAC unit.

Figure14 shows the velocity vectors and the pressure distribution on the horizontal plane just under the raised floor. The cold air exiting the CRAC unit impinges on the subfloor and expands horizontally. In the impingement region, the pressure levels are high, and they decrease rapidly as the air rushes out of these regions. As we move away from the CRAC unit, since the cold air is exiting the plenum, the horizontal velocity diminishes and the pressure rises. Note that the pressure under the perforated tiles next to the CRAC unit is negative (that is, below the pressure in the above-floor space) and produces a reverse flow through these tiles.

To illustrate the effect of plenum height on the airflow rates, the height for the base configuration is varied from 6 in. (0.1524 m) to 24 in. (0.6096 m).

The flow rates for different plenum heights are shown in Fig.15. It can be seen that the nonuniformity in flow rates is most pronounced for plenum height of 6 in. (0.1524 m) and diminishes as the height is increased. The intensity of reverse flow through the perforated tiles next to the CRAC unit also weakens as the plenum height is increased. Note that the curves for the plenum heights of 18 (0.4572 m) and 24 in. (0.6096 m) are almost coincident. This implies that, once the plenum height is large enough, any further increase does not affect the flow distribution. In Fig. 15, even for the 24-in. (0.6096 m) height, the flow distribution is not quite uniform. This is due to the complex flow and lateral spreading in the vicinity of the CRAC unit.

Figures16 shows the velocity vectors and pressure distribution on a horizontal plane just under the raised floor for plenum height of 6 in. (0.1524 m) For this smaller plenum height, the pressure variations are more significant (compared with Fig. 14), with an extensive region of negative pressure near the CRAC unit. Figure17 shows the same plot for plenum height of 24 in. (0.6096 m). Here, the pressure distribution is much more uniform; the variations are limited to a small region near the CRAC unit. These pressure plots explain the airflow distribution shown in Fig. 15.

To illustrate the effect of the open area of perforated tiles, the open area in the base configuration is varied from 10% to 60%.

The airflow distributions for different open areas are shown in Fig.18. For a fixed layout and plenum height, the nonuniformity in flow rates diminishes as the open area is reduced. A reduction in the open area also reduces the likelihood of reverse flow near the CRAC units. Note that there is no reverse flow for open area of 15% and 10%.

It may appear that using highly restrictive tiles (such as 10% open) is a good way of making the airflow distribution uniform. However, there is an undesirable side effect. At smaller open areas, the pressure levels in the plenum increase, and a large proportion of cold air escapes through extraneous openings on the floor, e.g., openings around cables and pipes and other leakage paths. (The flow resistance of these openings now becomes comparable to the flow resistance of the perforated tiles.) This wasted air will not be available for cooling of equipment.

A further simplification is possible by recognizing that, whereas $Ψ$ is an important parameter, the frictional parameter of $Φ$ is usually small and can be set equal to zero. Under these conditions, the governing one-dimensional differential equations can be solved to get an analytical exact solution. This solution reduces the complex problem of airflow in the under-floor space to a simple formula, which has surprisingly good validity.

While considering this simplification, we must remember that it applies to the simple configuration shown in Fig. 12. Whereas we get valuable guidance, understanding, and insight form the idealized model, it cannot be directly applied to real data centers with complex layouts, several CRAC units with intersecting flow streams, and irregular distribution of perforated tiles.

In practical data centers, the raised floor is not quite an impermeable surface as one would normally assume in an analysis. There are deliberate openings made in the tiles to bring cables to the server racks. (In fact, historically, the raised-floor arrangement was as much for bringing electrical connections as for supplying cooling air.) In addition, there are always small gaps between individual floor tiles. The leakage area due to these gaps is estimated to be 0.2% of the floor area.

In a normal data center, about 10% of the floor area is occupied by perforated tiles. If we assume that these tiles are 25% open, then the open area of the perforated tiles is 2.5% of the floor area. For this case, the leakage area mentioned above is 8% the total open area. If we reduce the number of perforated tiles on the floor, then the leakage area becomes an even larger proportion of the total open area.

If we ignore the leakage area, then we would expect that, as the number of perforated tiles decreases, the plenum pressure would increase and the average airflow rate through each perforated tile would be larger. These expectations are qualitatively in agreement with actual measurements, but quantitatively the pressure rise and the increase in airflow rate are not found to be as high. The reason for the discrepancy is the leakage area. As the number of perforated tiles decreases, the leakage area becomes comparable to the open area of the perforated tiles. As a result, the leakage airflow must be taken into account while estimating the plenum pressure and the average airflow rate through each perforated tile.

A detailed study of the effect of leakage is presented by Radmehr et al. (21) and Karki et al. (22). The study includes careful measurements performed in a data center with and without distributed leakage from the floor. The measurements are also compared with the results of CFD simulations.

In the under-floor space in a data center, usually there are many pipes, cables, structural beams, and other objects. Their presence reduces the area available for the airflow and creates nonuniformities in the pressure distribution. These are substantial effects and must be included in a CFD simulation.

In addition to the obstructions that are present for other reasons (bringing in chilled water or electrical power), we can consider placing deliberate obstructions for the purpose of controlling the airflow distribution. This is an inexpensive way of getting the desired flow distribution. This concept will be illustrated in this section.

As a demonstration of how an under-floor obstruction influences the airflow distribution through the perforated tiles, a modification to the base case is considered. As shown in Fig.19, a circular pipe of 6-in. (0.15 m) diameter is placed in the under-floor space as an obstruction. (Only the centerline of the pipe is shown in the figure. In the 12-in. height of the raised floor, the pipe is placed on the solid subfloor leaving a 6-in. clearance above it.) Fig.20 shows the resulting airflow distribution through the perforated tiles. The corresponding velocity vectors and pressure distribution just under the raised floor are displayed in Fig.21. In general, the pipe allows the pressure to build up in the upstream region. So, the flow rates close to the CRAC unit are not as small as in the base case. In the immediate upstream region of the pipe, air velocity increases (due to the blockage caused by the pipe) and pressure falls. Then, there is a further lowering of the pressure on the downstream side of the pipe. Thereafter, the pressure gradually builds up as the horizontal flow velocity diminishes. These pressure changes are reflected in the airflow distribution through the perforated tiles.

As mentioned above, deliberate obstructions can be used to control the airflow distribution. A convenient obstruction is a vertical partition placed in the under-floor space. The flow misdistribution in the base case occurs because the horizontal flow velocity deceases as the air emerges from the perforated tiles. If this velocity can be kept uniform (by deceasing the area available for the under-floor flow), then the pressure would remain uniform. Consequently, a uniform airflow distribution can be achieved through the perforated tiles.

Figure22 shows the base case modified by the placement of two inclined partitions in the under-floor space. The partitions attempt to reduce the available area for the horizontal flow in a linear fashion. However, in order to avoid intersecting the final perforated tiles, the final area at the right end is kept finite and not made zero. The resulting airflow distribution through the perforated tiles is presented in Fig.23, along with the distribution for the no-partition case. It can be seen that the use of the inclined partitions leads to a much more uniform airflow distribution compared with the base case. The only nonuniformities are in the region close to the CRAC unit (due to the rather complex flow there) and at the right end (due to our inability to make the area reduce to zero).

There is a valid criticism of the use of solid impermeable partitions that lead to closed regions in the under-floor space. For a data center with many CRAC units, the possibility of failure of one of the CRAC units is always present. In absence of solid partitions, other CRAC units would supply some of the air to the perforated tiles that are normally supplied by the failed CRAC unit. Solid partitions (such as the inclined ones used in Fig. 22) prevent such cross transfer and are likely to turn a normal failure into a catastrophic one.

From this point of view, a perforated plate can be a more suitable partition. A perforated partition does not completely stop the flow through it; however, the extra flow resistance offered by the partition can be used to discourage the flow in one place and consequently to encourage the flow in other places. The use of perforated partitions is discussed next.

The flow maldistribution in the base case delivers large airflow rates through the perforated tiles located far way from the CRAC unit. The flow distribution can be made more uniform by placing perforated partitions normal to the horizontal flow so that the flow will be somewhat discouraged from moving fast to the downstream region. This concept is shown in Fig.24 with two proposed locations for the partitions. The required percent open area of the partitions will be determined by computational experiments.

In Fig.25, the results of airflow distribution are shown for the case of the two perforated partitions set at 70% and 30% open, respectively. Also shown for reference are the results for the no-partition case. It can be seen that the use of the partitions has increased the airflow in the tiles near the CRAC unit. At each partition location, there is a drop in the airflow rate due to the pressure drop across the partition. Actually, the “discouragement” of the flow is so strong that the six tiles furthest away from the CRAC unit have a rather small flow through them. To get a uniform flow, we need to adjust the percent open area of the partitions. However, if we do desire a large airflow for tiles 1–9 and a small airflow for tiles 10–15, then the current arrangement is quite satisfactory. In general, the perforated partitions can be used to get any desired distribution of airflow rates, not just a uniform distribution.

Figure26 shows a further attempt with the partitions set at 75% and 50%. The airflow distribution has improved, but is still not quite uniform. The case in Fig.27 uses partitions at 80% and 65% open area. The resulting airflow distribution is quite uniform. In fact, except for tile 1 (which is affected by the very complex flow in the vicinity of the CRAC unit), all other tiles give nearly the same airflow.

As already mentioned above, in a data center, the goal of cooling is to ensure that the maximum inlet temperature to any server rack does not exceed the allowable inlet temperature. The purpose of measurement and/or simulation in the above-floor space is to verify whether this requirement is satisfied. The primary cause of high inlet temperatures is insufficient airflow supplied through the perforated tile at the foot of the server rack. If the airflow demand of the rack cannot be met by the perforated tile, the upper part of the rack draws in hot air. To illustrate this, a small section of a data center is shown in Fig.28, in which insufficient cooling airflow is supplied. For this situation, the calculated inlet temperatures for the racks are displayed in Fig.29. (The color scale used for plotting the temperature contours is included in Fig. 29. The same color scale is used for all the remaining figures that show a temperature distribution. For this reason, the color scale is not repeated in those figures.) Since the cooling airflow is insufficient, all racks have hot air entering the tops of their inlets. Figure30 shows where this hot air originates. It is the hot exhaust of the rack that gets recirculated into the inlet when the cooling air is insufficient. This was shown earlier in Fig. 5 via a simple example.

If sufficient cooling airflow is supplied at the foot of the server rack, usually there is no reason for unsatisfactory cooling. However, there are special locations and circumstances in which hot air can enter the inlets even when sufficient cold airflow is supplied at the perforated tiles. Some of these are described below.

Suppose that in Fig. 28, the airflow was increased to meet the demands of the racks. The corresponding calculated inlet temperatures are displayed in Fig.31. It can be seen that most of the racks in the middle have acceptable low inlet temperatures over the whole inlet face. However, higher inlet temperatures are seen for the racks near the CRAC unit and also for the racks furthest away from the CRAC unit. The behavior near the CRAC is easy to understand. As we have seen before, the usual maldistribution of airflow leads to small airflow rates at the perforated tiles near the CRAC. This insufficient cooling flow leads to the higher inlet temperatures for the racks in that region. Far away from the CRAC unit, the perforated tiles deliver the highest airflow rates. So, at first glance, the behavior of the racks in that region is difficult to understand. Figure32 shows a plot of temperature distribution on a horizontal plane near the top of the racks. Although the perforated tiles in the end region (away from the CRAC) deliver sufficient flow, the hot exhaust air finds its way around the last rack and enters the rack inlets.

One quick remedy for this effect is shown in Fig.33. Here, additional perforated tiles are placed in the end region to create an air curtain of cold air. The corresponding inlet temperatures in Fig. 33 can be seen to be an improvement over the distribution in Fig. 31. Figure34 shows how the air curtain is effective in preventing the recirculation of hot air into the rack inlets.

Normally, the racks are placed in a row in a contiguous manner. However, occasionally, there may be gaps between them. For example, in practice, gaps are created by removing a rack from a row. It is easy to see that the gaps provide additional places where the “end effects” can be observed. Hot air from the back of the racks can enter the cold aisle through the gaps and influence the inlet temperatures of the racks. An obvious remedy is to close the gaps by using impermeable plates or partitions.

The heat loads of modern server racks can be very high (10–20 kW) and the corresponding airflow demand may be of the order of $1.0m3/s$⁠. At these flow rates, air emerges from the perforated tile at a velocity of 3 m/s. When this high-velocity stream flows over the inlet face of the rack, would the cooling air enter the rack or simply flow past it? This is a valid concern. Radmehr et al. (25) considered this issue and performed a detailed analysis of the situation. It was shown that the high-velocity airflow does create a low-pressure region at the bottom of the rack. This means that the server fans in the bottom region deliver a lower flow rate compared with the uniform-pressure environment. Fortunately, this flow reduction is not large. The results in the paper indicate that, for realistic values of the flow resistance inside a server rack and for common fan curves, the flow reduction at the bottom of the rack is less than 15%.

The success of cooling in a data center depends on keeping the hot air away from the inlets of the server racks. This can be partially arranged by placing solid partitions in appropriate places. Figure35 shows some possible arrangements. In the two configurations at the back, the top or the sides of the cold aisle are closed. This would prevent hot air entering the cold aisle from the top or from the side. The arrangement in the middle tries to prevent both top and side recirculation. The arrangement at the right of the figure is an attempt to deal with the effects of a bad original layout. In this layout, the hot exhaust from one set of racks blows into the inlet faces of the next set of racks. (The hot aisle/cold aisle arrangement is not used.) The partition between the two rows of racks helps to keep the hot air away from the next set of inlets.

The use of partitions is an inexpensive way of keeping the hot air away from the inlets of the racks. An ultimate use of partitions is the arrangement known as the ceiling plenum or the use of a drop ceiling. This is described next.

Figure36 shows a schematic of the drop-ceiling arrangement, in which a false ceiling is created below the real ceiling. Above the hot aisles, vents are placed in the drop ceiling to take the hot air to the space above the drop ceiling. From this space, the hot air is ducted to the top of the CRAC units. A three-dimensional picture of the use of the drop ceiling is shown in Fig.37. This arrangement creates a near-perfect separation between the hot and cold air and ensures satisfactory cooling.

Some designs go one step further. Instead of using vents above the hot aisle to capture the hot air, exhaust ducts are attached to the racks to transport hot air to the space above the drop ceiling. This arrangement is schematically shown in Fig.38. It guarantees complete separation between hot and cold air. As long as the total amount of required airflow is supplied through the perforated tiles, proper cooling of all racks is assured. The flow distribution through the perforated tiles is no more important. The cold air can be supplied anywhere in the room; it will find its way to the inlets of the racks without getting “diluted” by any hot air.

Although the need for reliable and assured cooling has made the arrangements like drop ceiling and ducted racks quite popular, they create some concerns. Originally, the attractions of the raised-floor design were its simplicity and flexibility. One could easily move server racks to new locations and simply place perforated tiles next to them to provide cooling air. No ducting was involved. The use of a drop ceiling requires ducting of the return airflow to the CRAC units. This makes it difficult to relocate the CRACs. Moving any racks to new locations requires the relocation of the vents on the drop ceiling in addition to the movement of perforated tiles.

The use of ducted racks presents additional problems. Relocating the racks is now even more difficult. Further, the flow rate provided by the CRAC blowers should match the flow rate demanded by the internal fans in the server racks. Otherwise, the spaces above and below the drop ceiling will be at very different pressures. This will cause the CRAC blowers and the server fans to operate at abnormal conditions. If, for the same set of CRACs, one adds or removes a number of server racks, a significant disparity in flow rates will arise.

These problems can be partially handled by (i) ensuring that the total flow rate provided by the CRACs is always equal to or greater than the airflow demanded by the server fans and (ii) providing vents in the drop ceiling (in addition to the ducted returns from the racks). The excess cold airflow will pass through these vents and will help to reduce the pressure difference between the spaces above and below the drop ceiling.

The most serious concern is what happens when one or more CRAC units fail. Then the above-mentioned matching of the flow rates does not hold any more. Since the airflow exhausted by the server fans cannot be handled by the remaining CRAC units, hot air will flow from the space above the drop ceiling into the room below through the extra vents provided. This hot air can directly enter the server racks causing a catastrophic failure.

The increasing demands on data centers have led to a number of unconventional cooling solutions. These include in-row coolers, rear-door heat exchangers, and overhead-cooling units. An in-row cooler is similar to a rack and is placed in the middle of a row of racks. It draws in hot air from its back side (from the hot aisle), internally cools it, and exhausts cold air into the cold aisle. This cold air then enters the inlets of the surrounding server racks. A rear-door heat exchanger is mounted literally as a rear door for a rack. The normal hot exhaust from the rack is cooled in this heat exchanger so that the exhaust air is at an acceptably low temperature. This eliminates any hot-air streams in the data center and ensures proper cooling. There are also overhead-cooling units that draw in hot air from the hot aisle, cool it, and blow cold air downwards into the cold aisle.

Nearly all the discussion in this paper has focused on a raised-floor data center, which represents the most common design. However, there are non-raised-floor data centers used for special applications. For example, telecommunications equipment is so heavy that often it cannot be conveniently mounted on a raised floor. When there is no raised floor, the cooling air may come from upflow CRAC units (as opposed to the downflow units used in a raised-floor environment), from overhead ducts, and from the advanced cooling solutions mentioned above.

As a sample of how computed results agree with measurements, a comparison with the measurements by Schmidt (20) is shown here. Figure39 shows the layout of a data center at the National Center for Environmental Protection (NCEP), Bethesda, MD. Figure40 gives a plot of computed and measured flow rates through the perforated tiles in selected rows. Figure41 presents the comparison of computed and measured temperatures at the rack inlets at a height of 1.68 m (5.5 ft) from the floor (for selected rows). Finally, Fig.42 compares the computed and measured values of the CRAC return temperatures for the seven CRAC units in the data center. In general, the agreement can be seen to be very good.

This paper has described various fundamental and practical challenges in cooling a data center. To make a concerted effort on resolving many technical issues and to provide leadership and standardization, the Technical Committee TC 9.9 (Mission Critical Facilities, Technology Spaces, and Electronic Equipment) was established in 2002 under the auspices of American Society of Heating, Refrigerating, and Air-Conditioning Engineers (ASHRAE). The committee has representatives from over 110 organizations, which include manufacturers and users of data center equipment, consulting companies, and related agencies. The committee has made important contributions to ASHRAE handbooks and has published seven books (30,31,32,33,34,35 36) in the ASHRAE Datacom Series on topics directly related to data centers. These books represent a valuable resource for cooling and related issues in a data center.

This paper has dealt with a number of issues pertaining to the airflow and cooling in data centers. For a raised-floor data center, the flow field in the under-floor space holds the key to the distribution of airflow through the perforated tiles. If the airflow demand of each server rack is met by supplying the required airflow at the foot of the rack, proper cooling is, in general, assured. The airflow distribution through the perforated tiles is governed by the pressure variation under the raised floor. This is affected by the height of the raised floor, the locations of the CRAC units, the layout of the perforated tiles, their open area, and the presence of under-floor obstructions. Whereas some obstructions are present as a practical necessity, deliberate placement of obstructions (such as perforated partitions) can be used to influence the flow field in a desirable way.

In the above-floor space, the goal is to prevent any hot-air stream from reaching the inlets of the server racks. This is primarily achieved by supplying sufficient cold air at the perforated tiles. In addition, air curtains, partitions, drop ceiling, and ducted racks can be used to achieve a separation between the hot and cold air streams. In special circumstances, advanced cooling solutions that provide localized cooling can be used.

Cooling in a data center is a topic of enormous practical importance. It also leads to many physical, mathematical, and computational issues that are very fascinating.

The author would like to thank his colleague Dr. Kailash C. Karki for his significant help in the preparation of this paper.

$Atile$

total area of the perforated tiles

$Acs$

cross-sectional area of the plenum

$dh$

hydraulic diameter

$f$

friction factor

$F$

fractional open area for the perforated tile

$K$

pressure-loss factor

$L$

streamwise distance

$p$

pressure

$V$

velocity approaching the perforated tile

$η$

area ratio

$ρ$

density of air

$Φ$

frictional resistance parameter

$Ψ$

pressure variation parameter