Google Translate

Worldwide | United Kingdom

Best Practices for Data Centre Infrastructure Design

Cabling Distances and Space Planning

Best Practices for Data Centre Infrastructure Design

When designing and laying out a data centre, understanding best practices as well the pros and cons for each type of data centre is critical. The TIA 942 data centre guidelines are very specific that horizontal and vertical cabling should be run accommodating growth, so that these areas do not have to be revisited. It is also specific about equipment not being directly connected unless it is specifically required by the manufacturer. This is inline with other standards documents such as ANSI/TIA/EIA 568-B that design for opens systems architecture. So the question is raised: what is the best way to do this for a 10Gb/s environment?

There are considerations outside of the cable plant and number of connectors alone: usability, scalability, costs and the ability to perform Moves, Adds and Changes (MAC’s). Additionally, some limitations exist based on the category of the cabling system. Copper and fibre distances may vary with the type of cabling system selected. We will discuss some of those parameters and their potential impact on data centre designs.

All copper channels are based on a worst case, 100 meter, 4 connector model. ISO/IEC 24764 (draft), TIA- 942, ISO/IEC 11801 Ed2.0 and recommendations from electronics manufacturers suggest that the fixed horizontal portion of the channel be a minimum of 15m (50 ft.). While some shorter lengths may be supported in other portions of the channels, there is a requirement in zone distribution and consolidation points for this minimum distance. When moving to 10Gb/s electronics, the 15m minimum will likely exist for all horizontal cables due to recommendations from electronics manufacturers and the fact that all models within IEEE are based on a minimum 15m distance.

The 15m length is also dictated by signal strength issues, as your signal is strongest in those first 15m which can create issues with two connectors in close proximity. By providing at least 15m to the first connection point in the channel, you are allowing the attenuation to reduce the signal strength at the receiver or between components. In order to achieve the 15m distance, two options exist: either provide space in the pathway to take up the distance or create service loops under the floor. Service loops should not be a loop, but rather a loosely configured figure 8 for UTP systems, however this configuration is not a requirement for F/FUTP or F/STP systems. Bear in mind that the additional cable will consume more pathway space.

Copper distances for category 6A twisted pair cabling are limited to 100m for all channels with the exception of 10GBASE-T running on category 6/class E cabling. The distance for these channels will be limited to less than 37m depending upon the scope of potential mitigation practices to control alien crosstalk. It should be noted that the purpose of TSB 155 is to provide parameters for the qualification of existing Cat 6/Class E applications for use of 10GbaseT, TSB 155 should not be used for designing new installations.

Fibre channel lengths vary based on the grade and type of fibre and type of interface. Understanding these limitations will assist in the design and layout of the data centre space. If you are utilising 10GBASE-CX4 or Infiniband, you are distance limited to a maximum of 15m. The following chart summarises the distances for all 10G applications and their associated cabling systems.

Application Media Classification Max. Distance Wavelength
10GBASE-T Twisted Pair Copper Category 6/Class E UTP up to 55m*
10GBASE-T Twisted Pair Copper Category 6A/Class EA UTP 100m
10GBASE-T Twisted Pair Copper Category 6A/Class EA F/UTP 100m
10GBASE-T Twisted Pair Copper Class F/Class FA 100m
10GBASE-CX4 Manufactured N/A 10-15m
10GBASE-SX 62.5 MMF 160/500 28m 850nm
10GBASE-SX 62.5 MMF 200/500 28m 850nm
10GBASE-SX 50 MMF 500/500 86m 850nm
10GBASE-SX 50 MMF 2000/500 300m 850nm
10GBASE-LX SMF 10km 1310nm
10GBASE-EX SMF 40km 1550nm
10GBASE-LRM All MMF 220m 1300nm
10GBASE-LX4 All MMF 300m 1310nm
10GBASE-LX4 SMF 10km 1310nm

* As defined in 802.3an

THE LAYOUT...WHERE AND HOW TO CONNECT

When designing a cabling infrastructure, to often cost is the deciding characteristic of the channel selected. However, once all elements are considered, a design with higher initial cost may have a lower overall cost of ownership to a company that has a lot of MAC activity. The most important concern is that designers are familiar with all aspects of the different configurations available to make the best selection possible. A listing of cost, flexibility and performance is listed below.

Model Cost Flexibility Performance
2-Connector Lowest Lowest Highest
3-Connector with CP Medium Medium Medium
3-Connector with CC Medium Medium Medium
4-Connector Highest Highest Lowest

SPACE PLANNING OPTIONS

The MDA (Main Distribution Area) is considered the core of the data centre, connectivity will be needed to support the HDA (Horizontal Distribution Area). Following TIA-942 recommendations and utilising EDA’s (Equipment Distribution Areas) and ZDA’s(Zone Distribution Areas) we would like to present four design options for consideration.

OPTION ONE

Option One is to run all fibres and copper from the core horizontal distribution areas and equipment distribution areas to a central patching area. This provides one central area for patching all channels.

There are several benefits to this design. First, all cabinets can remain locked. As patching is done in a central area — there is no need to enter a cabinet at any time unless there is an actual hardware change. For industries that are governed by compliance and security related issues, this may provide a greater benefit by reducing physical access to connections. Intelligent patching can be added to the patching field to increase security by automatically monitoring and tracking moves, adds and changes in that environment.

Option 1

Provides Any to All connectivity. Patch cord changes in the patching area can connect any device to any device.

Option 1

Note: Black lines are Fibre, Blue lines are Copper

Another advantage is that all ports purchased for active gear can be utilised. With the ability to use VLANs, networks can be segmented as needed.

In other scenarios, entire switch blades are likely dedicated to a cabinet of servers. However, if there are insufficient server NICs to utilise all ports, then the idle ports become costly inefficient. For instance, if a 48 port blade was dedicated to a cabinet at location XY12, but there was only 6 servers with two connections each, then 36 ports were paid for and maintenance is being paid on those ports to remain idle. By utilising a central patching field, the additional 36 ports can be used as needed elsewhere in the network thereby lowering equipment and maintenance costs which are far more expensive than the cable channels.

OPTION TWO

Option Two is to place patch panels in server cabinets that correspond directly to their counterparts in the switch cabinets. In this scenario, switch blades/ports will be dedicated to server cabinets. This may be easier from a networking perspective, but may not provide the best usage of all ports in the active electronics. Extra ports can be used as spares or simply for future growth. However, if an enterprise is planning to implement blade technology where server density may decrease per cabinet, this may not be a cost effective option.

For the switch cabinets, the type of copper cabling chosen will be a significant factor due to the increased UTP cable diameters required to support 10GBASE-T. In reality, cabinets and cabling (both copper and fibre) are changed far less frequently than the active electronics. But with the new category 6A UTP cable‘s maximum diameter of 9.1mm (0.354 in.), pathways within the cabinets may not provide enough room to route cable and still provide the structural stability necessary. It is always recommended that percent fill calculations be addressed with the cabinet manufacturer. Moving the patch panels to adjacent locations or implementing a lower switch density may be required. While moving switches into open racks with adjacent patch panels provides a solution, this is only recommended if proper access security processes exist and some form of intelligent patching or other monitoring system is used so that network administrators can be notified immediately of any attempt to access switch ports.

Option 2

One to One patching for each port. Least flexibility

Option 2

Note: Black lines are Fibre, Blue lines are Copper

OPTION THREE

Option Three consists of providing consolidation points for connections. These can be either connecting blocks or patch panels. This allows for a zoned cabling approach, but may lead to higher moves, adds and changes costs. It is also difficult to design within the parameters of a 4 connector channel when using Zone distribution.

The other disadvantage to the consolidation point model is that the changes take more time than swapping a patch cord if the pair count changes. Depending on the location of the consolidation point, there may be additional risks from loss of static pressure under the floor when removing floor tiles ending up with more than 4 connectors in a channel, or harming existing channels during changes.

Option 3

Consolidation Points (must be 15m min. from horizontal patch panels). Can be patched from any CP to any server cabinet.

Option 3

Note: Black lines are Fibre, Blue lines are Copper

OPTION FOUR

A final option is to have all server cabinets and switch cabinets in a row, terminating to a single patching field for the row, rather than to a central location. Core connections from the MDA are brought into this patching field. This option can work well in ISP or other environments where cross department/customer functionality is not desirable or tolerated. This option provides a bit of best of both worlds in that there will be some spare ports, but also the floor tiles will not have to be lifted to perform MAC work. While this is very similar to the first option, the segmentation can make it easier for network administrators and physical plant technicians to coordinate efforts. Additionally this style of design provides for flexibility in the ever changing environment of shrinking and expanding storage/networking requirements over time.

Option 4

All Patching done within respective rows.

Option 4

Note: Black lines are Fibre, Blue lines are Copper

CONCLUSION

Whichever cabling choice or space option is made, the key step is planning. Siemon has resources to assist in the layout and planning or just as a second pair of eyes for any project.

ABOUT THE AUTHOR

Carrie Higbie has been involved in the computing and networking for 25+ years in executive and consultant roles. She is Siemon’s Global Network Applications Manager supporting end-users and active electronics manufacturers. She publishes columns and speaks at industry events globally. Carrie is an expert on TechTarget’s SearchNetworking, SearchVoIP, and SearchDataCenters and authors columns for these and SearchCIO and SearchMobile forums and is on the board of advisors. She is on the Board of Directors and former President of the BladeSystems Alliance. She participates in IEEE, the Ethernet Alliance and IDC Enterprise Expert Panels. She has one telecommunications patent and one pending.


Need Help?

Ask Siemon

info_uk@siemon.com

+44(0)1932 571 771 (UK)
8am-5:00pm (»Worldwide)


Innovate Magazine
Cisco and Siemon

Cisco Technology Developer Partner


See Siemon in Cisco Marketplace:

Category 7 Cabling?
Cat 7 for the real world Articles and case studies 48 pages
» Learn more

Case Studies - See how Siemon is connecting the world to a higher standard
Find Partners

» Find Siemon Authorized Distributors

» Find Certified Installers & Consultants