Proper management of network cables promotes the elimination of crosstalk and interference, cooler performance, improved maintenance, and easier upgrades. Incorrect cable management may result in damage or failure, which may lead to data transmission errors, performance issues, or system downtime. This section contains cable color and management recommendations. You can adapt the recommendations to suit your environment.
Regardless of the number of servers in each rack, cables must be in place for 32 servers. Ideally, data and power cables must be at opposite ends of the physical rack. If they are aggregated in a bundle or run parallel to each other, induction may introduce electromagnetic interference.
Using specific colors for cables from each device makes for easier troubleshooting.
All cables from the management switch (except those going to the ToRs): yellow
Management switch ports 49 and 50 going to the ToRs: black
ToR 1 cables to servers: blue
ToR 2 cables to servers: red
ToR 1 and ToR 2 connections to spine switches: orange
Console serial switch connections: grey
Cable Type and Length
The Telecommunications Industry Association (TIA) and the Electronic Industries association (EIA) structured cabling standards define how to design, build, and manage cabling systems. The specification is TIA/EIA-568-A. When used for 10/100/1000BASE-T Category 6 (Cat 6) cable length can be up to 100 meters (328 ft). This distance includes up to 90 meters (295 ft) of horizontal cabling between the patch panel and the wall jack, and up to 10 meters (33 ft) of patch cabling. When used for 10GBASE-T, Cat 6 cable length is reduced to 55 meters (180 ft) assuming minimal exposure to crosstalk. Category 6A (Cat 6A) does not have this limitation and can run at the same distances as 10/100/1000BASE-T.
Ensure that the cable type and length being used in your setup meet the following requirements.
The cable connecting the physical server baseboard management controller (BMC) port to the management switch is 10 ft.
The cable connecting the physical server 10 G interfaces to the ToR switches is 1-2 m (3.28-6.56 ft).
The cable connecting the ToR switches 40G interfaces to the Spine switches is 1-2 m (3.28-6.56 ft).
Cable Bend Radius
Modifying the geometry of a cable can impair data transmission and affect performance. When a cable is tied or tightly looped, the pairs within the cable jacket can be separated impacting the integrity of the cable. Therefore, bend radius should be considered when verifying cable management.
The minimum bend radius of a twisted pair patch cable is 4x the external cable diameter, and the minimum bend radius of an LC-type fiber optic cable is 0.8" (~2cm) and SC-type fiber optic cable is 1" (~3cm).
Where articulated arms or rail slides are used, there must be sufficient slack in the cable to allow operation.
No creases in the sheathing should be visible on any cable.
Improperly routed cables can contribute to thermal issues, make field replaceable units difficult to access, or impact performance.
Cable ties can damage cables due to excessive over tightening or by violating the bend radius of a cable. Cable ties also increase service time when an add, move, or change request is received. Cables should be bundled with Velcro straps where possible to avoid damage, simplify addition or removal of cables, and reduce service times.
Use velcro straps instead of cable ties.
Network cables should not be in an area where there is a chance of contacting sharp edges, excessive heat, or subject to pinching between sliding rails.
Cables must be free of tension. Where articulated arms or rail slides are used, there must be sufficient slack in the cable to prevent the cables from being stressed.
Forced air cooling is recommended to draw cool air from the front of the rack and push warm air out the back.
Ventilation slots, power supplies, and rear fans must be clear of cable obstructions.
Field replaceable units such as power supplies must be clear of any cable obstructions that may prevent access for service.
Partners must label the cables in their datacenter. Properly labeled cables reduce troubleshooting time since it is easier to trace and validate connections.
Cable testing ensures that the installed cabling links provide the transmission capability to support the data communication required.
Several tools are available for copper testing. Tests fall into three categories: Verification, Qualification, and Certification. Verification tools are used to perform basic continuity, cable length, and open connection verification. Qualification tools can provide information that details the cable capabilities, e.g. supports 10GBase-T. Certification tools determine whether the cable meets TIA standards such as TIA-568-B.
Options for testing SFP+ and QSFP+ cables are limited. Because handheld cable testers are not available, many network administrators typically reserve ports between two adjacent switches, then connect a suspect cable between active ports to determine if the cable is functional.
Test cables from the physical server baseboard management controller (BMC) port and the management switch.
Review the test print out to confirm that the cables passed the test.
Cables from the physical server BMC port to the management switch must be seated properly.
Cables from the physical server 10G interface to the ToR switches must be tested prior to installation. They must be seated properly.
Each 10G interface must be connected to a separate ToR switch.
Inter-switch SFP+ and QSFP+ cables must be tested prior to installation.
Each 40G QSFP+ cable from the ToR switch must be connected to a separate Spine switch.
There must be two 40G QSFP+ cable connections between each ToR switch and Spine switch.
Inter-switch SFP+ and QSFP+ cables must be seated properly.